Re: [libreoffice-l10n] Pootle migration

Hi *,

On Wed, Apr 6, 2011 at 9:58 AM, Dwayne Bailey <dwayne@translate.org.za> wrote:
> Please CC me on any replies as I'm not on the list.

Keeping this, so others will follow the request as well.

> > Mon, 4 Apr 2011 22:28:43 +0200
> >                              From:
> > Christian Lohmaier
> > On Mon, Apr 4, 2011 at 2:58 PM, Rimas Kudelis <rq@akl.lt> wrote:
> > Again, as you apparently still don't understand what I already wrote many
> > times:
> > * Adding more resources will /NOT/ make pootle run faster than it does
> > now.
> > The VM already has way more resources assigned than necessary. It is
> > /idle/ almost all the time.
>
> As far as I know the server hasn't really been used yet, so I guess
> we'll be collecting data from now on to see how things go.

Also from the avialble data from the old pootle server. But yes, the
new one doesn't have much data yet.

> During the
> setup of the server, we make tradeoffs between performance and memory
> use. If there is no memory available, we'll obviously try to optimise at
> all cost for minimising memory use,

Please explain those settings. Which settings, what is the effect, how
to see the effect (i.e. what UI actions to perform)

> and that is what I understand that
> Rimas said: things might be slower than necessary, since we are not
> optimising for performance, but for memory use.

No, and I repeat again: Memory is not an issue. The memory leak is.

> > * The only thing that is slow (when executed the first time) is
> > generation of the zips. So when you as translator request a zip: Don't
> > click the link multiple times because you don't immediately get the
> > zip. It can take 10 seconds for the files to be generated. Again:
> > * Adding more resources will /not/ make that time shorter. It is a
> > single-threaded process that can only uses one single CPU, thus
> > assigning more CPUs won't help at all (the VM has 4CPUs assigned
> > already)
> > Requesting that same zip another time (or different zips of the
> > project belonging to the same language is fast/instant, but requesting
> > the zip for another language again may take some seconds for the first
> > request (or again after the files did change in between).
> > * Pootle has a memory leak when creating the zips. It won't release
> > memory after processing the files.
> > This would be the only time where the assigned resources may run out
> > (the VM has 1GB or RAM assigned): Multiple different languages request
> > the zip at the same time. Then memory usage increases, memory runs out
> > and either it is crawling along or the process gets killed.
>
> Some stuff that is slow to load is cached for later use.

"Some stuff" maybe, but that little is not what I'm talking about.

> This is done
> for performance optimisation. This is one of the reasons you won't see
> the memory use go down immediately after generating a ZIP file.

No, this has nothing to do with caching. It is a memory leak. Rimas is
very good at not forwarding relevant information it seems. So here I
just copy and paste what I wrote to Rimas already.
#############################
> > > [Rimas]
> > > After generating, the zip files are cached on disk, so before
> > > redownloading the same file, you'll probably want to execute
> > > $ find /var/www/pootle/po/POOTLE_EXPORT/ -type f -iname *.zip -exec rm {}
> > > \;

No, this is not enough to do the same as what pootle does when
requesting the files the first time.
The first time it takes long (100% CPU while processing), and that is
also the time where the memory leak occurs. After it is finished
producing the file, the memory is not freed.

Requesting one of the zips interestingly also prepares the other files
at once, at least when selecting any of the other files on the same
language and project, the file will be served immediately, without the
lengthy processing step.
(And those requests also don't increase the memory usage further)

But switching to another language, and requesting a file results in
the processing with memory leak again.

But that memory is not needed at all, as is obvious when the process
has reached it's max-requests and is replaced by a new one, then it
serves all of the already covered languages without that increase in
memory-usage.

With that info at hand, I tweaked the apache-settings a bit (basically
reduced the amount of concurrent wsgi processes, and reducing their
lifetime) to make the accumulation less likely.

The machine will still run out-of memory when different language zips
are requested in a short amount of time with not enough regular
requests in between. In case people see too many "premature end of
script" or similar (i.e. the worker has been killed by the OOM
killer), reduce the number of requests before the worker is restarted.
Currently it is at 400, which is still pretty high, considering the
more or less non-existant regular load (but then again, this might be
because people know the server is in transition and because 3.3 is
done already)

Similarily, the very first page-load after a reboot also takes "ages",
but after that it runs smoothly,

1GB is enough for Pootle The rest is a matter of
configuration/tweaking. (or best: fixing the memory leak)
#######################
> Another
> reason is the way the garbage collector works in Python.

Well - how long should one wait for the memory to be freed? If it is
not freed when memory is about to run out, or after a hour or so, then
I don't assume it is a problem of running the garbage collector. (and
again that memory is not used/needed for further request of the same
kind, so I don't take your "it is caching stuff to accelerate future
actions" for granted.)

Please give me instructions on stuff to do in pootle to notice a difference.

> Deciding to cache something is a tradeoff. So we can disable or minimize
> some of the caching, which will simply make a few things slower,
> hopefully not by much, but we're guessing while the server hasn't been
> used much yet.

Numbers and examples please.

Caching for a more or less one-time operation that impacts the whole
server all the time surely is the wrong way to go. Even more so when
despite that "caching" (I still call it leak, as it doesn't cache
anything) the requesting of the zips is still CPU bound and slow.

> I suggested some customisations to the parse pool (to do exactly this).
> That affects the number of cached files and search indexes, both of
> which are very large on your server.

Please be more detailed on this. What setting, in what way to tweak,
and what should the effect be?
Numbers?

> > * I will NOT assign RAM to a VM (and thus block that ram for other
> > use) to satisfy a memory leak, when that RAM is unused 99% of the
> > time.
>
> I believe what you are seeing is the caching, not a memory leak.

And I strongly disagree, see above.

> We haven't seen the server used much yet. My educated guess from having
> worked on a few Pootle installations is that the RAM isn't enough,

And I strongly disagree. See above.

> > * The effects of the memory leak can be nullified by just restarting
> > the worker processes more frequently. Thus again:
>
> ...at the cost of making things slower, since more stuff needs to be
> loaded in memory afresh every time you restart a process.

I'm starting to belive you're in the same league as Rimas when it
comes to server administration (i.e. very much guesswork).

The starting/initializing of the processes is negligible. Just fire up
an ab bechmark and request 20000 with concurrency of 30 and look at
the distribution.
I'd gladly live with a few hundred of milliseconds, if that will save
hundreds of memory that can be freed by just restarting the worker
process.

> > * Adding more resources will /NOT/ make the VM run faster
>
> It most probably will, since we are sacrificing performance to minimise
> memory use.

No. As the times where it is slow is not bound by memory, but by CPU.
Please please please: Read what I wrote. I repeated myself so many
times, it really gets annoying.

> For example: we opted for more threads, rather than
> processes, that is known not to perform as well in Python, especially
> for CPU intensive tasks.

Who is "we"? Yes, I decided to (for now) only run with 2 wsgi
processes. But then again: how would that make execution faster?

If you provide me with hard numbers, concrete settings and
instructions on what to do in the UI to reproduce and compare the
settings, I'm willing to try them out.

> >  it will
> > /NOT/ allow it to handle more requests
>
> It most probably will, since we reduced the number of processes to
> minimise memory use, and slower serving of requests necessarily affects
> the number of requests you can serve in any given time.

Again this is wrong. it might allow 4 simultaneous "request the zip"
requests, but all 4 would be taking ages, and all would have the
memory leak, thus needing more RAM (that I'm not wiling to assign,
since regular operation just doesn't need it).
(and the users that are just using regular webinterface have to wait
then as well). Allowing more processes doesn't make sense when pootle
doesn't queue/limit those lengthy processes by itself.

> > Pootle is idling almost all of the time. There is less than one apache
> > request per second on average (and for regular requests (i.e.
> > non-"generate-a-zip" actions) it can easily serve >>50 simultaneous
> > requests per second.)
>
> Let's keep an eye on things when people actually start to use the
> server.

Whether the requests are generated by a benchmark tool or by regular
users doesn't make any difference. The number it can handle is way
beyond any real load to expect.

> > > Maybe if we
> > > manage to give enough load to the server, he'll change his mind (or
> > > we'll
> > > find other ways to deal with the problem).
> >
> > No, I won't change my mind, but depending on the load/effects of the
> > memory leak I'll reduce lifetime of the server-processes further.
>
> I hope you will be reasonable and look at the data as it becomes
> available, and at least consider changing your mind.

Again this is out of the question, but all *data*, real facts I
gathered so far don't suggest that it would be any different from my
current opinion.

> As mentioned, I'm
> pretty sure there is no memory leak.

I disagree, see above. Alternatively give a reasonable explanation on
on why the export still works fine even when the process that created
it (and thus the memory it occupied, and all possible caching it could
have down in its own space) is gone.

> If you reduce the lifetime of the
> server processes, you are just making performance worse, which is all
> that Rimas warned the users for.

Numbers/examples on how performance can be worse. Otherwise I cannot
take your comments seriously, as my measurements did show otherwise.

> [...]
> I agree. Generating the ZIP files is slow. Doing it multithreaded, will
> limit the performance for more users while doing that.

But the time they have to wait will be shorter.

> > * Requesting the other zips in the same project & language is fast.
> >
> > So there is no point in clicking all zip-URLs on a page at once (on
> > the contrary, than the request will all cause CPU to be burnt, while
> > all that is only needed one single time). If you want to download more
> > than one zip of a page, click the first one, wait until it is
> > generated and handed over to your browser, then click as many others
> > on the same page and get all of them quickly.
>
> Yes, we have optimised for several cases here that are likely, and I
> suggested some workarounds for some of the issues we're likely to hit
> with the little bit of RAM as Rimas has already started doing, as far as
> I know.

Don't suggest them only to Rimas, as he is not communicating well with
such technical matters it seems.

> > * Any other time, doing in-place-translation, just browsing along is
> > supposed to be fast.
>
> ... as long as we're not hitting the current imposed limits of
> concurrency.

Again: Numbers, no foggy guesswork please.

ciao
Christian

-- 
Unsubscribe instructions: E-mail to l10n+help@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted