Re: [libreoffice-l10n] Pootle migration

Hi Andre, *,

On Thu, Apr 7, 2011 at 1:37 PM, Andre Schnabel <Andre.Schnabel@gmx.net> wrote:

> Please leave the technical discussion aside for a minute.

No, cannot do that, as this is directly related to your point:

> What *we* as a team of localizers need is a working and performant
> setup to do our translation work.

Yes, and I fully agree, and that of course is also my highest measuremen/goal.

It is my true belief that the current setup will handle this just fine.

> - we had no full localization on the server (but we will have to provide
> this for 3.4 localization)

See the other messages. What is timeconsuming (and no memory will help
here, as it is is CPU bound) is importing of the new data, and time
until it is ready after a server-restart. Nothing to do about it,
apart from the admins actually doing the import using differently
formed comments that make use of all of the assigned CPUs.

> - hardly any team did full translation on the server, we "only" did
> bugfixes

See my post. I'm /sure/ the server can handle the full
online-translation within pootle.
I'm sure that the server will not get more than 10 apache requests per
seconds on average, and the server can handle about 200

> There were some good reasons to have a rather good equipped server for
> pootle.

A memory leak/wasteful setup is not a good reason for this.

> We knew, that pootle might be not the perfect solution regarding
> memory usage, multithreading ... That we now cannot use the initialy
> planned setup is very unfortunate. But we still need a system that
> provides similar performance.

Again: If performance is not satisfactory for translation work, I'll
reconsider. But again: I don't see any evidence that performance could
be increased by assigning more RAM to the VM.

> > I clearly explained why I'm convinced that it is a memory leak. It
> > doesn't free the memory, even when the machine is idling for hours. It
> > doesn't need that memory, since when the worker is replaced by a fresh
> > one, the functionality still works.
>
> No - it is just that this is totally irrelevant in the current situation.

No it is not.

> Even if pootle has a memory leak, we won't fix this within the next few
> days. But we need to start translating asap!

Again: It is /perfeclty working/ when restarting the worker threads.
And the translator will not notice that the worker-thread has been
replaced. The translator will not notice whether the few milliseconds
he has to wait are
* because the keep-alive expired and a new http-connection has to be negotiated
* a server process expired and thus is restarted
* seeking through the actual file for next/previous match took a little longer.

> So - if there is any way to provide more ressources (memory) we should do
> this and analyze the root cause later (I'm sure, pootle developers are
> interested to help with this).

/IF/ there are memory related problems, I'll assign more RAM. But
again: everything I experienced so far says: More memory won't help at
all.

> > Again:
> > * I don't have a problem with pootle taking half a day to import the
> > files for a new release (one time thing, no big deal)
> > * I don't have a problem with pootle taking very long for the very
> > first hit of the frontpage after a reboot (system is not rebooted that
> > often, also no big deal)
> > * I don't have a problem with restarting pootle server-processes to
> > workaround the memory-leak (whether you call it leak or not, I
> > definitely call it leak). It limits the amount of concurrent worker
> > processes, but that is not a problem since even with just the two as
> > in the VM, you can easily serve 50 concurrent requests per second
> > while benchmarking (resulting in being able to handle about 200
> > request/second of the frontpage, thus much reserves available. No
> > problem)
>
> Maybe you don't have, but the translators will have a problem with this.
> So either we can provide more ressources to the VM or we need a
> different solution.

Again: Those stuff that takes ages are /NOT/ solvable by assigning
more ressources to the VM. Those are CPU-bound, and all of them are
single-threaded.
If the process maxes out a single CPU, that is as much speed as you
can get, no matter how many RAM is sitting idle.

* Pootle has a very, very poor first-start performance.
→ not a problem as the server will not be rebooted every other day.
And in case this wasn't clear: Restarting the worker-processes will
/not/ have the same effect, the user will not notice anything,

* Pootle has poor performance when generating the zips for download
(fist trigger per language and project)
→ This again is CPU bound, and again: More RAM will not help.

This is the only case where the user can have a problem (and is part
of the problem).
It doesn't help to click multiple download links on one page to get
the zips faster, on the contrary. Click one, wait for the processing
to be done, and when you got the first one, feel free to download all
the remaining ones.
When multiple users request huge zips at the same time, all server
processes are busy. Currently there are two, can be increased to 4,
but that's not a big difference. It is CPU bound (and also does a
little disk i/o) and I repeat myself once again: MORE RAM WILL NOT
HELP!!!

If you want a dedicated pootle server, than order a 64 core system.
With that, you can hide pootle's weak spots. It will be idle 99% of
the time, but for the point of time where 10+ people get the idea to
download the zips at the same time, you won't run out of CPUs.

> All other discussion is very academic at the moment.

No - it is just very tiring that people still stick with their guesswork.

Again: The only problem is creation of zips for download. This takes
(in terms of web-responsiveness) /ages/, and blocks the server's
process while it is performed.
It is CPU-bound, and thus the amount of workers one can have is very limited.
Pootle must be reconfigured to limiit these actions, or (more simple
to implement as a workaround just create all the zips once per day in
a cronjob (or better distributed during the day, as creating them all
at once would again create the CPU-bottleneck) and only allow to
download those. Thus you don't get 100%up-to-date versions of the
file.
But when you download the zips, you're not interested in the current
state anyway, as just after you download someone else could have
edited in pootle thus creating the same issue.

Again: * Don't request additional zips of a project before the first
one is delivered to you.

ciao
Christian

-- 
Unsubscribe instructions: E-mail to l10n+help@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted