Re: [libreoffice-l10n] Pootle migration

Hi *

On Wed, Apr 6, 2011 at 6:11 PM, F Wolff <friedel@translate.org.za> wrote:
> Op Wo, 2011-04-06 om 10:50 +0200 skryf Christian Lohmaier:
>
> I find your responses rude and unhelpful.

Sorry for that. I'm just tired of writing the same stuff over and over again.

> Somehow we manage to provide this software on hundreds of sites running
> fine, ranging from hosting on a few hundred megabytes of RAM to machines
> with several gigabytes. All accidents?

No, on the contrary, especially because it is run on so many servers
with limited resources, I'm not willing to waste RAM on pootle-VM
alone, when pootle doesn't need it at all.
RAM assigned to the VM is not available to the host or other VMs, even
if the RAM is not used inside the VM.

> I could show you the specific lines of code caching big objects
> (expected to be tens to hundreds of megabytes in size), but I guess that
> won't convince you either. It is a leak, because someone who (I guess)
> never looked at the Pootle code, says so.

I clearly explained why I'm convinced that it is a memory leak. It
doesn't free the memory, even when the machine is idling for hours. It
doesn't need that memory, since when the worker is replaced by a fresh
one, the functionality still works.

And You're top-posting and fullquoting. This (again from my long
email/mailinglist experience) emprically is a sign that people don't
actually read what was written, don't answer the questions, and
basically discuss different topics all the way, leading to repetition
over and over again.

> Why can't you at least start
> by assuming that I might have an idea of how Pootle works?

That's not the point. Please take your time to actually read the post.

> When you are willing to discuss things under the good faith assumption
> that I _might_ not be talking nonsense, we can continue the
> conversation. I've been trying to help you in my free time based on my
> experience, but it seems you'd prefer to assume I don't know what I'm
> talking about.

Well - All I heard so far, and that was not from you personally, but
mainly via Rimas was just guesswork, no facts. And that guesswork is
contradicting hard numbers, real benchmarks. And I just hate when I
have to write the same thing over and over again. It surely did have
an impact on the tone of my response, apologies for that. But that
doesn't change anything.

> In the meanwhile, here is some recommended reading:
> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm

Sorry, but the information content of that article is near zero
compared to what has been written here already. It contains a hint for
programmers regarding the "builtin memory leak" when working with
large number of integers or floats. I also don't care whether it is
"python itself" or the allocator that doesn't give back memory, or
that memory cannot be returned because of memory fragmentation. In the
end it makes the system unusable. Not returning memory is a memory
leak, no matter whether it is by design or not. If you cannot free
memory, you have to spawn a childprocess and dismiss that child to
keep a runnable system. It is /not/ acceptable to accumulate more and
more memory and just say "but it is not my fault, it is the memory
allocators that don't return the memory to the system"
It's not a few kBs here and there we're talking about, but tens to
hundreds of MB.

In my definition when a (long-living) process doesn't return unneeded
memory. that process is leaking memory. Whether it is by design or not
doesn't matter to me.
* It consumes more and more memory (it would no problem if creating
the zips for another language would just reuse the memory that was
allocated when creating the zip for the first language)
* It doesn't release that memory when idle (it would also be no
problem if that memory would be returned to the system after a while -
now it has to be enforced by restarting the server-process itself)
* It doesn't release the memory when the machine runs out of available
memory (thus people cry "the machine needs more RAM", but there is no
need, as it can be easily circumvented)
* the allocated memory is not needed to perform any operation after
the memory has been used once. (it doesn't accelerate anything, having
that memory or not allocated does not make any difference at all to
subsequent requests. It is just blocking system resources - as is
obvious by replacing the process with a fresh one)


Again:
* I don't have a problem with pootle taking half a day to import the
files for a new release (one time thing, no big deal)
* I don't have a problem with pootle taking very long for the very
first hit of the frontpage after a reboot (system is not rebooted that
often, also no big deal)
* I don't have a problem with restarting pootle server-processes to
workaround the memory-leak (whether you call it leak or not, I
definitely call it leak). It limits the amount of concurrent worker
processes, but that is not a problem since even with just the two as
in the VM, you can easily serve 50 concurrent requests per second
while benchmarking (resulting in being able to handle about 200
request/second of the frontpage, thus much reserves available. No
problem)
* I don't have a problem with you
* I don't have a problem with Rimas (or anyone else here)

* I see a problem in pootle not having a queue or other limitation on
the "create zips" case. It is very easy to take a server out for at
least a couple of minutes by requesting the zips of a huge project for
multiple projects at once. 8 CPU system with 8 workers → just request
8 different languages and all workers will be using 100% each for
multiple minutes, additionally fighting a little over disk-i/o and
will not be available for other stuff, and the preparations are slow
(let alone the waste of memory resulting from not releasing it again).

* I have a problem with top-post & fullquote. As (with a very high
confidence) this is a clear sign of the person not reading the post,
not trying to understand the post
* I have a problem with having to write the same stuff over and over
again. I get angry when I have to, and my tone might then no longer be
100% appropriate.

ciao
Christian

-- 
Unsubscribe instructions: E-mail to l10n+help@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted