Re: [libreoffice-l10n] Pootle migration

On 2011-04-05 15:20, Christian Lohmaier wrote:
> Hi Andras, *,
>
> On Tue, Apr 5, 2011 at 9:42 AM, Andras Timar<timar74@gmail.com>  wrote:
> > 2011.04.04. 22:28 keltezéssel, Christian Lohmaier írta:
> >
> > > * If you experience any "slowness" on other operations than requesting
> > > zips: Report them.
> > Many thanks for your insightful explanations. I find the new server more
> > responsive than the old one. However, some operations are still slow.
> > When I add a new language to a project (e.g. LibreOffice 3.4.x UI) and
> > click Overview tab of that new language, I have to wait ~2-3 minutes.
> You don't write when exactly you did it, but from munin stats I see
> there is high CPU utilization between around the time you wrote that
> post (i.e. between 9 and 10). But that high utilization is again only
> on one single core, i.e. 100% out of 400% are used only.
> This is pootles's processing that is slow here (CPU-bound, thus
> nothing fixable unless you rewrite pootle to use multithreaded
> approach).
When adding a new language we don't do any caching until you go to the 
overview page.  Thus when first viewing we are doing all the updating 
for search and check failures.  A way around this would be to run 
refresh_stats manage.py command.

I doubt multithreading would help here because of Python's poor 
multithreading ability.  So more invasive changes to display stats as 
they are available would probably be the correct way to make the UI more 
responsive.  We're relying on Apache to start each mod_wsgi instance so 
while this is slow and hogging that one core it wouldn't prevent someone 
from translating elsewhere in Pootle.

> > Then I upload translations to that language and I have to wait another
> > ~5 minutes. This is not a convenient way to upload 100 languages. Can it
> > be done in a batch process from the shell?
> Oh, large-scale updates surely should be possible using the shell, but
> I don't know really know pootle, thus I cannot tell for sure (but I
> guess Rimas is doing just that right now...)
When uploading translations a few things are happening.

  1. Unzipping if needed
  2. Parsing the files that you supplied
  3. Merging your uploads with the current data in the db
  4. Refreshing the stats

We do this to prevent any data loss, but as you can imagine its a lot of 
work.  One thing worth trying on the LO Pootle server is to make use of 
the C PO parser, its much faster but not as widely tested as the Python 
PO parser.

You can do this from the command line update_stores followed by 
refresh_stats.  This of course if you know that the files on the files 
system are the ones that you want as it will override anything in the 
database.
> ciao
> Christian


--
regards
Dwayne


--
Unsubscribe instructions: E-mail to l10n+help@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted