Re: [Libreoffice] Mac Build with lang=ALL

On Tue, Jan 18, 2011 at 1:35 PM, Michael Meeks <michael.meeks@novell.com> wrote:
> On Wed, 2011-01-12 at 10:47 -0600, Norbert Thiebaud wrote:
> > indeed it is a dog. it took 15h20.
> > conservatively at least 13 hours of it were in helcontent2, mostly I/O
> > bound creating temp files and copying things around over and over.
[..]
>        A huge win. Personally, I find it hard to believe that the problem is
> not I/O related

Well as I said above, it _IS_ I/O related, no doubt about that.

> - ie. to do something -that- slow, a file-system has to
> fsync (ie. physically move the disk heads) I imagine, I don't think much
> else can explain it. Out of interest how fast is:
>
> mkdir /tmp/small ; time for (( a=1; a<=10000; a++ )) do touch /tmp/small/$a; rm /tmp/small/$a; 
> done
>
>        for you ? I get 22 to 26 seconds (on my two test machines), one has
> an SSD, the other a hard-disk, but you can see that neither write anything
> to the disk in that time.

real    0m30.254s
user    0m5.720s
sys     0m25.907s
on my mac (note: doing this in a ram_disk give about the same result)

real    0m20.760s
user    0m2.974s
sys     0m18.205s
on my linux

but that does not measure much since

mkdir /tmp/small ; time for (( a=1; a<=10000; a++ )) do touch
/tmp/small/$a;  done

real    0m17.662s
user    0m3.012s
sys     0m15.115s

time cp -r /tmp/small /tmp/small2

real    0m2.508s
user    0m0.119s
sys     0m2.256s

n_th@tpamac ~ $time rm -fr /tmp/small2

real    0m1.000s
user    0m0.014s
sys     0m0.808s

So the test you give is more about measuring bash and fork speed than
anything else

>        So - I'm simply surprised that a ramdisk makes much difference.
>
>        I was suspicious of the database / lucene building piece - which may
> well do fsyncs, and write a journal to try to ensure data integrity, I
> can believe that is horribly slow.

That is a distinct possibility. but the speed-up is there even for the
part that are not running lucene. (based on the cpu-load - io/load
monitoring while building on regular disk or on ramdisk)

for info
time for (( a=1; a<=1000; a++ )) do touch
/Volumes/ccache_ramdisk/xx$a; sync; rm /Volumes/ccache_ramdisk/xx$a;
sync; done

real    6m37.891s
user    0m1.448s
sys     6m33.465s

note: is 1,000 not  10,000 and it is the same order of magnitude when
using 'real' disk


Note: I'm building is -P6 -- -P6, that may also have an impact. some
I/O pattern involved may dog badly in a heavy
multi-threaded/multi-processed environment...

(yeah, yeah... on paper that sound crazy high, but in practice 6/6
give me the best result on this machine)

Norbert


> > Note: that if you do not have enough memory to do that, you could at
> > least move the TMPDIR do a ramdisk. you don't need a big one (not sure
> > exactly how much, but I bet 500M should probably do)
> > there are some step in helpcontent2 that do a LOT of creation/delete on TMP.
>
>        Right; I guess it might be nice to have an 'strace -f -ttt' log (or Mac
> equivalent) of a few minutes of the build - to see what is going on.
>
>        Thanks !
>
>                Michael.
>
> --
>  michael.meeks@novell.com  <><, Pseudo Engineer, itinerant idiot