Re: [Libreoffice] Bug 40673 - unify build parallelization and use gnu makes --load-average feature where available

On Wed, Sep 7, 2011 at 3:02 AM, Bjoern Michaelsen
<bjoern.michaelsen@canonical.com> wrote:
> Hi list,
>
> I just proposed the following at
> https://bugs.freedesktop.org/show_bug.cgi?id=40673 :
>
> The ways to specify
> build parallelization are way to complex and confusing.
>
> I propose the following solution:
>
> 1) We use --with-num-cpus and --with-max-jobs and --with-sense-load
> 2) --with-num-cpus defaults to the number of cpus on the machine and is
> stored in GMAKE_PARALELLISM only
> 3) --with-max-jobs defaults to the number of cpus divided by two on the
> machine and is stored in GMAKE_MODULE_PARALELLISM only
> 4) --with-sense-load defaults to "T" unless using icecream or on
> windows (where it is "") and is stored in GMAKE_SENSE_LOAD
> 5) all calls to build.pl are done as:
>   build.pl -P$((${GMAKE_PARALELLISM}/${GMAKE_MODULE_PARALELLISM})) --
> -P${GMAKE_MODULE_PARALELLISM}
> 6) gbuild calls (including tail_build) are done as:
>       make -srj$((${GMAKE_PARALELLISM}*4))
> -l$((${GMAKE_PARALELLISM}+1)) when GMAKE_SENSE_LOAD is not empty, or as:
>       make -srj${GMAKE_MODULE_PARALELLISM}
>   otherwise (exception: tail build might use GMAKE_PARALELLISM)
>
> The variables @BUILD_NCPUS@ and @BUILD_MAX_JOBS@ will be removed.
>
> Configure should warn, if GMAKE_MODULE_PARALELLISM is bigger than
> GMAKE_PARALELLISM and set it to GMAKE_PARALELLISM.
>
> Any objections to that implementation?

yep. it is even more complicated that the current scheme.
I mean:
  build.pl -P$((${GMAKE_PARALELLISM}/${GMAKE_MODULE_PARALELLISM})) --
> -P${GMAKE_MODULE_PARALELLISM}

really ? ok so on my Mac I build with max-job=8 num-cpu=6... what
value should I use to get the same result under the new scheme?
(today that means I get up to 8 jobs with up to 6 task per, except for
tail_build where I got 8 -- note that tail_build tend to run alone)

for info, right now the mode is

GMAKE_PARALLELISM=max(nb-cpu.max-jobs)
that value is used for tail_build only

GMAKE_MODULE_PARALLELISM=max-job (because that is how it was, so that
avoided 'regression'. that value is used for gmake module other than
tail_build.

in fine, only GMAKE_PARRALLELISM should be relevant as stuff get
accumulated under tail_build


Now, yes,  if it works (I never tested it), the load average sound
more promising than a pure -j n... but then if it works, then why even
bother with -j n at all...
if --with-sens-load is present then use the value specified or default
to nb_cpu + 1 like you said, but then use -j without value...

Actually, the end-game should be --with-sense-load=n that use -l n on
platform that support it and -j n on windows... n default to nb_cpu +
1


(btw: I've been recently experimenting with these value to find the
best combination on a 48 cores machines (4 x 12 core).

Norbert

PS: with ExternalLib.mk I was able to use the fact that sub-make can
coordinate with the main make for //ism. so eventually most of what we
build should end-up controlled by one //ism factor - with the
exception of ant base stuff I suppose... java, a pain as always...
actually ICU may also be a big pain...