Re: 1.6Gb of workdir/.../Dep - thoughts ...

On Mon, Jun 04, 2012 at 03:56:41PM +0200, Lubos Lunak wrote:
> On Friday 01 of June 2012, Michael Meeks wrote:
> >     And some thoughts on improving this; currently we add a ton of deps for
> > packages that are really internal and typically change en-masse or not
> > at all.
> >
> >     eg. 'boost' - it is installed, and then ~never changes again - people
> > don't edit a single boost header and expect a dependency clean
> > re-compile.
> ...
> >     IMHO we could - without significant loss of functionality reduce all
> > those deps to a single stamp file (which we prolly install anyway) in
> > the solver. Should be ~trivial to elide in our dep-re-writing anyway,
> > save > 300Mb of space etc.
> >
> >     Should just be a few lines in solenv/bin/concat-deps.c

But concat-deps is _after_ the creation dep files of the objects (which will
then still be huge).

> >     My question would be: do we want to continue to make 'over precise
> > deps' at this large computational, space and build performance cost
> > possible ?
>
>  I think there are several ways of reducing the size of .d files that are 
> safer:
>
> - a significant part of .d content is the LO build directory - defining that 
> one and some other common paths (solver, workdir) and sed /path/define/ 
> should save quite some space (50% or possibly even more)

We even did that once IIRC using $(OUTDIR)/$(WORKDIR) vars which are known to
gbuild already.

> - a significant part of .d content is the depend-on-nothing deps created 
> by -MP , if those would be merged into one dedicated .d file that'd save a 
> lot of space as well; not sure if this is easily doable though

Arent we doing that already when merging the .d files for one library?

Note however, that every bit of added complexity to the build system will bite
you back one day. Currently the Deps are ~10% of the working directory -- even
if you reduce them by 90%(*), nobody will jump with joy because his build now
fits on his drive by being 9% smaller.

So right now, I consider the topic premature optimization until proven otherwise.

Best,

Bjoern

(*) which you wont unless you gzip them (which is doable and shouldnt have too
big of an performance impact)