Re: 1.6Gb of workdir/.../Dep - thoughts ...

On 04/06/12 17:19, Bjoern Michaelsen wrote:
> On Mon, Jun 04, 2012 at 03:56:41PM +0200, Lubos Lunak wrote:
> > On Friday 01 of June 2012, Michael Meeks wrote:

> > >     IMHO we could - without significant loss of functionality reduce all
> > > those deps to a single stamp file (which we prolly install anyway) in
> > > the solver. Should be ~trivial to elide in our dep-re-writing anyway,
> > > save > 300Mb of space etc.
> > >
> > >     Should just be a few lines in solenv/bin/concat-deps.c
>
> But concat-deps is _after_ the creation dep files of the objects (which will
> then still be huge).

who cares how big the files are (disk is cheap), the relevant metric is:
how many seconds does make need to parse them?

> > >     My question would be: do we want to continue to make 'over precise
> > > deps' at this large computational, space and build performance cost
> > > possible ?
> >
> >  I think there are several ways of reducing the size of .d files that are 
> > safer:
> >
> > - a significant part of .d content is the LO build directory - defining that 
> > one and some other common paths (solver, workdir) and sed /path/define/ 
> > should save quite some space (50% or possibly even more)
>
> We even did that once IIRC using $(OUTDIR)/$(WORKDIR) vars which are known to
> gbuild already.

yes, but keep in mind that variables in the dep files will need to be
expanded by make, which will likely result in memory allocations, so
it's an open question whether that will actually improve performance or
slow it down.

> > - a significant part of .d content is the depend-on-nothing deps created 
> > by -MP , if those would be merged into one dedicated .d file that'd save a 
> > lot of space as well; not sure if this is easily doable though
>
> Arent we doing that already when merging the .d files for one library?

to some extent yes, but of course the same headers are included in many
libraries, so there is still some amount of duplication there; however i
don't know to improve this without breaking separate building of
modules, which requires the LinkTarget .d files to be self-contained.

> So right now, I consider the topic premature optimization until proven otherwise.

> (*) which you wont unless you gzip them (which is doable and shouldnt have too
> big of an performance impact)

sadly AFAIK make cannot include compressed files...