Quantifying the time overhead of Cygwin make

There were a few complaints on this list about the performance of the
build system on Windows; this mail provides some data on the problem.

I. Why is Cygwin make slow?

1. GNU make calls vfork(2), which is implemented[1] as fork(2) on
   Cygwin which, among other things, copies the entire process memory
   of the parent, and does several context switches.

   According to Task Manager, our non-recursive make process uses 110MB
   of memory, and copying that for every spawned process adds up.

2. to check time stamps, make calls stat(2) which is implemented by
   Cygwin as a series of Win32 system calls, followed perhaps by some
   mapping of Win32 user-ids to something more POSIXy, or opening the
   file to see if it starts with "#!", none of which matters to make.

3. Another problem of Cygwin make is token deflation:  the job-server
   implementation (by which spawned make processes in bundled externals
   can build in parallel) is not reliable and is notorious for losing
   tokens, which is why it is currently recommended to use a higher -jN
   than on other platforms to avoid idle CPUs.


II. GNU make configurations

The following configurations were tested, pointing out any performance
relevant differences from upstream releases:

1. LO patched Cygwin GNU make 3.82 downloaded from dev-www.libreoffice

 (purely guessing here, based on the git log:)
 - back-ported upstream 4.0 glob performance regression fix

2. LO patched Cygwin GNU make 3.82 from gnu-make-lo-3.82 branch

 - back-ported upstream 4.0 $(file) function
 - back-ported upstream 4.0 glob performance regression fix
 - built-in "cp" commands: heuristic to recognize the particular
   invocations of cp(1) that happen during the build and open-code
   these directly in make's C code

 git clone git://gerrit.libreoffice.org/gnu-make-lo
 git checkout gnu-make-lo-3.82
 ./configure
 make

3. LO patched Cygwin GNU make 4.0 from gnu-make-lo-4.0 branch

 - built-in "cp" commands (as above)
 - "depcache" feature to speed-up incremental builds with a more space-
   efficient generated dependency file format

 git clone git://gerrit.libreoffice.org/gnu-make-lo
 git checkout gnu-make-lo-4.0
 ./configure
 make

4. upstream Win32 GNU make 4.0+ (upstream current master
   85047eb9044d4b72d50e6620c505c675d55ab98b)

 This is built with Visual Studio 2012; it's not entirely obvious
 how to build it, for example it requires a sed program in the PATH;
 this should work:

 git clone git://git.savannah.gnu.org/make.git
 [go to Start menu, click on "VS2012 x86 Native Tools Command Prompt"]
 set PATH=%PATH%;C:\Cygwin\bin
 cd .../path/to/make
 build_w32.bat --without-guile

 should result in a WinRel/gnumake.exe

 There were quite a few issues with both shell quoting and over-long
 command lines; one advantage of Cygwin is apparently that it uses
 a different way to transfer the argument vector to spawned processes;
 with Win32 CreateProcess the limit appears to be 8k and there is no
 argument vector, but simply a single command line string that the
 called process then has to split up again somehow; see also [3].

 These issues are fixed on current LO master (and "gengal" can now
 read arguments from a response file).

 The only required patch that is not yet in master is:
 https://gerrit.libreoffice.org/#/c/9698/

 Note1: a Win32 build of the upstream 4.0 release crashed with memory
        corruption, that's why i switched to current master, which does
        not have this problem.

 Note2: one would think that defining HAVE_CYGWIN_SHELL is a good idea
        when using Cygwin's shell, but experience indicates otherwise.
        In particular, it resulted in more quoting issues, contrary
        to what is stated in the documentation.

 Note3: there were problems with the very old Cygwin DLL 1.7.18 i had
        installed, crashing in sh.exe with message "fork: can't reserve
        memory for stack ... Win32 error 487" similar to [2], that went
        away after upgrading to  current Cygwin DLL 1.7.30.


III. Measurements

These were taken on a Fedora 20 KVM with 12 logical cores and 16GB RAM,
running Windows 7, with Cygwin DLL 1.7.30 and Visual Studio 2012.

autogen.input:

MAKE=E:/make/make-85047eb-msvc.exe
#MAKE=/cygdrive/e/make/make-3.82-lo-cygwin.exe
#MAKE=/cygdrive/e/make/make-4.0-lo-cygwin.exe
--with-external-tar=/cygdrive/e/lo/ext_sources
--with-lang=fr es
--disable-ccache
--enable-dbgutil
--enable-werror
--enable-pch
--enable-ext-languagetool
--enable-ext-wiki-publisher
--enable-ext-nlpsolver
--enable-ext-numbertext
--with-ant-home=/cygdrive/c/Program Files/apache-ant-1.8.4
--with-junit=/home/mstahl/junit-4.10.jar
--enable-crashdump
--with-help
--with-myspell-dicts
--with-parallelism=20
--with-doxygen=E:/doxygen/doxygen/bin/doxygen
--with-package-format=msi

We measure build from scratch on an otherwise idle system of revision
aa8b268935b07c9bf95a596ee84c76eed4c03634 (plus necessary build system
fixes) via "time make check", taking note of how many job-server tokens
were lost, and then the make overhead of an incremental rebuild by
invoking make with --dry-run (so no commands are actually executed, make
only checks which targets need to be rebuilt) via:

  sed -i Makefile -e "s/(build_goal)/(build_goal) --dry-run/"
  time make check

legend:
  F = from-scratch build time
  T = job-server tokens at end of build
  I = incremental build make overhead time (4x)

1. "old" LO patched Cygwin GNU make 3.82

  F:  111m49s   T: 17/20      I: 3m24s 1m30s 1m31s 1m31s
  F:  113m58s   T: 19/20      I: 3m32s 1m25s 1m22s 1m26s
  F:  112m23s   T: 18/20      I: 3m25s 1m30s 1m27s 1m31s
  F:  116m15s

2. LO patched Cygwin GNU make 3.82 from gnu-make-lo-3.82 branch

  F:   88m12s   T: 18/20      I: 2m56s 1m02s 1m01s 1m00s
  F: (112m58s)  T: 19/20      I: 2m57s 1m00s 1m01s 1m01s
     ^^^^^^^^^ measurement error, failed test waits for click on dialog
  F:   89m40s   T: 20/20      I: 3m02s 1m00s 1m00s 1m00s

3. LO patched Cygwin GNU make 4.0 from gnu-make-lo-4.0 branch

  F:   89m25s   T: 16/20      I: 2m54s 0m48s 0m49s 0m50s
  F:   90m15s   T: 15/20      I: 2m50s 0m48s 0m49s 0m49s
  F:   94m32s   T: 20/20      I: 2m51s 0m49s 0m50s 0m49s

4. upstream Win32 GNU make 4.0+

  F:   83m40s
  F:   85m19s
  F:   85m53s                 I: 0m49s 0m27s 0m28s 0m27s
  F:   80m17s                 I: 0m47s 0m34s 0m34s 0m34s
  F:   84m34s                 I: 0m45s 0m28s 0m29s 0m29s
  F:   84m23s                 I: 0m44s 0m29s 0m29s 0m28s

  Win32 make did not emit messages about missing tokens; hopefully that
  indicates no tokens were lost, and not that it wouldn't print an
  error if this happens.


IV. Conclusion

Using a native Win32 GNU make provides faster from-scratch rebuilds than
any previous attempt to improve build performance, and significantly
reduces the incremental re-build overhead of make on Windows by a factor
of 2x-3x.

Since there are not many developers using Windows anyway, i'd propose to
have a flag day to switch to requiring Win32 make on master, dropping
support for Cygwin make.

This gerrit patch does the switch:
https://gerrit.libreoffice.org/#/c/9698/

Of course, this first requires preparing tinderboxes, and testing by
others if the Win32 make works well everywhere; please do so if you are
able to do a Windows build of LO.


[1]
http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/fork.cc?rev=1.244&content-type=text/x-cvsweb-markup&cvsroot=src

[2] https://cygwin.com/ml/cygwin/2011-02/msg00416.html

[3]
http://blogs.msdn.com/b/twistylittlepassagesallalike/archive/2011/04/23/everyone-quotes-arguments-the-wrong-way.aspx