User:Gszorc/Build System Experimentation
The Mozilla build system, specifically all the makefiles and the things they do, is scary. I have a personal project (which will inevitably bleed into bugs and patches) to improve it any way I can.
Impetus
User:ally and I were opining how developing mozilla-central on Windows isn't very Windows-y. Specifically, there is no official Visual Studio story. Windows developers love Visual Studio and it is really unfortunate we can't give them what they love. So, all the work on this page started as a way to explore generating Visual Studio project files automatically with the goal of eventually bundling them in the source tree (or at least providing a way for them to be generated automatically on checkout).
PyMake Tracing
I've implemented tracing to PyMake. When you construct a Makefile instance in PyMake, you can supply an object which will receive callbacks when various build events occur, such as:
- Start/end of new Makefile processing
- Start/end of target processing
- Start/end of command execution
These callbacks all pass information about the object they are operating on.
I've implemented a basic tracing logger and integrated it with the PyMake make.py driver. It takes the callback data, converts it to JSON, and writes it to a file. This stream contains very detailed records, including dependencies and variables. From that stream, we should be able to perform magic not previously conceivable.
This work can be found at https://github.com/indygreg/mozilla-central/tree/pymake_hacking.
Build System Observations
Lots of Unnecessary Work
I would argue that in the ideal build system based on automake that the output of a simple make for an already completely built tree would only consist of the following messages:
Make: Entering directory A Make: Entering directory B Make: Nothing to be done for B Make: Leaving directory B Make: Leaving directory B
Actually, a fully ideal make system would just say nothing to be done and exit. But, make's little brain can't grok the state of the world like that, since it has to parse Makefiles in other directories (which is what the Entering directory messages convey.
Anyway, when you build mozilla-central on a tree that's already fully built, we see tons of other output. e.g.
make.py[6]: Entering directory '/home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi' /home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi/Makefile:1035:0$ depbase=`echo src/debug.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\ /bin/bash ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/debug.lo -MD -MP -MF $depbase.Tpo -c -o src/debug.lo ../../../../../js/src/ctypes/libffi/src/debug.c &&\ mv -f $depbase.Tpo $depbase.Plo libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/debug.lo -MD -MP -MF src/.deps/debug.Tpo -c ../../../../../js/src/ctypes/libffi/src/debug.c -fPIC -DPIC -o src/debug.o /home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi/Makefile:1035:0$ depbase=`echo src/prep_cif.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\ /bin/bash ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/prep_cif.lo -MD -MP -MF $depbase.Tpo -c -o src/prep_cif.lo ../../../../../js/src/ctypes/libffi/src/prep_cif.c &&\ mv -f $depbase.Tpo $depbase.Plo libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/prep_cif.lo -MD -MP -MF src/.deps/prep_cif.Tpo -c ../../../../../js/src/ctypes/libffi/src/prep_cif.c -fPIC -DPIC -o src/prep_cif.o /home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi/Makefile:1035:0$ depbase=`echo src/types.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\ /bin/bash ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/types.lo -MD -MP -MF $depbase.Tpo -c -o src/types.lo ../../../../../js/src/ctypes/libffi/src/types.c &&\ mv -f $depbase.Tpo $depbase.Plo
Wha!? I just built the tree! y u repeat work?
On Linux, if I trace pymake, I find that 3037 non-pymake.py commands were executed during make evaluation! A number are simple shell one-liners, like tests, which is expected, I suppose. Still, the majority seem to be commands with side-effects. This is bad, because it means redundant work.
If we look at the targets evaluated during a build on a built tree, we notice that a number are evaluated more than once. Actually, 14266 are evaluated multiple times.
The top 20 are:
36 /home/gps/src/mozilla-central/obj-ff-debug/../toolkit/toolkit-tiers.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../toolkit/mozapps/installer/package-name.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../testing/testsuite-targets.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../config/version.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../config/static-checking-config.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../config/rules.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../config/nspr/build.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/./config/myrules.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/./config/myconfig.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../config/js/build.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../config/config.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/./config/autoconf.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../build/binary-location.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../browser/build.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../browser/app-rules.mk 36 /home/gps/src/mozilla-central/obj-ff-debug/../browser/app-config.mk 35 /home/gps/src/mozilla-central/obj-ff-debug/Makefile.in 35 /home/gps/src/mozilla-central/obj-ff-debug/Makefile 6 /home/gps/src/mozilla-central/obj-ff-debug/widget/src/gtkxtbinMakefile.in 6 /home/gps/src/mozilla-central/obj-ff-debug/widget/src/gtkxtbinMakefile
What does redundant target evaluation mean? Well, I can't make any hard or fast rules because I believe some redundant evaluation will occur as a result of parallel evaluation. But, I can say that any time a target is processes more than once, there is some extra work involved.