User:Gszorc/Build System Experimentation: Difference between revisions

User:Gszorc/Build System Experimentation (view source)

Revision as of 01:42, 12 August 2011

5,509 bytes added , 12 August 2011

add some observations

Gszorc

canmove, Confirmed users

409

edits

@@ Line 16: / Line 16: @@
 This work can be found at https://github.com/indygreg/mozilla-central/tree/pymake_hacking.
+==Build System Observations==
+===Lots of Unnecessary Work===
+I would argue that in the ideal build system based on automake that the output of a simple ''make'' for an already completely built tree would only consist of the following messages:
+ Make: Entering directory A
+ Make: Entering directory B
+ Make: Nothing to be done for B
+ Make: Leaving directory B
+ Make: Leaving directory B
+Actually, a fully ideal make system would just say ''nothing to be done'' and exit. But, make's little brain can't grok the state of the world like that, since it has to parse Makefiles in other directories (which is what the ''Entering directory'' messages convey.
+Anyway, when you build mozilla-central on a tree that's already fully built, we see tons of other output. e.g.
+<pre>
+make.py[6]: Entering directory '/home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi'
+/home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi/Makefile:1035:0$ depbase=`echo src/debug.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
+/bin/bash ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi  -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src  -Wall -g -fexceptions  -O2  -MT src/debug.lo -MD -MP -MF $depbase.Tpo -c -o src/debug.lo ../../../../../js/src/ctypes/libffi/src/debug.c &&\
+mv -f $depbase.Tpo $depbase.Plo
+libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/debug.lo -MD -MP -MF src/.deps/debug.Tpo -c ../../../../../js/src/ctypes/libffi/src/debug.c  -fPIC -DPIC -o src/debug.o
+/home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi/Makefile:1035:0$ depbase=`echo src/prep_cif.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
+/bin/bash ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi  -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src  -Wall -g -fexceptions  -O2  -MT src/prep_cif.lo -MD -MP -MF $depbase.Tpo -c -o src/prep_cif.lo ../../../../../js/src/ctypes/libffi/src/prep_cif.c &&\
+mv -f $depbase.Tpo $depbase.Plo
+libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MT src/prep_cif.lo -MD -MP -MF src/.deps/prep_cif.Tpo -c ../../../../../js/src/ctypes/libffi/src/prep_cif.c  -fPIC -DPIC -o src/prep_cif.o
+/home/gps/src/mozilla-central/obj-ff-debug/js/src/ctypes/libffi/Makefile:1035:0$ depbase=`echo src/types.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
+/bin/bash ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../../../js/src/ctypes/libffi  -I. -I../../../../../js/src/ctypes/libffi/include -Iinclude -I../../../../../js/src/ctypes/libffi/src  -Wall -g -fexceptions  -O2  -MT src/types.lo -MD -MP -MF $depbase.Tpo -c -o src/types.lo ../../../../../js/src/ctypes/libffi/src/types.c &&\
+mv -f $depbase.Tpo $depbase.Plo
+</pre>
+Wha!? I just built the tree! y u repeat work?
+On Linux, if I trace pymake, I find that '''3037''' non-pymake.py commands were executed during make evaluation! A number are simple shell one-liners, like tests, which is expected, I suppose. Still, the majority seem to be commands with side-effects. This is bad, because it means redundant work.
+If we look at the targets evaluated during a build on a built tree, we notice that a number are evaluated more than once. Actually, '''14266''' are evaluated multiple times.
+The top 20 are:
+<pre>
+      /home/gps/src/mozilla-central/obj-ff-debug/../toolkit/toolkit-tiers.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../toolkit/mozapps/installer/package-name.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../testing/testsuite-targets.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../config/version.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../config/static-checking-config.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../config/rules.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../config/nspr/build.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/./config/myrules.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/./config/myconfig.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../config/js/build.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../config/config.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/./config/autoconf.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../build/binary-location.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../browser/build.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../browser/app-rules.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/../browser/app-config.mk
+      /home/gps/src/mozilla-central/obj-ff-debug/Makefile.in
+      /home/gps/src/mozilla-central/obj-ff-debug/Makefile
+       /home/gps/src/mozilla-central/obj-ff-debug/widget/src/gtkxtbinMakefile.in
+       /home/gps/src/mozilla-central/obj-ff-debug/widget/src/gtkxtbinMakefile
+</pre>
+What does redundant target evaluation mean? Well, I can't make any hard or fast rules because I believe some redundant evaluation will occur as a result of parallel evaluation. But, I can say that any time a target is processes more than once, there is some extra work involved.