Build config/Projects: Difference between revisions

MozillaBuild is the Windows development environment. We have a few projects for it.

MozillaBuild 1.9.0 with GNU Make 4 should be released.

Tracked by {{bug|928594}}.

*MozillaBuild 2.0* will be an overhaul of MozillaBuild to use msys2, a decent terminal, and a bunch of other improvements.

See https://groups.google.com/d/msg/mozilla.dev.builds/X760-Y4DSRc/LTlo39uNKSEJ

These are projects that relate to how the core build system interacts with automation.

We want all in-tree calls in automation to be proxied through well-defined points so we have a clearly supported API.

{{bug|978211}} tracks.

Automation should use GNU Make 4.0 on Windows.

A follow-up goal is likely to remove code for supporting pymake and eventually pymake itself from the tree.

This  is a request from Release Engineering. For capacity reasons, we  want  to perform Mac builds on Linux. We should add support for this.  

{{bug|927061}}

{{bug|928196}}

This work will enable Tup, Ninja, etc building.

We have a number of build actions that run unconditionally, either due to clownshoes (like removing large parts of dist/) or due to bad build rules. These build actions add excessive overhead to light builds. Light builds are so slow that developers commonly perform a partial tree build and hope they build everything they needed to. This is playing with fire and this practice needs to be curtailed through us providing faster light builds.  

Progress towards this goal is measured by comparing the number of  non-make processes launched during no-op and light builds over time. This number needs to go down. Drastically.  

ICU  has been a world of pain for the build system. It makes builds significantly slower and it is often the long pole, especially for builds close to the no-op side of the spectrum.  

We need to mitigate the impact of ICU on the build system. This might mean loudly recommending people not build it (although more and more  features want to start using it, so likely not a long term solution). This might mean someone contributing patches back to ICU to make its build system suck less.

The  way SpiderMonkey integrates with the build system is very hacky.  Nobody likes it - including the SpiderMonkey developers. Furthermore, the way things are slows down builds. e.g. SpiderMonkey has its own  configure script, which is extremely similar to the one in m-c. This slows down configure time, especially on Windows.  

The  goal here is to integrate SpiderMonkey into the main build system. We'll also provide mach command(s) for building just SpiderMonkey (they've asked me for this).  

The Android make rules are horrible. They are slow and do a lot of actions when not needed. There is much room for improvement here.

Our current Python unit testing solution is quite fragile. Failures will result in all tests being aborted. We don't get as much context from failures as we'd like. It's somewhat difficult to run specific tests in isolation (not whole test files).

This will likely involve defining Python unit tests in moz.build files somehow. We can probably define things as a directory containing test files plus an optional regex used to discover them. Generally speaking, nose's algorithm for finding tests is pretty robust. Worst case, we can adapt our existing tests that don't conform to its expectations. This is probably a good idea anyway.

client.mk is a glorified shell script. The way it interacts with the world (especially mozconfigs) is very hacky. client.mk's days are numbered. Everything it does should be replaced with mach commands.

This requires release automation to not use client.mk (another project).

We want to be able to build the tree with Tup, not make.

Benefits: Faster and proper builds. Eliminate clobbers.

We would support this as a local development mode first then move to automation when it is stable.

There is a longstanding desire for mach to support config files and for some aspects of mozconfigs to be reinvented. These can be independent, but they may overlap.

There are many options being landed in mach commands that are effectively user preferences. There is no good way to define sticky  defaults for those options. mach needs config files so people can tailor their environments to their liking. We should add config file support to mach.

Mach has become slower as more commands are added. We should look at making it faster.  

Support for detecting what commands are runnable at run-time has been added recently. When you run |mach help| it even prunes the list to commands that should work. We should hook up most of the commands to this mechanism so people can't do things like attempt to run tests when the tree isn't built.

Mach integration with comm-central and b2g is fragile. We should make that work so we don't have to continue supporting legacy interfaces and so developers there don't complain so much about not being able to use mach. This is more important than ever since functionality like test running is starting to require mach.

In theory we can provide a build mode so developers can build just the browser app without having to build all of Gecko/libxul. This will dramatically reduce the horsepower required to obtain a working build from source.

{{bug|901840}}

Release engineering is now publishing archives of their chroot environments at https://s3.amazonaws.com/mozilla-releng-mock-archive/mock_archives.json. It would be rad if someone hooked this up to mach and the build system so local builds could be performed in containers using the exact same bits that automation uses.

It's always nice to make data-driven decisions. Currently, we have no reliable way of knowing what hardware people are building on, what mach commands they use, which parts of the tree are getting built the most locally, etc.

While other companies that do this kind of instrumentation can get away with having it enabled by default because all developers are under the corporate banner and thus not susceptible to privacy concerns because they signed them away as part of employment, Mozilla can't. Anyone can download the source code and we can't have things reporting automatically without notification. It's likely not legal and it's definitely not the Mozilla way. However, for this data to be useful and not skewed, we'd want near 100% participation. We obviously need to work with privacy and legal to flush out how this would work.