Gaia/Build/NewBuildSystem
Overview
After surveying, we decide to split build system into 2 steps: configure step and build step [1]. For detailed information, please visit change proposal in dev-gaia Update on Gaia build changes: A configure step, and Make v.s. Gulp.
New architecture plan is going to construct a powerful configure to generate an appropriate Makefile for build back-end as know as build step to consume. Thus, GNU Make will be adopted as our firstly choose build back-end tool. We're going to reference the architecture of CMake, they also have adopted the concept of splitting configure step and build step early and it's a fully fledged tool already have good solution to build up a powerful configure such as detecting the changing of ENV variable and computing those complicated file dependency to decide whether to re-generate Makefile or not.
Learning from CMake could be a huge help for us mitigate circumstances while we're fighting in new architecture transition. A study of CMake is ongoing within our implementation phase, and moreover we are trying to dig into CMake from it's source https://github.com/Kitware/CMake and play with those popular CMake projects such as opencv or libwebsockets
Our goal is to fasten incremental build and easier to customize and manage new/current building step. However, new architecture would be slow down clobber build which means first time build would take more time than old build system. We still believe that the only thing developers are concerned is the time of incremental build . As a result, the benefit of incremental build improvement is more importantly than clobber build. All of above glossary can be referred to Gecko build system document Glossary.
The architecture of configure:
Configure
This plan focuses on constructing a powerful configure, it will separate current modules such as webapp-shared, webapp-optimiz, multilocale...etc to configure and builder.
Build config / Environment variable manager
A manager to provide an elegant way to handle build config.
Cache configure
A cache mechanism to keep the status of last time configuring and speed up re-configure process. The configure-decider could take responsibility of keeping these status.
Plan
- Developer:
- Ricky (MoCoTPE) (irc #gaia: rickychien)
- George (MoCoTPE) (irc #gaia: gduan)
- Total expected schedule: 10 - 12 weeks
Milestone 1
Initial landing step. Since our new build system plan is around splitting a build process into two steps including configuring and building. This stage will focus on building up a configuration modules - configure.js and using GNU make as our build back-end to produce final target. It means configure.js part will run on XPCShell and require existed build modules at this stage in order to keep build system working properly. Finally, the new build system is going to be landed in code-base and replace current build system.
- Deliverable: A new Gaia Makefile target to trigger configure.js modules should be landed in Gaia code base. The configure.js can generate an all.mk (a temporary new Makefile) and produce a same PROFILE_DIR as usual.
- Expected schedule: 4 weeks
- Breakdowns:
- New config step: adding configure.js to generate strong dependency all.mk and execute it.
- New re-run step: in 2nd make, new build will check the change files and ENV to decide whether to regenerate all.mk or not, and then execute it.
- Refactoring webapp-shared.js and copyToStage: based on new build system architecture, we'll refactor webapp-shared module and copyToStage into config and execute task in order to save time when re-run make and check if any possible risk before Milestone 2 (refactoring all build modules and app/build.js).
| ID | Summary | Assigned to | Status | Resolution |
|---|---|---|---|---|
| 1154582 | Build system configure - adding configure.js to generate strong dependency all.mk and execute it | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | FIXED |
| 1154583 | Configure re-run step - configure.js check the change files and ENV to decide whether to regenerate all.mk or not | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | WONTFIX |
| 1154585 | Refactoring webapp-shared.js for new build system | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Milestone 2
After initial landing step being done, we should abandon GNU make's PHONY targets as many as possible, so we could have a faster build by taking the advantage of GNU make's incremental build. The configure.js should be able to connect with different build back-end. In order to achieve this aim, we eventually come out a new phase - config.status [2] for preparing the next step (replacing build back-end). The idea of config.status is charged with the task of tell back-end tool how to build the tree which includes scanning the build system definition and constructing a DAG data structure for representing task dependency and supporting parallel build natively. It means config.status will be generated by configure.js and output a build-file for back-end consumption (e.g. a `make` back-end will write a `Makefile`).
- Deliverable: Abandon GNU make's PHONY targets as many as possible to speed up incremental build and implement config.status phase.
- Expected schedule: 6 - 8 weeks
- Breakdowns:
- Refactoring all modules and app/build.js into small tasks and split each module into configuring + building steps.
- Implement config.status which will introduce DAG structure to configure step.
| ID | Summary | Assigned to | Status | Resolution |
|---|---|---|---|---|
| 1154586 | [meta] Refactoring all modules and app/build.js into small tasks and split each module into configuring + building steps | RESOLVED | INVALID | |
| 1154587 | Implement config.status which will introduce DAG structure to configure step | RESOLVED | WONTFIX |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Milestone 3
We're going to measure incremental build performance from GNU make back-end. If the result is as good as our expectation, the decision of choosing build back-end would be GNU make. Otherwise, the new milestone would be targeting on constructing the new build back-end tool. After surveying in milestone 3 of previous plan RefactoringToNode, we believe that the Gulp or streaming base build back-end could offer a better performance especially in clobber build. Therefore, in this step, a streaming base build system which will be implemented.
- Deliverable: A blazing fast JavaScript build system
- Expected schedule: ? weeks
- Breakdowns:
- Implement a JavaScript build system as our build back-end tool instead of GNU make
Q & A
Why new build system is quicker than current one?
- Clobber build
- Current build: It manages process runner by app, which means the there would be idle processes if the last app takes longer time to complete.
- New build: It manages process runner by file. The idle processes won't remain too long until all tasks complete.
- Incremental build
- Current build: If one file has changed, it will rebuild all the app (see below).
- New build: If one file has changed, it will only trigger all the related tasks (see below).
