Gaia/Build/RefactoringToNodejs: Difference between revisions

 
(45 intermediate revisions by 2 users not shown)
Line 21: Line 21:


* '''Developer:'''
* '''Developer:'''
** Ricky (MoCoTPE) (irc #gaia: rickychien)
** Ricky Chien (MoCoTPE)
** George (MoCoTPE) (irc #gaia: gduan)
** Scott Wu (MoCoTPE)


* '''Total expected schedule:''' 18 - 22 weeks (5 months)
* '''Total expected schedule:''' 18 - 22 weeks (5 months)
Line 33: Line 33:
* '''Expected schedule:''' 2 week
* '''Expected schedule:''' 2 week
* '''Breakdowns:'''
* '''Breakdowns:'''
** {{Bug|1131469}} - Add RUN_ON_NODE switcher in Makefile for switching xpcshell / node.js build script (Switch run-js-command or run-node-command in Makefile)
 
** {{Bug|1131471}} - Enable Gbn & Gbun for all platform including Linux/Mac/Windows in Try-server (gaia-try / try / b2g-inbound) and set invisible until M1 complete
<bugzilla>
{
  "id": "1131469, 1207073",
  "include_fields": "id, summary, priority, assigned_to, status, resolution"
}
</bugzilla>


== Milestone 1 ==
== Milestone 1 ==
Line 43: Line 48:
* '''Expected schedule:''' 4 - 6 weeks
* '''Expected schedule:''' 4 - 6 weeks
* '''Breakdowns:'''
* '''Breakdowns:'''
** {{Bug|955988}} - [Meta] Running build scripts on node.js
*** {{Bug|1130934}} - Refactoring uitls.js, utils-node.js, utils-xpc.js
*** {{Bug|955989}} - Running additional-extensions.js on node.js
*** {{Bug|1131468}} - Running app.js on node.js
*** {{Bug|1131496}} - Running build-app.js on node.js
*** {{Bug|1131497}} - Running build-test.js on node.js
*** {{Bug|1131499}} - Running clean-stage-app.js on node.js
*** {{Bug|1131500}} - Running contacts-import-services.js on node.js
*** {{Bug|1131501}} - Running copy-build-stage-data.js on node.js
*** {{Bug|1131502}} - Running copy-common-files.js on node.js
*** {{Bug|1131503}} - Running csslint.js and csslinter.js on node.js
*** {{Bug|1131504}} - Running download-manager.js on node.js
*** {{Bug|1131505}} - Running homescreen-manager.js on node.js
*** {{Bug|1131506}} - Running import-config.js on node.js
*** {{Bug|1131510}} - Running settings.js on node.js
*** {{Bug|1131511}} - Running jsonlint.js on node.js
*** {{Bug|1131512}} - Running keyboard-layouts.js on node.js
*** {{Bug|1131513}} - Running media-resolution.js on node.js
*** {{Bug|1131514}} - Running monitor.js on node.js
*** {{Bug|1131515}} - Running multilocale.js and l10n.js on node.js
*** {{Bug|1131516}} - Running post-app.js on node.js
*** {{Bug|1131517}} - Running post-manifest.js on node.js
*** {{Bug|1131518}} - Running pre-app.js on node.js
*** {{Bug|955994}} - Running preferences.js on node.js
*** {{Bug|1131519}} - Running push-to-device.js on node.js
*** {{Bug|1131521}} - Running r-wrapper.js on node.js
*** {{Bug|1131522}} - Running rebuild.js on node.js
*** {{Bug|1131524}} - Running shared-utils.js on node.js
*** {{Bug|955996}} - Running svoperapps.js on node.js
*** {{Bug|1131525}} - Running watcher.js on node.js
*** {{Bug|955997}} - Running webapp-manifests.js on node.js
*** {{Bug|955998}} - Running webapp-optimize.js on node.js
*** {{Bug|1131526}} - Running webapp-shared.js on node.js
*** {{Bug|955999}} - Running webapp-zip.js on node.js
*** {{Bug|1131527}} - Set tryserver Gbn / Gbun visible


<bugzilla>
{
  "id": "955988, 1130934, 955989, 1131468, 1131496, 1131497, 1131499, 1131500, 1131501, 1131502, 1131503, 1131504, 1131505, 1131506, 1131510, 1131511, 1131512, 1131513, 1131514, 1131515, 1131516, 1131517, 1131518, 955994, 1131519, 1131521, 1131522, 1131524, 955996, 955997, 955998, 1131526, 955999, 1131527, 1138773, 1131471, 1240678, 1240740, 1242326, 1242327, 1243351",
  "include_fields": "id, summary, priority, assigned_to, status, resolution"
}
</bugzilla>
<!--
== Milestone 2 ==
== Milestone 2 ==


Line 86: Line 64:
** Keep tracking build system status and fix any regression.
** Keep tracking build system status and fix any regression.
** Bug ? - Turn on node build script by default
** Bug ? - Turn on node build script by default
* Backlog:
** Bug ? - Refactor utils-node.js API and remove utils-xpc.js and utils.js
** Bug ? - Refactor utils-node.js API and remove utils-xpc.js and utils.js
** Bug ? - Refactor all build/*.js to conform to node-style
** Bug ? - Refactor all build/*.js to conform to node-style
 
-->
== Milestone 3 (Draft) ==
 
The M3 goal is to simplify make Makefile into merely a compatibility layer for the existing commands, and move all the logic into external JavaScript-based build tools. By doing so it would improve the maintainability and readability of the build system. We tentatively choose [http://gulpjs.com/ Gulp.js] as target currently as it offers [https://github.com/gulpjs/gulp#incremental-builds incremental builds] and stream workflow.
 
* '''Deliverable:''' Purge complex logic from Makefile and rely on an external JavaScript-based build tools for most of the complexities.
* '''Expected schedule:''' 8 weeks
* '''Breakdowns:'''
** Extract Makefile logic into JS modules
** Rewrite Makefile targets into gulp tasks
** Rewrite parallel build
*** [https://www.npmjs.com/package/gulp gulp]
*** [https://www.npmjs.com/package/gulp-spawn gulp-spawn]
** Building parameters management
*** [https://www.npmjs.com/package/parse-env parse-env]
*** [https://www.npmjs.com/package/config-manager config-manager]
** Design per app build config (?)
* '''Reference:'''
** https://github.com/fxos/build


= Q & A =
= Q & A =
Line 152: Line 113:


In short, targeting io.js is not the scope of the current plan.
In short, targeting io.js is not the scope of the current plan.
== File dependency resolution to improve incremental build ==
It would work by using existed gulp plugins (ex: gulp-resolve-dependencies) or adding new feature into plugins.
== Task dependency and parallel build ==
Gulp is not as much convenience as Makefile and doesn't support parallel building natively. Therefore, although plugins could help us build up a same parallel architecture of our current build system, it’s hard to split every task into dependency tree to take full advantage of multi-core CPU (like "make -j8").
== If we decide to build up our build system by GNU Make tool chain, considering to split up every task into small tasks would be very hard to implement. Furthermore, splitting into many small tasks mean that you have to deal with more I/O operations because you need to write more temporary file for communicating within each task, such thing will slow down  performance ==
It would work by condition plugins for instance gulp-if, gulp-filter, gulp-ignore…, but introducing these plugins mean that lots of condition logic will pollute inside gulpfile.js. (Can we solve this problem in GNU make tool chain?)
Repack zip could be achieved by adding a new feature in gulp-zip (Issue is filed [https://github.com/sindresorhus/gulp-zip/issues/45 gulp-zip/issues/45]) and customize [https://www.npmjs.com/package/gulp-changed#haschanged hasChanged] comparator callback.
Confirmed users
429

edits