Gaia/Build/RefactoringToNodejs

From MozillaWiki
< Gaia‎ | Build
Jump to: navigation, search

Overview

Our current build system comes with a Makefile, with each target took up to JavaScript scripts running in the Jetpack environment (XPCShell + CommonJS module loader). The JS runtime environment is quite unique and therefore comes with a few downsides:

  • A lot of OS-specific detail leaks through, making modification hard to debug
  • While providing a CommonJS interface, the environment does not come with common Node.js modules (e.g. fs, path, etc.), making reuse of code hard and creating new features harder.

In contrast, Node.js and the NPM packages runs on it can do things entirely simply in JavaScript. By substitute runtime dependency from XPCShell to Node.js, we would get a stable and common runtime environment, and gain access to NPM packages.

It's worthy to pointing out not building Gaia in XPCShell means we no longer considering making Firefox the sole dependency to build Gaia an eventual goal (i.e. develop Gaia with WebIDE & a "build Gaia" add-on and no extra dependency). Yet, considering the reward we decided using Node.js is the better way moving forward.

Profit from Node.js

  • Easier to catch up: Web developers are familiar with JavaScript, also Node.js is the most popular and acceptable for those who are familiar with client-side JavaScript. It could attract more contributors to involve and improve Firefox OS.
  • Abundant tool: NPM is a well-known build-in package manager of node, such tool is commonly used for web developers for years. It will make your life easier that developers can install third-party libraries and prevent us to reinvent the wheel if there is no such library provided by XULRunner.
  • Pure JS: It would be great if we build up a pure JS system. To re-implement Makefile into JS is a hard work without third-party library, NPM provides a bunch of packages that let us build our pure JS system more quickly, flexible and maintainable.

Plan

Node.js refactoring plan will be split into 4 milestones .

  • Developer:
    • Ricky Chien (MoCoTPE)
    • Scott Wu (MoCoTPE)
  • Total expected schedule: 18 - 22 weeks (5 months)

Milestone 0

To ensure our work works on all environments and operating systems, we would like to create test suites on Gaia-Try/Try/b2g-inbound, namely (Gbn/Gbun) tests. They will be keep hidden and red until M1 is complete, as a proof for achieving M1 (explaining below).

  • Deliverable: Enable test suites for node build script
  • Expected schedule: 2 week
  • Breakdowns:
ID Summary Priority Assigned to Status Resolution
1131469 Add RUN_ON_NODE switcher in Makefile -- Ricky Chien [:rickychien] RESOLVED FIXED
1207073 RUN_ON_NODE=1 make is broken -- Ricky Chien [:rickychien] RESOLVED FIXED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Milestone 1

Tackle each of the build system core functions (parallel build, webapp-manifests, preferences, webapp-optimize, webapp-zip... etc). All of them exist certain specific code only use for XPCShell. We would also like to finish the previous attempt to wrap Node.js APIs into high-level Utils in (build/node-utils.js) (See also: bug 955988). During this milestone, we will ensure the build system continue to work on both XPCShell setup and Node.js. Once we complete all the work for the milestone, Gbn and Gbun should be green and set visible.

  • Deliverable: Complete the implementation of node build scripts that coexist with XPCShell and passes Gbn / Gbun tests
  • Expected schedule: 4 - 6 weeks
  • Breakdowns:
ID Summary Priority Assigned to Status Resolution
955988 [meta] Running build scripts on node.js -- Ricky Chien [:rickychien] RESOLVED FIXED
955989 Running additional-extensions.js and download-manager.js on node.js P2 Ricky Chien [:rickychien] RESOLVED FIXED
955994 Running preferences.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
955996 Running svoperapps.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
955997 Running webapp-manifests.js on node.js P1 (Inactive after June) George Duan [:gduan] [:喬智] RESOLVED FIXED
955998 Running webapp-optimize.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
955999 Running webapp-zip.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1130934 Refactoring uitls.js, utils-node.js, utils-xpc.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131468 Running app.js and build-app.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131471 Enable Gbn & Gbun for Node.js build script P2 Ricky Chien [:rickychien] RESOLVED WONTFIX
1131496 Running build-app.js on node.js P1 Ricky Chien [:rickychien] RESOLVED DUPLICATE
1131497 Running build-test.js on node.js P2 Scott Wu [:scottwu] RESOLVED FIXED
1131499 Running clean-stage-app.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131500 Running contacts-import-services.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131501 Running copy-build-stage-data.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131502 Running copy-common-files.js on node.js P1 (Inactive after June) George Duan [:gduan] [:喬智] RESOLVED FIXED
1131503 Running csslint.js on node.js P2 Ricky Chien [:rickychien] RESOLVED FIXED
1131504 Running download-manager.js on node.js P2 Ricky Chien [:rickychien] RESOLVED DUPLICATE
1131505 Running homescreen-manager.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131506 Running import-config.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131510 Running settings.js on node.js P1 (Inactive after June) George Duan [:gduan] [:喬智] RESOLVED FIXED
1131511 Running jsonlint.js on node.js P2 Scott Wu [:scottwu] RESOLVED FIXED
1131512 Running keyboard-layouts.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131513 Running media-resolution.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131514 Remove deprecated watcher.js and monitor.js P3 Scott Wu [:scottwu] RESOLVED FIXED
1131515 Running multilocale.js on node.js P1 (Inactive after June) George Duan [:gduan] [:喬智] RESOLVED FIXED
1131516 Running post-app.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131517 Running post-manifest.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131518 Running pre-app.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131519 Running push-to-device.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131521 Running r-wrapper.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131522 Running rebuild.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131524 Running shared-utils.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1131526 Running webapp-shared.js on node.js P1 (Inactive after June) George Duan [:gduan] [:喬智] RESOLVED FIXED
1131527 Set tryserver Gbn / Gbun visible P2 Ricky Chien [:rickychien] RESOLVED WONTFIX
1138773 Running search-provider.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1240678 Running email build.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1240740 Running all app's build.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1242326 Running keyboard build.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1242327 Running operatorvariant build.js on node.js P1 Ricky Chien [:rickychien] RESOLVED FIXED
1243351 Running scan-appdir on node.js P2 Scott Wu [:scottwu] RESOLVED FIXED

41 Total; 0 Open (0%); 41 Resolved (100%); 0 Verified (0%);


Q & A

Which Gaia build task will continue to rely on b2g_sdk after Node.js transition?

b2g_sdk run on ...

  • Building Gaia (including sub-commands)
  • Running build integration test
  • Running push to device
  • Running marionette integration test (must)

All of the above can be replaced with node.js smoothly except marionette test. Since the B2G Desktop is the test target for these tests, it's impossible to remove from our dependency. However, it's not necessary to download a b2g_sdk package if someone do not execute marionette test.

What are the requirements removed/added of building Gaia before and after the transition?

  • Removed:
    • Build Gaia without downloading b2g (but we still need to download b2g while launching marionette tests)
  • Added:
    • Requiring Node.js as a build prerequisite -- User must make available node before building Gaia.
    • We will need to download NPM packages and other NPM command-line tools

Why are we giving up making Firefox as the only Gaia development dependency? Isn't that better and more inviting?

Weighting the benefits, we believe requiring Node.js won't frustrate many users to get started with hacking Gaia, since Node.js comes with installers for all platforms. Again, using Node.js means we can build up new features quickly by NPM and also speed up rewriting Makefile to pure JS.

On the other hand, try to rewrite Makefile from scratch in XULRunner is proven to be hard. In sum, we believe that NPM ecosystem will bring lots of benefits more than XULRunner.

How do we ensure NPM package access to all parties, e.g. partner in China or Try servers

It should continue to work if we keep hosting our gaia-node-modules mirror to solve such situation. Replacing the make-shift gaia-node-modules mirror is not the scope of this work, but we are looking forward to find a way to do that.

Buildbot should pre-install proper command tools (node.js, gulp) if we decide to use gulp, right?

Buildbot has already installed node environment except gulp. For gulp, we can pre-install gulp -g in buildbot, or download gulp while first time build then launch it by relative path.

Would the new build system slower than we have right now?

Although we believe that M1 & M2 won't effect obviously on performance, it probably happen on M3. So we should ensure that we take full advantage of multi-core systems to speed up our workflow in Node.js environment (with gulp).

Should we choose io.js instead of node.js for developing new build system?

In my experience, build scripts doesn't take too many ES6 features, may be able to pass "node --harmony" in node.js environment. So there are no such immediate reasons to choose io.js in current phase. However, io.js provides more complete ES6 features and we'll consider to transfer to io.js in the future. We would have to figure out if buildbots can work with multiple Node.js versions (maybe with nvm) before using io.js.

In short, targeting io.js is not the scope of the current plan.

File dependency resolution to improve incremental build

It would work by using existed gulp plugins (ex: gulp-resolve-dependencies) or adding new feature into plugins.

Task dependency and parallel build

Gulp is not as much convenience as Makefile and doesn't support parallel building natively. Therefore, although plugins could help us build up a same parallel architecture of our current build system, it’s hard to split every task into dependency tree to take full advantage of multi-core CPU (like "make -j8").

If we decide to build up our build system by GNU Make tool chain, considering to split up every task into small tasks would be very hard to implement. Furthermore, splitting into many small tasks mean that you have to deal with more I/O operations because you need to write more temporary file for communicating within each task, such thing will slow down performance

It would work by condition plugins for instance gulp-if, gulp-filter, gulp-ignore…, but introducing these plugins mean that lots of condition logic will pollute inside gulpfile.js. (Can we solve this problem in GNU make tool chain?) Repack zip could be achieved by adding a new feature in gulp-zip (Issue is filed gulp-zip/issues/45) and customize hasChanged comparator callback.