Gaia/Build/RefactoringToNodejs

Goal

Build system has run on XULRunner(XPCShell) since a long time ago, also requiring Make and shell script to shuffle, optimize or compress files around. However, all features are able to be done by Node.js so that everyone can benefit from the good part of Node.js. Our plan is rewrite/refactoring existed XULRunner-style JavaScript into Node.js.

Profit from Node.js

• Easier to catch up: Web developers are familiar with JavaScript, also Node.js is the most popular and acceptable for those who are familiar with client side JavaScript. It could attract more contributors to involve and improve Firefox OS.
• Abundant tool: NPM is a well-known build-in package manager of node, such tool is commonly used for web developers for years. It will make your life easier that developers can install third-party libraries and prevent us to reinvent the wheel if there is no such library provided by XULRunner.
• Pure JS: It would be great if we build up a pure JS system. To re-implement Makefile into JS is a hard work without third-party library, NPM provides a bunch of packages that let us build our pure JS system more quickly, flexible and maintainable.

Overviews

Our current build system comes with a Makefile front-end, and most logic are JavaScript running in the Jetpack environment (XPCShell + CommonJS module loader). The JS runtime environment is quite unique and therefore comes with a few downsides:

• A lot of OS-specific detail leaks through, making modification hard to debug
• While providing a CommonJS interface, the environment does not come with common NodeJS modules (e.g. fs, path, etc.), making reuse of code hard and creating new features harder.

It's therefore obvious to run build system with NodeJS instead. By substitute runtime dependency from XPCShell to NodeJS, we would get a stable and common runtime environment, and gain access to NPM packages.

It's worthy to pointing out not building Gaia in XPCShell means we no longer considering making Firefox the sole dependency to build Gaia an eventual goal (i.e. develop Gaia with WebIDE & a "build Gaia" add-on and no extra dependency). Yet, considering the reward we decided using NodeJS is the better way moving forward.

Plan

Node.js refactoring plan will be split into 4 milestones .

Milestone 0

To ensure our work works on all environments and operating systems, we would like to create test suites on Gaia-Try/Try/b2g-inbound, namely (Gbn/Gbun) tests. They will be keep hidden and red until M1 is complete, as a proof for achieving M1 (explaining below).

• Deliverable: Enable test suites for node build script
• Expected schedule: 2 week
• Breakdowns:
  • Bug ? - Add RUN_ON_NODE switcher in Makefile for switching xpcshell / node.js build script (Switch run-js-command or run-node-command in Makefile)
  • Bug ? - Enable Gbn & Gbun in Try-server (gaia-try / try / b2g-inbound) and set invisible until M1 complete

Milestone 1

Tackle each of the build system core functions (parallel build, webapp-manifests, preferences, webapp-optimize, webapp-zip... etc). All of them exist certain specific code only use for XPCShell. We would also like to finish the previous attempt to wrap NodeJS APIs into high-level Utils in (build/node-utils.js) (See also: bug 955988). During this milestone, we will ensure the build system continue to work on both XPCShell setup and NodeJS. Once we complete all the work for the milestone, Gbn and Gbun should be green and set visible.

• Deliverable: Complete the implementation of node build scripts that coexist with XPCShell and passes Gbn / Gbun tests
• Expected schedule: 4 - 6 weeks
• Breakdowns:
  • Bug ? - [Meta] Running build scripts on node.js
    • Bug ? - Complete utils-node.js
    • Bug ? - Running additional-extensions.js on node.js
    • Bug ? - Running app.js on node.js
    • Bug ? - Running build-app.js on node.js
    • Bug ? - Running build-test.js on node.js
    • Bug ? - Running clean-stage-app.js on node.js
    • Bug ? - Running contacts-import-services.js on node.js
    • Bug ? - Running copy-build-stage-data.js on node.js
    • Bug ? - Running copy-common-files.js on node.js
    • Bug ? - Running csslint.js on node.js
    • Bug ? - Running csslinter.js on node.js
    • Bug ? - Running download-manager.js on node.js
    • Bug ? - Running homescreen-manager.js on node.js
    • Bug ? - Running import-config.js on node.js
    • Bug ? - Running jsmin.js on node.js
    • Bug ? - Running jsonlint.js on node.js
    • Bug ? - Running keyboard-layouts.js on node.js
    • Bug ? - Running l10n.js on node.js
    • Bug ? - Running media-resolution.js on node.js
    • Bug ? - Running monitor.js on node.js
    • Bug ? - Running multilocale.js on node.js
    • Bug ? - Running optimize-clean.js on node.js
    • Bug ? - Running post-app.js on node.js
    • Bug ? - Running post-manifest.js on node.js
    • Bug ? - Running pre-app.js on node.js
    • Bug ? - Running preferences.js on node.js
    • Bug ? - Running push-to-device.js on node.js
    • Bug ? - Running r-wrapper.js on node.js
    • Bug ? - Running rebuild.js on node.js
    • Bug ? - Running settings.js on node.js
    • Bug ? - Running shared-utils.js on node.js
    • Bug ? - Running svoperapps.js on node.js
    • Bug ? - Running utils-node.js on node.js
    • Bug ? - Running utils-xpc.js on node.js
    • Bug ? - Running utils.js on node.js
    • Bug ? - Running watcher.js on node.js
    • Bug ? - Running webapp-manifests.js on node.js
    • Bug ? - Running webapp-optimize.js on node.js
    • Bug ? - Running webapp-shared.js on node.js
    • Bug ? - Running webapp-zip.js on node.js
    • Bug ? - Set Gbn / Gbnu visible

Milestone 2 (Draft)

The M2 goal is to simplify Makefile into merely a compatibility layer for the existing commands, and move all the logic into external JavaScript-based build tools. By doing so it would improve the maintainability and readability of the build system. We tentatively choose Gulp.js as target currently as it offers incremental builds and stream workflow.

• Deliverable: Purge complex logic from Makefile and rely on an external JavaScript-based build tools for most of the complexities.
• Expected schedule: TBD
• Breakdowns: TBD

Milestone 3 (Draft)

Milestone: Turn on node build script by default and remove all legacy APIs.

Schedule: 2 sprints

Until stage 1 and stage 2 are stable, we're going to enable node build script by default and remove all legacy APIs.

To do list

• Turn on node build script by default
• Refactor utils-node.js API and remove utils-xpc.js and utils.js
• Refactor all build/*.js to conform to node-style

Q & A

1. Which Gaia build task will continue to rely on b2g_sdk after NodeJS transition?

   b2g_sdk run on ...

   • Building Gaia (including sub-commands)
   • Running build integration test
   • Running push to device
   • Running marionette integration test (must)

   All of the above can be replaced with node.js smoothly except marionette test. Since the B2G Desktop is the test target for these tests, it's impossible to remove from our dependency. However, it's not necessary to download a b2g_sdk package if someone do not execute marionette test.

2. What of the effect on Gaia if build script run on node.js instead of b2g_sdk?

   Pros:

   • Build Gaia without downloading b2g (but we still download b2g while launching marionette tests)

   Cons:

   • We need to download NPM packages and pre-install node.js or other NPM command-line tools (probably be gulp?)

3. WebIDE is trying to make developing Gaia more easily by clicking a button without any command line tricks. It triggers building process by invoking "make" currently. In XULRunner build system, after rewriting Makefile to pure JS module to take over it's job, WebIDE can build Gaia without any dependencies. (such as make and node)

   For node build system, requiring Node.js may be a little bit inconvenient. However, it won't frustrate many users to get started with hacking Gaia since Node provides a great installer for all platforms. After Node.js installed, we can build up new features quickly by NPM and also speed up rewriting Makefile to pure JS. On the other hand, try to rewrite Makefile from scratch in XULRunner seems not so easy. I'm pretty sure and still believe that NPM ecosystem will bring lots of benefits more than XULRunner.

4. Try-server or some countries have network issues when pulling packages from NPM.

   It should work if we keep hosting our gaia-node-modules mirror to solve such situation.

5. In stage 2, buildbot should pre-install proper command tools (node.js, gulp) if we decide to use gulp.js

   Buildbot has already installed node environment except gulp. We can pre-install gulp -g in buildbot, or download gulp while first time build then launch it by relative path.

6. We might take performance into account, the new build system shouldn't slower than the current.

   Although I believe that stage 1 & 3 won't effect obviously on performance, it probably happen on stage 2. So we should ensure that we take full advantage of multi-core systems to speed up our workflow in node.js environment (in gulp.js).