Gaia/Build/RefactoringToNodejs

Goal

Build system has run on XULRunner(XPCShell) since a long time ago, also requiring Make and shell script to shuffle, optimize or compress files around. However, all features are able to be done by Node.js so that everyone can benefit from the good part of Node.js. Our plan is rewrite/refactoring existed XULRunner-style JavaScript into Node.js.

Profit from Node.js

• Easier to catch up: Web developers are familiar with JavaScript, also Node.js is the most popular and acceptable for those who are familiar with client side JavaScript. It could attract more contributors to involve and improve Firefox OS.
• Abundant tool: NPM is a well-known build-in package manager of node, such tool is commonly used for web developers for years. It will make your life easier that developers can install third-party libraries and prevent us to reinvent the wheel if there is no such library provided by XULRunner.
• Pure JS: It would be great if we build up a pure JS system. To re-implement Makefile into JS is a hard work without third-party library, NPM provides a bunch of packages that let us build our pure JS system more quickly, flexible and maintainable.

Plan

Node.js refactoring plan will split into 4 stages.

Stage 0

Milestone: Enable test suites for node build script

Schedule: 1 sprint

For finding out potential risks during node refactoring, we're going to enable build test suites for node.js beforehand, namely (Gbn/Gbun) in Treeherder and target on gaia-try, try and b2g-inbound. After that, the light of Gbn and Gbun will keep red for a while until stage 1 is finished.

To do list

• Add RUN_ON_NODE switcher in Makefile for switching node.js build script (Switch run-js-command to run-node-command in Makefile)
• Enable Gbn & Gbun in Treeherder (gaia-try / try / b2g-inbound) and set invisible until stage 1 complete

Stage 1

Milestone: Complete the implementation of node build scripts that coexist with XPCShell and pass Gbn / Gbun tests (2 sprints)

Schedule: 2 ~ 3 sprints

Focus on refactoring build system core functionality (parallel build, webapp-manifests, preferences, webapp-optimize, webapp-zip... etc). All of them exist certain specific code only use for XPCShell. Unfinished Node.js build script (build/node-utils.js) is already located in Gaia for a while (See also: bug 955988. We should keep working on it and keep both Node and XPCShell compatible in build system. After that, Gbn and Gbun should be green and set visible.

To do list

• [Meta] Running build scripts on node.js
  • Complete utils-node.js
  • Running additional-extensions.js on node.js
  • Running app.js on node.js
  • Running build-app.js on node.js
  • Running build-test.js on node.js
  • Running clean-stage-app.js on node.js
  • Running contacts-import-services.js on node.js
  • Running copy-build-stage-data.js on node.js
  • Running copy-common-files.js on node.js
  • Running csslint.js on node.js
  • Running csslinter.js on node.js
  • Running download-manager.js on node.js
  • Running homescreen-manager.js on node.js
  • Running import-config.js on node.js
  • Running jsmin.js on node.js
  • Running jsonlint.js on node.js
  • Running keyboard-layouts.js on node.js
  • Running l10n.js on node.js
  • Running media-resolution.js on node.js
  • Running monitor.js on node.js
  • Running multilocale.js on node.js
  • Running optimize-clean.js on node.js
  • Running post-app.js on node.js
  • Running post-manifest.js on node.js
  • Running pre-app.js on node.js
  • Running preferences.js on node.js
  • Running push-to-device.js on node.js
  • Running r-wrapper.js on node.js
  • Running rebuild.js on node.js
  • Running settings.js on node.js
  • Running shared-utils.js on node.js
  • Running svoperapps.js on node.js
  • Running utils-node.js on node.js
  • Running utils-xpc.js on node.js
  • Running utils.js on node.js
  • Running watcher.js on node.js
  • Running webapp-manifests.js on node.js
  • Running webapp-optimize.js on node.js
  • Running webapp-shared.js on node.js
  • Running webapp-zip.js on node.js

Stage 2 (Draft)

Milestone: Using NPM modules to build up a pure JS build system instead of Makefile.

Schedule: 4 sprints

In stage 2, we're going to rewrite Makefile into JS module. Currently, we'll choose Gulp.js for our new build system. In order to rewrite Makefile into JS module, Gulp's incremental builds and stream workflow seems to meet our needs. I'm looking forward to seeing a readable and maintainable gulpfile.js in Firefox OS.

Stage 3 (Draft)

Milestone: Turn on node build script by default and remove all legacy APIs.

Schedule: 2 sprints

Until stage 1 and stage 2 are stable, we're going to enable node build script by default and remove all legacy APIs.

To do list

• Turn on node build script by default
• Refactor utils-node.js API and remove utils-xpc.js and utils.js
• Refactor all build/*.js to conform to node-style

Q & A

• Which Gaia components rely on b2g_sdk?

b2g_sdk run on ...

• Building Gaia (including sub-commands)
• Running build integration test
• Running push to device
• Running marionette integration test (must)

All of the above can be replaced with node.js smoothly except marionette test. Since marionette must be run on b2g simulator, it's impossible to remove from our dependency. However, we're not necessary to download a b2g_sdk package if someone do not execute marionette test.

• WebIDE is trying to make developing Gaia more easily by clicking a button without any command line tricks. It triggers building process by invoking "make" currently. In XULRunner build system, after rewriting Makefile to pure JS module to take over it's job, WebIDE can build Gaia without any dependencies. (such as make and node)

For node build system, requiring Node.js may be a little bit inconvenient. However, it won't frustrate many users to get started with hacking Gaia since Node provides a great installer for all platforms. After Node.js installed, we can build up new features quickly by NPM and also speed up rewriting Makefile to pure JS. On the other hand, try to rewrite Makefile from scratch in XULRunner seems not so easy. I'm pretty sure and still believe that NPM ecosystem will bring lots of benefits more than XULRunner.

• Treeherder or some countries have network issues when pulling packages from NPM.

It should work if we keep hosting our gaia-node-modules mirror to solve such situation.

• In stage 2, buildbot should pre-install proper command tools (node.js, gulp) if we decide to use gulp.js

Buildbot has already installed node environment except gulp. We can pre-install gulp -g in buildbot, or download gulp while first time build then launch it by relative path.