Building Firefox/SURF

This page is to serve as a guide for interested researchers (primarily academic researchers) who would like to experiment with Firefox for research purposes. Such experiments might include:

• Developing new compiler features, performance improvements, or security mechanisms
• Altering the behavior of the browser with respect to the web platform
• Creating a web scanning platform based on the browser (similar to OpenWPM)

Or something entirely different. Be creative.

Building Firefox Locally

The first step will almost certainly be to build the browser locally. Firefox has a well maintained mechanism for obtaining the dependencies needed to build it, even on Windows. Follow the Developer guide at MDN to build Firefox for the first time locally.

Once you've built it, use ./mach run to start the browser and take it for a spin.

Running Tests (Locally)

./mach test path/to/test will run most types of tests; for example ./mach test dom/security/test/csp/test_upgrade_insecure.html

You can add --headless (after 'test') to run them without launching the browser window. This is useful when working on a remote server. Note however that while --headless works most of the time, if a test fails to run, it is worth re-running it without --headless in a GUI environment.

gtests are run differently, for example ./mach gtest "*ReducePrecision_*" to run tests with 'ReducePrecision_' in their name (which would be tests like these).

While you can use ./mach test mochitest to run all mochitests; or ./mach test to run all tests - doing this locally is going to produce false positives and generally be unpleasant. Instead the next step is to move to try.

Building and Testing in Mozilla Infrastructure

Note that all try runs are public, and presently Mozilla does not have a mechanism to do private builds or test runs.

The TryServer is a way to submit a one or more patches and have them built on multiple Operating Systems, run tests, and download builds and logs. Because the entire build system is specified in the mozilla-central tree as configuration files it is possible to write a patch that changes the compiler used (either to a different version, or actually patching the compiler), have the tryserver check out the compiler source from a remote repository, build the compiler, and then use the compiler to build Firefox. All in the cloud. https://wiki.mozilla.org/ReleaseEngineering/TryServer has more information about the TryServer, but we will aim to have a functional quickstart tutorial in this document.

Getting Access

Before submitting to try, you will need to contact SURF at the email surf@mozilla.com and provide us with some information about who you are and roughly what type of research you are doing. We will treat this as privileged information and not share it outside the SURF initiative, except with other Mozilla engineers to help you debug or diagnose issues. (Note above; however, that try runs including the patches, builds, and logs are public.)

Once we've had a conversation, we will help you through the process of getting Level 1 commit access, which will give you access to try. This process is documented at https://www.mozilla.org/en-US/about/governance/policies/commit/

The TryServer represents a significant cost for Mozilla - especially for special Operating Systems. We expect that you will be judicious about your use of try. In particular:

• Build jobs for Windows (except the mingw-clang builds) and all Windows tests occur on Windows hardware and are more expensive.
• While OSX build jobs run on Linux; tests for OSX run on OSX and are expensive and in-demand
• Android builds also run on Linux; but tests run on Android hardware and emulators and are also in-demand and expensive
• Talos tests (which are performance tests) are expensive because they run on native hardware (Android, Windows, and OSX especially so) and need to be retriggered multiple times.

A good rule of thumb is to get thing working with linux64 debug builds (running the full unit test suite is generally fine) and after that works; move on to building and running debug+optimized builds and unit tests on OSX and Windows. Until you are ready to measure performance, we ask you omit Talos jobs. Unless your research is specifically related to Android (and we would request you tell us that ahead of time) - we ask you not to run Android jobs.

Your First Try Run

Let's get the preliminaries out of the way:

1. Ensure that ssh is properly configured with the ssh key you created during your commit access process (e.g. using ssh-agent + ssh-add, or editing .ssh/config)
2. Create a test commit
3. Ensure your working directory is clear (that you have have no modified files with hg status).

After that there are multiple ways to push to try; but the two most common are:

Fuzzy

Fuzzy is the currently supported and encouraged method for pushing to try.

./mach try fuzzy

Fuzzy allows you to very selectively choose what tasks to run. Its documentation is located here. It will automatically figure out dependencies; so if you choose to only run linux64-mochitests, it will schedule the needed linux64 build job as well.

Once you have pushed using ./mach try fuzzy once; ./mach try again will submit the configuration of the last push. Fuzzy has lots more additional options.

Legacy Try Syntax

This try syntax is deprecated; but still commonly used for its terse-ness.

./mach try -b do -p linux64 -u all -t none

• -b means 'build type' and valid values are 'd', 'o', or 'do' and they stand for 'debug' and 'optimized'.
• -p means 'platform'. The most commonly used values are 'linux' (meaning x86), 'linux64', 'win32', 'win64', and 'macosx64'. They are separated by commas, no spaces.
• -u means 'unit tests'. If you are using this syntax you would typically choose either 'all' or 'none'
• -t means 'talos tests' (which are our performance tests.) Choose 'none'

There are a couple of other flags that are commonly used:

./mach try -b do -p linux64,win64 -u none -t all --rebuild-talos=5
./mach try -b do -p linux64 -u all -t none --setenv="MOZ_LOG=CSMLog:3"
./mach try -b do -p linux64 dom testing/web-platform/tests/dom

• --rebuild-talos=5 indicates that Talos jobs should be run 5 times (so you can average the performance.) Talos jobs only run on opt and pgo builds.
• --setenv="MOZ_LOG=CSMLog:3" is how to pass environment variables to the test runs. Typically this is used to get logging. (More on this later)
• 'dom testing/web-platform/tests/dom' (positional arguments) are a way to specify that only tests in these directories should be run.

Viewing your try run

After you submit your try run, it will give you a link that looks like https://treeherder.mozilla.org/#/jobs?repo=try&revision=2827e91340e5b62da31628d8f27546e3a7d53d82 (this link may not work after this job ages out of try; but https://treeherder.mozilla.org will show a view of lots of recent jobs.

Here's a quick overview of the most commonly used features.

Here's what the red numbers mean:

1. Here is the list of commits in your push to try. Note: Only *new* commits never seen by the try repository are listed here. So if you send a job to try with Commit A, and then a subsequent job with a new Commit B - you will only see Commit B listed. If you `hg histedit` and edit A, then submit again - you will see commit A' (the edited commit A) and commit B' (B wasn't edited, but it has a new parent so it becomes B'.)
2. This is the job view. If you send a lot of builds and tests up, this can get pretty crowded. Mouse over things to get a tooltip for what they are. +6 means a group containing 6 sub-items. Build jobs begin with B, and typically the 'normal' Build for this platform will just be a plain B. Here we've selected the Linux x64 Debug Build Job. It's caused a pane at the bottom to open.
3. There are two ways to view logs from a job. The Log icon that is below and slightly to the left of the (3) icon - this log view is a web-based reader that is fast to load. The icon directly below the (3) takes you to the full, raw log. I prefer this log personally, even if it takes 45 seconds to load.
4. The ... menu directly below the (4) contains an option to "Create an Interactive Task". This lets you connect to the build job and, if it fails, take a look at the error and poke around the system to see what might have gone wrong. Very handy.
5. If you select the Job Details tab on a Build ob, you'll see a list of files here like (5). Here you can download your build. It will be in a archive named 'target'. Other built objects - like tests - will be target.foo archives.

Adding stuff to a Try Run, or re-running jobs

If you have an existing try run that you want to add jobs to (for example additional build jobs, or to re-run tests) this can be done from the treeherder web interface without needing to send in a new try job. The option to do this is behind the downward-facing arrow, at the top of the treeherder UI on the left-hand side, next to 'View Tests' and the pin icon.

For example, if you want to see if a test always fails, or only some of the time, you can re-trigger the test on an existing job, rather than submit a whole new try push that will perform a build step also. To retrigger a job, select it in the UI (so its details show up in the bottom pane) and press the 'r' key. It's a shortcut. Pressing the '?' key will show you the shortcut list.

For another example, if your build and tests worked successfully for one platform; and you want to test a second one - you can add the tests on the web interface. This has the advantage of keeping all your results for a single patchset in one place. See Scheduling jobs with Treeherder and Scheduling jobs with Treeherder (Search).

The only caveat to this method is that it can be easy to add tests that are not typically run in a normal try run and may currently be failing in general - independent of your patchset. So keep that in mind if you get unexpected failures.

Frequently Asked Questions

How do I modify the compiler used to build Firefox?

There's two answers to this question.

Locally: (to be answered)

In Try: (to be answered)

How do I add logging to my code to figure out what is going on?

(to be answered)