Building Firefox/SURF

This page is to serve as a guide for interested researchers (primarily academic researchers) who would like to experiment with Firefox for research purposes. Such experiments might include:

• Developing new compiler features, performance improvements, or security mechanisms
• Altering the behavior of the browser with respect to the web platform
• Creating a web scanning platform based on the browser (similar to OpenWPM)

Or something entirely different. Be creative.

Building Firefox Locally

The first step will almost certainly be to build the browser locally. Firefox has a well maintained mechanism for obtaining the dependencies needed to build it, even on Windows. Follow the Developer guide at MDN to build Firefox for the first time locally.

Once you've built it, use ./mach run to start the browser and take it for a spin.

Running Tests (Locally)

./mach test path/to/test will run most types of tests; for example ./mach test dom/security/test/csp/test_upgrade_insecure.html

You can add --headless (after 'test') to run them without launching the browser window. This is useful when working on a remote server. Note however that while --headless works most of the time, if a test fails to run, it is worth re-running it without --headless in a GUI environment.

gtests are run differently, for example ./mach gtest "*ReducePrecision_*" to run tests with 'ReducePrecision_' in their name (which would be tests like these).

While you can use ./mach test mochitest to run all mochitests; or ./mach test to run all tests - doing this locally is going to produce false positives and generally be unpleasant. Instead the next step is to move to try.

Building and Testing in Mozilla Infrastructure

Note that all try runs are public, and presently Mozilla does not have a mechanism to do private builds or test runs.

The TryServer is a way to submit a one or more patches and have them built on multiple Operating Systems, run tests, and download builds and logs. Because the entire build system is specified in the mozilla-central tree as configuration files it is possible to write a patch that changes the compiler used (either to a different version, or actually patching the compiler), have the tryserver check out the compiler source from a remote repository, build the compiler, and then use the compiler to build Firefox. All in the cloud. https://wiki.mozilla.org/ReleaseEngineering/TryServer has more information about the TryServer, but we will aim to have a functional quickstart tutorial in this document.

Getting Access

Before submitting to try, you will need to contact SURF at the email surf@mozilla.com and provide us with some information about who you are and roughly what type of research you are doing. We will treat this as privileged information and not share it outside the SURF initiative, except with other Mozilla engineers to help you debug or diagnose issues. (Note above; however, that try runs including the patches, builds, and logs are public.)

Once we've had a conversation, we will help you through the process of getting Level 1 commit access, which will give you access to try. This process is documented at https://www.mozilla.org/en-US/about/governance/policies/commit/

The TryServer represents a significant cost for Mozilla - especially for special Operating Systems. We expect that you will be judicious about your use of try. In particular:

• Build jobs for Windows (except the mingw-clang builds) and all Windows tests occur on Windows hardware and are more expensive.
• While OSX build jobs run on Linux; tests for OSX run on OSX and are expensive and in-demand
• Android builds also run on Linux; but tests run on Android hardware and emulators and are also in-demand and expensive
• Talos tests (which are performance tests) are expensive because they run on native hardware (Android, Windows, and OSX especially so) and need to be retriggered multiple times.

A good rule of thumb is to get thing working with linux64 debug builds (running the full unit test suite is generally fine) and after that works; move on to building and running debug+optimized builds and unit tests on OSX and Windows. Until you are ready to measure performance, we ask you omit Talos jobs. Unless your research is specifically related to Android (and we would request you tell us that ahead of time) - we ask you not to run Android jobs.

Your First Try Run

Let's get the preliminaries out of the way:

1. Ensure that ssh is properly configured with the ssh key you created during your commit access process (e.g. using ssh-agent + ssh-add, or editing .ssh/config)
2. Create a test commit
3. Ensure your working directory is clear (that you have have no modified files with hg status).

After that there are multiple ways to push to try; but the two most common are:

Fuzzy

Fuzzy is the currently supported and encouraged method for pushing to try.

./mach try fuzzy

Fuzzy allows you to very selectively choose what tasks to run. Its documentation is located here. It will automatically figure out dependencies; so if you choose to only run linux64-mochitests, it will schedule the needed linux64 build job as well.

Once you have pushed using ./mach try fuzzy once; ./mach try again will submit the configuration of the last push. Fuzzy has lots more additional options.

Legacy Try Syntax

This try syntax is deprecated; but still commonly used for its terse-ness.

./mach try -b do -p linux64 -u all -t none

• -b means 'build type' and valid values are 'd', 'o', or 'do' and they stand for 'debug' and 'optimized'.
• -p means 'platform'. The most commonly used values are 'linux' (meaning x86), 'linux64', 'win32', 'win64', and 'macosx64'. They are separated by commas, no spaces.
• -u means 'unit tests'. If you are using this syntax you would typically choose either 'all' or 'none'
• -t means 'talos tests' (which are our performance tests.) Choose 'none'

There are a couple of other flags that are commonly used:

./mach try -b do -p linux64,win64 -u none -t all --rebuild-talos=5
./mach try -b do -p linux64 -u all -t none --setenv="MOZ_LOG=CSMLog:3"
./mach try -b do -p linux64 dom testing/web-platform/tests/dom

• --rebuild-talos=5 indicates that Talos jobs should be run 5 times (so you can average the performance.) Talos jobs only run on opt and pgo builds.
• --setenv="MOZ_LOG=CSMLog:3" is how to pass environment variables to the test runs. Typically this is used to get logging. (More on this later)
• 'dom testing/web-platform/tests/dom' (positional arguments) are a way to specify that only tests in these directories should be run.

Viewing your try run

After you submit your try run, it will give you a link that looks like https://treeherder.mozilla.org/#/jobs?repo=try&revision=2827e91340e5b62da31628d8f27546e3a7d53d82 (this link may not work after this job ages out of try; but https://treeherder.mozilla.org will show a view of lots of recent jobs.

Here's a quick overview of the most commonly used features.

Here's what the red numbers mean:

1. Here is the list of commits in your push to try. Note: Only *new* commits never seen by the try repository are listed here. So if you send a job to try with Commit A, and then a subsequent job with a new Commit B - you will only see Commit B listed. If you `hg histedit` and edit A, then submit again - you will see commit A' (the edited commit A) and commit B' (B wasn't edited, but it has a new parent so it becomes B'.)
2. This is the job view. If you send a lot of builds and tests up, this can get pretty crowded. Mouse over things to get a tooltip for what they are. +6 means a group containing 6 sub-items. Build jobs begin with B, and typically the 'normal' Build for this platform will just be a plain B. Here we've selected the Linux x64 Debug Build Job. It's caused a pane at the bottom to open.
3. There are two ways to view logs from a job. The Log icon that is below and slightly to the left of the (3) icon - this log view is a web-based reader that is fast to load. The icon directly below the (3) takes you to the full, raw log. I prefer this log personally, even if it takes 45 seconds to load.
4. The ... menu directly below the (4) contains an option to "Create an Interactive Task". This lets you connect to the build job and, if it fails, take a look at the error and poke around the system to see what might have gone wrong. Very handy.
5. If you select the Job Details tab on a Build ob, you'll see a list of files here like (5). Here you can download your build. It will be in a archive named 'target'. Other built objects - like tests - will be target.foo archives.

Adding stuff to a Try Run, or re-running jobs

If you have an existing try run that you want to add jobs to (for example additional build jobs, or to re-run tests) this can be done from the treeherder web interface without needing to send in a new try job. The option to do this is behind the downward-facing arrow, at the top of the treeherder UI on the left-hand side, next to 'View Tests' and the pin icon.

For example, if you want to see if a test always fails, or only some of the time, you can re-trigger the test on an existing job, rather than submit a whole new try push that will perform a build step also. To retrigger a job, select it in the UI (so its details show up in the bottom pane) and press the 'r' key. It's a shortcut. Pressing the '?' key will show you the shortcut list.

For another example, if your build and tests worked successfully for one platform; and you want to test a second one - you can add the tests on the web interface. This has the advantage of keeping all your results for a single patchset in one place. See Scheduling jobs with Treeherder and Scheduling jobs with Treeherder (Search).

The only caveat to this method is that it can be easy to add tests that are not typically run in a normal try run and may currently be failing in general - independent of your patchset. So keep that in mind if you get unexpected failures.

Frequently Asked Questions

How do I modify the compiler used to build Firefox?

There's two answers to this question.

Locally: Inside your mozconfig, you can set the following like export FOO="bar"

• CC - C compiler used to compile firefox
• CFLAGS
• CPPFLAGS
• HOST_CC - used to compile utilities that will run on the build machine
• HOST_CFLAGS
• CXX / HOST_CXX - C++ compiler
• CXXFLAGS / HOST_CXXFLAGS
• LLVM_CONFIG
• RUSTC / CARGO / RUSTDOC / RUSTFMT / CBINDGEN

Less commonly you may need to set

• AR / AS / RANLIB / STRIP / OTOOL / OBJCOPY / LIPO / NASM / NODEJS / MAKECAB
• BINDGEN_CFLAGS - for rust bindgen

You may also need to set:

• ac_add_options --with-toolchain-prefix=foo - used for a cross compile. example
• mk_add_options "export PATH=directory:$PATH" - to add more PATHs
• LD_LIBRARY_PATH - as shown here

In Try: In try, you'll need to build your compiler and then use it. I'm going to assume you're building clang and working on linux64.

Here's a quick tour of where things are first though. The linux64 builds are defined in this file. Look for 'linux64/debug'. Some important things to note are mozconfig-variant and fetches. You match the mozconfig-variant with the corresponding file in this directory. Observe that the debug mozconfig pulls in additional mozconfigs. But modifying this mozconfig will modify the linux64 debug build.

fetches->toolchain will match up with the keywords defined in the toolchain definitions. So if we're looking for 'linux64-clang' we can search for it and see that while there is nothing defining 'linux64-clang' and a key (it would end in a : (colon)) - there is something saying toolchain-alias: linux64-clang. If we follow that, we can see at time of writing, it is defined as linux64-clang-9. (This link is a permalink to an old file, it may be different when you read this.)

Let's figure out how we build this toolchain. We want to look at build-clang.sh, which receives the argument build/build-clang/clang-linux64.json. It fetches the clang source using a keyword defined as clang-9 (under fetches->fetch) and this compiler is itself built using the toolchains linux64-binutils and linux64-gcc-7. (We could follow binutils and gcc-7 to find where they come from also, but we won't do that in this tutorial.)

First let's match clang-9 to the source code. We do that by searching for clang-9 in the toolchain's fetch file. We see the repository it comes from and the git revision we check out.

Then we can find build-clang.sh - it looks like most of the heavy lifting is done by a script called build-clang.py. We can also find clang-linux64.json. Of special note, is that the json file specifies where to find the compiler to compile clang, as well as patches to apply to clang before compiling it. (It might be simpler for you to point the source code fetch to your own repo, rather than adding patches in the json file though.)

So - using permanent links to a mozilla-central repo as of 11/4/2019 - the steps to build a compiler to compile Firefox with are

1. Download the clang source code as specified here
2. Build the source code using the job specified here
   1. This calls build-clang.sh with clang-linux64.json which passes that argument to build-clang.py
   2. We patch the clang source code using the patches specified in the json file before building it.
3. We build the linux64 debug job using the job specified here (which defines what toolchains to use) and the we use the mozconfig specified here.
   1. If we follow the mozconfig include tree, we find that mozconfig.unix is where we set the CC and CXX variables.

Modifying components of those steps is how you'd customize the compiler used to build Firefox.

You may also need to set

ac_add_options --disable-warnings-as-errors - if your compiler produces new warnings, we may treat them as errors and abort. This will stop that.

How do I add logging to my code to figure out what is going on?

With MOZ_LOG. nsRFPService.cpp has many examples of it being used. Here's some notes:

1. You'll need to declare something like static mozilla::LazyLogModule gResistFingerprintingLog("nsResistFingerprinting");
2. The format is odd. It's MOZ_LOG(module, severity (format string, args)). Note that second set of parens.
3. The list of severities are defined here
4. To see the logfile output, run Firefox with the environment variable MOZ_LOG="moduleName:5,secondModule:3"
5. You can send the logs to files by setting MOZ_LOG_FILE="path-to-file.txt"
6. On Windows, due to weirdness, to see the log output from sub-processes you need to either redirect them to a file (or tee) in the console or specify MOZ_LOG_FILE
7. This page has more details. As always, be wary that things may be out of date.