Auto-tools/AutoTestingGuide

This page should be moved to MDN when we have a reasonable first version.

Automated Testing at Mozilla

You've just written a feature and (hopefully!) want to test it. Or you've decided that an existing feature doesn't have enough tests and want to contribute some. But where do you start? You've looked around and found references to things like "xpcshell" or "mozmill" or "talos". What do they all do? What's the overlap? In short, where should your new tests go?

This document aims to answer that question. There's a very short summary of each framework, and a bit of Q&A to help you pick your framework. This may only narrow down your choices, however, in which case you should read more about the frameworks and/or hop on #ateam, #qa, or one of the development forums and ask questions.

Generally, you should pick the lowest-level framework that you can. If you are testing JavaScript but don't need a window, use xpcshell. If you're testing page layout, try to use reftest. The advantage is that you don't drag in a lot of other components that might have their own problems, so you can home in quickly on any bugs in what you are specifically testing.

In production

buildbot tests

These tests are found within the mozilla-central tree, along with the product code. They are all run when a changeset is pushed to mozilla-central, mozilla-inbound, or try, with the results showing up on tbpl. They can also be run on their own.

The letters in parentheses are the abbreviations used by tbpl.

compiled-code (B)

Written in C++, compiled-code tests can test pretty much anything but are difficult to write properly. In general, this should be your last option for a new test.

xpcshell (B)

xpcshell are console JavaScript tests. There is no chrome, no content, no window. xpcshell is useful for testing low-level things, such as XPCOM components. If you don't need a window, use this.

JS engine regression (J)

Tests specifically for the JavaScript engine, and that's about it.

crashtest (C)

Really simple: open a web page and see if it causes a crash. If you've found pages that crash Firefox, add a test here to make sure future versions don't experience this crash again.

reftest (R)

A reftest verifies that two web pages are rendered identically. There is generally more than one way to write a web page, and a reftest can be used to make sure that they are rendered identically. If you want to test layout or graphics, reftest can be a very easy way to do so.

Mochitest (M, Moth)

Mochitest uses JavaScript to test features. Anything piece that has its functionality exposed in JavaScript can theoretically be tested with Mochitest. A subsection of Mochitest, chrome tests, run with high privileges and hence can test a lot of the browser's functionality. And then there are browser chrome tests, which run in the scope of the browser window. (FIXME: what are the differences between chrome and browser-chrome tests?).

If you can use JavaScript to test your feature, Mochitests are not a bad place to start. (FIXME: or are there now better places?)

Talos (T)

Talos is a framework for performance testing. It is a more complicated beast than other Mozilla frameworks, but there is a standalone version which is easier to set up than the full automated system. If you're measuring performance, Talos is the place to go.

Mozmill

Mozmill is an extensive framework for browser automation. It is an extension with an extensive API to help you write functional tests that simulate user interactions, as well as a full unit test API. It is not yet part of the main automated test suite.

Speedtests

SpeedTests is a framework for executing arbitrary tests, which report results via JavaScript calls, in several browsers. Originally this framework was designed for modified versions of Microsoft's speed demos but has now been expanded to include conformance tests such as test262 and Kraken. SpeedTests are good for cross-browser comparisons when tests don't need to dig too deep into the guts of the browsers.

In progress

Marionette

Marionette is a new framework designed with B2G Boot2Gecko in mind but applies to all flavours of Gecko. It uses Selenium-like tests that send and receive JSON data directly to the lower layers of the browser. Marionette has the potential to replace Mozmill and various other Gecko-specific test frameworks.

Robocop

Auto-tools/Projects/Robocop is a framework to test the Firefox's native Android UI. It is based on the Robotium framework. You'll need this if you want to test the UI on Android.

Peptest

Peptest measures responsiveness, how "snappy" Firefox/Thunderbird feels, by issuing alerts when the event loop is stuck for more than 50 ms.