Auto-tools/AutoTestingGuide

This page should be moved to MDN when we have a reasonable first version.

Automated Testing at Mozilla

You've just written a feature and (hopefully!) want to test it. Or you've decided that an existing feature doesn't have enough tests and want to contribute some. But where do you start? You've looked around and found references to things like "xpcshell" or "mozmill" or "talos". What do they all do? What's the overlap? In short, where should your new tests go?

This document aims to answer that question. There's a very short summary of each framework, and a bit of Q&A to help you pick your framework. This may only narrow down your choices, however, in which case you should read more about the frameworks and/or hop on #ateam, #qa, or one of the development forums and ask questions.

In production

buildbot tests

These tests are found within the mozilla-central tree, along with the product code. They are all run when a changeset is pushed to mozilla-central, mozilla-inbound, or try, with the results showing up on [tbpl]. They can also be run on their own.

The letters in parentheses are the abbreviations used by tbpl.

compiled-code (B)

Written in C++, compiled-code tests can test pretty much anything but are difficult to write properly. In general, this should be your last option for a new test.

xpcshell (B)

xpcshell are console JavaScript tests. There is no chrome, no content, no window. xpcshell is useful for testing low-level things, such as XPCOM components. If you don't need a window, use this.

JS engine regression (J)

Tests specifically for the JavaScript engine, and that's about it.

crashtest (C)

Really simple: open a web page and see if it causes a crash. If you've found pages that crash Firefox, add a test here to make sure future version don't experience this crash again.

reftest (R)

A reftest verifies that two web pages are rendered identically. There is generally more than one way to write a web page, and a reftest can be used to make sure that