Personal tools

Jetpack/Testing

From MozillaWiki

Jump to: navigation, search

This page explains how to run the SDK's unit tests, and how to interpret the results. It's aimed at people who don't have any familiarity with the SDK, but nevertheless have to run the tests to diagnose any test failures, for example because of regressions introduced by the underlying platform.

For any help with these tests come to the #jetpack channel and ask the team, Gozala is the module owner.

Contents

Running tests from compiled Firefox

You have two options to run tests from a built version of Firefox. Both do basically the same thing and are quite limited right now, if you want more functionality (like being able to select specific tests) you'll need to checkout the SDK repository separately and follow the later instructions.

Running with mach

From the source directory or object directory run:

 mach jetpack-test

Running with make

From the object directory run:

 make jetpack-tests

Disabling tests

If tests need to be disabled then you can do so by adding the following to the end of the test file:

 // Test disabled because of bug XXXXXX
 module.exports = {};

If you need to disable for a particular platform:

 // Test disabled on Windows because of bug XXXXXX
 if (require("sdk/system/runtime").OS == "WINNT")
   module.exports = {};

The OS values are the normal OS_TARGET values.

Running tests from the SDK repository

First, get the SDK from git:

 git clone git://github.com/mozilla/addon-sdk.git
 cd addon-sdk

Or go into the SDK directory in mozilla-central:

 cd addon-sdk

Activate the SDK (this isn't necessary but puts the cfx tool into your path, you can instead run it from addon-sdk/bin/cfx):

 source bin/activate

To run all tests:

 cfx testall

To run only the API tests (typically the ones that break when Firefox changes), invoke testpkgs instead of testall:

 cfx testpkgs

To run only certain test suites, specify the --filter flag with a regular expression that matches the filenames of the test suites you want to run:

 cfx testpkgs --filter panel

Specifying the Firefox Build

To run tests on a specific build of Firefox, specify the build via the --binary flag:

 cfx testall --binary /Applications/Nightly.app/
 cfx testall --binary ~/Mozilla/source/obj-dir/dist/bin/firefox
 cfx testall --binary /c/Program\ Files\Nightly\firefox.exe

Attaching a Debugger

To attach gdb before running tests, specify the --no-run flag:

 cfx testpkgs --no-run

--no-run doesn't actually run the command, it just prints out the shell command that would be invoked in order to run it. So the above command would produce output like:

To launch the application, enter the following command:
/Applications/Firefox.app/Contents/MacOS/firefox-bin -profile /var/folder
/me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmp72nW_f.mozrunner -foreground -no-remote

You can then run the debugger using that command:

gdb --args /Applications/Firefox.app/Contents/MacOS/firefox-bin -profile /var/folder
/me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmp72nW_f.mozrunner -foreground -no-remote

Gory Details of Testing With cfx

There's cfx documentation here, and command line help is available by typing:

cfx --help

There are five different test commands:

 test       - run tests
 testcfx    - test the cfx tool
 testex     - test all example code
 testpkgs   - test all installed packages
 testall    - test whole environment

cfx test

cfx test runs the tests implemented for a single SDK "package". A package is a collection of JavaScript modules, and is defined as such by the presence of a file called "package.json" in its root directory. Among other things, "package.json" includes an string called "tests" which describes where the package's test suites live. If this is not defined, the default is the "tests" or "test" directory under the package root.

So if you navigate to a package's root directory (containing a package.json file) and execute cfx test, cfx will:

  1. initialize the tests directory using the "tests" value inside "package.json", or the top-level "tests/" or "test/" directory
  2. load every module it finds in the tests directory whose name starts with "test"
  3. call each function exported by that module, passing it a test object as an argument.

An add-on is a package, and the SDK's APIs are currently collected into two packages called "addon-kit" and "api-utils" (you can see them under the top-level "packages" directory).

Testing an Add-on

From the SDK root, navigate to "examples/reading-data" and explore the "tests" directory:

(addon-sdk)~/mozilla/addon-sdk/examples > cd reading-data/
(addon-sdk)~/mozilla/addon-sdk/examples/reading-data > ls tests
test-main.js
(addon-sdk)~/mozilla/addon-sdk/examples/reading-data > more tests/test-main.js 
const m = require("main");
const self = require("self");

exports.testReplace = function(test) {
  const input = "Hello World";
  const output = m.replaceMom(input);
  test.assertEqual(output, "Hello Mom");
  var callbacks = { quit: function() {} };

  // Make sure it doesn't crash...
  m.main({ staticArgs: {} }, callbacks);
};

exports.testID = function(test) {
  // The ID is randomly generated during tests, so we cannot compare it against
  // anything in particular.  Just assert that it is not empty.
  test.assert(self.id.length > 0);
  test.assertEqual(self.data.url("sample.html"),
                   "resource://reading-data-example-at-jetpack-dot-mozillalabs-dot-com-reading-data-data /sample.html");
};

So this add-on implements two unit tests. You can see that each test expects a test object as an argument, and that test defines functions like assert, assertEqual and timeout functionality like waitUntilDone. To learn more about what you can do with a test object check the api documentation.

If you type cfx test it will execute these two tests. We'll use the -v option to make it show us the test names:

(addon-sdk)~/mozilla/addon-sdk/examples/reading-data > cfx test -v
Using binary at '/Applications/Firefox.app/Contents/MacOS/firefox-bin'.
Using profile at '/var/folders/me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmpASJfvx.mozrunner'.
Running tests on Firefox 6.0/Gecko 6.0 ({ec8030f7-c20a-464f-9b0e-13a3a9e97384}) under Darwin/x86_64-gcc3.
info: executing 'test-main.testReplace'
info: pass: a == b == "Hello Mom"
info: My ID is reading-data-example@jetpack.mozillalabs.com
info: executing 'test-main.testID'
info: pass: assertion successful
info: pass: a == b == "resource://reading-data-example-at-jetpack-dot-mozillalabs-dot-com-reading-data-data /sample.html"

3 of 3 tests passed.
OK
Total time: 6.058659 seconds
Program terminated successfully.

Testing an SDK Package

An SDK package like "addon-kit" or "api-utils" is a package too, so the procedure is the same. Executing cfx test from the root of "addon-kit" will execute all the following test suites:

(addon-sdk)~/mozilla/addon-sdk > 
(addon-sdk)~/mozilla/addon-sdk > cd packages/addon-kit/
(addon-sdk)~/mozilla/addon-sdk/packages/addon-kit > ls tests
pagemod-test-helpers.js  test-notifications.js  test-request.js
test-clipboard.js  test-page-mod.js  test-selection.js
test-context-menu.html  test-page-worker.js  test-simple-storage.js
test-context-menu.js  test-panel.js  test-tabs.js
test-hotkeys.js  test-passwords.js  test-widget.js
test-module.js  test-private-browsing.js  test-windows.js

This isn't ideal if you only want to test a single module, so you can use the -f option to cfx to filter the set of tests to run. This option takes a regex and runs only the test suites whose names match the regex. This example runs only the tests for the "clipboard" module:

(addon-sdk)~/mozilla/addon-sdk/packages/addon-kit > cfx test -f clipboard -v
Using binary at '/Applications/Firefox.app/Contents/MacOS/firefox-bin'.
Using profile at '/var/folders/me/me2B1lFDE0WZCgd33s2OTE+++TU/-Tmp-/tmpVLFIqe.mozrunner'.
Running tests on Firefox 6.0/Gecko 6.0 ({ec8030f7-c20a-464f-9b0e-13a3a9e97384}) under Darwin/x86_64-gcc3.
info: executing 'test-clipboard.testWithNoFlavor'
info: pass: assertion successful
info: pass: a == b == "text"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: executing 'test-clipboard.testWithFlavor'
info: pass: assertion successful
info: pass: a == b == "html"
info: pass: a == b == null
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: executing 'test-clipboard.testWithRedundantFlavor'
info: pass: assertion successful
info: pass: a == b == "text"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: pass: a == b == "hello there"
info: executing 'test-clipboard.testNotInFlavor'
info: pass: assertion successful
info: pass: a == b == null

17 of 17 tests passed.
OK
Total time: 2.909257 seconds
Program terminated successfully.

cfx testcfx

cfx testcfx tests the cfx tool itself. cfx is written in Python, and testcfx uses Python's unittest framework.

cfx test code is implemented by a collection of Python modules under "python-lib/cuddlefish/tests". Each module defines a subclass of the unittest module's TestCase class, and implements one or more test functions, each of which begins with the string "test". For example, here's what the "test_rdf" module looks like:

(addon-sdk)~/mozilla/addon-sdk > cd python-lib/cuddlefish/tests
(addon-sdk)~/mozilla /addon-sdk/python-lib/cuddlefish/tests > more test_rdf.py
import unittest
import xml.dom.minidom

from cuddlefish import rdf


class RDFTests(unittest.TestCase):
    def testBug567660(self):
        obj = rdf.RDF()
        data = u'\u2026'.encode('utf-8')
        x = '<?xml version="1.0" encoding="utf-8"?><blah>%s</blah>' % data
        obj.dom = xml.dom.minidom.parseString(x)
        self.assertEqual(obj.dom.documentElement.firstChild.nodeValue,
                         u'\u2026')
        self.assertEqual(str(obj).replace("\n",""), x.replace("\n",""))

cfx testcfx builds a TestSuite out of all the test cases it finds, then uses TestTestRunner to execute them.

It also looks for DocTests in the documentation files, and executes any that it finds. But there aren't any, at the moment.

cfx testex

cfx testex runs cfx test for all the add-ons which ship with the SDK under the examples/ directory.

cfx testpkgs

cfx testpkgs finds all packages installed under the packages/ directory, and executes cfx test for each package it finds.

cfx testall

cfx testall tests everything:

cfx testcfx
cfx testex
cfx testpkgs