Mobile/Fennec Unittests/On Device

Overview

Currently these techniques are only used on Maemo based devices due to the full support that environment provides. We can also use these techniques on a desktop build.

Requirements

To run the unit tests, we need to have a build of Fennec and a build of the test bits (do this by adding "ac_add_options --enable-tests" to your mozconfig). Once you have the tests and Fennec binaries, we also need:

• Python (preferably 2.5) - and these modules:
  • subprocess: Popen (wait, stdout.readline, communicate, returncode), PIPE, STDOUT
  • os.path: normpath, expanduser, join, exists, dirname, basename, abspath, realpath, splitext, isdir
  • os: chdir, getcwd, remove, kill, mkdir, listdir, unlink, environ
  • shutil: rmdir, rmtree, copytree, copy
  • tempfile: mkdtemp, gettempdir
  • file: open, read, write, close
  • sys: stderr, platform, exit
• maemkit (see below - same python requirements, also using Popen to launch another python script)
• available RAM (>=16MB) to run local http server in xpcshell and test data
• available disk (250MB - can be on storage card)

I list these requirements out as we learned on Windows Mobile there were a lot of Python libraries not available. In trying to work around this, we have ran into many other issues, so take caution in trying to find another solution for a missing library. Note that getting --enable-tests to build is not always an easy thing as we discovered in [Windows Mobile].

Maemkit

Maemkit is a toolset that wraps around the unittest harness drivers. This handles:

• extra configurations (extending [timeouts], [page caches], etc..) to the browser
• running the tests in smaller chunks to reduce the memory footprint
• splitting up the tests to run on a series of devices to reduce run time
• make the general configuration easier by using a config file to reduce command line parameters.

The biggest win is the splitting up of the tests into smaller chunks. Without this we would not be able to even run any tests. During our first steps, we found that we could run a test pass with a few tests, but not the full set of tests. As we continued our development, it became obvious that we needed to limit the number of tests ran in a given test run. Now we chunk up the test based on the directories (find the directory that is a leaf node) and sometimes split those directories up into smaller tests. This is done by either file system manipulation or manifest file manipulation.

The original list of requirements for maemkit can be found [here]. With it running in the field, we have it [checked into hg] and a [tracking bug] for any adjustments.

Assumptions / Decisions

We are making these assumption:

* tests will run on the device, not an emulator
* python is available 
* device will store all tests information
* device will run local httpd.js server via xpcshell
* all tests provided from make package-tests will be run on device
* there is a method to copy files to/from the device
* the device has access to the internet (not required)
* we have access to launch fennec from a command line (or a shortcut)