Auto-tools/B2G/Running Emulator Tests

From MozillaWiki
Jump to navigation Jump to search

This page details requirements for running B2G Emulator tests in buildbot/mozharness.

WebAPI Tests

python runtests.py --emulator arm --homedir /path/to/emulator/on/test/slave --type b2g 
--install-gecko /path/to/gecko/build/to/install /path/to/unit-tests.ini
  • Note that the --install-gecko argument doesn't exist yet; it's being worked on in bug 789976.
  • We can probably repurpose the existing mozharness Marionette script for this. I think the only changes needed are:
  1. Change the --marionette-address default to None, and don't pass it to runtests.py if None
  2. Add an --emulator argument
  3. Add an --install-gecko argument
  4. Add a --homedir argument
  5. If --emulator is specified, don't pass to --binary to runtests.py

Mochitests

  • Python package requirements: manifestparser, mozhttpd, mozinfo, mozinstall, mozprofile, mozprocess, mozrunner. These can all be installed from the tests.zip package, see http://hg.mozilla.org/build/mozharness/file/f96bd9a4cacd/scripts/marionette.py#l76 for reference. The 'marionette' package also needs to be installed from tests.zip.
  • Command-line should be invoked from the 'mochitest' directory of the downloaded tests.zip package.
  • Command-line is:
python runtestsb2g.py --emulator arm --console-level INFO --b2gpath /path/to/emulator/on/test/slave  
--remote-webserver 10.0.2.2 --total-chunks 9 --this-chunk $CHUNK --run-only-tests b2g.json 
--xre-path /path/to/directory/containing/desktop/xpcshell --adbpath /path/to/adb/on/test/slave
--install-gecko /path/to/downloaded/gecko/build

A few notes about the argument:

  • --adbpath specifies the path to adb on the test slave. It is normally part of the emulator package, at $EMULATOR_HOME/out/host/linux_x86/bin/adb
  • --xre-path is the path to the directory containing the desktop version of xpcshell; it doesn't point to a path inside the b2g build, as the xpcshell there will be an arm version that can't run on the linux test slave. We'll have to download an xpcshell as part of the mozharness script for this.
  • --remote-webserver is always set to 10.0.2.2; this is a special IP that has meaning to the emulator; it doesn't matter what the IP of the test slave is
  • --run-only-tests should point to a json file in the mochitest directory that contains known-green tests for mochitest. This file doesn't exist yet, I'll file a bug to create it.
  • --install-gecko doesn't exist yet; I'll add it in a separate bug.

Reftests

  • Python package requirements: same as mochitest
  • Command-line should be invoked from the 'reftest' directory of the downloaded tests.zip package.
  • Command-line is:
python runreftestb2g.py --emulator arm --emulator-res 800x1000 --ignore-window-size 
--b2gpath /path/to/emulator/on/test/slave --remote-webserver 10.0.2.2 --total-chunks <total_chunks> 
--this-chunk <chunk> --xre-path /path/to/directory/containing/desktop/xpcshell --adbpath /path/to/adb/on/test/slave
--install-gecko <path_to_reftest_manifest>

A few notes. Same as mochitest notes but also:

  • There is no run-only-tests option in reftest. We need to figure out how to get a list of passing tests (either disable tests in m-c manifests or use a separate set somewhere)
  • As a side effect of how the remote reftest server is implemented the tests need to be inside the reftest directory. I get around this by linking the gecko directory inside, e.g:
cd _tests/reftest
ln -s /path/to/gecko tests
python runreftestb2g.py [options] tests/layout/reftests/reftest.list

XPCShell

  • Not ready to run yet due to some packaging problems.
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/B2G/Running_Emulator_Tests&oldid=472445"