WebExtensions/Try Server

From MozillaWiki
Jump to: navigation, search

Try Server for WebExtensions

WebExtension code is covered by unit tests, which are described at WebExtensions/Hacking#Unit_tests and WebExtensions/Contribution_Onramp#Testing. These tests can be run on the Try Server. Contributors with access to the try server can push try jobs to run tests. Try jobs can be scheduled via the web interface or via the "mach try" command.

The following platforms (-p) are of interest:

  • win32,win64
  • linux,linux64
  • macosx64
  • android-api-16,android-x86 (android-x86 is faster)

Symbols for the following test suites are shown in the test results on Treeherder. Test suite identifiers for use with "-t" in "mach try" are listed below the main bullet points:

  • M (mochitests) and M-1proc (mochitests in single-process mode)
    • mochitests (all mochitests, including many non-extension test suites). Not to be confused with "mochitest" (which only runs plain e10s mochitests).
    • mochitest-bc (browser chrome tests; desktop only - Windows, Linux and macOS)
    • mochitest-o (Android only)
  • X (xpcshell tests) and X-1proc (xpcshell tests in single-process mode)
    • xpcshell
  • TV (test verification mode; same effect as "mach test path/to/your/test --verify" locally) (desktop platforms only; not available on Android)
    • test-verify
  • ES (ESlint)

Pass --artifact to speed up builds (only if your changes do not require recompilation of C++/Rust code, such as Java (Android) and JavaScript-only changes).

Tests can be run in a debug build ("-b d"), an optimized ("-b o") or both ("-b do"). It is recommended to run both, because together they can catch more potential failures:

  • Some tests are only enabled in one of the two build types because of intermittent/permanent test failures ("skip-if" in the .ini files that describe the tests).
  • Debug builds usually run slower than optimized builds, and reveal weaknesses in tests (e.g. time-sensitive race conditions).
  • Debug builds have additional checks (e.g. validation of extension API schema against the actual implementation).

Examples

Examples of "mach try" commands (add "--no-push" to see the generated try syntax without actually triggering the try job):

  • Run all relevant tests. This is expensive and should only be used when you don't know which tests should be run, because it also includes many non-extension tests.
 mach try -b do -p win64,linux64,macosx64,android-api-16 -u mochitests,xpcshell,test-verify --artifact
  • Run all relevant tests, without non-extension tests. The command with the `--tag=webextensions` flag uses your local checkout and .mozconfig configuration to identify the tests. Some Android-only tests may not run if your current configuration targets desktop.
 mach try -b do -p win64,linux64,macosx64,android-api-16 -u test-verify --artifact --tag=webextensions
  • Run browser-chrome extension tests (and many other browser-chrome tests, but at least less expensive than just "mochitests"). This test type is only used on desktop, so Android was omitted from the platform flag ("-p").
 mach try -b do -p win64,linux64,macosx64 -u mochitest-bc,test-verify --artifact
  • Run extension tests on Android (requires a mozconfig file for Android; otherwise tests in mobile/android/components/extensions/test/ will not be run. The MOZCONFIG environment variable is used to select a different mozconfig file, so that you can develop for multiple platforms with a single checkout):
 MOZCONFIG=/path/to/mozconfig-for-android mach try -b do -p android-api-16 --artifact --tag=webextensions

It may take several seconds before a "mach try" command with the "--tag=webextensions" flag completes. If you already know the directory that contains the test, it is recommended to use the directory instead of "--tag=webextensions", e.g.:

  • Run tests in specific directories for desktop:
 mach try -b do -p win64,linux64,macosx64 -u mochitests,mochitest-bc,xpcshell --artifact --and toolkit/components/extensions/test/ browser/components/extensions/test/
  • Run tests in specific directories for Android:
 MOZCONFIG=/path/to/mozconfig-for-android mach try -b do -p android-api-16 -u mochitests,mochitest-o,xpcshell --artifact --and toolkit/components/extensions/test/ mobile/android/components/extensions/test/

By default, a test that match any specified test tags ("--tag") or suite ("-u") will be scheduled. To only run tests that match all specified rules, include "--and" in the try syntax. When the "--and" flag is used, the set of tests is computed locally. To see what tests are going to be scheduled, add "--no-push".

Test failures

After running the tests, test results are displayed in Treeherder. Green tests mean success, orange means failure. Orange results can be resolved as follows.

  • Your test or implementation caused a failure.
    • Fix your code and push a new try job.
  • Your test or some other test failed unexpectedly (test timed out or cannot be reproduced locally).
    • Log in to Treeherder and click on the "Retrigger" button at your specific test to repeat the test run.
  • Intermittent failures: An existing test without any relation to your code has failed.
    • Ignore the failure, especially if there is an existing bug report for it.

A set of tests can be split in different chunks to ensure that individual test tasks complete within a reasonable time frame. There is no obvious way to identify the chunk in which a test will be run, since it may depend on the build type, platform and the ordering of tests. To save server resources, you can run "mach try empty" to create an empty try job, and schedule specific jobs by selecting them in the Treeherder interface.