Auto-tools/Reporting Test Results to Treeherder for non-buildbot Test Frameworks

From MozillaWiki
Jump to: navigation, search

Reporting Test Results to Treeherder for non-buildbot Test Frameworks

Mozilla has an extensive system for building and testing Firefox and Firefox OS. See Mozilla automated testing and Automated Testing of Firefox OS on MDN for details.

The purpose of this document is to outline the steps necessary in creating a non-buildbot test framework which reports to Treeherder and which has the capability of being managed by mozilla's sheriffs. Autophone is used extensively as an example and also provides code examples which can be reused and modified for use in other test frameworks. You can see Autophone tests reporting to Treeherder today.

Buildbot managed tests

Historically, Buildbot has been used to manage the systems which build and test at mozilla. TaskCluster is a new system which is being used to manage tests for Firefox OS and may replace buildbot in the future. For the most part, tests at mozilla are executed by buildbot.

Buildbot monitors the source repositories and schedules build and test jobs for each changeset pushed to the repositories. Executing tests using buildbot requires that the tests meet several requirements which may not be possible to meet for all test frameworks. Some of the requirements for a test framework to be supported by buildbot are:

  • The tests must reside in the same repository as the code being tested.
  • The tests must not rely on external network resources.
  • The machines which are used for testing must be controllable from buildbot.
  • The tests must be run for each changeset pushed to the repository which results in a build being created.
  • The tests must produce log files using a standard format created by mozlog which can be parsed for errors by mozharness and treeherder.

Buildbot managed tests have distinct advantages over non-buildbot managed tests:

  • The tests are managed centrally via mozharness.
  • Buildbot handles the detection of new builds, uploading logs, initiating retries without any involvement of the test framework.
  • Test logs are uploaded to the build directory on
  • Treeherder automatically creates reports on the status of builds and tests by parsing the test logs without any need for the test framework to become involved.

Non-buildbot managed tests

Autophone is an example of a test framework that can not be managed by buildbot since it runs tests on actual phones and tablets which can not be placed in mozilla's co-location facilities and can not be managed by buildbot.

While non-buildbot managed tests allow increased flexibility in terms not being required to fulfill the requirements for buildbot jobs, they require the test framework to perform actions which are normally handled by buildbot.

  • Detection of new builds.
  • Job Scheduling.
  • Downloading and installing builds, symbols, and tests.
  • Uploading logs to an accessible location.
  • Reporting of test results.
  • Responding to user-initiated requests to cancel or retrigger tests.
  • Responding to user-initiated try builds.


Treeherder is a web application where the results of builds and tests can be viewed.

As might be expected, Treeherder has to a great extent been designed around the current operation of buildbot managed tests. Treeherder collects information on buildbot managed tests by polling buildbot for recently completed builds and tests, parsing the logs for metadata and test results, then creating resultsets which show each changeset pushed to the repositories along with links for the corresponding build and test runs.

Non-buildbot test frameworks are responsible for reporting to Treeherder directly using the Treeherder client as well as uploading test logs.


Running tests serves no purpose unless the results are used to prevent faulty code changes from being introduced. Sheriffs at mozilla are responsible for monitoring test results for each changeset pushed to the source repositories, backing out changesets which cause build or test failures, closing and opening the repositories and in general preventing code changes from regressing Firefox and Firefox OS. Sheriffs use Treeherder to keep track of the changesets being pushed to the repositories, whether the changesets break the building of Firefox or Firefox OS, cause test failures or if infrastructure issues are causing problems with actually running the tests.

Sheriffs have created a set of requirements which test frameworks must meet in order for the Sheriffs to use the test results to determine if a changeset is responsible for test failures. Test frameworks which meet these requirements are referred to as Tier 1 jobs.

Tier 1 Jobs

Some of the requirements for a Tier 1 test framework are:

  • Has an active owner.
  • Produces Usable job logs.
  • Produces a Low intermittent failure rate.
  • Avoid patterns known to cause non deterministic failures.
  • Has sufficient documentation.
  • Breakage is expected to be followed by tree closure or backout.
  • Tests run on mozilla-central and all trees that merge into it.
  • Scheduled on every push.
  • Easily run on try server.
  • Supports job retriggers and cancels.

Tier 2 Jobs

Not all test frameworks can meet the Tier 1 requirements. If a test framework can meet all of the Tier 1 requirements except:

  • Breakage does not cause a tree closure or backout;
  • Not scheduled on every push;
  • Does not run on mozilla-central or all trees that merge into it;

then the test framework can still receive limit sheriffing as a Tier 2 test framework. In the event that a Tier 2 test fails, sheriffs will file a bug on the failure but will not close the tree nor back out the changes.

Developing a Non-buildbot Test Framework

There are several prerequisites before beginning a non-buildbot test framework.


Treeherder Credentials

You will need to obtain credentials in order for your test framework to authenticate and submit results to Treeherder. File a bug under product Tree Management, component Treeherder asking for the necessary credentials.

You can use Treeherder staging for developing and testing your framework before deploying the production version which will report to Treeherder production. You will need separate credentials for Treeherder staging and Treeherder production.

Pulse Credentials

You will need to create Pulse users in order to listen to pulse messages. Visit, login with your ldap credentials and create separate pulse users and passwords for production and development.

S3 Credentials

Non-buildbot test frameworks typically upload logs to Amazon's S3. File a bug under product Testing, component General asking for development and production buckets, key ids and keys to be created for you.

Treeherder Groups and Jobs

Treeherder uses Groups and Jobs to organize the display of related tests. You will need to decide which names and symbols to associate with your test framework in the Treeherder display.

If your test framework executes more than one type of test or if your tests execute in chunks, you will need to specify a Group name and Group symbol for your framework. The Group name is the name of the test framework. The group symbol is a shortened version of the group name and is typically a single uppercase letter which is used in Treeherder to identify tests associated with the group. For example, the group name for Autophone is "Autophone" while the group symbol is "A".

You will need a Job name, symbol and an optional description for each distinct type of test that your framework can execute. The Job name is the name of a job type submitted by the test framework. This usually consists of one to a few words which can be used to identify a test type. The Job symbol is a shortened version of the Job name and is typically a single lowercase letter which is used in Treeherder to identify the test. The Job description is an optional short phrase or sentence that can be used to provide more information about the test.

For example, Autophone uses the following:

Group Name Group Symbol Job Name Job Symbol
Autophone A Autophone Throbber t
Autophone A Autophone Webappstartup w

Treeherder displays Autophone test jobs as A(t w).

Try Chooser is a web page which helps developers construct the appropriate try commit message in order to construct the builds and execute the tests they wish to "try" before landing their changes on one of the official source repositories.

File a bug under product Release Engineering, component Tools with keyword trychooser to add your test framework to try chooser.

You will need to create a patch for which adds your test framework to the choices a developer can make.

An example patch from adds the following for Autophone:

<div class='option-subgroup'>
<li><label><input type="checkbox"
  class="subgroup-selector subgroup-all-selector"
  value="autophone-tests">Autophone-tests (all)</label></li>
<li><label><input type="checkbox"
<li><label><input type="checkbox"
<li><label><input type="checkbox"

An example try commit message for Autophone might contain:

try: -b o -p android-api-9,android-api-11 -u autophone-smoketest,autophone-s1s2,autophone-webapp -t none

It is important that you choose values for your test choices which can be used to determine if your test framework should execute tests for a particular try build. Make the values unique but not so complicated that it is difficult to remember.

Implementing a Non-buildbot Test Framework

Pulse monitor

Listening to build pulse messages

Most test frameworks will be driven by the availability of builds.

Listening to the build pulse exchange for finished builds can be used to initiate tests. In addition, listening for builds with commit messages related to the try build commits for your test framework can be used to initiate try build testing.

The data from the finished build pulse message can be used to determine the url to the build, the build's platform, which repository it was built from, and whether it is a debug or opt build. Using this data, you can easily construct a filter which only processes build messages of relevance to your test platform.

The build url can also be transformed into urls for:

  • The build's txt file: This file contains the build id and the link to the changeset from which the build created.
  • The build's crash reporter symbols zip file: This file is normally available for tinderbox builds, but is not available for official nightly builds.
  • The build's tests zip file: This file is only necessary if your test framework intends to run the unit tests that are contained in the build's source repository.
Listening to job action pulse messages

Job action messages are used to detect retrigger and cancel requests initiated by users using the Treeherder UI. Job action messages only contain a limited amount of information and do not directly contain the build url. Storing required information such as build urls in Treeherder artifacts which can be retrieved as needed is one means of maintaining state.

Note these job action messages are only available if initiated from the Treeherder UI and not from buildbot self serve

Implementing a pulse monitor

In order to monitor both build and job action pulse messages, it is necessary to simultaneously listen to messages from two different pulse exchanges. mozillapulse can not be used due to issues with threading which can cause one or more of the pulse consumers to stop processing new messages.

Autophone implements a pulse monitor which can successfully listen to messages from both the build exchange and the job actions exchange. Although it is specific to Autophone, it illustrates several necessary features and can be used as a basis for developing a pulse monitor for your test framework:

  • ability to select either the staging (exchange/treeherder-stage/v1/job-actions) or production (exchange/treeherder/v1/job-actions) Treeherder job action pulse exchanges.
  • ability to filter builds and job actions based on the build's repository.
  • ability to filter builds and job actions based on the build's platform.
  • ability to filter builds and job actions based on the build type.
  • ability to filter try builds based on the tests specified by the check in comment.
  • ability to specify build and job action callback functions which will be called on the selected builds and job actions.

Necessary Build information

Once the url to the build to be tested is available, you can prepare to run your tests.

The urls for builds, build txt files, tests zip files and crash reporter symbols files follow a pattern which can be used to transform one to another.

Replace the build url suffix

Operating System Build url suffix
Android .apk
Linux .tar.bz2
OS X .dmg
Windows .installer.exe
Windows .zip

with the appropriate suffix to obtain the url to the desired file.

File Target suffix
Build txt file .txt
Tests zip file
Crashreporter symbols file

You will need to obtain the repository and revision from which the build was created. These items can be parsed from the downloaded build txt file corresponding to the build. Autophone's contains the function get_build_data which can be used to download and parse the file given the build url.

When you have the repository and revision for the build, you will need to look up Treeherder's revision hash for the build. The revision hash is necessary to associate your test results with the appropriate build on Treeherder. Autophone's contains the function get_treeherder_revision_hash which can be used to obtain the revision hash.

Running tests and reporting to treeherder

Each test, which will be executed for the build, will require a unique guid. You will need to generate the guid and save it for future use as it will be required when updating the status of a test on Treeherder. Autophone's contains the function generate_guid which can be used for this purpose.

I recommend that the following sequence be followed when executing your tests:

1. Submit pending jobs to Treeherder for each of the tests which your framework will execute on the build. This will update Treeherder almost immediately after the build becomes available that your tests are pending. Treeherder artifacts can be used to save information such as the build url that may be needed when processing retrigger or cancel requests. 2. Immediately before running each test, send an update to Treeherder to change the state of the job to running. 3. Execute the test. Note that it should be possible to process the cancel_job job action while the test is running. 4. Once the test completes, upload the logs to S3 and send an update to Treeherder to change the state of the job to complete.

It is important that the final state of each test job you submit to Treeherder be marked as completed. If your test terminates abnormally and does not mark the test completed, it will continue to be displayed on Treeherder as a pending or running job.

If a test job fails to run due to infrastructure or device errors and you wish to retry the job, first update the failed test job on Treeherder as completed with a result of RETRY then start over for this build and test by creating a new guid for the test and submitting a new pending job to Treeherder.

Treeherder submission

Non-buildbot test frameworks submit and update Treeherder using the Treeherder client.

Autophone uses the AutophoneTreeherder class in to submit updates to Treeherder. While AutophoneTreeherder is specific to Autophone, it illustrates several details that will be necessary in any test framework.

  • You can submit updates to a Treeherder job so long as the TreeherderJobCollection was created with job_type='update'. Once you have submitted a TreeherderJobCollection that was not created with job_type='update', you will no longer be able to update the job.
  • Treeherder job updates are not cummulative. You need to specify the full set of information during the initial submission of the job as well as during any subsequent updates.
  • The use of a 'buildapi' artifact is required if you wish Treeherder to make the retrigger and cancel job features available for your test framework. Although non-buildbot test frameworks do not have the concept of buildbot's buildername, it can be emulated using a string of the form: "platform repository buildtype job_name job_symbol".
  • Arbitrary artifacts can be created and stored with your job in order to save any additional information concerning the job that you may wish to retrieve at a later time. Autophone uses artifact named 'privatebuild' to store build urls for use when processing retrigger and cancel job actions.

AutophoneTreeherder reimplements the logic from for obtaining the data for the Bug Suggestions artifact. Bug 1149570 - Treeherder should provide a REST api to obtain bug suggestions has requested that an an external api to retrieve the bug suggestions be added as an enhancement to Treeherder's api.

S3 submission

Autophone uses S3Bucket from to upload files to Amazon's S3 service. also can be used from the command line to list and delete files from an S3 bucket. It should be fully reusable.

Crash processing

It will be necessary to handle crash processing. Unit tests in the source repositories use mozbase's to process crashes and produce a crash signature. Autophone reproduces the same logic using

Crash processing also requires that a build of Google Breakpad be available in order to process the minidumps and crash symbols.

Putting a Test Framework into Production

You should use Treeherder staging ( for testing while you are developing or modifying your test framework.

Once you are ready to begin reporting to Treeherder production (, you need to contact the Sheriffs in order to hide your test results. By default, new tests are visible in Treeherder production. You must first submit test results to Treeherder production before your test framework can be hidden from the default view. Before doing so, arrange for the sheriff to hide the test once it begins submitting results.

Once your test framework meets the Sheriffing Job Visibility Policy requirements, file a bug in product Tree Management, component Treeherder to change the visibility and set the tier status to either Tier 1 or Tier 2.

Even if your test framework does not meet the Tier 1 or Tier 2 job requirements, it can still be useful to submit test results to Treeherder production as a hidden job. In this case, someone other than the sheriffs will need to be responsible for triaging the test results and filing the appropriate bugs.