Overview

Test Informant is a service designed to monitor test manifests of check-ins as they come in. For each test suite and platform combination, it keeps track of the total number of tests being run, as well as the total number of tests that are skipped or disabled. It saves the results in a database which can later be queried to generate reports that contain additional information, such as how many tests have been enabled or disabled over a given period of time.

The goal of Test Informant is to provide a higher level of visibility into the current state of our tests. When someone asks questions like "How many reftests are we running on linux64 debug builds?" or "What percentage of mochitests are skipped on android?" we should be able to answer them by directing them towards a Test Informant report.

Implementation

Test informant uses pulse to listen for the completion of builds on mozilla-central. It then downloads the tests.zip of the associated build and parses as many test suite manifests as it knows about. It saves the results in a mongo database and continues indefinitely.

A sibling script is used to query the database and generate the html formatted reports. Finally a cron job is set up to run the report generator on a daily/weekly basis and upload the resulting report to brasstacks.

Limitations

Test Informant currently has several limitations:

Only test suites with manifestparser style manifests (i.e the .ini format) are supported at the moment. Additional kinds of manifests may be supported at a later date.
This tool just parses manifests, it doesn't actually execute the tests. So any configuration used by manifests that gets added by the harness at runtime, will not be taken into account. All of reftest's filter values are calculated at runtime, so this tool likely won't work with reftest anytime soon.

See the configuration file for a list of supported suites and platforms.

Contribute

Please take a look at the source repository. For the future, we'd like to:

Switch to consuming structured logs instead of downloading tests and parsing manifests.
Implement a dashboard with graphs over time in addition to the report generator.
Add some basic unit tests.

If you have any questions or comments about Test Informant, please contact:

Andrew Halberstadt, ahalberstadt@mozilla.com, ahal on irc

Getting set up for Development

In order to work on Test-Informant bugs, you'll first need to clone the repository:

   $ git clone https://github.com/mozilla/test-informant.git

You may want to first fork it to your personal github account and clone that instead. If you do this, you can add an upstream remote:

   $ git clone https://github.com/<username>/test-informant.git
   $ git remote add mozilla https://github.com/mozilla/test-informant.git

Then you can update your local copy as follows:

   $ git pull mozilla master
   $ git push origin master

Testing Code

In order to test out your changes, you'll need to install the test-informant package:

   $ cd test-informant
   $ python setup.py develop

I strongly recommend doing the above in a virtualenv using virtualenvwrapper.

Test Informant uses MongoDB as a backend, so you'll also need to install that. If you want to run test-informant, you'll need to have a local database server running. To do this simply run:

   $ mongod

Now that you have the database process running, you'll need to get set up to use pulse. Head over to the pulse website, sign in with Persona and create a new pulse user. You'll first need to create a Persona account if you don't have one already (in addition to a pulse user). Finally you'll need to create a runtime configuration file to tell Test Informant about what user to use. In a file called `~/.testinrc`, add the following contents:

   [settings]
   db_name = test-informant
   db_host = localhost
   db_port = 27017
   num_workers = 4
   max_build_queue_size = 1000
   max_tests_cache_size = 0
   
   [pulse]
   user = <your pulse username>
   password = <your pulse password>
   durable = False
   applabel = test-informant
   topic = build.mozilla-inbound.#

You can adjust these settings to taste, but these are good defaults. You're finally ready to start test-informant! To start it simply run:

   $ test-informant

You should see a bunch of output at the start and eventually see some builds getting processed. Once builds have been processed, you can verify them by inspecting the database. For information on how to do this, see the official mongodb documentation.

Submitting Changes

The preferred way of contributing code to Test Informant is via github pull requests. If you are unfamiliar with pull requests, please read the official documentation. Once you have created a pull request, you'll need to ask for review in the bug you are currently working on. To do this:

Create a new attachment in the bug by clicking 'Add an attachment'.
Click 'paste text as an attachment'.
Copy the link to the github pull request into the text box.
Fill out the description, reviewer and comment as normal and submit.

Bugzilla will automatically detect that a pull request was pasted and create a link to it for you. Once you have an r+, you may need to remind the reviewer to merge your pull request for you!

Bugs

Open bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Fixed bugs

Full Query

ID	Priority	Summary	Assigned to
1124689	--	Test informant should use structured logs instead of parsing manifests	Alice Scarpa [:adusca]
1124720	--	Make test-informant listen for test jobs instead of build jobs	Alice Scarpa [:adusca]

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Auto-tools/Projects/Test-Informant

Contents