Auto-tools/Projects/Test-Informant: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
 
Line 1: Line 1:
= Overview =
= Overview =


Test Informant is a service designed to monitor test manifests of check-ins as they come in. For each test suite and platform combination, it keeps track of the total number of tests being run, as well as the total number of tests that are skipped or disabled. It saves the results in a database which can later be queried to generate reports that contain additional information, such as how many tests have been enabled or disabled over a given period of time.
Test Informant is a tool designed to report on the state of tests at Mozilla. For each test suite and platform combination, it can display the total number of tests being run, as well as the total number of tests that are skipped or disabled. It can also show how many tests have been added/enabled or removed/disabled over a given period of time.


The goal of Test Informant is to provide a higher level of visibility into the current state of our tests. When someone asks questions like "How many reftests are we running on linux64 debug builds?" or "What percentage of mochitests are skipped on android?" we should be able to answer them by directing them towards a Test Informant report.
The goal of Test Informant is to provide a higher level of visibility into the current state of our tests. When someone asks questions like "How many reftests are we running on linux64 debug builds?" or "What percentage of mochitests are skipped on android?" we should be able to answer them by directing them towards a Test Informant report.
Line 7: Line 7:
= Implementation =
= Implementation =


Test informant uses [https://pulse.mozilla.org pulse] to listen for the completion of builds on mozilla-central. It then downloads the tests.zip of the associated build and parses as many test suite manifests as it knows about. It saves the results in a mongo database and continues indefinitely.
Test informant uses [https://wiki.mozilla.org/Auto-tools/Projects/ActiveData ActiveData] to query which tests have run on the given branch on the given days. It then uses the data to generate an HTML formatted report. The entire tool is written in JS and is executed on the client side, anyone can host and use a static copy of test-informant. It uses JQuery, bootstrap and handlebars to format the reports.
 
A sibling script is used to query the database and generate the html formatted reports. Finally a cron job is set up to run the report generator on a daily/weekly basis and upload the resulting report to [http://brasstacks.mozilla.com/testreports brasstacks].
 
== Limitations ==
 
Test Informant currently has several limitations:
 
* Only test suites with [http://mozbase.readthedocs.org/en/latest/manifestparser.html manifestparser] style manifests (i.e the .ini format) are supported at the moment. Additional kinds of manifests may be supported at a later date.
* This tool just parses manifests, it doesn't actually execute the tests. So any configuration used by manifests that gets added by the harness at runtime, will not be taken into account. All of reftest's filter values are calculated at runtime, so this tool likely won't work with reftest anytime soon.
 
See the [https://github.com/ahal/test-informant/blob/master/informant/config.py configuration file] for a list of supported suites and platforms.


= Contribute =
= Contribute =
Line 24: Line 13:
Please take a look at the [https://github.com/mozilla/test-informant source repository]. For the future, we'd like to:
Please take a look at the [https://github.com/mozilla/test-informant source repository]. For the future, we'd like to:


* Switch to consuming structured logs instead of downloading tests and parsing manifests.
* Export reports into various formats (starting with a simple standalone HTML report)
* Implement a dashboard with graphs over time in addition to the report generator.
* Add more detailed information (i.e distinguish between 'added' and 'removed')
* Improve UI
* Add some basic unit tests.
* Add some basic unit tests.


Line 50: Line 40:
== Testing Code ==
== Testing Code ==


In order to test out your changes, you'll need to install the test-informant package:
In order to test out your changes, you'll need to set up a local webserver to host the test-informant. There are many ways to do this, but if you have python installed, the easiest is:


     $ cd test-informant
     $ cd test-informant
     $ python setup.py develop
     $ python -m SimpleHTTPServer
 
I strongly recommend doing the above in a virtualenv using [http://virtualenvwrapper.readthedocs.org/en/latest/install.html virtualenvwrapper].
 
Test Informant uses MongoDB as a backend, so you'll also need to [http://docs.mongodb.org/manual/installation/ install that]. If you want to run test-informant, you'll need to have a local database server running. To do this simply run:
 
    $ mongod
 
Now that you have the database process running, you'll need to get set up to use pulse. Head over to the [https://pulse.mozilla.org pulse website], sign in with Persona and create a new pulse user. You'll first need to create a Persona account if you don't have one already (in addition to a pulse user). Finally you'll need to create a runtime configuration file to tell Test Informant about what user to use. In a file called `~/.testinrc`, add the following contents:
 
    [settings]
    db_name = test-informant
    db_host = localhost
    db_port = 27017
    num_workers = 4
    max_build_queue_size = 1000
    max_tests_cache_size = 0
   
    [pulse]
    user = <your pulse username>
    password = <your pulse password>
    durable = False
    applabel = test-informant
    topic = build.mozilla-inbound.#
 
You can adjust these settings to taste, but these are good defaults. You're finally ready to start test-informant! To start it simply run:
 
    $ test-informant


You should see a bunch of output at the start and eventually see some builds getting processed. Once builds have been processed, you can verify them by inspecting the database. For information on how to do this, see the [http://docs.mongodb.org/manual/tutorial/getting-started/ official mongodb documentation].
Then browse to http://localhost:8000 (or whatever port was dumped to the console).


== Submitting Changes ==
== Submitting Changes ==

Latest revision as of 13:41, 30 July 2015

Overview

Test Informant is a tool designed to report on the state of tests at Mozilla. For each test suite and platform combination, it can display the total number of tests being run, as well as the total number of tests that are skipped or disabled. It can also show how many tests have been added/enabled or removed/disabled over a given period of time.

The goal of Test Informant is to provide a higher level of visibility into the current state of our tests. When someone asks questions like "How many reftests are we running on linux64 debug builds?" or "What percentage of mochitests are skipped on android?" we should be able to answer them by directing them towards a Test Informant report.

Implementation

Test informant uses ActiveData to query which tests have run on the given branch on the given days. It then uses the data to generate an HTML formatted report. The entire tool is written in JS and is executed on the client side, anyone can host and use a static copy of test-informant. It uses JQuery, bootstrap and handlebars to format the reports.

Contribute

Please take a look at the source repository. For the future, we'd like to:

  • Export reports into various formats (starting with a simple standalone HTML report)
  • Add more detailed information (i.e distinguish between 'added' and 'removed')
  • Improve UI
  • Add some basic unit tests.

If you have any questions or comments about Test Informant, please contact:

  • Andrew Halberstadt, ahalberstadt@mozilla.com, ahal on irc

Getting set up for Development

In order to work on Test-Informant bugs, you'll first need to clone the repository: 

   $ git clone https://github.com/mozilla/test-informant.git

You may want to first fork it to your personal github account and clone that instead. If you do this, you can add an upstream remote: 

   $ git clone https://github.com/<username>/test-informant.git
   $ git remote add mozilla https://github.com/mozilla/test-informant.git

Then you can update your local copy as follows: 

   $ git pull mozilla master
   $ git push origin master

Testing Code

In order to test out your changes, you'll need to set up a local webserver to host the test-informant. There are many ways to do this, but if you have python installed, the easiest is: 

   $ cd test-informant
   $ python -m SimpleHTTPServer

Then browse to http://localhost:8000 (or whatever port was dumped to the console).

Submitting Changes

The preferred way of contributing code to Test Informant is via github pull requests. If you are unfamiliar with pull requests, please read the official documentation. Once you have created a pull request, you'll need to ask for review in the bug you are currently working on. To do this:

  1. Create a new attachment in the bug by clicking 'Add an attachment'.
  2. Click 'paste text as an attachment'.
  3. Copy the link to the github pull request into the text box.
  4. Fill out the description, reviewer and comment as normal and submit.

Bugzilla will automatically detect that a pull request was pasted and create a link to it for you. Once you have an r+, you may need to remind the reviewer to merge your pull request for you!

Bugs

Open bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Fixed bugs

Full Query
ID Priority Summary Assigned to
1124689 -- Test informant should use structured logs instead of parsing manifests Alice Scarpa [:adusca]
1124720 -- Make test-informant listen for test jobs instead of build jobs Alice Scarpa [:adusca]

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Test-Informant&oldid=1087338"