QA/Mozmill Test Automation/Dashboard/Documentation
With the continuous efforts to raise the popularity of automated ui testing with Mozmill, a central shared storage is needed to collect and visualize results of executed test-runs. The software to implement has to cover all the different types of possible Mozmill test types and should also be flexible enough to support other additions at a later stage. For the visualization of the test results, different views and sub-views will have to be added. Each view will cover a supported report type.
The central storage mentioned above will be a CouchDB database located on one of our internal servers. The dashboard itself will be implemented as a Sammy application and is a part of the database itself.
The target audience for the dashboard is the testing community around Firefox. Those people will run tests to check the basic functionality of Firefox. But also localizers, who want to verify their localized builds, and add-on authors, who want to check the compatibility of their add-ons, are on our focus. Beside that the Mozilla QA team will use a copy of the dashboard to track the results of release testing.
Objective
- Collect test results for all supported test-runs
- Collect l10n related test results across different localized builds
- Offer an easy way to analyze results of automated Mozmill tests
- Improve the collaboration with the community in the automation perspective
- Offering a free service to support localizers and add-on authors
Product Components
Report Collector
Whenever a test-run is executed and test results are available and are uploaded to the couchdb database, the collector script of the database has to make sure that the correct data is contained in the report.
The following tasks listed below have to be run:
- Check the type of the report. If the type is ‘mozmill’ or ‘mozmill-restart’ change it to ‘firefox-general’. This is needed as a fallback solution if the automation scripts cannot be used for a testrun. If such a transformation would not happen, those results wouldn’t be visible in the dashboard.
- Please note that we will not update the current report version number, because the collector cannot add or update additionally information. That means that some additional features would not be available for those reports.
- Update the entry for time_uploaded to the current time. This timestamp is not visible in the ui.
- Convert the full path of the test file name into a canonical path. Therefore backslashes have to be replaced with slashes first, followed by the removal of all hierarchies above the firefox folder for general tests. For update, add-ons, l10n, and accessibility tests a slight modification is necessary because the root path will be different.
Result Views
The visualization of test results will be divided into different views. Each view will cover a supported test-run. That allows an easier navigation between results of different test-runs. Each view will be directly accessible via a canonical URL.
Information for reports of all supported test-runs can be found in this spreadsheet.
- List of views
