QA/Automation/Projects/Mozmill Automation/Mozmill2Scripts
Contents
Overview
| Name: | MozMill 2 Script Updates |
| Leads: | Dave Hunt, Henrik Skupin |
| Contributors: | David Guo |
| Repository: | https://github.com/whimboo/mozmill-automation |
| Bugzilla: | https://bugzilla.mozilla.org/show_bug.cgi?id=732134 |
Requirements
- Remove Mozmill wrapper dependency
- Remove Mozmill CLI dependency
- All past tests must be able to be run on both Mozmill 2.0 and 1.5
- Must be compatible with Thunderbird and Firefox
- Must be maintainable
Project Milestones
| Milestone | Status | Description |
| Remove dependency of Mozmill wrapper | Complete; Implementation under review | Strategy: https://etherpad.mozilla.org/mozmill-wrapper-removal, Patch: https://bugzilla.mozilla.org/show_bug.cgi?id=732134 |
| Determine if current automation scripts can adapt to Mozmill 2.0 infrastructure | Complete | The conclusion was that there are no viable options without rewriting some parts of TestRun; Detailed information: https://etherpad.mozilla.org/mozmill2strategydraft |
| Draft a spec and design the TestRun module remodel | Complete; Needs review | This effort will outline how TestRun will look inheriting from Mozrunner opposed to MozmillCLI |
| Implement the new TestRun | Not started | This effort will swap the Mozmill CLI inheritance with Mozrunner |
| Refactor and testing | Not started | After implementation is complete, the new TestRun module will be tested against previous tests and possible ways to refactor will be looked at |
Architecture
Class Diagram
Data Flow
Components
To allow a relative flexible module usage and to make code re-usable for other projects most of the current code should be moved to mozbase or made available as packages on PyPI.
| Module | Bug | Description |
| application | bug 769606 | Application specific utilities. While updates will remain in our repository, the handling of the application.ini file could be part of mozrunner. |
| errors | n/a | Module which contains all the supported error classes. This module will remain in our repository. |
| json_file | n/a | Utilities for JSON files. This module will remain in our repository but should probably be moved to a generic file handling module. |
| install | |
Installer and uninstaller for Gecko based applications across platforms. This module should be merged with mozinstall. |
| rdf_parser | n/a | Parsing RDF files, e.g. install.rdf, with correct namespace handling. This module should be merged with mozprofile. |
| reports | |
Module for custom reports. It will remain in our repository and will work as plugin for Mozmill 2.0 |
| repository | done | Utilities for handling repositories. We could make it generic to be able to switch between hg and git later. This module will remain in our repository. |
| scraper | |
Scraper for Gecko based applications from ftp.mozilla.org. We have to unify all the existing tools, e.g. getlatesttinderbox. This module will get its own github repository mozdownload. |
| testrun | n/a | Contains available test-runs to be used by the command line scripts |
Proposal
Inherit from Mozrunner with Mozmill object composition
Mozmill 2.0 essentially has the functionality of 1.5 and yet the solutions are not trivial. This issue stems from the fact that TestRun currently inherits implicitly (via object composition) from Mozmill CLI. It enforces us to write TestRun in such a way that we are trying to keep the structure consistent with the CLI even though program flow and the structure is very different between 1.5 and 2.0.
Inheriting from Mozrunner is viable as we can use Mozrunner for Mozilla applications and use Mozmill's API as an object. These two modules will offer the same control that the CLI offers without the maintainability issue.
It is possible to inherit from Mozmill API with Mozrunner object composition or use multiple inheritance, but these relationships are less logical. The TestRun module should be a Mozrunner instance logically due to the fact that it handles the preparation and configuration of tests and the Mozilla application. Another reason would be because it helps to be able to override the init in Mozrunner as we do not want to use any of the Mozrunner parser options.
Changes to existing methods/new additions
The impact of removing Mozmill CLI will affect the following methods within TestRun:
- __init__
- Added: logging capability
- _generate_custom_report
- prepare_binary
- prepare_repository
- prepare_tests
- run_tests
- run
- update_report
These methods will offer the same functionality to the consumer, but will be changed to offer support for Mozmill 2.
There will also be three new methods:
- get_report
- send_report(report_url)
- get_mozmill_version
These two reporting methods offers the TestRun the ability to get and send reports since this concept is very different in Mozmill 1.5 and 2.0. get_mozmill_version will use the Mozmill's metadata to return the version number. This might be useful for classes that inherit TestRun to be able to not run for specific versions.
Another possible feature is to have a constraint "allow_legacy_api" that will prevent systems with the Mozmill 1.5 API to run the tests.
Verification
To verify that the implementation is correctly working for Mozmill 1.5:
- The results of an older TestRun must match the new one for all the current restart and non-restart tests
- Ensure that the results are reported correctly
Likewise, for Mozmill 2.0:
- Results of Mozmill 1.5 API restart and non-restart tests are the same for both Mozmill 2.0 and Mozmill 1.5
- Ensure that tests using only Mozmill 2.0 API are working
- Ensure that the new features (e.g. screenshot) are working by creating simple tests
- Ensure that the results are reported correctly
