QA/Automation/Projects/Mozmill Automation/Mozmill2Scripts

From MozillaWiki
Jump to: navigation, search


Name: MozMill 2 Script Updates
Leads: Dave Hunt, Henrik Skupin
Contributors: David Guo


  • Remove Mozmill wrapper dependency
  • Remove Mozmill CLI dependency
  • All past tests must be able to be run on both Mozmill 2.0 and 1.5
  • Must be compatible with Thunderbird and Firefox
  • Must be maintainable

Project Milestones

Milestone Status Description
Remove dependency of Mozmill wrapper Complete; Implementation under review Strategy:, Patch:
Determine if current automation scripts can adapt to Mozmill 2.0 infrastructure Complete The conclusion was that there are no viable options without rewriting some parts of TestRun; Detailed information:
Draft a spec and design the TestRun module remodel Complete; Needs review This effort will outline how TestRun will look inheriting from Mozrunner opposed to MozmillCLI
Implement the new TestRun Not started This effort will swap the Mozmill CLI inheritance with Mozrunner
Refactor and testing Not started After implementation is complete, the new TestRun module will be tested against previous tests and possible ways to refactor will be looked at


Class Diagram


Data Flow



To allow a relative flexible module usage and to make code re-usable for other projects most of the current code should be moved to mozbase or made available as packages on PyPI.

Module Bug Description
application bug 769606 Application specific utilities. While updates will remain in our repository, the handling of the application.ini file could be part of mozrunner.
errors n/a Module which contains all the supported error classes. This module will remain in our repository.
json_file n/a Utilities for JSON files. This module will remain in our repository but should probably be moved to a generic file handling module.
install bug 757411 (done) Installer and uninstaller for Gecko based applications across platforms. This module should be merged with mozinstall.
rdf_parser n/a Parsing RDF files, e.g. install.rdf, with correct namespace handling. This module should be merged with mozprofile.
reports Github (done) Module for custom reports. It will remain in our repository and will work as plugin for Mozmill 2.0
repository done Utilities for handling repositories. We could make it generic to be able to switch between hg and git later. This module will remain in our repository.
scraper Github (done) Scraper for Gecko based applications from We have to unify all the existing tools, e.g. getlatesttinderbox. This module will get its own github repository mozdownload.
testrun n/a Contains available test-runs to be used by the command line scripts


Inherit from Mozrunner with Mozmill object composition

Mozmill 2.0 essentially has the functionality of 1.5 and yet the solutions are not trivial. This issue stems from the fact that TestRun currently inherits implicitly (via object composition) from Mozmill CLI. It enforces us to write TestRun in such a way that we are trying to keep the structure consistent with the CLI even though program flow and the structure is very different between 1.5 and 2.0.

Inheriting from Mozrunner is viable as we can use Mozrunner for Mozilla applications and use Mozmill's API as an object. These two modules will offer the same control that the CLI offers without the maintainability issue.

It is possible to inherit from Mozmill API with Mozrunner object composition or use multiple inheritance, but these relationships are less logical. The TestRun module should be a Mozrunner instance logically due to the fact that it handles the preparation and configuration of tests and the Mozilla application. Another reason would be because it helps to be able to override the init in Mozrunner as we do not want to use any of the Mozrunner parser options.

Changes to existing methods/new additions

The impact of removing Mozmill CLI will affect the following methods within TestRun:

  • __init__
    • Added: logging capability
  • _generate_custom_report
  • prepare_binary
  • prepare_repository
  • prepare_tests
  • run_tests
  • run
  • update_report

These methods will offer the same functionality to the consumer, but will be changed to offer support for Mozmill 2.

There will also be three new methods:

  • get_report
  • send_report(report_url)
  • get_mozmill_version

These two reporting methods offers the TestRun the ability to get and send reports since this concept is very different in Mozmill 1.5 and 2.0. get_mozmill_version will use the Mozmill's metadata to return the version number. This might be useful for classes that inherit TestRun to be able to not run for specific versions.

Another possible feature is to have a constraint "allow_legacy_api" that will prevent systems with the Mozmill 1.5 API to run the tests.


To verify that the implementation is correctly working for Mozmill 1.5:

  1. The results of an older TestRun must match the new one for all the current restart and non-restart tests
  2. Ensure that the results are reported correctly

Likewise, for Mozmill 2.0:

  1. Results of Mozmill 1.5 API restart and non-restart tests are the same for both Mozmill 2.0 and Mozmill 1.5
  2. Ensure that tests using only Mozmill 2.0 API are working
  3. Ensure that the new features (e.g. screenshot) are working by creating simple tests
  4. Ensure that the results are reported correctly