Introduction

m21s is the code name of the mozmill-marionette-e10s project, the project whose aim is to replace mozmill with e10s-compliant marionette tests. The project consists roughly of the following pieces:

• adapt marionette to e10s
• add features to marionette required by mozmill tests
• rewrite existing mozmill tests as marionette tests
• update the mozmill infrastructure and CI to utilize the new tests and harness

Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver.

Team

• Jonathan Griffin (jgriffin) - project management
• Henrik Skupin (whimboo) - mozmill maintainer and infrastructure/CI owner
• David Burns (AutomatedTester) - Marionette lead
• Andrew Halberstadt (ahal) - automation developer
• Chris Manchester (chmanchester) - automation developer
• Andreas Tolfsen (ato) - automation developer

Problem

Every release we do of Firefox need to have testing done to ensure installation, upgrades, security fixes, and l10n work. Mozmill is a framework that was created in 2008 to do this and we found many test cases that fit into that model. We now need to do this with a modern Firefox (e10s) and with modern tools (Marionette).

Goals & Considerations

Mozmill is currently used for release and update testing by QA. It provides coverage not duplicated elsewhere, and sometimes finds important bugs. Mozmill was written long before e10s and features a model which includes promiscuous chrome-content interactions implemented in a way which is not trivial to port to an e10s-compatible model. Making mozmill e10s-compatible would involve updating the harness, the libraries and the test themselves.

Although this is achievable, there has been momentum building for the idea of migrating mozmill on top of marionette, to benefit from the stability of marionette's protocol. An evaluation of this idea has been performed, and it was determined that this solution would be difficult to implement and likely problematic as well. Combined with the requirement of e10s-compatibility, this idea was determined to be not worth the effort and has been abandoned in favor of simply rewriting all of the existing mozmill tests as marionette tests, after the necessary feature work to marionette has been accomplished.

Non-Goals

We are not looking to duplicate tests between mochitest browser-chrome and Marionette Firefox-UI-Tests. There is a good chance we have some duplicated tests. We are also not trying to automate the entire front end of Firefox.

Dependencies / Who will use this

M21s will be run in buildbot automation for as many tests as possible. For tests which won't work in buildbot automation (due to dependencies and requirements of network access, process uptime, accessing resources, etc.) we will run those remaining tests in a similar fashion to how we run Mozmill on a pool of machines/vms.

This data will be looked at regularly by release management, and eventually possibly a sheriff role for Tier 2 jobs.

Design and Approach

High-level design ideas and concepts. The "how" in a general sense.

Milestones and Dates

• finish writing Marionette update and security tests [whimboo, chmanchester]
• get the above tests running in mozmill-CI and work on stabilizing them [whimboo]
  • rewrite mozmill automation scripts ( https://github.com/mozilla/mozmill-automation/ ), [whimboo]
  • update the pre-configured environments ( https://github.com/whimboo/mozmill-environment ), [whimboo]
  • update mozmill-ci to handle marionette instead of mozmill, [whimboo]
  • revise documentation, [whimboo]
• uplift recent Marionette changes to aurora (38) so that we won't have to depend on mozmill for release testing of the next ESR [AutomatedTester]
• implement method of running Marionette update tests that span different gecko releases when the marionette client becomes incompatible with sequential versions of the marionette server [chmanchester] (Given that these incompatibilities may not actually arise in practice, this has been deferred until it's established this will save work over patching the marionette client on a case-by-case basis to be compatible with a prior gecko release.)
• work with Releng to get Marionette update tests running in buildbot release automation [owner: TBD]
  • in parallel, work with Treeherder to enable it to display jobs from buildbot release automation
• work with Releng to get security and other Marionette tests running in buildbot release automation
• update the mozmill-crowd add-on to let community still participate in testing (probably low priority), [whimboo]
• rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo]
• gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo]

Implementation

M21s is using Marionette as a technology to drive the tests. This involves adding a handful of features into Marionette while assessing and converting the existing libraries and tests from Mozmill into javascript and python code that uses Marionette.

Getting Involved

Test conversion

We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit.

The repository this work is happening can be found at firefox-ui-tests.

Patch Expectations

Bugs

Open bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Fixed bugs

45 Total; 45 Open (100%); 0 Resolved (0%); 0 Verified (0%);