Confirmed users
3,376
edits
Auto-tools/Projects/m21s: Difference between revisions
m
converted to the new format, added some verbage
m (converted to the new format, added some verbage) |
|||
| Line 9: | Line 9: | ||
Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver. | Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver. | ||
= Team = | = Team = | ||
| Line 25: | Line 19: | ||
* Andreas Tolfsen (ato) - automation developer | * Andreas Tolfsen (ato) - automation developer | ||
= | = Problem = | ||
Every release we do of Firefox need to have testing done to ensure installation, upgrades, security fixes, and l10n work. Mozmill is a framework that was created in 2008 to do this and we found many test cases that fit into that model. We now need to do this with a modern Firefox (e10s) and with modern tools (Marionette). | |||
= Goals & Considerations= | |||
Mozmill is currently used for release and update testing by QA. It provides coverage not duplicated elsewhere, and sometimes finds important bugs. Mozmill was written long before e10s and features a model which includes promiscuous chrome-content interactions implemented in a way which is not trivial to port to an e10s-compatible model. Making mozmill e10s-compatible would involve updating the harness, the libraries and the test themselves. | |||
Although this is achievable, there has been momentum building for the idea of migrating mozmill on top of marionette, to benefit from the stability of marionette's protocol. An evaluation of this idea has been performed, and it was determined that this solution would be difficult to implement and likely problematic as well. Combined with the requirement of e10s-compatibility, this idea was determined to be not worth the effort and has been abandoned in favor of simply rewriting all of the existing mozmill tests as marionette tests, after the necessary feature work to marionette has been accomplished. | |||
= Non-Goals = | |||
We are not looking to duplicate tests between mochitest browser-chrome and Marionette Firefox-UI-Tests. There is a good chance we have some duplicated tests. We are also not trying to automate the entire front end of Firefox. | |||
= Dependencies / Who will use this = | |||
M21s will be run in buildbot automation for as many tests as possible. For tests which won't work in buildbot automation (due to dependencies and requirements of network access, process uptime, accessing resources, etc.) we will run those remaining tests in a similar fashion to how we run Mozmill on a pool of machines/vms. | |||
This data will be looked at regularly by release management, and eventually possibly a sheriff role for Tier 2 jobs. | |||
= Design and Approach = | |||
''High-level design ideas and concepts. The "how" in a general sense.'' | |||
= Milestones and Dates = | |||
* finish writing Marionette update and security tests [whimboo, chmanchester] | * finish writing Marionette update and security tests [whimboo, chmanchester] | ||
| Line 41: | Line 59: | ||
* rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo] | * rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo] | ||
* gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo] | * gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo] | ||
= Implementation = | |||
M21s is using Marionette as a technology to drive the tests. This involves adding a handful of features into Marionette while assessing and converting the existing libraries and tests from Mozmill into javascript and python code that uses Marionette. | |||
= Getting Involved = | |||
== Test conversion == | == Test conversion == | ||
We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit. | We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit. | ||
The repository this work is happening can be found at https://github.com/mozilla/firefox-ui-tests. | The repository this work is happening can be found at [https://github.com/mozilla/firefox-ui-tests firefox-ui-tests]. | ||
== Patch Expectations == | |||
= Bugs = | == Bugs == | ||
== [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1080766 Open bugs] == | == [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocked%3A1080766 Open bugs] == | ||