Auto-tools/Projects/m21s

From MozillaWiki
Jump to: navigation, search

Introduction

m21s is the code name of the mozmill-marionette-e10s project, the project whose aim is to replace mozmill with e10s-compliant marionette tests. The project consists roughly of the following pieces:

  • adapt marionette to e10s
  • add features to marionette required by mozmill tests
  • rewrite existing mozmill tests as marionette tests
  • update the mozmill infrastructure and CI to utilize the new tests and harness

Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver.

Team

  • Jonathan Griffin (jgriffin) - project management
  • Henrik Skupin (whimboo) - mozmill maintainer and infrastructure/CI owner
  • David Burns (AutomatedTester) - Marionette lead
  • Andrew Halberstadt (ahal) - automation developer
  • Chris Manchester (chmanchester) - automation developer
  • Andreas Tolfsen (ato) - automation developer

Problem

Every release we do of Firefox need to have testing done to ensure installation, upgrades, security fixes, and l10n work. Mozmill is a framework that was created in 2008 to do this and we found many test cases that fit into that model. We now need to do this with a modern Firefox (e10s) and with modern tools (Marionette).

Goals & Considerations

Mozmill is currently used for release and update testing by QA. It provides coverage not duplicated elsewhere, and sometimes finds important bugs. Mozmill was written long before e10s and features a model which includes promiscuous chrome-content interactions implemented in a way which is not trivial to port to an e10s-compatible model. Making mozmill e10s-compatible would involve updating the harness, the libraries and the test themselves.

Although this is achievable, there has been momentum building for the idea of migrating mozmill on top of marionette, to benefit from the stability of marionette's protocol. An evaluation of this idea has been performed, and it was determined that this solution would be difficult to implement and likely problematic as well. Combined with the requirement of e10s-compatibility, this idea was determined to be not worth the effort and has been abandoned in favor of simply rewriting all of the existing mozmill tests as marionette tests, after the necessary feature work to marionette has been accomplished.

Non-Goals

We are not looking to duplicate tests between mochitest browser-chrome and Marionette Firefox-UI-Tests. There is a good chance we have some duplicated tests. We are also not trying to automate the entire front end of Firefox.

Dependencies / Who will use this

M21s will be run in buildbot automation for as many tests as possible. For tests which won't work in buildbot automation (due to dependencies and requirements of network access, process uptime, accessing resources, etc.) we will run those remaining tests in a similar fashion to how we run Mozmill on a pool of machines/vms.

This data will be looked at regularly by release management, and eventually possibly a sheriff role for Tier 2 jobs.

Design and Approach

High-level design ideas and concepts. The "how" in a general sense.

Milestones and Dates

  • finish writing Marionette update and security tests [whimboo, chmanchester]
  • get the above tests running in mozmill-CI and work on stabilizing them [whimboo]
  • uplift recent Marionette changes to aurora (38) so that we won't have to depend on mozmill for release testing of the next ESR [AutomatedTester]
  • implement method of running Marionette update tests that span different gecko releases when the marionette client becomes incompatible with sequential versions of the marionette server [chmanchester] (Given that these incompatibilities may not actually arise in practice, this has been deferred until it's established this will save work over patching the marionette client on a case-by-case basis to be compatible with a prior gecko release.)
  • work with Releng to get Marionette update tests running in buildbot release automation [owner: TBD]
    • in parallel, work with Treeherder to enable it to display jobs from buildbot release automation
  • work with Releng to get security and other Marionette tests running in buildbot release automation
  • update the mozmill-crowd add-on to let community still participate in testing (probably low priority), [whimboo]
  • rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo]
  • gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo]


Implementation

M21s is using Marionette as a technology to drive the tests. This involves adding a handful of features into Marionette while assessing and converting the existing libraries and tests from Mozmill into javascript and python code that uses Marionette.

Getting Involved

Test conversion

We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit.

The repository this work is happening can be found at firefox-ui-tests.

Patch Expectations

Bugs

Open bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Fixed bugs

Full Query
ID Priority Summary Assigned to
1129702 P1 Add support for doubleclicking to marionette's actions Chris Manchester (:chmanchester)
1097705 P1 Add ability to right or middle click an element Chris Manchester (:chmanchester)
906712 P1 implement tab modal dialog handling for desktop with Marionette (alert, confirm, prompt) Chris Manchester (:chmanchester)
940954 P1 Let Marionette support shutdown/restart tests David Burns :automatedtester
1103196 P1 Support for untrusted/self-signed TLS certificates Andreas Tolfsen 「:ato」
941749 P1 getWindows should return tabs Chris Manchester (:chmanchester)
1106913 P1 Detect cyclic object references when marshaling objects in evaluate.toJSON Andreas Tolfsen 「:ato」
1107336 P1 Split marionette automation API into its own package and not have a monolithic package David Burns :automatedtester
1169180 P1 Intermittent failure for update tests and unit tests in test_about_window.py due to Marionette exception: "win.document.documentElement is null" Chris Manchester (:chmanchester)
1046512 P3 Marionette breaks in e10s enabled Nightly Chris Manchester (:chmanchester)
1125209 -- Sending a string with capital letters should not trigger a shift key
1094441 -- Enable 'css selector' strategy in chrome scope David Burns :automatedtester
1125377 -- Marionette fails during getWindowHandles with e10s turned on when accessing a CPOW Chris Manchester (:chmanchester)
1096488 -- Marionette support for navigating between remote and non-remote pages with e10s enabled Chris Manchester (:chmanchester)
755036 -- Save globals between execute_script calls in 'chrome' contexts Chris Manchester (:chmanchester)
1096571 -- Don't call into Webdriver's isShown() method when attempting to manipulate content xul elements Andrew Halberstadt [:ahal]
1135846 -- Re-organize marionette actions code to expose actions to chrome scope Chris Manchester (:chmanchester)
1099331 -- Marionette's navigate errors on loading an about:blocked or about:error page Chris Manchester (:chmanchester)
1137388 -- Marionette hard-kills the browser for restarts which causes lots of trouble Chris Manchester (:chmanchester)
1141483 -- After a restart Marionette doesn't restore the previous context Henrik Skupin (:whimboo) [⌚️UTC+2]
1104742 -- Stopping Marionette with Ctrl-C does not quit the application Andrew Halberstadt [:ahal]
1141519 -- Marionette hangs if exceptions are happening in content scope David Burns :automatedtester
1028407 -- Try to make marionette connect to browser sessions quicker Chris Manchester (:chmanchester)
1141679 -- Marionette doesn't clean up its socket following a restart initiated inside the application Chris Manchester (:chmanchester)
1144907 -- Marionette's gather_debug shouldn't leave calling code in content scope if it didn't start there Chris Manchester (:chmanchester)
1080764 -- Add support for anonymous nodes in chrome Andrew Halberstadt [:ahal]
1114623 -- Implement closeChromeWindow() to close the currently active chrome window Chris Manchester (:chmanchester)
1080772 -- Convert a few mozmill tests to Marionette as POC Henrik Skupin (:whimboo) [⌚️UTC+2]
1120630 -- Turn on Mn-e10s Chris Manchester (:chmanchester)
1169600 -- "MarionetteException: this.browserForTab is null" when closing the first tab of a chrome window Chris Manchester (:chmanchester)
1083131 -- Marionette does not always remove temporary profiles Chris Manchester (:chmanchester)
1120809 -- Marionette sometimes fails to register a content script with e10s enabled Chris Manchester (:chmanchester)
1169798 -- MarionetteException in switch_to_window() with chrome scope enabled: "this.browser is undefined" Chris Manchester (:chmanchester)
1084412 -- searchTimeout doesn't work when using findElements Andrew Halberstadt [:ahal]
1122609 -- execute_script() called with new_sandbox=False in chrome context does not use current script_args but those from a former call Chris Manchester (:chmanchester)
1176663 -- [e10s] Marionette.navigate() hangs when first tab gets closed with: NS_ERROR_ILLEGAL_VALUE in nsIMessageSender.sendAsyncMessage
1088116 -- Marionette support for testing l10n builds at parity with mozmill Chris Manchester (:chmanchester)
1123683 -- Set default testing preferences (e.g. disabling FHR and other metric pings) Barbara Miller (:galgeek)
1176758 -- Allow mozprocess to detect and kill detached child processes Henrik Skupin (:whimboo) [⌚️UTC+2]
1091662 -- Ensure shortcuts work in Marionette tests Chris Manchester (:chmanchester)
1124604 -- Add `focus` parameter to switch_to_window() to stay in compat with switch_to_frame() Henrik Skupin (:whimboo) [⌚️UTC+2]
1202392 -- Improve exception handling and message details in Marionette Henrik Skupin (:whimboo) [⌚️UTC+2]
1092223 -- Marionette tests fail with e10s enabled
1124667 -- Release marionette-driver 0.2 and marionette-client 0.9 Henrik Skupin (:whimboo) [⌚️UTC+2]
1093707 -- Using navigate() in chrome scope locks up the browser Chris Manchester (:chmanchester)

45 Total; 45 Open (100%); 0 Resolved (0%); 0 Verified (0%);