Auto-tools/Projects/m21s

From MozillaWiki
Jump to: navigation, search

Introduction

m21s is the code name of the mozmill-marionette-e10s project, the project whose aim is to replace mozmill with e10s-compliant marionette tests. The project consists roughly of the following pieces:

  • adapt marionette to e10s
  • add features to marionette required by mozmill tests
  • rewrite existing mozmill tests as marionette tests
  • update the mozmill infrastructure and CI to utilize the new tests and harness

Many of these pieces can be done in parallel, but this is a significant project which will likely take multiple months to fully deliver.

Team

  • Jonathan Griffin (jgriffin) - project management
  • Henrik Skupin (whimboo) - mozmill maintainer and infrastructure/CI owner
  • David Burns (AutomatedTester) - Marionette lead
  • Andrew Halberstadt (ahal) - automation developer
  • Chris Manchester (chmanchester) - automation developer
  • Andreas Tolfsen (ato) - automation developer

Problem

Every release we do of Firefox need to have testing done to ensure installation, upgrades, security fixes, and l10n work. Mozmill is a framework that was created in 2008 to do this and we found many test cases that fit into that model. We now need to do this with a modern Firefox (e10s) and with modern tools (Marionette).

Goals & Considerations

Mozmill is currently used for release and update testing by QA. It provides coverage not duplicated elsewhere, and sometimes finds important bugs. Mozmill was written long before e10s and features a model which includes promiscuous chrome-content interactions implemented in a way which is not trivial to port to an e10s-compatible model. Making mozmill e10s-compatible would involve updating the harness, the libraries and the test themselves.

Although this is achievable, there has been momentum building for the idea of migrating mozmill on top of marionette, to benefit from the stability of marionette's protocol. An evaluation of this idea has been performed, and it was determined that this solution would be difficult to implement and likely problematic as well. Combined with the requirement of e10s-compatibility, this idea was determined to be not worth the effort and has been abandoned in favor of simply rewriting all of the existing mozmill tests as marionette tests, after the necessary feature work to marionette has been accomplished.

Non-Goals

We are not looking to duplicate tests between mochitest browser-chrome and Marionette Firefox-UI-Tests. There is a good chance we have some duplicated tests. We are also not trying to automate the entire front end of Firefox.

Dependencies / Who will use this

M21s will be run in buildbot automation for as many tests as possible. For tests which won't work in buildbot automation (due to dependencies and requirements of network access, process uptime, accessing resources, etc.) we will run those remaining tests in a similar fashion to how we run Mozmill on a pool of machines/vms.

This data will be looked at regularly by release management, and eventually possibly a sheriff role for Tier 2 jobs.

Design and Approach

High-level design ideas and concepts. The "how" in a general sense.

Milestones and Dates

  • finish writing Marionette update and security tests [whimboo, chmanchester]
  • get the above tests running in mozmill-CI and work on stabilizing them [whimboo]
  • uplift recent Marionette changes to aurora (38) so that we won't have to depend on mozmill for release testing of the next ESR [AutomatedTester]
  • implement method of running Marionette update tests that span different gecko releases when the marionette client becomes incompatible with sequential versions of the marionette server [chmanchester] (Given that these incompatibilities may not actually arise in practice, this has been deferred until it's established this will save work over patching the marionette client on a case-by-case basis to be compatible with a prior gecko release.)
  • work with Releng to get Marionette update tests running in buildbot release automation [owner: TBD]
    • in parallel, work with Treeherder to enable it to display jobs from buildbot release automation
  • work with Releng to get security and other Marionette tests running in buildbot release automation
  • update the mozmill-crowd add-on to let community still participate in testing (probably low priority), [whimboo]
  • rework Mozmill tests for TPS to become Marionette tests (low priority), [whimboo]
  • gradually turn off tests in mozmill-CI that are covered by m21s tests running in buildbot, with the goal of being completely off of mozmill-CI within two quarters [whimboo]


Implementation

M21s is using Marionette as a technology to drive the tests. This involves adding a handful of features into Marionette while assessing and converting the existing libraries and tests from Mozmill into javascript and python code that uses Marionette.

Getting Involved

Test conversion

We are prioritizing the conversion of update and security tests; others will follow later as time and resources permit.

The repository this work is happening can be found at firefox-ui-tests.

Patch Expectations

Bugs

Open bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Fixed bugs

Full Query
ID Priority Summary Assigned to
1129702 P1 Add support for doubleclicking to marionette's actions Chris Manchester (Offline Aug 16-22)
906712 P1 implement tab modal dialog handling for desktop with Marionette (alert, confirm, prompt) Chris Manchester (Offline Aug 16-22)
1097705 P1 Add ability to right or middle click an element Chris Manchester (Offline Aug 16-22)
940954 P1 Let Marionette support shutdown/restart tests David Burns :automatedtester
941749 P1 getWindows should return tabs Chris Manchester (Offline Aug 16-22)
1103196 P1 Support for untrusted/self-signed TLS certificates Andreas Tolfsen 「:ato」
1106913 P1 Detect cyclic object references when marshaling objects in evaluate.toJSON Andreas Tolfsen 「:ato」
1107336 P1 Split marionette automation API into its own package and not have a monolithic package David Burns :automatedtester
1169180 P1 Intermittent failure for update tests and unit tests in test_about_window.py due to Marionette exception: "win.document.documentElement is null" Chris Manchester (Offline Aug 16-22)
1046512 P3 Marionette breaks in e10s enabled Nightly Chris Manchester (Offline Aug 16-22)
755036 -- Save globals between execute_script calls in 'chrome' contexts Chris Manchester (Offline Aug 16-22)
1096571 -- Don't call into Webdriver's isShown() method when attempting to manipulate content xul elements Andrew Halberstadt [:ahal]
1135846 -- Re-organize marionette actions code to expose actions to chrome scope Chris Manchester (Offline Aug 16-22)
1099331 -- Marionette's navigate errors on loading an about:blocked or about:error page Chris Manchester (Offline Aug 16-22)
1137388 -- Marionette hard-kills the browser for restarts which causes lots of trouble Chris Manchester (Offline Aug 16-22)
1141483 -- After a restart Marionette doesn't restore the previous context Henrik Skupin (:whimboo) [⌚️UTC+2]
1028407 -- Try to make marionette connect to browser sessions quicker Chris Manchester (Offline Aug 16-22)
1104742 -- Stopping Marionette with Ctrl-C does not quit the application Andrew Halberstadt [:ahal]
1141519 -- Marionette hangs if exceptions are happening in content scope David Burns :automatedtester
1141679 -- Marionette doesn't clean up its socket following a restart initiated inside the application Chris Manchester (Offline Aug 16-22)
1080764 -- Add support for anonymous nodes in chrome Andrew Halberstadt [:ahal]
1144907 -- Marionette's gather_debug shouldn't leave calling code in content scope if it didn't start there Chris Manchester (Offline Aug 16-22)
1080772 -- Convert a few mozmill tests to Marionette as POC Henrik Skupin (:whimboo) [⌚️UTC+2]
1114623 -- Implement closeChromeWindow() to close the currently active chrome window Chris Manchester (Offline Aug 16-22)
1083131 -- Marionette does not always remove temporary profiles Chris Manchester (Offline Aug 16-22)
1120630 -- Turn on Mn-e10s Chris Manchester (Offline Aug 16-22)
1169600 -- "MarionetteException: this.browserForTab is null" when closing the first tab of a chrome window Chris Manchester (Offline Aug 16-22)
1084412 -- searchTimeout doesn't work when using findElements Andrew Halberstadt [:ahal]
1120809 -- Marionette sometimes fails to register a content script with e10s enabled Chris Manchester (Offline Aug 16-22)
1169798 -- MarionetteException in switch_to_window() with chrome scope enabled: "this.browser is undefined" Chris Manchester (Offline Aug 16-22)
1088116 -- Marionette support for testing l10n builds at parity with mozmill Chris Manchester (Offline Aug 16-22)
1122609 -- execute_script() called with new_sandbox=False in chrome context does not use current script_args but those from a former call Chris Manchester (Offline Aug 16-22)
1176663 -- [e10s] Marionette.navigate() hangs when first tab gets closed with: NS_ERROR_ILLEGAL_VALUE in nsIMessageSender.sendAsyncMessage
1091662 -- Ensure shortcuts work in Marionette tests Chris Manchester (Offline Aug 16-22)
1123683 -- Set default testing preferences (e.g. disabling FHR and other metric pings) Barbara Miller (:galgeek)
1176758 -- Allow mozprocess to detect and kill detached child processes Henrik Skupin (:whimboo) [⌚️UTC+2]
1092223 -- Marionette tests fail with e10s enabled
1124604 -- Add `focus` parameter to switch_to_window() to stay in compat with switch_to_frame() Henrik Skupin (:whimboo) [⌚️UTC+2]
1202392 -- Improve exception handling and message details in Marionette Henrik Skupin (:whimboo) [⌚️UTC+2]
1093707 -- Using navigate() in chrome scope locks up the browser Chris Manchester (Offline Aug 16-22)
1124667 -- Release marionette-driver 0.2 and marionette-client 0.9 Henrik Skupin (:whimboo) [⌚️UTC+2]
1094441 -- Enable 'css selector' strategy in chrome scope David Burns :automatedtester
1125209 -- Sending a string with capital letters should not trigger a shift key
1096488 -- Marionette support for navigating between remote and non-remote pages with e10s enabled Chris Manchester (Offline Aug 16-22)
1125377 -- Marionette fails during getWindowHandles with e10s turned on when accessing a CPOW Chris Manchester (Offline Aug 16-22)

45 Total; 45 Open (100%); 0 Resolved (0%); 0 Verified (0%);