QA/Electrolysis

From MozillaWiki
< QA
Jump to: navigation, search

This is the test plan for Electrolysis, also referred to as the e10s project or "multiprocess". With e10s enabled, Firefox content runs in a different process than the browser itself. This will help improve performance and security. Our goals in testing are to identify and communicate problems, and to be able to give a recommendation for when e10s may be ready to stay on by default for non-Nightly channels.

People

  • Juan Becerra - QA lead
  • Tracy Walker
  • e10s team

Goal

Help E10S to achieve the same level of quality as non-e10s nightly Firefox by the time it goes to beta.

Priorities - Q1 2015

  1. Define and establish baseline quality for e10s vs. non-e10s versions in terms of stability and memory footprint.
  2. Identify and help triage bugs to address the quality gap
  3. Define quality requirements for ship to aurora/dev edition
  4. Work to identify areas of unknown bugs and risk by experimenting with expanding automated test coverage to previously untested platforms required for ship to aurora/dev edition.

How to test e10s

Please test e10s in Nightly.

To enable e10s, check the "Enable e10s" or "multiprocess" box in about:preferences, and restart Nightly. If you have any problems or questions, drop by #e10s on IRC.

Reporting bugs

When you find a problem, please report a bug. Please include the word "[e10s]" in the summary and set the "tracking-e10s" flag to '?' to ensure your bugs get triaged. (If you are certain a bug is present with e10s enabled, but not present with it turned off.)

Identifying e10s crashes

Find and file e10s crashes by searching crash-stats for reports that include domipcenabled.

Testing addons

You can test popular addons from this list: http://arewee10syet.com

Status

Milestones

Check the tracking-e10s field. ? means it needs triage.

  • Milestone 1: e10s is useable by average Nightly users. bug 997456
  • m4: high priority, needs to be fixed asap
  • m6: blocking uplift to aurora.
  • m8: blocking uplift to beta.
  • +: tracking-e10s+ blocks uplift to release.
  • later: this bug has been triaged, and will not block release.
Releases

Goals for releases so far:

  • Firefox 36: enable e10s by default in Nightly, then disabled in Aurora.
  • Firefox 37: (Nov. 24) e10s enabled by default in Nightly.
  • Firefox 38: This may end up being the target version for e10s to ride the trains to release.

Readiness

Testing e10s really means testing all the functionality of Firefox. There are some key areas in a feature checklist where QA feature owners have expertise in testing, or where we think it will be useful to focus exploratory testing, along with our test coverage of each feature. Here are our current feature testing priorities.

Current features to explore:

  • Developer Tools

Do manual and exploratory testing. Report bugs, and associate them with the Dev Tools e10s tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=875871

  • Session store

Triage

The Electrolysis team meets regularly to triage bugs which are tagged tracking-e10s:?. (TODO: add meeting times)

Blocking Aurora ride (m4)

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Triage incoming bugs to look for likely e10s issues

Once e10s is enabled by default on Nightly, we should see an influx of e10s bugs.

How can we tell they may be e10s related? We will need to evaluate incoming bugs in Nightly with e10s turned on and turned off. Anything that is e10s related should get a whiteboard tag and we may also want to rate them for severity or mark them as blockers. We can ask bug reporters to check in about:support and search for the word "multiprocess" (which means e10s is on).

If you are able to confirm a bug and can tell it is an e10s issue, please nominate the bug for the e10s tracking flag.

Incoming bugs on Nightly 36, not marked with a tracking-e10s flag yet: https://bugzilla.mozilla.org/buglist.cgi?f1=cf_tracking_e10s&list_id=11591695&o1=isempty&o2=notsubstring&query_format=advanced&f2=short_desc&bug_status=UNCONFIRMED&bug_status=NEW&version=36%20Branch&version=36%20Branch&v2=intermittent&product=Core&product=Firefox&product=Toolkit

QA wanted

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Fixed bug triage

e10s bugs marked as RESOLVED FIXED.

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


2015 work

Q1 strategy and goals for QA: https://etherpad.mozilla.org/e10s-strategy-qa

e10s test metrics
Tests enabled on treeherder for e10s

Some marionette, mochitests, reftests for Linux are enabled. A few sets of mochitests are enabled for Win7 and 8.

  • Linux opt Mn-e10s M-e10s( 1 2 3 4 5 bc1 bc2 bc3 dt ) R-e10s( C R-e10s )
  • Linux debug M-e10s( 1 2 3 4 5 )
  • Linux x64 opt M-e10s( 1 2 3 4 5 bc1 bc2 bc3 dt ) R-e10s( C R-e10s )
  • Linux x64 asan M-e10s( 1 2 3 4 5 bc1 bc2 bc3 )
  • Linux x64 debug M-e10s( 1 2* 3 4 5 )
  • Windows 7 opt M-e10s( bc1 bc2 bc3 )
  • Windows 8 x64 opt M-e10s( bc1 bc2 bc3 )
Stability metrics.

This is not as much of a problem as we thought it might be. Having e10s doesn't appear to be causing bad problems with crashes.

Near Horizon Issues and Notes
  • Fixing crasher bug 1116884 will increase stability in e10s.
  • There seems to be traction in platform bugs that are needed for developer tools compatibility in e10s
  • Add-on compatibility work is ongoing.
  • We still don't have a community onboarding story for e10s.
  • Current ship plan:
    • e10s in Aurora 40
    • e10s in Beta 41
    • e10s in Release 42 or later
    • (FYI: Firefox 42 is last release of 2015)

Sprint Notes

Feb 19 - Feb 26

  • [Juan] Followed up with Florin to prioritize session restore and tab browsing regression testing
  • [Juan] Will add a means to track results of that work
  • [Tracy] Dig into crash spike in nightly and make sure bugs are filed
  • [Juan] Follow up with releng about getting the automated tests turned on on central that are passing on holly.
  • [Juan & Tracy] Work on getting test day together for addons
  • [Tracy] continue investigating addon breakage
  • [clint] send Tracy and Juan a set of recent features that landed with known e10s deficiencies