Electrolysis/Experiments
Electrolysis Experiments
This page intends to document the A/B experiments that were done during the development of e10s.
Experiment mechanisms
Telemetry Experiments
The first experiments were done through the Telemetry Experiments system, which is an A/B mechanism meant to test new features to telemetry-enabled users. The experiments were published in the production server for this mechanism.
Staged Rollout - System add-on
With the approach of releasing e10s to the release audience, it was necessary to have a more detailed mechanism specific to e10s in order to be able to do a staged rollout of the feature and quickly increase/decrease the rollout rate. To do that, the new System Add-on mechanism was chosen, because it can be updated out-of-band from the release process.
After the system add-on was developed, we started using it to run new phases of e10s experiments on Beta, in order to also test the add-on itself in preparation for Release.
Note: all dates are an approximation from the end-user's point of view, because they may take a couple of days to be updated to the build where the experiment was started/terminated.
List of e10s-multi experiments
Beta 54
Phase 2
Start: beta 2 (or 3?) (Apr 27)
Sample: 10% non-e10s, of the 90%, 50% 1 content process (control) 50% 4 content processes (test)
Phase 1
Start: beta 1 (Apr 19)
End: beta 2 (or 3) (Apr 26)
Sample: 10% non-e10s of the 90%, 25% 1 content process 25% 2 content processes 25% 4 content processes 25% 8 content processes
List of e10s experiments
Beta 49
Single phase
Start: beta 1 (Aug 2 or 3)
End: still running
Range: [beta 1, ...)
Sample: 50% test/control for each group
Details:
- APZ is enabled by default (conditional to e10s) starting in 47
- No accessibility on Mac, Windows, Linux
- No Mac OS X 10.6 - 10.8
- No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
- Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)
Addons Experiment Details:
- This experiment is the first to test some add-ons with e10s. The cohorts are divided like so:
- Test/Control:
- Users with no add-ons, and no other disqualifying criteria (same as previous experiments)
- addons-set2a-test / addons-set2a-control
- Users with any number > 0 of add-ons, where all of their add-ons are part of the experiment list (see below)
- disqualified-test / disqualified-control
- Users with any number > 0 of add-ons, where any of their add-ons are not part of the experiment list. (Or may be disqualified by other criteria such as accessibility, OS, etc)
- Test/Control:
Addons Experiment List:
- All WebExtension-style add-ons
- And the following 10 bootstrap/jetpack add-ons:
- Greasemonkey, Download YouTube Videos as MP4, Video Download Helper, Lightbeam, Flashblock, Adblock Plus, uBlock Origin, Emoji Cheatsheet, Awesome Screenshot Plus, Personas Plus
Related bugs: Tracking bug, automatically started (no bug), mechanism to control addons experiment list,
Analyses:
Beta 48
Single phase
Start: beta 1 (built on Jun 6, shipped Jun 8)
End: ran through the entire beta (ended on Aug 2)
Range: [beta 1, ...)
Sample: 50% of eligible users (yield ~20%)
Details:
- APZ is enabled by default (conditional to e10s) starting in 47
- No add-ons
- No accessibility on Mac and Windows
- No accessibility on Linux starting on build 2 (bug 1277882)
- No Mac OS X 10.6 - 10.8
- GTK+ 3.20+ users reincluded
- No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
- Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)
Related bugs: Tracking bug, automatically started (no bug), unblock GTK+ 3.20+, block on OSX 10.6-10.8, block on release WinXP, block on linux a11y, block on WinXP+Accel
Analyses:
Beta 47
Single phase
Start: beta 1 (built on Apr 26, shipped Apr 27)
End: ran through the entire beta (Jun 8)
Range: [beta 1, ...)
Sample: 50% of eligible users (yield ~20%)
Details:
- APZ is enabled by default (conditional to e10s) starting in 47
- No add-ons, no a11y
- No GTK+ 3.20+
- E10SCohort annotation added to crash report
- "disqualified" cohort is now split into "disqualified-test" and "disqualified-control" which should help verify the even-ness of the data
Related bugs: Tracking bug, automatically started (no bug), block GTK+ 3.20+, disqualified -> disqualified-test/disqualified-control, E10SCohort annotation on crash reporter
Analyses:
Beta 46
2nd phase
Start: beta 3 (built on March 21, shipped March 22)
End: beta 8 (built on April 4, shipped April 5)
Range: [beta 3, beta 8) (15 days)
Sample: 50% of ~40% eligible users (yield ~20%)
Details:
- No add-ons, no a11y
- APZ enabled (for the first time)
- Through the system add-on (for the first time)
Related bugs: bugs related to the system add-on, stop it
Analyses:
1st phase
Start: March 9
End: March 21
Range: [March 9, March 21) (12 days)
Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
Details:
- No add-ons
- APZ disabled
- Through Telemetry Experiments
Related bugs: start it, stop it
Analyses:
Beta 45
2nd phase
Start: February 12
End: February 22
Range: [February 12, February 22) (11 days)
Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
Details:
- No add-ons (add-on users filtered for the first time)
- APZ disabled
- Through Telemetry Experiments
Related bugs: start it, stop it, [filter add-on users from e10s https://bugzilla.mozilla.org/show_bug.cgi?id=1234675]
Analyses:
1st phase
Start: January 25
End: February 11
Range: [January 25, February 11) (17 days)
Sample: 50% targeted by experiment * 50% A/B split (yield ~25%)
Details:
- Add-ons enabled
- LastPass 3.* and Skype Click-to-Call users removed from the experiment after a few days due to crash
- APZ disabled
- No a11y
- Through Telemetry Experiments
Related bugs: start it, automatically stopped by date, block LastPass 3.* users, Block Skype Click-to-Call users
Analyses:
