Electrolysis/Experiments

From MozillaWiki
Jump to: navigation, search

Electrolysis Experiments

This page intends to document the A/B experiments that were done during the development of e10s.

Experiment mechanisms

Telemetry Experiments

The first experiments were done through the Telemetry Experiments system, which is an A/B mechanism meant to test new features to telemetry-enabled users. The experiments were published in the production server for this mechanism.

Staged Rollout - System add-on

With the approach of releasing e10s to the release audience, it was necessary to have a more detailed mechanism specific to e10s in order to be able to do a staged rollout of the feature and quickly increase/decrease the rollout rate. To do that, the new System Add-on mechanism was chosen, because it can be updated out-of-band from the release process.

After the system add-on was developed, we started using it to run new phases of e10s experiments on Beta, in order to also test the add-on itself in preparation for Release.

List of experiments

Note: all dates are an approximation from the end-user's point of view, because they may take a couple of days to be updated to the build where the experiment was started/terminated.

Beta 49

Single phase

Start: beta 1 (Aug 2 or 3)

End: still running

Range: [beta 1, ...)

Sample: 50% test/control for each group

Details:

  • APZ is enabled by default (conditional to e10s) starting in 47
  • No accessibility on Mac, Windows, Linux
  • No Mac OS X 10.6 - 10.8
  • No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
  • Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)

Addons Experiment Details:

  • This experiment is the first to test some add-ons with e10s. The cohorts are divided like so:
    • Test/Control:
      • Users with no add-ons, and no other disqualifying criteria (same as previous experiments)
    • addons-set2a-test / addons-set2a-control
      • Users with any number > 0 of add-ons, where all of their add-ons are part of the experiment list (see below)
    • disqualified-test / disqualified-control
      • Users with any number > 0 of add-ons, where any of their add-ons are not part of the experiment list. (Or may be disqualified by other criteria such as accessibility, OS, etc)

Addons Experiment List:

  • All WebExtension-style add-ons
  • And the following 10 bootstrap/jetpack add-ons:
    • Greasemonkey, Download YouTube Videos as MP4, Video Download Helper, Lightbeam, Flashblock, Adblock Plus, uBlock Origin, Emoji Cheatsheet, Awesome Screenshot Plus, Personas Plus

Related bugs: Tracking bug, automatically started (no bug), mechanism to control addons experiment list,

Analyses:

Beta 48

Single phase

Start: beta 1 (built on Jun 6, shipped Jun 8)

End: ran through the entire beta (ended on Aug 2)

Range: [beta 1, ...)

Sample: 50% of eligible users (yield ~20%)

Details:

  • APZ is enabled by default (conditional to e10s) starting in 47
  • No add-ons
  • No accessibility on Mac and Windows
  • No accessibility on Linux starting on build 2 (bug 1277882)
  • No Mac OS X 10.6 - 10.8
  • GTK+ 3.20+ users reincluded
  • No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
  • Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)


Related bugs: Tracking bug, automatically started (no bug), unblock GTK+ 3.20+, block on OSX 10.6-10.8, block on release WinXP, block on linux a11y, block on WinXP+Accel

Analyses:

Beta 47

Single phase

Start: beta 1 (built on Apr 26, shipped Apr 27)

End: ran through the entire beta (Jun 8)

Range: [beta 1, ...)

Sample: 50% of eligible users (yield ~20%)

Details:

  • APZ is enabled by default (conditional to e10s) starting in 47
  • No add-ons, no a11y
  • No GTK+ 3.20+
  • E10SCohort annotation added to crash report
  • "disqualified" cohort is now split into "disqualified-test" and "disqualified-control" which should help verify the even-ness of the data

Related bugs: Tracking bug, automatically started (no bug), block GTK+ 3.20+, disqualified -> disqualified-test/disqualified-control, E10SCohort annotation on crash reporter

Analyses:

Beta 46

2nd phase

Start: beta 3 (built on March 21, shipped March 22)

End: beta 8 (built on April 4, shipped April 5)

Range: [beta 3, beta 8) (15 days)

Sample: 50% of ~40% eligible users (yield ~20%)

Details:

  • No add-ons, no a11y
  • APZ enabled (for the first time)
  • Through the system add-on (for the first time)

Related bugs: bugs related to the system add-on, stop it

Analyses:

1st phase

Start: March 9

End: March 21

Range: [March 9, March 21) (12 days)

Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)

Details:

  • No add-ons
  • APZ disabled
  • Through Telemetry Experiments

Related bugs: start it, stop it

Analyses:

Beta 45

2nd phase

Start: February 12

End: February 22

Range: [February 12, February 22) (11 days)

Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)

Details:

  • No add-ons (add-on users filtered for the first time)
  • APZ disabled
  • Through Telemetry Experiments

Related bugs: start it, stop it, [filter add-on users from e10s https://bugzilla.mozilla.org/show_bug.cgi?id=1234675]

Analyses:

1st phase

Start: January 25

End: February 11

Range: [January 25, February 11) (17 days)

Sample: 50% targeted by experiment * 50% A/B split (yield ~25%)

Details:

  • Add-ons enabled
    • LastPass 3.* and Skype Click-to-Call users removed from the experiment after a few days due to crash
  • APZ disabled
  • No a11y
  • Through Telemetry Experiments

Related bugs: start it, automatically stopped by date, block LastPass 3.* users, Block Skype Click-to-Call users

Analyses: