Confirm
371
edits
Changes
First version of page
= Electrolysis Experiments =
This page intends to document the A/B experiments that were done during the development of e10s.
== Experiment mechanisms ==
=== Telemetry Experiments ===
The first experiments were done through the [[QA/Telemetry|Telemetry Experiments]] system, which is an A/B mechanism meant to test new features to telemetry-enabled users. The experiments were published in the [http://telemetry-experiment.cdn.mozilla.net/ production server] for this mechanism.
=== Staged Rollout - System add-on ===
With the approach of releasing e10s to the release audience, it was necessary to have a more detailed mechanism specific to e10s in order to be able to do a staged rollout of the feature and quickly increase/decrease the rollout rate. To do that, the new System Add-on mechanism was chosen, because it can be updated out-of-band from the release process.
After the [https://bugzilla.mozilla.org/show_bug.cgi?id=1249845 system add-on was developed], we started using it to run new phases of e10s experiments on Beta, in order to also test the add-on itself in preparation for Release.
= List of experiments =
Note: all dates are an approximation from the end-user's point of view, because they may take a couple of days to be updated to the build where the experiment was started/terminated.
== Beta 47 ==
=== 1st phase ===
'''Start''': beta 1 (built on May 26, shipped May 27)
'''End''': still running
'''Range''': '''['''beta 1, ...''')'''
'''Sample''': 50% of eligible users (yield ~20%)
'''Details''':
* APZ is enabled by default (conditional to e10s) starting in 47
* No add-ons, no a11y
* No GTK+ 3.20+
* E10SCohort annotation added to crash report
* "disqualified" cohort is now split into "disqualified-test" and "disqualified-control" which should help verify the even-ness of the data
'''Related bugs''': automatically started (no bug), [https://bugzilla.mozilla.org/show_bug.cgi?id=1266213 block GTK+ 3.20+], [https://bugzilla.mozilla.org/show_bug.cgi?id=1264437 disqualified -> disqualified-test/disqualified-control], [https://bugzilla.mozilla.org/show_bug.cgi?id=1259846 E10SCohort annotation on crash reporter]
== Beta 46 ==
=== 2nd phase ===
'''Start''': beta 3 (built on March 21, shipped March 22)
'''End''': beta 8 (built on April 4, shipped April 5)
'''Range''': '''['''beta 3, beta 8''')''' (15 days)
'''Sample''': 50% of ~40% eligible users (yield ~20%)
'''Details''':
* No add-ons, no a11y
* APZ enabled (for the first time)
* Through the system add-on (for the first time)
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1257265 bugs related to the system add-on], [https://bugzilla.mozilla.org/show_bug.cgi?id=1261422 stop it]
=== 1st phase ===
'''Start''': March 9
'''End''': March 21
'''Range''': '''['''March 9, March 21''')''' (12 days)
'''Sample''': 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
'''Details''':
* No add-ons
* APZ disabled
* Through Telemetry Experiments
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1253669 start it], [https://bugzilla.mozilla.org/show_bug.cgi?id=1258592 stop it]
== Beta 45 ==
=== 2nd phase ===
'''Start''': February 12
'''End''': February 22
'''Range''': '''['''February 12, February 22''')''' (11 days)
'''Sample''': 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
'''Details''':
* No add-ons (add-on users filtered for the first time)
* APZ disabled
* Through Telemetry Experiments
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1244187 start it], [https://bugzilla.mozilla.org/show_bug.cgi?id=1250338 stop it], [filter add-on users from e10s https://bugzilla.mozilla.org/show_bug.cgi?id=1234675]
=== 1st phase ===
'''Start''': January 25
'''End''': February 11
'''Range''': '''['''January 25, February 11''')''' (17 days)
'''Sample''': 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
'''Details''':
* Add-ons enabled
** LastPass 3.* and Skype Click-to-Call users removed from the experiment after a few days due to crash
* APZ disabled
* Through Telemetry Experiments
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1241336 start it], automatically stopped by date, [https://bugzilla.mozilla.org/show_bug.cgi?id=1241336 block LastPass 3.* users], [https://bugzilla.mozilla.org/show_bug.cgi?id=1245192 Block Skype Click-to-Call users]
This page intends to document the A/B experiments that were done during the development of e10s.
== Experiment mechanisms ==
=== Telemetry Experiments ===
The first experiments were done through the [[QA/Telemetry|Telemetry Experiments]] system, which is an A/B mechanism meant to test new features to telemetry-enabled users. The experiments were published in the [http://telemetry-experiment.cdn.mozilla.net/ production server] for this mechanism.
=== Staged Rollout - System add-on ===
With the approach of releasing e10s to the release audience, it was necessary to have a more detailed mechanism specific to e10s in order to be able to do a staged rollout of the feature and quickly increase/decrease the rollout rate. To do that, the new System Add-on mechanism was chosen, because it can be updated out-of-band from the release process.
After the [https://bugzilla.mozilla.org/show_bug.cgi?id=1249845 system add-on was developed], we started using it to run new phases of e10s experiments on Beta, in order to also test the add-on itself in preparation for Release.
= List of experiments =
Note: all dates are an approximation from the end-user's point of view, because they may take a couple of days to be updated to the build where the experiment was started/terminated.
== Beta 47 ==
=== 1st phase ===
'''Start''': beta 1 (built on May 26, shipped May 27)
'''End''': still running
'''Range''': '''['''beta 1, ...''')'''
'''Sample''': 50% of eligible users (yield ~20%)
'''Details''':
* APZ is enabled by default (conditional to e10s) starting in 47
* No add-ons, no a11y
* No GTK+ 3.20+
* E10SCohort annotation added to crash report
* "disqualified" cohort is now split into "disqualified-test" and "disqualified-control" which should help verify the even-ness of the data
'''Related bugs''': automatically started (no bug), [https://bugzilla.mozilla.org/show_bug.cgi?id=1266213 block GTK+ 3.20+], [https://bugzilla.mozilla.org/show_bug.cgi?id=1264437 disqualified -> disqualified-test/disqualified-control], [https://bugzilla.mozilla.org/show_bug.cgi?id=1259846 E10SCohort annotation on crash reporter]
== Beta 46 ==
=== 2nd phase ===
'''Start''': beta 3 (built on March 21, shipped March 22)
'''End''': beta 8 (built on April 4, shipped April 5)
'''Range''': '''['''beta 3, beta 8''')''' (15 days)
'''Sample''': 50% of ~40% eligible users (yield ~20%)
'''Details''':
* No add-ons, no a11y
* APZ enabled (for the first time)
* Through the system add-on (for the first time)
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1257265 bugs related to the system add-on], [https://bugzilla.mozilla.org/show_bug.cgi?id=1261422 stop it]
=== 1st phase ===
'''Start''': March 9
'''End''': March 21
'''Range''': '''['''March 9, March 21''')''' (12 days)
'''Sample''': 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
'''Details''':
* No add-ons
* APZ disabled
* Through Telemetry Experiments
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1253669 start it], [https://bugzilla.mozilla.org/show_bug.cgi?id=1258592 stop it]
== Beta 45 ==
=== 2nd phase ===
'''Start''': February 12
'''End''': February 22
'''Range''': '''['''February 12, February 22''')''' (11 days)
'''Sample''': 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
'''Details''':
* No add-ons (add-on users filtered for the first time)
* APZ disabled
* Through Telemetry Experiments
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1244187 start it], [https://bugzilla.mozilla.org/show_bug.cgi?id=1250338 stop it], [filter add-on users from e10s https://bugzilla.mozilla.org/show_bug.cgi?id=1234675]
=== 1st phase ===
'''Start''': January 25
'''End''': February 11
'''Range''': '''['''January 25, February 11''')''' (17 days)
'''Sample''': 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
'''Details''':
* Add-ons enabled
** LastPass 3.* and Skype Click-to-Call users removed from the experiment after a few days due to crash
* APZ disabled
* Through Telemetry Experiments
'''Related bugs''': [https://bugzilla.mozilla.org/show_bug.cgi?id=1241336 start it], automatically stopped by date, [https://bugzilla.mozilla.org/show_bug.cgi?id=1241336 block LastPass 3.* users], [https://bugzilla.mozilla.org/show_bug.cgi?id=1245192 Block Skype Click-to-Call users]
