Electrolysis/Release Criteria: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(add some more)
m (Fix layout)
 
(292 intermediate revisions by 8 users not shown)
Line 1: Line 1:
Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.
Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.
= Rollout Criteria =
We plan to ship e10s in Firefox 48 using a staged rollout. See [https://docs.google.com/document/d/150XUlvXi13mliWIixS5x5DoLE14ySxItBhQfCIKg1NI/edit elan's e10s release plan].
* '''Stability dashboards''':
** [https://sql.telemetry.mozilla.org/queries/613/source  Firefox 48 Release]
** [https://sql.telemetry.mozilla.org/queries/607/source Firefox 48 Beta]
** "M+(c-s)" is the metric that matters: main (chrome) process crash rate + content process crash rate - content process shutdown crash rate.
* '''Engagement Ratio dashboards''': (DAU/MAU)
** [https://sql.telemetry.mozilla.org/dashboard/client-count-firefox-e10s-release%2D Firefox 48 Release]
** [https://sql.telemetry.mozilla.org/dashboard/e10s-client-count-beta-e10scontrol%2D Firefox 48 Beta]
* chutten's [https://github.com/vitillo/e10s_analyses/blob/master/beta/48/week3/e10s_experiment.ipynb '''telemetry report'''] for Firefox 48 '''Beta'''. ''(FIXME: fix URL to point to Firefox 48 GA report when available!)''
= Release Criteria =
== Stability ==


{| class="wikitable"
{| class="wikitable"
! Responsible
| bsmedberg
|-
|-
! Release Criteria !! Responsible !! Status
! Accountable
| bsmedberg
|-
|-
| P1: [[Electrolysis/Release Criteria/Stability|Stability]] || poiru || AT RISK
! Supporting
| Kairo, rvitillo
|-
|-
| P1: [[Electrolysis/Release Criteria/Jank|Jank]] || chutten || ?
! Consulted
| Barbara, release management
|-
|-
| P1: [[Electrolysis/Release Criteria/Scrolling|Scrolling]] || ? || ?
! Informed
| cpeterson, elan
|}
 
{| class="wikitable"
|-
! Description
! Metric
! Analysis
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| chrome + content crash rate
| (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours
| "M+(c-s)" on [https://sql.telemetry.mozilla.org/queries/607/source 48 Beta]
| jimm
| 5.34 crashes/khour
| style="background-color: PaleGreen;" | 5.20 crashes/khour
| style="background-color: PaleGreen;" | YES
| style="background-color: PaleGreen;" | Barbara (2016-06-29), bsmedberg (2016-07-01)
|-
| plugin process crash rate
| plugin crashes per 1000 use hours
| "PLUGIN" on [https://sql.telemetry.mozilla.org/queries/607/source 48 Beta]
| jimm
| 9.53 plugin crashes/khour
| 12.59 plugin crashes/khour
| NO, but blassey says plugin crashes do not block e10s release.
| N/A
|}
 
Crash rates exclude XP and deprecated OS X operating systems
 
* [https://telemetry.mozilla.org/crashes/office-dashboard.html Stability "office dashboard"]
* [[Electrolysis/Release Criteria/Stability|More information on Stability analysis]]
* Stability analysis meta {{bug|1251376}}
 
== User Engagement ==
 
{| class="wikitable"
! Responsible
| rvitillo
|-
! Accountable
| TBD
|-
! Supporting
| measurement/data teams
|-
! Consulted
| Barbara, release management
|-
! Informed
| cpeterson, elan
|}
 
{| class="wikitable"
|-
! Metric
! Analysis
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| Engagement Ratio (DAU/MAU)
| [https://sql.telemetry.mozilla.org/dashboard/e10s-client-count-beta-e10scontrol- Engagement Ratio graphs] ({{bug|1251259}})
| rvitillo
| style="background-color: PaleGreen;" | ~0.22
| style="background-color: PaleGreen;" | ~0.35
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | rvitillo, bsmedberg (2016-05-27)
|}
 
== Jank ==
 
{| class="wikitable"
! Responsible
| chutten
|-
! Accountable
| bsmedberg
|-
! Supporting
| data team, RyanVM, rvitillo, avih, Softvision
|-
! Consulted
| jgriffiths, billm
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Description
! Metric
! Analysis
! Analysis By
! Criteria Met?
! Criteria Met Signed-off By
|-
| GC pauses
| GC_MAX_PAUSE_MS (parent, children)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
| Cycle Collector pauses
| CYCLE_COLLECTOR_MAX_PAUSE (parent, children)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
| Event loop lag
| INPUT_EVENT_RESPONSE_MS
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
[http://mzl.la/21WwhFj Nightly 48 telemetry]
| chutten
| style="background-color: PaleGreen;" | e10s is different but not worse
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
| Browser responsiveness during page load
| Talos tp5o_responsiveness
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20responsiveness Perfherder]
|
| style="background-color: PaleGreen;" | YES: e10s is 39%–88% faster
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
|}
 
* [[Electrolysis/Release Criteria/Jank|More information on Jank analysis]]
* Jank analysis meta {{bug|1251377}}
 
== Page Load ==
 
{| class="wikitable"
! Responsible
| jimm
|-
! Accountable
| RyanVM
|-
! Supporting
| jmaher
|-
! Consulted
| jgriffiths, rvitillo
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Metric
! Analysis
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| tp5o summary ±5%
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20summary Perfherder]
|
| 248.32 (Linux)
230.35 (Win8)
| style="background-color: PaleGreen;" | 246.28 (Linux)
213.51 (Win8)
| style="background-color: PaleGreen;" | YES: -0.82% to -7.31% improvement
| style="background-color: PaleGreen;" | jgriffiths, RyanVM (2016-04-01)
jimm (2016-04-04)
|-
| FX_PAGE_LOAD_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] ({{bug|1228020}})
| rvitillo
|
| style="background-color: PaleGreen;" | e10s is much better.
| style="background-color: PaleGreen;" | YES: e10s is much better.
| style="background-color: PaleGreen;" | jgriffiths, RyanVM (2016-04-01)
jimm (2016-04-04)
|}
 
* [[Electrolysis/Release Criteria/Pageload|More information on Page Load analysis]]
* Page Load analysis meta {{bug|1251549}}
 
== Startup/Shutdown Time ==
 
{| class="wikitable"
! Responsible
| rvitillo
|-
! Accountable
|
|-
! Supporting
|
|-
! Consulted
| jgriffiths, jmaher
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Release Criteria
! Metric
! Analysis
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| Talos tests
| ts_paint ±10%
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=ts_paint Perfherder]
|
| 863 (Win7)
| style="background-color: PaleGreen;" | 433 (Win7)
| style="background-color: PaleGreen;" | YES: 43–55% better ({{bug|1174767}})
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|-
| Time to first paint
| SIMPLE_MEASURES_FIRSTPAINT ±10%
("simpleMeasurements/firstPaint")
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| 4468 ms
| style="background-color: PaleGreen;" | 4391 ms
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|-
|-
| P1: [[Electrolysis/Release Criteria/Slow Script|Slow Script]] || azhang || ?
| Shutdown time
| SIMPLE_MEASURES_SHUTDOWNDURATION ±25%
("simpleMeasurements/shutdownDuration")
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| 1156 ms
| style="background-color: PaleGreen;" | 1190 ms
| style="background-color: PaleGreen;" | YES: e10s is slightly worse, but within ±25% tolerance.
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|}
 
* In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".
 
== Scrolling ==
 
{| class="wikitable"
! Responsible
| kats
|-
|-
| P1: no blocker bugs || blassey || ?
! Accountable
| RyanVM
|-
|-
| P1: automated tests running and passing || mrbkap? || AT RISK
! Supporting
| avih, jmaher, graphics team (mstange)
|-
! Consulted
| jgriffiths, jmaher
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Metric
! Analysis
! Analysis By
! Criteria Met?
! Criteria Met Signed-off By
|-
| FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| avih
| style="background-color: PaleGreen;" | YES: e10s is faster
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
|-
| tp5o_scroll
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o_scroll Perfherder]
| kats
| style="background-color: PaleGreen;" | YES: e10s is 6% – 53% better
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
|-
| tscrollx
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tscrollx Perfherder]
| kats
| style="background-color: PaleGreen;" | YES: e10s is 7% – 58% better
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
|}
 
* [[Electrolysis/Release Criteria/Scrolling|More information on Scrolling analysis]]
* Scrolling analysis meta {{bug|1251388}}
 
== Plugin Jank ==
 
{| class="wikitable"
! Responsible
| chutten
|-
! Accountable
| aklotz
|-
! Supporting
| bsmedberg
|-
! Consulted
| Barbara
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Description
! Metric
! Analysis
! Analysis By
! Criteria Met?
! Criteria Met Signed-off By
|-
| Plugin module initialization jank time
| BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| chutten
| style="background-color: PaleGreen;" | '''Flash''': YES. e10s is better.
'''Java''': YES. e10s is better.
 
'''Silverlight''': YES. e10s is better.
 
'''Unity''': YES. e10s is better.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|-
| Plugin initialization jank time
| BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| chutten
| style="background-color: PaleGreen;" | '''Flash''': MAYBE? e10s is slightly worse.
'''Java''': YES. e10s is slightly better.
 
'''Silverlight''': MAYBE? e10s is slightly worse.
 
'''Unity''': YES! e10s is better.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|-
| Plugin destruction jank time
| BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| chutten
| style="background-color: PaleGreen;" | '''Flash''': YES. e10s is different, but not worse.
'''Java''': MAYBE? e10s is slightly worse.
 
'''Silverlight''': MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)
 
'''Unity''': YES. e10s is different, but not worse.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|}
 
* [[Electrolysis/Release Criteria/Plugin Jank|More information on Plugin Jank analysis]]
* Plugin jank analysis meta {{bug|1251551}}
 
== Memory Usage ==
 
{| class="wikitable"
! Responsible
| erahm
|-
! Accountable
| blassey
|-
! Supporting
| jmaher (Talos)
|-
! Consulted
| Barbara
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Metric
! Analysis
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| tp5o Main_RSS
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=Main_RSS Perfherder]
| 401623141.33 (OS X)
| style="background-color: PaleGreen;" | 331418381.64 (OS X)
| style="background-color: PaleGreen;" | e10s is 17%–35% better
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
|-
| MEMORY_TOTAL ±25% (browser + content process)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] and [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ erahm's blog post]
|
|
| style="background-color: PaleGreen;" | e10s is 10–20% worse, but within our ±25% tolerance
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
|-
| VSIZE_MAX_CONTIGUOUS (browser + content process)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
|
|
| style="background-color: PaleGreen;" | e10s is slightly different, but not worse?
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
|}
 
* We expect e10s MEMORY_TOTAL to regress compared to non-e10s because we are using more processes.
* erahm's blog post: [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ Memory Usage of Firefox with e10s Enabled]
 
== UI Smoothness ==
 
{| class="wikitable"
! Responsible
| mconley
|-
! Accountable
| RyanVM
|-
! Supporting
| jmaher, avih, gabor
|-
! Consulted
| Barbara
|-
! Informed
| cpeterson, elan, release management
|}
 
Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.
 
{| class="wikitable"
|-
! Metric
! Metric Owner
! Analysis
! Measurements
! Bugs
! Criteria Met?
! Criteria Met Signed-off By
|-
| TART ±5% (Tab Animation Regression Test)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tart Perfherder]
| style="background-color: PaleGreen;" |
e10s / APZ<br>
xp = 4.4% combined<br>
win7 = 1.8% / 9%<br>
win8 = -0.6% / 10% (improvement)<br>
osx = 6% / 3.5%<br>
linux = 0.5% / 16%
| style="background-color: PaleGreen;" |
e10s: {{bug|1251699}}<br>
apz:{{bug|1253412}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
|-
| tpaint ±5% (first paint for a new window)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tpaint Perfherder]
| style="background-color: PaleGreen;" |
xp = 0%<br>
win7 = 9.3%<br>
win8 = 3.4%<br>
osx = -11% (improvement)<br>
linux = 7%
| style="background-color: PaleGreen;" | {{bug|1174770}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
|-
| tresize ±5% (window resize responsiveness)
| jimm
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tresize Perfherder]
| style="background-color: PaleGreen;" |
e10s / APZ<br>
xp = 1.0% combined<br>
win7 = 0.0% <br>
win8 = -22% (improvement overall)<br>
osx = 1.0% combined<br>
linux = 7% / 2.0%
| style="background-color: PaleGreen;" | {{bug|1252947}}
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | jgriffiths (2016-03-17, {{bug|1252947#c23}}), mconley (2016-04-21)
|-
| tps ±5% (tab switch speed)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tps Perfherder]
| style="background-color: PaleGreen;" |
xp = -27%<br>
win7 = -28%<br>
win8 = -34%<br>
osx = -30%<br>
linux = -19%
| style="background-color: PaleGreen;" |
e10s: {{bug|1186585}}<br>
apz: {{bug|1216924}}
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | mconley (2016-04-19), Barbara (2016-04-20)
|-
| FX_TAB_SWITCH_TOTAL_MS
| Gabor
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: PaleGreen;" | 52.66 ms (median) < 82.61 ms (median)
| style="background-color: PaleGreen;" | [https://bugzilla.mozilla.org/show_bug.cgi?id=1252031#c2 bug 1252031]
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | gabor (2016-03-23, {{bug|1252031#c2}}), Barbara (2016-04-21)
|-
| CART ±5% (Customize Animation Regression Test)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=cart Perfherder]
| style="background-color: PaleGreen;" |
e10s / APZ<br>
xp = improvement<br>
win7 = 4.0% / 2.5%<br>
win8 = 2.0% / 2.5%<br>
osx = 3.7% / 2.7%<br>
linux = 6.8% / 6.8%
| style="background-color: PaleGreen;" |
e10s: {{bug|1252768}}<br>
apz: {{bug|1253380}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)
|}
 
* [[Electrolysis/Release Criteria/UI Smoothness|More information on UI Smoothness analysis]]
* UI smoothness analysis meta {{bug|1251547}}
 
== Graphics Performance ==
 
{| class="wikitable"
! Responsible
| Milan
|-
! Accountable
| RyanVM
|-
! Supporting
| gw280
|-
! Consulted
| Barbara, jmaher, graphics team
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Metric
! Metric Owner
! Analysis
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| glterrain ±5%
| jgilbert
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=glterrain Perfherder]
| 18
| style="background-color: PaleGreen;" | 19
| style="background-color: PaleGreen;" | Mixed results from 3% worse to 66% better
| style="background-color: PaleGreen;" | Barbara (2016-05-06), Milan (2016-05-06)
|-
| tcanvasmark ±5%
|
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tcanvasmark Perfherder]
| 6444
| style="background-color: PaleGreen;" | 6301
| style="background-color: PaleGreen;" | Mixed results from 2% worse to 3% better
| style="background-color: PaleGreen;" | Barbara (2016-05-06), Milan (2016-05-06)
|}
 
* [[Electrolysis/Release Criteria/Graphics|More information on Graphics Performance analysis]]
* Graphics analysis meta {{bug|1251555}}
 
== Slow Scripts ==
 
{| class="wikitable"
! Responsible
| azhang
|-
! Accountable
| bsmedberg
|-
! Supporting
|
|-
! Consulted
| Barbara, billm
|-
! Informed
| cpeterson, elan, release management
|}
 
{| class="wikitable"
|-
! Description
! Metric
! Analysis
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met Signed-off By
|-
| Slow script warnings
| SLOW_SCRIPT_PAGE_COUNT
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/47/week1/e10s_experiment.ipynb beta 47 experiment]
| azhang
| 0.00117087669174 warnings per use hour (aka 854 hours/warning)
| 0.00158655016424 warnings per use hour (aka 630 hours/warning)
({{bug|1260769}})
| style="background-color: PaleGreen;" | Barbara and Jim are OK with this regression because display rates are extremely low in both modes.
| style="background-color: PaleGreen;" | Barbara, jimm, billm (2016-05-25)
|}
 
* [[Electrolysis/Release Criteria/Slow Script|More information on slow script analysis]]
* Slow script analysis meta {{bug|1251545}}
 
== Tests ==
 
All tests disabled for e10s must be triaged and:
 
* reenabled (and fixed, if necessary)
* or annotated that the test is irrelevant for e10s?
* or deleted?
 
We want each test directory to be signed-off by the feature area's owner.
 
{| class="wikitable"
! Responsible
| Blake, Felipe
|-
! Accountable
| jimm
|-
! Supporting
| dolske (coordinating people fixing front-end tests)
|-
! Consulted
| jgriffiths
|-
! Informed
| cpeterson, elan, release management
|}
 
* e10s test meta {{bug|e10s-tests}}
* [https://treeherder.mozilla.org/perf.html#/e10s Perfherder e10s dashboard]
 
{| class="wikitable"
|-
! Task
! Owner
! Criteria Met?
! Criteria Met Signed-off By
|-
| [https://wiki.mozilla.org/QA/E10s_Test_Plan Write test plan]
| RyanVM
| style="background-color: LightYellow;" | In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan.
| style="background-color: SkyBlue;" | TODO
|-
| [https://wiki.mozilla.org/QA/E10s_Test_Plan Execute test plan]
| RyanVM
| style="background-color: LightYellow;" | In progress
| style="background-color: SkyBlue;" | TODO
|-
| Re-enable [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/ disabled e10s tests]
| Felipe, Blake
| style="background-color: LightYellow;" | In progress
| style="background-color: SkyBlue;" | TODO
|}
 
* Notes from jimm: This looks good. I've removed XP from the operating system lists.
 
== Release Blocking Bugs ==
 
All M9 bugs must be fixed before we release.
 
{| class="wikitable"
! Responsible
| jimm
|-
|-
| P2: [[Electrolysis/Release Criteria/UI Smoothness]] || ? || ?
! Accountable
| blassey
|-
|-
| P2: [[Electrolysis/Release Criteria/Pageload]] || ? || ?
! Supporting
| release management
|-
|-
| P2: [[Electrolysis/Release Criteria/Plugin Jank|Plugin Jank]] || ? || ?
! Consulted
| jgriffiths
|-
|-
| P2: [[Electrolysis/Release Criteria/Graphics|Graphics]] || ? || ?
! Informed
| cpeterson, elan
|}
|}
* Bugzilla query for e10s blocking bugs: '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=e10s-release-blockers&sharer_id=430528 is.gd/e10s_blockers]'''
=== M9 bugs ===
<bugzilla>
{
    "f1": "cf_tracking_e10s",
    "o1": "anywords",
    "v1": "m9+,m8+,m7+,m6+,m5+,m4+,m3+,m2+,",
    "resolution": "---",
    "include_fields": "id, summary, whiteboard, keywords, assigned_to, cf_tracking_e10s"
}
</bugzilla>
=== Release Criteria bugs ===
<bugzilla>
{
    "blocks": "1198187,1251376, 1251377, 1251388, 1251545, 1251547, 1251549, 1251551, 1251555",
    "resolution": "---",
    "include_fields": "id, summary, whiteboard, keywords, assigned_to"
}
</bugzilla>
=== APZ Bugs ===
<bugzilla>
{
    "blocks": "1254668",
    "resolution": "---",
    "include_fields": "id, summary, whiteboard, keywords, assigned_to"
}
</bugzilla>
== Accessibility ==
* Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.
== Add-ons ==
* Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.

Latest revision as of 01:38, 15 July 2016

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Rollout Criteria

We plan to ship e10s in Firefox 48 using a staged rollout. See elan's e10s release plan.

Release Criteria

Stability

Responsible bsmedberg
Accountable bsmedberg
Supporting Kairo, rvitillo
Consulted Barbara, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
chrome + content crash rate (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours "M+(c-s)" on 48 Beta jimm 5.34 crashes/khour 5.20 crashes/khour YES Barbara (2016-06-29), bsmedberg (2016-07-01)
plugin process crash rate plugin crashes per 1000 use hours "PLUGIN" on 48 Beta jimm 9.53 plugin crashes/khour 12.59 plugin crashes/khour NO, but blassey says plugin crashes do not block e10s release. N/A

Crash rates exclude XP and deprecated OS X operating systems

User Engagement

Responsible rvitillo
Accountable TBD
Supporting measurement/data teams
Consulted Barbara, release management
Informed cpeterson, elan
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Engagement Ratio (DAU/MAU) Engagement Ratio graphs (bug 1251259) rvitillo ~0.22 ~0.35 Yes rvitillo, bsmedberg (2016-05-27)

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Event loop lag INPUT_EVENT_RESPONSE_MS beta 46 experiment

Nightly 48 telemetry

chutten e10s is different but not worse chutten and jgriffiths (2016-04-08)
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder YES: e10s is 39%–88% faster chutten and jgriffiths (2016-04-08)

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o summary ±5% Perfherder 248.32 (Linux)

230.35 (Win8)

246.28 (Linux)

213.51 (Win8)

YES: -0.82% to -7.31% improvement jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

FX_PAGE_LOAD_MS ±5% beta 46 experiment (bug 1228020) rvitillo e10s is much better. YES: e10s is much better. jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

Startup/Shutdown Time

Responsible rvitillo
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder 863 (Win7) 433 (Win7) YES: 43–55% better (bug 1174767) jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10%

("simpleMeasurements/firstPaint")

beta 46 experiment rvitillo 4468 ms 4391 ms YES: e10s is slightly better jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25%

("simpleMeasurements/shutdownDuration")

beta 46 experiment rvitillo 1156 ms 1190 ms YES: e10s is slightly worse, but within ±25% tolerance. jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

  • In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS beta 46 experiment avih YES: e10s is faster kats and jgriffiths (2016-04-08)
tp5o_scroll Perfherder kats YES: e10s is 6% – 53% better kats and jgriffiths (2016-04-08)
tscrollx Perfherder kats YES: e10s is 7% – 58% better kats and jgriffiths (2016-04-08)

Plugin Jank

Responsible chutten
Accountable aklotz
Supporting bsmedberg
Consulted Barbara
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
Plugin module initialization jank time BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is better.

Java: YES. e10s is better.

Silverlight: YES. e10s is better.

Unity: YES. e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% Beta 46 experiment chutten Flash: MAYBE? e10s is slightly worse.

Java: YES. e10s is slightly better.

Silverlight: MAYBE? e10s is slightly worse.

Unity: YES! e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is different, but not worse.

Java: MAYBE? e10s is slightly worse.

Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)

Unity: YES. e10s is different, but not worse.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Memory Usage

Responsible erahm
Accountable blassey
Supporting jmaher (Talos)
Consulted Barbara
Informed cpeterson, elan, release management
Metric Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder 401623141.33 (OS X) 331418381.64 (OS X) e10s is 17%–35% better erahm (2016-04-15), Barbara (2016-04-18)
MEMORY_TOTAL ±25% (browser + content process) beta 46 experiment and erahm's blog post e10s is 10–20% worse, but within our ±25% tolerance erahm (2016-04-15), Barbara (2016-04-18)
VSIZE_MAX_CONTIGUOUS (browser + content process) beta 46 experiment e10s is slightly different, but not worse? erahm (2016-04-15), Barbara (2016-04-18)

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted Barbara
Informed cpeterson, elan, release management

Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.

Metric Metric Owner Analysis Measurements Bugs Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder

e10s / APZ
xp = 4.4% combined
win7 = 1.8% / 9%
win8 = -0.6% / 10% (improvement)
osx = 6% / 3.5%
linux = 0.5% / 16%

e10s: bug 1251699
apz:bug 1253412

Borderline mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
tpaint ±5% (first paint for a new window) mconley Perfherder

xp = 0%
win7 = 9.3%
win8 = 3.4%
osx = -11% (improvement)
linux = 7%

bug 1174770 Borderline mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
tresize ±5% (window resize responsiveness) jimm Perfherder

e10s / APZ
xp = 1.0% combined
win7 = 0.0%
win8 = -22% (improvement overall)
osx = 1.0% combined
linux = 7% / 2.0%

bug 1252947 Yes jgriffiths (2016-03-17, bug 1252947#c23), mconley (2016-04-21)
tps ±5% (tab switch speed) mconley Perfherder

xp = -27%
win7 = -28%
win8 = -34%
osx = -30%
linux = -19%

e10s: bug 1186585
apz: bug 1216924

Yes mconley (2016-04-19), Barbara (2016-04-20)
FX_TAB_SWITCH_TOTAL_MS Gabor beta 46 experiment 52.66 ms (median) < 82.61 ms (median) bug 1252031 Yes gabor (2016-03-23, bug 1252031#c2), Barbara (2016-04-21)
CART ±5% (Customize Animation Regression Test) mconley Perfherder

e10s / APZ
xp = improvement
win7 = 4.0% / 2.5%
win8 = 2.0% / 2.5%
osx = 3.7% / 2.7%
linux = 6.8% / 6.8%

e10s: bug 1252768
apz: bug 1253380

Borderline avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)

Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted Barbara, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder 18 19 Mixed results from 3% worse to 66% better Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5% Perfherder 6444 6301 Mixed results from 2% worse to 3% better Barbara (2016-05-06), Milan (2016-05-06)

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted Barbara, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT beta 47 experiment azhang 0.00117087669174 warnings per use hour (aka 854 hours/warning) 0.00158655016424 warnings per use hour (aka 630 hours/warning)

(bug 1260769)

Barbara and Jim are OK with this regression because display rates are extremely low in both modes. Barbara, jimm, billm (2016-05-25)

Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?

We want each test directory to be signed-off by the feature area's owner.

Responsible Blake, Felipe
Accountable jimm
Supporting dolske (coordinating people fixing front-end tests)
Consulted jgriffiths
Informed cpeterson, elan, release management
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan RyanVM In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan. TODO
Execute test plan RyanVM In progress TODO
Re-enable disabled e10s tests Felipe, Blake In progress TODO
  • Notes from jimm: This looks good. I've removed XP from the operating system lists.

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1063169 [meta] tracking performance issues with e10s [webvr] meta
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


APZ Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Accessibility

  • Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

  • Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Release_Criteria&oldid=1139838"