Electrolysis/Release Criteria: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎User Engagement: glind is not working on bug 1249665)
m (Fix layout)
 
(222 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.
Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.
= Rollout Criteria =
We plan to ship e10s in Firefox 48 using a staged rollout. See [https://docs.google.com/document/d/150XUlvXi13mliWIixS5x5DoLE14ySxItBhQfCIKg1NI/edit elan's e10s release plan].
* '''Stability dashboards''':
** [https://sql.telemetry.mozilla.org/queries/613/source  Firefox 48 Release]
** [https://sql.telemetry.mozilla.org/queries/607/source Firefox 48 Beta]
** "M+(c-s)" is the metric that matters: main (chrome) process crash rate + content process crash rate - content process shutdown crash rate.
* '''Engagement Ratio dashboards''': (DAU/MAU)
** [https://sql.telemetry.mozilla.org/dashboard/client-count-firefox-e10s-release%2D Firefox 48 Release]
** [https://sql.telemetry.mozilla.org/dashboard/e10s-client-count-beta-e10scontrol%2D Firefox 48 Beta]
* chutten's [https://github.com/vitillo/e10s_analyses/blob/master/beta/48/week3/e10s_experiment.ipynb '''telemetry report'''] for Firefox 48 '''Beta'''. ''(FIXME: fix URL to point to Firefox 48 GA report when available!)''


= Release Criteria =
= Release Criteria =


 
== Stability ==
== User Engagement ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| dzeber? hulmer?
| bsmedberg
|-
|-
! Accountable
! Accountable
| TBD
| bsmedberg
|-
|-
! Supporting
! Supporting
| measurement/data teams
| Kairo, rvitillo
|-
|-
! Consulted
! Consulted
| jgriffiths, release management
| Barbara, release management
|-
|-
! Informed
! Informed
Line 25: Line 37:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Description
! Metric
! Metric
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| User Retention
| chrome + content crash rate
| Experiment user drop-off rate (?)
| (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours
| style="background-color: #ffc;" | {{bug|1249665}}
| "M+(c-s)" on [https://sql.telemetry.mozilla.org/queries/607/source 48 Beta]
| style="background-color: #ffc;" | TBD
| jimm
| style="background-color: #ffc;" | TBD
| 5.34 crashes/khour
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 5.20 crashes/khour
| style="background-color: #ffc;" | NO ONE
| style="background-color: PaleGreen;" | YES
| style="background-color: PaleGreen;" | Barbara (2016-06-29), bsmedberg (2016-07-01)
|-
|-
| User Retention
| plugin process crash rate
| e10s/non-10s engagement ratio
| plugin crashes per 1000 use hours
| style="background-color: #ffc;" | {{bug|1251259}}
| "PLUGIN" on [https://sql.telemetry.mozilla.org/queries/607/source 48 Beta]
| style="background-color: #ffc;" | dzeber, hulmer?
| jimm
| style="background-color: #ffc;" | TBD
| 9.53 plugin crashes/khour
| style="background-color: #ffc;" | TBD
| 12.59 plugin crashes/khour
| style="background-color: #ffc;" | NO ONE
| NO, but blassey says plugin crashes do not block e10s release.
| N/A
|}
|}


== Stability ==
Crash rates exclude XP and deprecated OS X operating systems
 
* [https://telemetry.mozilla.org/crashes/office-dashboard.html Stability "office dashboard"]
* [[Electrolysis/Release Criteria/Stability|More information on Stability analysis]]
* Stability analysis meta {{bug|1251376}}
 
== User Engagement ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| chutten
| rvitillo
|-
|-
! Accountable
! Accountable
| bsmedberg
| TBD
|-
|-
! Supporting
! Supporting
| Kairo, measurement/data teams
| measurement/data teams
|-
|-
! Consulted
! Consulted
| jgriffiths, release management
| Barbara, release management
|-
|-
! Informed
! Informed
Line 71: Line 92:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Metric
! Metric
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| chrome+content process crash rate
| Engagement Ratio (DAU/MAU)
| chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength
| [https://sql.telemetry.mozilla.org/dashboard/e10s-client-count-beta-e10scontrol- Engagement Ratio graphs] ({{bug|1251259}})
| [https://gist.github.com/poiru/1ceec405172bcafa7305 IPython notebook]
| rvitillo
| style="background-color: #0f0;" | rvitillo
| style="background-color: PaleGreen;" | ~0.22
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | ~0.35
| style="background-color: #f00;" | NO
| style="background-color: PaleGreen;" | Yes
| style="background-color: #ffc;" | NO ONE
| style="background-color: PaleGreen;" | rvitillo, bsmedberg (2016-05-27)
|-
| plugin process crash rate
| SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength
| [https://gist.github.com/poiru/1ceec405172bcafa7305 IPython notebook]
| style="background-color: #0f0;" | rvitillo
| style="background-color: #ffc;" | TBD
| style="background-color: #f00;" | NO
| style="background-color: #ffc;" | NO ONE
|}
|}
* [[Electrolysis/Release Criteria/Stability|More information on Stability analysis]]
* Stability analysis meta {{bug|1251376}}


== Jank ==
== Jank ==
Line 120: Line 130:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Description
! Metric
! Metric
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| GC pauses
| GC pauses
| GC_MAX_PAUSE_MS
| GC_MAX_PAUSE_MS (parent, children)
| style="background-color: #0f0;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: #ffc;" | TBD
| rvitillo
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
| style="background-color: #ffc;" | TBD
|-
|-
| Cycle Collector pauses
| Cycle Collector pauses
| CYCLE_COLLECTOR_MAX_PAUSE
| CYCLE_COLLECTOR_MAX_PAUSE (parent, children)
| style="background-color: #0f0;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: #ffc;" | TBD
| rvitillo
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
| style="background-color: #ffc;" | TBD
|-
|-
| Event loop lag
| Event loop lag
| INPUT_EVENT_RESPONSE_MS
| INPUT_EVENT_RESPONSE_MS
| style="background-color: #ffc;" | TBD
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: #ffc;" | TBD
[http://mzl.la/21WwhFj Nightly 48 telemetry]
| style="background-color: #ffc;" | TBD
| chutten
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | e10s is different but not worse
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
|-
| Browser responsiveness during page load
| Browser responsiveness during page load
| Talos tp5o_responsiveness
| Talos tp5o_responsiveness
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20responsiveness Perfherder]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20responsiveness Perfherder]
| style="background-color: #ffc;" | TBD
|
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is 39%–88% faster
| style="background-color: #0f0;" | About -87% faster!
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
| style="background-color: #ffc;" | TBD
|-
|-
| Frame delay
| FX_REFRESH_DRIVER_{CHROME,CONTENT}_FRAME_DELAY_MS
| style="background-color: #0f0;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
|-
| BHR/chrome hangs
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
|}
|}


Line 180: Line 171:
* Jank analysis meta {{bug|1251377}}
* Jank analysis meta {{bug|1251377}}


== Scrolling ==
== Page Load ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| kats
| jimm
|-
|-
! Accountable
! Accountable
Line 190: Line 181:
|-
|-
! Supporting
! Supporting
| avih, jmaher, graphics team (mstange)
| jmaher
|-
|-
! Consulted
! Consulted
| jgriffiths, jmaher
| jgriffiths, rvitillo
|-
|-
! Informed
! Informed
Line 202: Line 193:
|-
|-
! Metric
! Metric
! Metric Owner
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS
| tp5o summary ±5%
| style="background-color: #ffc;" | kats?
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20summary Perfherder]
| style="background-color: #ffc;" | {{bug|1228147}}?
|
| style="background-color: #ffc;" | kats?
| 248.32 (Linux)
| style="background-color: #ffc;" | TBD
230.35 (Win8)
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 246.28 (Linux)
| style="background-color: #ffc;" | TBD
213.51 (Win8)
| style="background-color: PaleGreen;" | YES: -0.82% to -7.31% improvement
| style="background-color: PaleGreen;" | jgriffiths, RyanVM (2016-04-01)
jimm (2016-04-04)
|-
|-
| tp5o_scroll
| FX_PAGE_LOAD_MS ±5%
| style="background-color: #ffc;" | avih? mstange?
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] ({{bug|1228020}})
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o_scroll Perfherder] ({{bug|1179735}})
| rvitillo
| style="background-color: #ffc;" | kats?
|
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | e10s is much better.
| style="background-color: #f00;" | Up to 105% worse!
| style="background-color: PaleGreen;" | YES: e10s is much better.
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | jgriffiths, RyanVM (2016-04-01)
|-
jimm (2016-04-04)
| tp5r
| style="background-color: #ffc;" | kats?
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | kats?
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
|-
| tscrollx
| style="background-color: #ffc;" | avih? mstange?
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tscrollx Perfherder] ({{bug|1174772}}? {{bug|1249976}}?)
| style="background-color: #ffc;" | kats?
| style="background-color: #ffc;" | TBD
| style="background-color: #f00;" | Up to 32% worse!
| style="background-color: #ffc;" | TBD
|}
|}


* [[Electrolysis/Release Criteria/Scrolling|More information on Scrolling analysis]]
* [[Electrolysis/Release Criteria/Pageload|More information on Page Load analysis]]
* Scrolling analysis meta {{bug|1251388}}
* Page Load analysis meta {{bug|1251549}}


== Slow Scripts ==
== Startup/Shutdown Time ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| azhang
| rvitillo
|-
|-
! Accountable
! Accountable
| bsmedberg
|  
|-
|-
! Supporting
! Supporting
Line 258: Line 237:
|-
|-
! Consulted
! Consulted
| jgriffiths, billm
| jgriffiths, jmaher
|-
|-
! Informed
! Informed
Line 270: Line 249:
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Slow script warnings
| Talos tests
| SLOW_SCRIPT_PAGE_COUNT ({{bug|1251667}})
| ts_paint ±10%
| style="background-color: #ffc;" | TBD
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=ts_paint Perfherder]
| style="background-color: #ffc;" | TBD
|
| style="background-color: #ffc;" | TBD
| 863 (Win7)
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 433 (Win7)
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: 43–55% better ({{bug|1174767}})
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|-
| Time to first paint
| SIMPLE_MEASURES_FIRSTPAINT ±10%
("simpleMeasurements/firstPaint")
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| 4468 ms
| style="background-color: PaleGreen;" | 4391 ms
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|-
| Shutdown time
| SIMPLE_MEASURES_SHUTDOWNDURATION ±25%
("simpleMeasurements/shutdownDuration")
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| 1156 ms
| style="background-color: PaleGreen;" | 1190 ms
| style="background-color: PaleGreen;" | YES: e10s is slightly worse, but within ±25% tolerance.
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|}
|}


* [[Electrolysis/Release Criteria/Slow Script|More information on slow script analysis]]
* In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".
* Slow script analysis meta {{bug|1251545}}


== UI Smoothness ==
== Scrolling ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| mconley
| kats
|-
|-
! Accountable
! Accountable
Line 296: Line 299:
|-
|-
! Supporting
! Supporting
| jmaher, avih, gabor
| avih, jmaher, graphics team (mstange)
|-
|-
! Consulted
! Consulted
| jgriffiths
| jgriffiths, jmaher
|-
|-
! Informed
! Informed
Line 308: Line 311:
|-
|-
! Metric
! Metric
! Metric Owner
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| TART ±5% (Tab Animation Regression Test)
| FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS
| mconley
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tart Perfherder]
| avih
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is faster
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
| style="background-color: #f00;" | Up to 17% worse
| style="background-color: #ffc;" | TBD
|-
|-
| tpaint ±5% (first paint for a new window)
| tp5o_scroll
| jimm
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o_scroll Perfherder]
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tpaint Perfherder]
| kats
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is 6% – 53% better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
| style="background-color: #f00;" | Up to 46% worse
| style="background-color: #ffc;" | TBD
|-
|-
| tresize ±5% (window resize responsiveness)
| tscrollx
| jimm
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tscrollx Perfherder]
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tresize Perfherder]
| kats
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is 7% – 58% better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
| style="background-color: #f00;" | Mixed results: up to 5.44% worse
| style="background-color: #ffc;" | TBD
|-
| tps ±5% (tab switch speed)
| mconley
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tps Perfherder]
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #f00;" | Up to 107% worse!
| style="background-color: #ffc;" | TBD
|-
| FX_TAB_SWITCH_TOTAL_MS
| style="background-color: #ffc;" | mconley?
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
|-
| CART ±5% (Customize Animation Regression Test)
| mconley
| style="background-color: #f00;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=cart Perfherder]
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #f00;" | Up to 14% worse
| style="background-color: #ffc;" | TBD
|}
|}


* [[Electrolysis/Release Criteria/UI Smoothness|More information on UI Smoothness analysis]]
* [[Electrolysis/Release Criteria/Scrolling|More information on Scrolling analysis]]
* UI smoothness analysis meta {{bug|1251547}}
* Scrolling analysis meta {{bug|1251388}}


== Page Load ==
== Plugin Jank ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| jimm
| chutten
|-
|-
! Accountable
! Accountable
| RyanVM
| aklotz
|-
|-
! Supporting
! Supporting
| jmaher
| bsmedberg
|-
|-
! Consulted
! Consulted
| jgriffiths, rvitillo
| Barbara
|-
|-
! Informed
! Informed
Line 388: Line 359:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Description
! Metric
! Metric
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Talos tests for page load time
| Plugin module initialization jank time
| tp5o ±5%
| BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5%
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| style="background-color: #0f0;" | no change
| chutten
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | '''Flash''': YES. e10s is better.
| style="background-color: #ffc;" | TBD
'''Java''': YES. e10s is better.
| style="background-color: #ffc;" | TBD
 
'''Silverlight''': YES. e10s is better.
 
'''Unity''': YES. e10s is better.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|-
| Plugin initialization jank time
| BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| chutten
| style="background-color: PaleGreen;" | '''Flash''': MAYBE? e10s is slightly worse.
'''Java''': YES. e10s is slightly better.
 
'''Silverlight''': MAYBE? e10s is slightly worse.
 
'''Unity''': YES! e10s is better.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|-
|-
| Page load time telemetry
| Plugin destruction jank time
| FX_PAGE_LOAD_MS ±5%
| BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%
| style="background-color: #0f0;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| style="background-color: #0f0;" | rvitillo
| chutten
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | '''Flash''': YES. e10s is different, but not worse.
| style="background-color: #ffc;" | TBD
'''Java''': MAYBE? e10s is slightly worse.
| style="background-color: #ffc;" | NO ONE
 
'''Silverlight''': MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)
 
'''Unity''': YES. e10s is different, but not worse.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|}
|}


* [[Electrolysis/Release Criteria/Pageload|More information on Page Load analysis]]
* [[Electrolysis/Release Criteria/Plugin Jank|More information on Plugin Jank analysis]]
* Scrolling analysis meta {{bug|1251549}}
* Plugin jank analysis meta {{bug|1251551}}


== Plugin Jank ==
== Memory Usage ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| chutten
| erahm
|-
|-
! Accountable
! Accountable
| bsmedberg
| blassey
|-
|-
! Supporting
! Supporting
| aklotz
| jmaher (Talos)
|-
|-
! Consulted
! Consulted
| jgriffiths
| Barbara
|-
|-
! Informed
! Informed
Line 437: Line 433:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Metric
! Metric
! Analysis
! Analysis
! Analysis By
! non-e10s value
! Analysis Signed-off By
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Plugin initialization jank time
| tp5o Main_RSS
| BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=Main_RSS Perfherder]
| style="background-color: #ffc;" | TBD
| 401623141.33 (OS X)
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 331418381.64 (OS X)
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | e10s is 17%–35% better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
| style="background-color: #ffc;" | TBD
|-
| MEMORY_TOTAL ±25% (browser + content process)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] and [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ erahm's blog post]
|
|
| style="background-color: PaleGreen;" | e10s is 10–20% worse, but within our ±25% tolerance
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
|-
|-
| Plugin destruction jank time
| VSIZE_MAX_CONTIGUOUS (browser + content process)
| BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: #ffc;" | TBD
|
| style="background-color: #ffc;" | TBD
|
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | e10s is slightly different, but not worse?
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
| style="background-color: #ffc;" | TBD
|}
|}


* [[Electrolysis/Release Criteria/Plugin Jank|More information on Plugin Jank analysis]]
* We expect e10s MEMORY_TOTAL to regress compared to non-e10s because we are using more processes.
* Scrolling analysis meta {{bug|1251551}}
* erahm's blog post: [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ Memory Usage of Firefox with e10s Enabled]


 
== UI Smoothness ==
== Graphics Performance ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| Milan
| mconley
|-
|-
! Accountable
! Accountable
Line 476: Line 475:
|-
|-
! Supporting
! Supporting
| gw280
| jmaher, avih, gabor
|-
|-
! Consulted
! Consulted
| jgriffiths, jmaher, graphics team
| Barbara
|-
|-
! Informed
! Informed
| cpeterson, elan, release management
| cpeterson, elan, release management
|}
|}
Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.


{| class="wikitable"
{| class="wikitable"
Line 490: Line 491:
! Metric Owner
! Metric Owner
! Analysis
! Analysis
! Analysis By
! Measurements
! Analysis Signed-off By
! Bugs
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| glterrain ±5%
| TART ±5% (Tab Animation Regression Test)
| jgilbert
| mconley
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=glterrain Perfherder]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tart Perfherder]
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" |
| style="background-color: #ffc;" | TBD
e10s / APZ<br>
| style="background-color: #0f0;" | Mixed results: from 4% worse to 63% better
xp = 4.4% combined<br>
| style="background-color: #ffc;" | TBD
win7 = 1.8% / 9%<br>
win8 = -0.6% / 10% (improvement)<br>
osx = 6% / 3.5%<br>
linux = 0.5% / 16%
| style="background-color: PaleGreen;" |
e10s: {{bug|1251699}}<br>
apz:{{bug|1253412}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
|-
| tpaint ±5% (first paint for a new window)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tpaint Perfherder]
| style="background-color: PaleGreen;" |
xp = 0%<br>
win7 = 9.3%<br>
win8 = 3.4%<br>
osx = -11% (improvement)<br>
linux = 7%
| style="background-color: PaleGreen;" | {{bug|1174770}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
|-
| tresize ±5% (window resize responsiveness)
| jimm
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tresize Perfherder]
| style="background-color: PaleGreen;" |
e10s / APZ<br>
xp = 1.0% combined<br>
win7 = 0.0% <br>
win8 = -22% (improvement overall)<br>
osx = 1.0% combined<br>
linux = 7% / 2.0%
| style="background-color: PaleGreen;" | {{bug|1252947}}
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | jgriffiths (2016-03-17, {{bug|1252947#c23}}), mconley (2016-04-21)
|-
|-
| tcanvasmark ±5%
| tps ±5% (tab switch speed)
| style="background-color: #f00;" | TBD
| mconley
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tcanvasmark Perfherder]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tps Perfherder]
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" |
| style="background-color: #ffc;" | TBD
xp = -27%<br>
| style="background-color: #0f0;" | From 2% worse to 3% better
win7 = -28%<br>
| style="background-color: #ffc;" | TBD
win8 = -34%<br>
osx = -30%<br>
linux = -19%
| style="background-color: PaleGreen;" |
e10s: {{bug|1186585}}<br>
apz: {{bug|1216924}}
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | mconley (2016-04-19), Barbara (2016-04-20)
|-
|-
| tsvgx ±5%
| FX_TAB_SWITCH_TOTAL_MS
| style="background-color: #ff0;" | mstange?
| Gabor
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tsvgx Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 52.66 ms (median) < 82.61 ms (median)
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | [https://bugzilla.mozilla.org/show_bug.cgi?id=1252031#c2 bug 1252031]
| style="background-color: #0f0;" | Up to 51% better
| style="background-color: PaleGreen;" | Yes
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | gabor (2016-03-23, {{bug|1252031#c2}}), Barbara (2016-04-21)
|-
|-
| tsvg_opacity ±5%
| CART ±5% (Customize Animation Regression Test)
| style="background-color: #ff0;" | avih says no real owner. mattwoodrow? seth?
| mconley
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tsvgr_opacity Perfherder]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=cart Perfherder]
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" |
| style="background-color: #ffc;" | TBD
e10s / APZ<br>
| style="background-color: #0f0;" | Up to -79% better
xp = improvement<br>
| style="background-color: #ffc;" | TBD
win7 = 4.0% / 2.5%<br>
win8 = 2.0% / 2.5%<br>
osx = 3.7% / 2.7%<br>
linux = 6.8% / 6.8%
| style="background-color: PaleGreen;" |
e10s: {{bug|1252768}}<br>
apz: {{bug|1253380}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)
|}
|}


* [[Electrolysis/Release Criteria/Graphics|More information on Graphics Performance analysis]]
* [[Electrolysis/Release Criteria/UI Smoothness|More information on UI Smoothness analysis]]
* Scrolling analysis meta {{bug|1251555}}
* UI smoothness analysis meta {{bug|1251547}}


== Startup/Shutdown Time ==
== Graphics Performance ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| rvitillo?
| Milan
|-
|-
! Accountable
! Accountable
|  
| RyanVM
|-
|-
! Supporting
! Supporting
|  
| gw280
|-
|-
! Consulted
! Consulted
| jgriffiths, jmaher
| Barbara, jmaher, graphics team
|-
|-
! Informed
! Informed
Line 552: Line 603:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Metric
! Metric
! Metric Owner
! Analysis
! Analysis
! Analysis By
! non-e10s value
! Analysis Signed-off By
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Talos tests
| glterrain ±5%
| ts_paint ±10%
| jgilbert
| style="background-color: #0f0;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=ts_paint Perfherder]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=glterrain Perfherder]
| style="background-color: #ffc;" | TBD
| 18
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 19
| style="background-color: #0f0;" | Up to 55% better
| style="background-color: PaleGreen;" | Mixed results from 3% worse to 66% better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | Barbara (2016-05-06), Milan (2016-05-06)
|-
|-
| Time to first paint
| tcanvasmark ±5%
| SIMPLE_MEASURES_FIRSTPAINT ±10%
|
| style="background-color: #ffc;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tcanvasmark Perfherder]
| style="background-color: #ffc;" | bad probability #'s
| 6444
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | 6301
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | Mixed results from 2% worse to 3% better
| style="background-color: #ffc;" | TBD
| style="background-color: PaleGreen;" | Barbara (2016-05-06), Milan (2016-05-06)
|-
| Session restore time
| SIMPLE_MEASURES_SESSIONRESTORED ±10%
| style="background-color: #ffc;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| style="background-color: #ffc;" | no data?
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
|-
| Shutdown time
| SIMPLE_MEASURES_SHUTDOWNDURATION ±25%
| style="background-color: #ffc;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| style="background-color: #ffc;" | no data?
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
|}
|}


* Startup times may regress up to 10% and shutdown time up to 25% compared to non-e10s.
* [[Electrolysis/Release Criteria/Graphics|More information on Graphics Performance analysis]]
* Graphics analysis meta {{bug|1251555}}


== Memory Usage ==
== Slow Scripts ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| chutten
| azhang
|-
|-
! Accountable
! Accountable
Line 608: Line 644:
|-
|-
! Consulted
! Consulted
| jgriffiths
| Barbara, billm
|-
|-
! Informed
! Informed
Line 616: Line 652:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Description
! Metric
! Metric
! Metric Owner
! Analysis
! Analysis
! Analysis By
! Analysis By
! Analysis Signed-off By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| tp5o Main_RSS
| Slow script warnings
|  
| SLOW_SCRIPT_PAGE_COUNT
| style="background-color: #ffc;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=Main_RSS Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/47/week1/e10s_experiment.ipynb beta 47 experiment]
| style="background-color: #ffc;" | TBD
| azhang
| style="background-color: #ffc;" | TBD
| 0.00117087669174 warnings per use hour (aka 854 hours/warning)
| style="background-color: #ffc;" | 4% to 35% better
| 0.00158655016424 warnings per use hour (aka 630 hours/warning)
| style="background-color: #ffc;" | TBD
({{bug|1260769}})
| style="background-color: PaleGreen;" | Barbara and Jim are OK with this regression because display rates are extremely low in both modes.
| style="background-color: PaleGreen;" | Barbara, jimm, billm (2016-05-25)
|}
 
* [[Electrolysis/Release Criteria/Slow Script|More information on slow script analysis]]
* Slow script analysis meta {{bug|1251545}}
 
== Tests ==
 
All tests disabled for e10s must be triaged and:
 
* reenabled (and fixed, if necessary)
* or annotated that the test is irrelevant for e10s?
* or deleted?
 
We want each test directory to be signed-off by the feature area's owner.
 
{| class="wikitable"
! Responsible
| Blake, Felipe
|-
! Accountable
| jimm
|-
! Supporting
| dolske (coordinating people fixing front-end tests)
|-
! Consulted
| jgriffiths
|-
|-
| tp5o Private Bytes
! Informed
|
| cpeterson, elan, release management
| style="background-color: #ffc;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=Private%20Bytes Perfherder]
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | TBD
| style="background-color: #ffc;" | 20% to 50% worse
| style="background-color: #ffc;" | TBD
|}
|}
* e10s test meta {{bug|e10s-tests}}
* [https://treeherder.mozilla.org/perf.html#/e10s Perfherder e10s dashboard]


{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Task
! Metric
! Owner
! Analysis
! Analysis By
! Analysis Signed-off By
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Swap Rate (TBD)
| [https://wiki.mozilla.org/QA/E10s_Test_Plan Write test plan]
| {{bug|1198211}}
| RyanVM
| style="background-color: #f00;" | TBD
| style="background-color: LightYellow;" | In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan.
| style="background-color: #ffc;" | chutten
| style="background-color: SkyBlue;" | TODO
| style="background-color: #ffc;" | TBD
| style="background-color: #0f0;" | TBD
| style="background-color: #ffc;" | TBD
|-
|-
| Overall memory usage (browser + content)
| [https://wiki.mozilla.org/QA/E10s_Test_Plan Execute test plan]
| {{bug|1198209}}
| RyanVM
| style="background-color: #ff0;" | TBD
| style="background-color: LightYellow;" | In progress
| style="background-color: #ffc;" | chutten
| style="background-color: SkyBlue;" | TODO
| style="background-color: #ffc;" | TBD
|-
| style="background-color: #ffc;" | TBD
| Re-enable [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/ disabled e10s tests]
| style="background-color: #ffc;" | TBD
| Felipe, Blake
| style="background-color: LightYellow;" | In progress
| style="background-color: SkyBlue;" | TODO
|}
|}
* Notes from jimm: This looks good. I've removed XP from the operating system lists.


== Release Blocking Bugs ==
== Release Blocking Bugs ==


All M8 and M9 bugs must be fixed before we release.
All M9 bugs must be fixed before we release.


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| jimm?
| jimm
|-
|-
! Accountable
! Accountable
| blassey?
| blassey
|-
|-
! Supporting
! Supporting
Line 689: Line 751:
|}
|}


M8/M9 bugs:
* Bugzilla query for e10s blocking bugs: '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=e10s-release-blockers&sharer_id=430528 is.gd/e10s_blockers]'''
 
=== M9 bugs ===


<bugzilla>
<bugzilla>
{
{
     "f1": "cf_tracking_e10s",
     "f1": "cf_tracking_e10s",
     "o1": "anywordssubstr",
     "o1": "anywords",
     "v1": "m8+,m9+",
     "v1": "m9+,m8+,m7+,m6+,m5+,m4+,m3+,m2+,",
     "resolution": "---",
     "resolution": "---",
     "include_fields": "id, summary, whiteboard, keywords, assigned_to, cf_tracking_e10s"
     "include_fields": "id, summary, whiteboard, keywords, assigned_to, cf_tracking_e10s"
Line 701: Line 765:
</bugzilla>
</bugzilla>


Release Criteria meta bugs' blockers:
=== Release Criteria bugs ===


<bugzilla>
<bugzilla>
{
{
     "blocks": "1251376, 1251377, 1251388, 1251545, 1251547, 1251549, 1251551, 1251555",
     "blocks": "1198187,1251376, 1251377, 1251388, 1251545, 1251547, 1251549, 1251551, 1251555",
     "resolution": "---",
     "resolution": "---",
     "include_fields": "id, summary, whiteboard, keywords, assigned_to"
     "include_fields": "id, summary, whiteboard, keywords, assigned_to"
Line 711: Line 775:
</bugzilla>
</bugzilla>


* Bugzilla query for [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=e10s-release-blockers e10s-release-blockers]
=== APZ Bugs ===


== Tests ==
<bugzilla>
 
{
All tests disabled for e10s must be triaged and:
    "blocks": "1254668",
 
    "resolution": "---",
* reenabled (and fixed, if necessary)
    "include_fields": "id, summary, whiteboard, keywords, assigned_to"
* or annotated that the test is irrelevant for e10s?
}
* or deleted?
</bugzilla>
 
{| class="wikitable"
! Responsible
| Blake, Felipe
|-
! Accountable
| blassey?
|-
! Supporting
|
|-
! Consulted
| jgriffiths
|-
! Informed
| cpeterson, elan, release management
|}


* [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/ List of disabled e10s tests]
== Accessibility ==
* e10s test meta {{bug|e10s-tests}}
* [https://treeherder.mozilla.org/perf.html#/e10s Perfherder e10s dashboard]


* Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.


= Detail Pages =
== Add-ons ==


{| class="wikitable"
* Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
|-
! Release Criteria !! Responsible !! Status !! Tracking Bug
|-
| P1: [[Electrolysis/Release Criteria/Stability|Stability]] || poiru || {{mrisk}} || {{bug|1251376}}
|-
| P1: [[Electrolysis/Release Criteria/Jank|Jank]] || chutten || TBD || {{bug|1251377}}
|-
| P1: [[Electrolysis/Release Criteria/Scrolling|Scrolling]] || kats || {{mrisk}} || {{bug|1251388}}
|-
| P1: [[Electrolysis/Release Criteria/Slow Script|Slow Script]] || azhang || TBD || {{bug|1251545}}
|-
| P1: no blocker bugs || blassey || {{mok}} || [https://is.gd/1a6onf m9 bug query]
|-
| P1: automated tests running and passing || mrbkap? || {{mrisk}} || {{bug|e10s-tests}}
|-
| P2: [[Electrolysis/Release Criteria/UI Smoothness|UI Smoothness]] || mconley || {{mrisk}} || {{bug|1251547}}
|-
| P2: [[Electrolysis/Release Criteria/Pageload|Page Load Time]] || jimm || {{mok}} || {{bug|1251549}}
|-
| P2: [[Electrolysis/Release Criteria/Plugin Jank|Plugin Jank]] || chutten || TBD || {{bug|1251551}}
|-
| P2: [[Electrolysis/Release Criteria/Graphics|Graphics]] || Milan || {{mok}} || {{bug|1251555}}
|}

Latest revision as of 01:38, 15 July 2016

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Rollout Criteria

We plan to ship e10s in Firefox 48 using a staged rollout. See elan's e10s release plan.

Release Criteria

Stability

Responsible bsmedberg
Accountable bsmedberg
Supporting Kairo, rvitillo
Consulted Barbara, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
chrome + content crash rate (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours "M+(c-s)" on 48 Beta jimm 5.34 crashes/khour 5.20 crashes/khour YES Barbara (2016-06-29), bsmedberg (2016-07-01)
plugin process crash rate plugin crashes per 1000 use hours "PLUGIN" on 48 Beta jimm 9.53 plugin crashes/khour 12.59 plugin crashes/khour NO, but blassey says plugin crashes do not block e10s release. N/A

Crash rates exclude XP and deprecated OS X operating systems

User Engagement

Responsible rvitillo
Accountable TBD
Supporting measurement/data teams
Consulted Barbara, release management
Informed cpeterson, elan
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Engagement Ratio (DAU/MAU) Engagement Ratio graphs (bug 1251259) rvitillo ~0.22 ~0.35 Yes rvitillo, bsmedberg (2016-05-27)

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Event loop lag INPUT_EVENT_RESPONSE_MS beta 46 experiment

Nightly 48 telemetry

chutten e10s is different but not worse chutten and jgriffiths (2016-04-08)
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder YES: e10s is 39%–88% faster chutten and jgriffiths (2016-04-08)

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o summary ±5% Perfherder 248.32 (Linux)

230.35 (Win8)

246.28 (Linux)

213.51 (Win8)

YES: -0.82% to -7.31% improvement jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

FX_PAGE_LOAD_MS ±5% beta 46 experiment (bug 1228020) rvitillo e10s is much better. YES: e10s is much better. jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

Startup/Shutdown Time

Responsible rvitillo
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder 863 (Win7) 433 (Win7) YES: 43–55% better (bug 1174767) jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10%

("simpleMeasurements/firstPaint")

beta 46 experiment rvitillo 4468 ms 4391 ms YES: e10s is slightly better jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25%

("simpleMeasurements/shutdownDuration")

beta 46 experiment rvitillo 1156 ms 1190 ms YES: e10s is slightly worse, but within ±25% tolerance. jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

  • In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS beta 46 experiment avih YES: e10s is faster kats and jgriffiths (2016-04-08)
tp5o_scroll Perfherder kats YES: e10s is 6% – 53% better kats and jgriffiths (2016-04-08)
tscrollx Perfherder kats YES: e10s is 7% – 58% better kats and jgriffiths (2016-04-08)

Plugin Jank

Responsible chutten
Accountable aklotz
Supporting bsmedberg
Consulted Barbara
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
Plugin module initialization jank time BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is better.

Java: YES. e10s is better.

Silverlight: YES. e10s is better.

Unity: YES. e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% Beta 46 experiment chutten Flash: MAYBE? e10s is slightly worse.

Java: YES. e10s is slightly better.

Silverlight: MAYBE? e10s is slightly worse.

Unity: YES! e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is different, but not worse.

Java: MAYBE? e10s is slightly worse.

Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)

Unity: YES. e10s is different, but not worse.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Memory Usage

Responsible erahm
Accountable blassey
Supporting jmaher (Talos)
Consulted Barbara
Informed cpeterson, elan, release management
Metric Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder 401623141.33 (OS X) 331418381.64 (OS X) e10s is 17%–35% better erahm (2016-04-15), Barbara (2016-04-18)
MEMORY_TOTAL ±25% (browser + content process) beta 46 experiment and erahm's blog post e10s is 10–20% worse, but within our ±25% tolerance erahm (2016-04-15), Barbara (2016-04-18)
VSIZE_MAX_CONTIGUOUS (browser + content process) beta 46 experiment e10s is slightly different, but not worse? erahm (2016-04-15), Barbara (2016-04-18)

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted Barbara
Informed cpeterson, elan, release management

Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.

Metric Metric Owner Analysis Measurements Bugs Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder

e10s / APZ
xp = 4.4% combined
win7 = 1.8% / 9%
win8 = -0.6% / 10% (improvement)
osx = 6% / 3.5%
linux = 0.5% / 16%

e10s: bug 1251699
apz:bug 1253412

Borderline mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
tpaint ±5% (first paint for a new window) mconley Perfherder

xp = 0%
win7 = 9.3%
win8 = 3.4%
osx = -11% (improvement)
linux = 7%

bug 1174770 Borderline mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
tresize ±5% (window resize responsiveness) jimm Perfherder

e10s / APZ
xp = 1.0% combined
win7 = 0.0%
win8 = -22% (improvement overall)
osx = 1.0% combined
linux = 7% / 2.0%

bug 1252947 Yes jgriffiths (2016-03-17, bug 1252947#c23), mconley (2016-04-21)
tps ±5% (tab switch speed) mconley Perfherder

xp = -27%
win7 = -28%
win8 = -34%
osx = -30%
linux = -19%

e10s: bug 1186585
apz: bug 1216924

Yes mconley (2016-04-19), Barbara (2016-04-20)
FX_TAB_SWITCH_TOTAL_MS Gabor beta 46 experiment 52.66 ms (median) < 82.61 ms (median) bug 1252031 Yes gabor (2016-03-23, bug 1252031#c2), Barbara (2016-04-21)
CART ±5% (Customize Animation Regression Test) mconley Perfherder

e10s / APZ
xp = improvement
win7 = 4.0% / 2.5%
win8 = 2.0% / 2.5%
osx = 3.7% / 2.7%
linux = 6.8% / 6.8%

e10s: bug 1252768
apz: bug 1253380

Borderline avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)

Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted Barbara, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder 18 19 Mixed results from 3% worse to 66% better Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5% Perfherder 6444 6301 Mixed results from 2% worse to 3% better Barbara (2016-05-06), Milan (2016-05-06)

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted Barbara, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT beta 47 experiment azhang 0.00117087669174 warnings per use hour (aka 854 hours/warning) 0.00158655016424 warnings per use hour (aka 630 hours/warning)

(bug 1260769)

Barbara and Jim are OK with this regression because display rates are extremely low in both modes. Barbara, jimm, billm (2016-05-25)

Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?

We want each test directory to be signed-off by the feature area's owner.

Responsible Blake, Felipe
Accountable jimm
Supporting dolske (coordinating people fixing front-end tests)
Consulted jgriffiths
Informed cpeterson, elan, release management
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan RyanVM In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan. TODO
Execute test plan RyanVM In progress TODO
Re-enable disabled e10s tests Felipe, Blake In progress TODO
  • Notes from jimm: This looks good. I've removed XP from the operating system lists.

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1063169 [meta] tracking performance issues with e10s [webvr] meta
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


APZ Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Accessibility

  • Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

  • Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Release_Criteria&oldid=1139838"