Electrolysis/Release Criteria: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(clear incorrect RASCI for memory)
m (Fix layout)
 
(146 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.
Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.


= APZ Regressions =
= Rollout Criteria =


Talos numbers listed on this page are pulled from the [https://treeherder.mozilla.org/perf.html#/e10s Perfherder e10s dashboard]. These results include the affects of enabling APZ on mozilla central. The component of individual regressions and improvements associated with APZ is listed below and is based on a [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=b2b111fece96&newProject=mozilla-central&newRevision=dd1abe874252&framework=1&filter=e10s&showOnlyConfident=1 Perfherder comparison] between a try run with APZ disabled and mozilla-central. Test that showed no change are not listed.
We plan to ship e10s in Firefox 48 using a staged rollout. See [https://docs.google.com/document/d/150XUlvXi13mliWIixS5x5DoLE14ySxItBhQfCIKg1NI/edit elan's e10s release plan].


{| class="wikitable"
* '''Stability dashboards''':
|-
** [https://sql.telemetry.mozilla.org/queries/613/source  Firefox 48 Release]
! Talos Metric
** [https://sql.telemetry.mozilla.org/queries/607/source Firefox 48 Beta]
! Change
** "M+(c-s)" is the metric that matters: main (chrome) process crash rate + content process crash rate - content process shutdown crash rate.
! Related bugs
* '''Engagement Ratio dashboards''': (DAU/MAU)
|-
** [https://sql.telemetry.mozilla.org/dashboard/client-count-firefox-e10s-release%2D Firefox 48 Release]
| CART
** [https://sql.telemetry.mozilla.org/dashboard/e10s-client-count-beta-e10scontrol%2D Firefox 48 Beta]
| style="background-color: Tomato;" | 3%-5%
* chutten's [https://github.com/vitillo/e10s_analyses/blob/master/beta/48/week3/e10s_experiment.ipynb '''telemetry report'''] for Firefox 48 '''Beta'''. ''(FIXME: fix URL to point to Firefox 48 GA report when available!)''
| {{bug|1253380}}
|-
| DAMP
| style="background-color: Tomato;" | 5%-7%
|
|-
| TART
| style="background-color: Tomato;" | 10%-13%
| {{bug|1253412}}
|-
| tp5o %proc time
| style="background-color: Tomato;" | 5%-7% (windows)
|
|-
| tp5o main_RSS
| style="background-color: Tomato;" | 3%-15%
| {{bug|1254273}} (OS X)
|-
| tp5o
| style="background-color: LightYellow;" | mixed
| {{bug|1254273}} (OS X)
|-
| tp5o_scroll
| style="background-color: PaleGreen;" | 16%-40%
| {{bug|1179735}}, {{bug|1253860}}
|-
| tpaint
| style="background-color: LightYellow;" | 2% (Win7)
|
|-
| tps
| style="background-color: Tomato;" | 9%-11%
| {{bug|1216924}}
|-
| tresize
| style="background-color: Tomato;" | 4% (WinXP)
|
|-
| ts_paint
| style="background-color: LightYellow;" | 3% (WinXP)
|
|-
| tscrollx
| style="background-color: PaleGreen;" | 18%-42%
|
|-
| tsvgr_opacity
| style="background-color: Tomato;" | 3% (Linux)
|
|-
| tsvgx
| style="background-color: Tomato;" | 3%-4%
|
|}


= Release Criteria =
= Release Criteria =


== User Engagement ==
== Stability ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| rvitillo
| bsmedberg
|-
|-
! Accountable
! Accountable
| TBD
| bsmedberg
|-
|-
! Supporting
! Supporting
| measurement/data teams
| Kairo, rvitillo
|-
|-
! Consulted
! Consulted
| jgriffiths, release management
| Barbara, release management
|-
|-
! Informed
! Informed
Line 100: Line 46:
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Firefox Usage
| chrome + content crash rate
| style="background-color: Tomato;" | TBD ({{bug|1249665}})
| (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours
| style="background-color: LightYellow;" | TBD
| "M+(c-s)" on [https://sql.telemetry.mozilla.org/queries/607/source 48 Beta]
| style="background-color: LightYellow;" | glind?
| jimm
| style="background-color: LightYellow;" | TBD
| 5.34 crashes/khour
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | 5.20 crashes/khour
| style="background-color: Tomato;" | TBD
| style="background-color: PaleGreen;" | YES
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | Barbara (2016-06-29), bsmedberg (2016-07-01)
|-
|-
| User engagement (DAU/MAU)
| plugin process crash rate
| e10s/non-10s engagement ratio
| plugin crashes per 1000 use hours
| style="background-color: LightYellow;" | {{bug|1251259}}
| "PLUGIN" on [https://sql.telemetry.mozilla.org/queries/607/source 48 Beta]
| rvitillo
| jimm
| style="background-color: LightYellow;" | TBD
| 9.53 plugin crashes/khour
| style="background-color: LightYellow;" | TBD
| 12.59 plugin crashes/khour
| style="background-color: Tomato;" | TBD
| NO, but blassey says plugin crashes do not block e10s release.
| style="background-color: SkyBlue;" | TODO
| N/A
|}
|}


== Stability ==
Crash rates exclude XP and deprecated OS X operating systems
 
* [https://telemetry.mozilla.org/crashes/office-dashboard.html Stability "office dashboard"]
* [[Electrolysis/Release Criteria/Stability|More information on Stability analysis]]
* Stability analysis meta {{bug|1251376}}
 
== User Engagement ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| bsmedberg
| rvitillo
|-
|-
! Accountable
! Accountable
| bsmedberg
| TBD
|-
|-
! Supporting
! Supporting
| Kairo, rvitillo
| measurement/data teams
|-
|-
! Consulted
! Consulted
| jgriffiths, release management
| Barbara, release management
|-
|-
! Informed
! Informed
Line 140: Line 92:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Description
! Metric
! Metric
! Analysis
! Analysis
Line 149: Line 100:
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| chrome+content process crash rate
| Engagement Ratio (DAU/MAU)
| chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength
| [https://sql.telemetry.mozilla.org/dashboard/e10s-client-count-beta-e10scontrol- Engagement Ratio graphs] ({{bug|1251259}})
| [https://gist.github.com/poiru/1ceec405172bcafa7305 IPython notebook]
| rvitillo
| 84,390 (Beta 45)
| style="background-color: Tomato;" | 109,348 (Beta 45)
| style="background-color: Tomato;" | NO
| style="background-color: SkyBlue;" | TODO
|-
| plugin process crash rate
| SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength
| [https://gist.github.com/poiru/1ceec405172bcafa7305 IPython notebook]
| rvitillo
| rvitillo
| 41,897 (Beta 45)
| style="background-color: PaleGreen;" | ~0.22
| style="background-color: Tomato;" | 55,809 (Beta 45)
| style="background-color: PaleGreen;" | ~0.35
| style="background-color: Tomato;" | NO
| style="background-color: PaleGreen;" | Yes
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | rvitillo, bsmedberg (2016-05-27)
|}
|}
* [[Electrolysis/Release Criteria/Stability|More information on Stability analysis]]
* Stability analysis meta {{bug|1251376}}


== Jank ==
== Jank ==
Line 201: Line 139:
| GC pauses
| GC pauses
| GC_MAX_PAUSE_MS (parent, children)
| GC_MAX_PAUSE_MS (parent, children)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| rvitillo
| style="background-color: PaleGreen;" | e10s is slightly better
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
|-
| Cycle Collector pauses
| Cycle Collector pauses
| CYCLE_COLLECTOR_MAX_PAUSE (parent, children)
| CYCLE_COLLECTOR_MAX_PAUSE (parent, children)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| rvitillo
| style="background-color: PaleGreen;" | e10s is slightly better
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
|-
| Event loop lag
| Event loop lag
| INPUT_EVENT_RESPONSE_MS
| INPUT_EVENT_RESPONSE_MS
| [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_responsiveness.ipynb beta 45 experiment]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
[http://mzl.la/21WwhFj Nightly 48 telemetry]
| chutten
| chutten
| style="background-color: PaleGreen;" | about the same
| style="background-color: PaleGreen;" | e10s is different but not worse
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
|-
| Browser responsiveness during page load
| Browser responsiveness during page load
Line 224: Line 163:
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20responsiveness Perfherder]
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20responsiveness Perfherder]
|
|
| style="background-color: PaleGreen;" | 80%-90% faster!
| style="background-color: PaleGreen;" | YES: e10s is 39%–88% faster
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | chutten and jgriffiths (2016-04-08)
|-
|-
|}
|}
Line 232: Line 171:
* Jank analysis meta {{bug|1251377}}
* Jank analysis meta {{bug|1251377}}


== Scrolling ==
== Page Load ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| kats
| jimm
|-
|-
! Accountable
! Accountable
Line 242: Line 181:
|-
|-
! Supporting
! Supporting
| avih, jmaher, graphics team (mstange)
| jmaher
|-
|-
! Consulted
! Consulted
| jgriffiths, jmaher
| jgriffiths, rvitillo
|-
|-
! Informed
! Informed
Line 256: Line 195:
! Analysis
! Analysis
! Analysis By
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS
| tp5o summary ±5%
| style="background-color: LightYellow;" | TBD
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20summary Perfherder]
| avih
|
| style="background-color: Tomato;" | regressed
| 248.32 (Linux)
| style="background-color: SkyBlue;" | TODO
230.35 (Win8)
| style="background-color: PaleGreen;" | 246.28 (Linux)
213.51 (Win8)
| style="background-color: PaleGreen;" | YES: -0.82% to -7.31% improvement
| style="background-color: PaleGreen;" | jgriffiths, RyanVM (2016-04-01)
jimm (2016-04-04)
|-
|-
| tp5o_scroll
| FX_PAGE_LOAD_MS ±5%
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o_scroll Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] ({{bug|1228020}})
| kats
| rvitillo
| style="background-color: Tomato;" | results are mixed, but Linux is 46% better ({{bug|1253860}} fixes this)
|
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | e10s is much better.
|-
| style="background-color: PaleGreen;" | YES: e10s is much better.
| tscrollx
| style="background-color: PaleGreen;" | jgriffiths, RyanVM (2016-04-01)
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tscrollx Perfherder]
jimm (2016-04-04)
| kats
| style="background-color: Tomato;" | Up to 60% better on non-XP, but 10% worse on XP. ({{bug|1253860}} fixes this)
e10s is better on other platforms.
| style="background-color: SkyBlue;" | TODO
|}
|}


* [[Electrolysis/Release Criteria/Scrolling|More information on Scrolling analysis]]
* [[Electrolysis/Release Criteria/Pageload|More information on Page Load analysis]]
* Scrolling analysis meta {{bug|1251388}}
* Page Load analysis meta {{bug|1251549}}


== Slow Scripts ==
== Startup/Shutdown Time ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| azhang
| rvitillo
|-
|-
! Accountable
! Accountable
| bsmedberg
|  
|-
|-
! Supporting
! Supporting
Line 295: Line 237:
|-
|-
! Consulted
! Consulted
| jgriffiths, billm
| jgriffiths, jmaher
|-
|-
! Informed
! Informed
Line 303: Line 245:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Description
! Release Criteria
! Metric
! Metric
! Analysis
! Analysis
Line 312: Line 254:
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Slow script warnings
| Talos tests
| SLOW_SCRIPT_PAGE_COUNT
| ts_paint ±10%
| style="background-color: LightYellow;" | TBD (waiting for Beta 46 experiment)
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=ts_paint Perfherder]
| azhang
|
| style="background-color: LightYellow;" | TBD (waiting for Beta 46 experiment)
| 863 (Win7)
| style="background-color: LightYellow;" | TBD (waiting for Beta 46 experiment)
| style="background-color: PaleGreen;" | 433 (Win7)
| style="background-color: Tomato;" | TBD
| style="background-color: PaleGreen;" | YES: 43–55% better ({{bug|1174767}})
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|-
| Time to first paint
| SIMPLE_MEASURES_FIRSTPAINT ±10%
("simpleMeasurements/firstPaint")
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| 4468 ms
| style="background-color: PaleGreen;" | 4391 ms
| style="background-color: PaleGreen;" | YES: e10s is slightly better
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|-
| Shutdown time
| SIMPLE_MEASURES_SHUTDOWNDURATION ±25%
("simpleMeasurements/shutdownDuration")
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| rvitillo
| 1156 ms
| style="background-color: PaleGreen;" | 1190 ms
| style="background-color: PaleGreen;" | YES: e10s is slightly worse, but within ±25% tolerance.
| style="background-color: PaleGreen;" | jgriffiths, rvitillo (2016-04-01)
jimm (2016-04-04)
|}
|}


* [[Electrolysis/Release Criteria/Slow Script|More information on slow script analysis]]
* In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".
* Slow script analysis meta {{bug|1251545}}


== UI Smoothness ==
== Scrolling ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| mconley
| kats
|-
|-
! Accountable
! Accountable
Line 335: Line 299:
|-
|-
! Supporting
! Supporting
| jmaher, avih, gabor
| avih, jmaher, graphics team (mstange)
|-
|-
! Consulted
! Consulted
| jgriffiths
| jgriffiths, jmaher
|-
|-
! Informed
! Informed
Line 347: Line 311:
|-
|-
! Metric
! Metric
! Metric Owner
! Analysis
! Analysis
! Analysis By
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| TART ±5% (Tab Animation Regression Test)
| FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS
| mconley
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tart Perfherder]
| avih
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | YES: e10s is faster
| 3.96 (Linux)
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
| style="background-color: Tomato;" | 4.55 (Linux)
|-
| style="background-color: Tomato;" | 7%–15% ({{bug|1251699}})
| tp5o_scroll
| style="background-color: SkyBlue;" | TODO
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o_scroll Perfherder]
| kats
| style="background-color: PaleGreen;" | YES: e10s is 6% – 53% better
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
|-
| tscrollx
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tscrollx Perfherder]
| kats
| style="background-color: PaleGreen;" | YES: e10s is 7% – 58% better
| style="background-color: PaleGreen;" | kats and jgriffiths (2016-04-08)
|}
 
* [[Electrolysis/Release Criteria/Scrolling|More information on Scrolling analysis]]
* Scrolling analysis meta {{bug|1251388}}
 
== Plugin Jank ==
 
{| class="wikitable"
! Responsible
| chutten
|-
! Accountable
| aklotz
|-
! Supporting
| bsmedberg
|-
! Consulted
| Barbara
|-
|-
| tpaint ±5% (first paint for a new window)
! Informed
| mconley
| cpeterson, elan, release management
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tpaint Perfherder]
|}
| style="background-color: LightYellow;" | TBD
 
| 197 (Linux)
{| class="wikitable"
| style="background-color: Tomato;" | 286 (Linux
| style="background-color: Tomato;" | 29%–45% ({{bug|1174770}})
| style="background-color: SkyBlue;" | TODO
|-
|-
| tresize ±5% (window resize responsiveness)
! Description
| style="background-color: LightYellow;" | jimm?
! Metric
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tresize Perfherder]
! Analysis
| style="background-color: LightYellow;" | TBD
! Analysis By
| 9.92 (XP)
! Criteria Met?
| style="background-color: Tomato;" | 11.21 (XP)
! Criteria Met Signed-off By
| style="background-color: Tomato;" | 13% (XP) ({{bug|1252947}})
| style="background-color: SkyBlue;" | TODO
|-
|-
| tps ±5% (tab switch speed)
| Plugin module initialization jank time
| mconley
| BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5%
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tps Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| style="background-color: LightYellow;" | TBD
| chutten
| 43.27 (Linux)
| style="background-color: PaleGreen;" | '''Flash''': YES. e10s is better.
| style="background-color: Tomato;" | 72.27 (Linux)
'''Java''': YES. e10s is better.
| style="background-color: Tomato;" | 45%-70% ({{bug|1186585}})
 
| style="background-color: SkyBlue;" | TODO
'''Silverlight''': YES. e10s is better.
 
'''Unity''': YES. e10s is better.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|-
|-
| FX_TAB_SWITCH_TOTAL_MS
| Plugin initialization jank time
| style="background-color: LightYellow;" | mconley?
| BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%
| style="background-color: LightYellow;" | 46 experiment
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| style="background-color: LightYellow;" | TBD
| chutten
| TBD
| style="background-color: PaleGreen;" | '''Flash''': MAYBE? e10s is slightly worse.
| style="background-color: Tomato;" | TBD
'''Java''': YES. e10s is slightly better.
| style="background-color: Tomato;" | TBD
 
| style="background-color: SkyBlue;" | TODO
'''Silverlight''': MAYBE? e10s is slightly worse.
 
'''Unity''': YES! e10s is better.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|-
|-
| CART ±5% (Customize Animation Regression Test)
| Plugin destruction jank time
| mconley
| BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=cart Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/apz/e10s_plugin_lag.ipynb Beta 46 experiment]
| style="background-color: LightYellow;" | TBD
| chutten
| 20.76 (Linux)
| style="background-color: PaleGreen;" | '''Flash''': YES. e10s is different, but not worse.
| style="background-color: Tomato;" | 23.53 (Linux)
'''Java''': MAYBE? e10s is slightly worse.
| style="background-color: Tomato;" | 5%–13% ({{bug|1252768}})
 
| style="background-color: SkyBlue;" | TODO
'''Silverlight''': MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)
 
'''Unity''': YES. e10s is different, but not worse.
| style="background-color: PaleGreen;" | chutten (2016-04-08, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c4 bug 1261362])
aklotz (2016-04-09, [https://bugzilla.mozilla.org/show_bug.cgi?id=1261362#c5 bug 1261362])
Barbara (2016-04-18)
|}
|}


* [[Electrolysis/Release Criteria/UI Smoothness|More information on UI Smoothness analysis]]
* [[Electrolysis/Release Criteria/Plugin Jank|More information on Plugin Jank analysis]]
* UI smoothness analysis meta {{bug|1251547}}
* Plugin jank analysis meta {{bug|1251551}}


== Page Load ==
== Memory Usage ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| jimm
| erahm
|-
|-
! Accountable
! Accountable
| RyanVM
| blassey
|-
|-
! Supporting
! Supporting
| jmaher
| jmaher (Talos)
|-
|-
! Consulted
! Consulted
| jgriffiths, rvitillo
| Barbara
|-
|-
! Informed
! Informed
Line 434: Line 433:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Description
! Metric
! Metric
! Analysis
! Analysis
! Analysis By
! non-e10s value
! non-e10s value
! e10s value
! e10s value
Line 443: Line 440:
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Talos page load time
| tp5o Main_RSS
| tp5o ±5%
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=Main_RSS Perfherder]
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o Perfherder]
| 401623141.33 (OS X)
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | 331418381.64 (OS X)
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | e10s is 17%–35% better
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
| style="background-color: PaleGreen;" | no change
|-
| style="background-color: SkyBlue;" | TODO
| MEMORY_TOTAL ±25% (browser + content process)
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] and [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ erahm's blog post]
|
|
| style="background-color: PaleGreen;" | e10s is 10–20% worse, but within our ±25% tolerance
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
|-
|-
| Page load time telemetry
| VSIZE_MAX_CONTIGUOUS (browser + content process)
| FX_PAGE_LOAD_MS ±5%
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]  
| style="background-color: LightYellow;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
|
| style="background-color: LightYellow;" | rvitillo
|
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | e10s is slightly different, but not worse?
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | erahm (2016-04-15), Barbara (2016-04-18)
| style="background-color: PaleGreen;" | improvement
| style="background-color: SkyBlue;" | TODO
|}
|}


* [[Electrolysis/Release Criteria/Pageload|More information on Page Load analysis]]
* We expect e10s MEMORY_TOTAL to regress compared to non-e10s because we are using more processes.
* Scrolling analysis meta {{bug|1251549}}
* erahm's blog post: [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ Memory Usage of Firefox with e10s Enabled]


== Plugin Jank ==
== UI Smoothness ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| chutten
| mconley
|-
|-
! Accountable
! Accountable
| bsmedberg
| RyanVM
|-
|-
! Supporting
! Supporting
| aklotz
| jmaher, avih, gabor
|-
|-
! Consulted
! Consulted
| jgriffiths
| Barbara
|-
|-
! Informed
! Informed
| cpeterson, elan, release management
| cpeterson, elan, release management
|}
|}
Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.


{| class="wikitable"
{| class="wikitable"
|-
|-
! Description
! Metric
! Metric
! Metric Owner
! Analysis
! Analysis
! Analysis By
! Measurements
! non-e10s value
! Bugs
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Plugin initialization jank time
| TART ±5% (Tab Animation Regression Test)
| BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%
| mconley
| style="background-color: LightYellow;" | TBD
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tart Perfherder]
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" |
| style="background-color: LightYellow;" | TBD
e10s / APZ<br>
| style="background-color: LightYellow;" | TBD
xp = 4.4% combined<br>
| style="background-color: Tomato;" | TBD
win7 = 1.8% / 9%<br>
| style="background-color: SkyBlue;" | TODO
win8 = -0.6% / 10% (improvement)<br>
osx = 6% / 3.5%<br>
linux = 0.5% / 16%
| style="background-color: PaleGreen;" |
e10s: {{bug|1251699}}<br>
apz:{{bug|1253412}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
|-
| tpaint ±5% (first paint for a new window)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tpaint Perfherder]
| style="background-color: PaleGreen;" |
xp = 0%<br>
win7 = 9.3%<br>
win8 = 3.4%<br>
osx = -11% (improvement)<br>
linux = 7%
| style="background-color: PaleGreen;" | {{bug|1174770}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
|-
| tresize ±5% (window resize responsiveness)
| jimm
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tresize Perfherder]
| style="background-color: PaleGreen;" |
e10s / APZ<br>
xp = 1.0% combined<br>
win7 = 0.0% <br>
win8 = -22% (improvement overall)<br>
osx = 1.0% combined<br>
linux = 7% / 2.0%
| style="background-color: PaleGreen;" | {{bug|1252947}}
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | jgriffiths (2016-03-17, {{bug|1252947#c23}}), mconley (2016-04-21)
|-
| tps ±5% (tab switch speed)
| mconley
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tps Perfherder]
| style="background-color: PaleGreen;" |
xp = -27%<br>
win7 = -28%<br>
win8 = -34%<br>
osx = -30%<br>
linux = -19%
| style="background-color: PaleGreen;" |
e10s: {{bug|1186585}}<br>
apz: {{bug|1216924}}
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | mconley (2016-04-19), Barbara (2016-04-20)
|-
| FX_TAB_SWITCH_TOTAL_MS
| Gabor
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment]
| style="background-color: PaleGreen;" | 52.66 ms (median) < 82.61 ms (median)
| style="background-color: PaleGreen;" | [https://bugzilla.mozilla.org/show_bug.cgi?id=1252031#c2 bug 1252031]
| style="background-color: PaleGreen;" | Yes
| style="background-color: PaleGreen;" | gabor (2016-03-23, {{bug|1252031#c2}}), Barbara (2016-04-21)
|-
|-
| Plugin destruction jank time
| CART ±5% (Customize Animation Regression Test)
| BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%
| mconley
| style="background-color: LightYellow;" | TBD
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=cart Perfherder]
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" |
| style="background-color: LightYellow;" | TBD
e10s / APZ<br>
| style="background-color: LightYellow;" | TBD
xp = improvement<br>
| style="background-color: Tomato;" | TBD
win7 = 4.0% / 2.5%<br>
| style="background-color: SkyBlue;" | TODO
win8 = 2.0% / 2.5%<br>
osx = 3.7% / 2.7%<br>
linux = 6.8% / 6.8%
| style="background-color: PaleGreen;" |
e10s: {{bug|1252768}}<br>
apz: {{bug|1253380}}
| style="background-color: PaleGreen;" | Borderline
| style="background-color: PaleGreen;" | avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)
|}
|}


* [[Electrolysis/Release Criteria/Plugin Jank|More information on Plugin Jank analysis]]
* [[Electrolysis/Release Criteria/UI Smoothness|More information on UI Smoothness analysis]]
* Plugin jank analysis meta {{bug|1251551}}
* UI smoothness analysis meta {{bug|1251547}}


== Graphics Performance ==
== Graphics Performance ==
Line 530: Line 595:
|-
|-
! Consulted
! Consulted
| jgriffiths, jmaher, graphics team
| Barbara, jmaher, graphics team
|-
|-
! Informed
! Informed
Line 551: Line 616:
| 18
| 18
| style="background-color: PaleGreen;" | 19
| style="background-color: PaleGreen;" | 19
| style="background-color: PaleGreen;" | Mixed results from 5% worse to 64% better
| style="background-color: PaleGreen;" | Mixed results from 3% worse to 66% better
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | Barbara (2016-05-06), Milan (2016-05-06)
|-
|-
| tcanvasmark ±5%
| tcanvasmark ±5%
Line 560: Line 625:
| style="background-color: PaleGreen;" | 6301
| style="background-color: PaleGreen;" | 6301
| style="background-color: PaleGreen;" | Mixed results from 2% worse to 3% better
| style="background-color: PaleGreen;" | Mixed results from 2% worse to 3% better
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | Barbara (2016-05-06), Milan (2016-05-06)
|-
| tsvgx ±5%
| mstange
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tsvgx Perfherder]
| 122
| style="background-color: PaleGreen;" | 92
| style="background-color: PaleGreen;" | 25%–51% better
| style="background-color: SkyBlue;" | TODO
|-
| tsvg_opacity ±5%
| style="background-color: LightYellow;" | avih says no real owner. mattwoodrow? seth?
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=tsvgr_opacity Perfherder]
| 536
| style="background-color: Tomato;" | 865
| style="background-color: Tomato;" | about 65% slower on 32-bit Windows ({{bug|1255936}})
| style="background-color: SkyBlue;" | TODO
|}
|}


* [[Electrolysis/Release Criteria/Graphics|More information on Graphics Performance analysis]]
* [[Electrolysis/Release Criteria/Graphics|More information on Graphics Performance analysis]]
* Scrolling analysis meta {{bug|1251555}}
* Graphics analysis meta {{bug|1251555}}


== Startup/Shutdown Time ==
== Slow Scripts ==


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| rvitillo
| azhang
|-
|-
! Accountable
! Accountable
|  
| bsmedberg
|-
|-
! Supporting
! Supporting
Line 595: Line 644:
|-
|-
! Consulted
! Consulted
| jgriffiths, jmaher
| Barbara, billm
|-
|-
! Informed
! Informed
Line 603: Line 652:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Release Criteria
! Description
! Metric
! Metric
! Analysis
! Analysis
Line 612: Line 661:
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| Talos tests
| Slow script warnings
| ts_paint ±10%
| SLOW_SCRIPT_PAGE_COUNT
| [https://treeherder.mozilla.org/perf.html#/e10s?filter=ts_paint Perfherder]
| [https://github.com/vitillo/e10s_analyses/blob/master/beta/47/week1/e10s_experiment.ipynb beta 47 experiment]
|
| azhang
| 863 (Win7)
| 0.00117087669174 warnings per use hour (aka 854 hours/warning)
| style="background-color: PaleGreen;" | 433 (Win7)
| 0.00158655016424 warnings per use hour (aka 630 hours/warning)
| style="background-color: PaleGreen;" | 43–55% better
({{bug|1260769}})
| style="background-color: SkyBlue;" | TODO
| style="background-color: PaleGreen;" | Barbara and Jim are OK with this regression because display rates are extremely low in both modes.  
|-
| style="background-color: PaleGreen;" | Barbara, jimm, billm (2016-05-25)
| Time to first paint
| SIMPLE_MEASURES_FIRSTPAINT
| [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| rvitillo
| 4636 ms (median)
| style="background-color: PaleGreen;" | 4620 ms (median)
| style="background-color: PaleGreen;" | no change
| style="background-color: SkyBlue;" | TODO
|-
| Shutdown time
| SIMPLE_MEASURES_SHUTDOWNDURATION
| [https://github.com/vitillo/e10s_analyses/blob/master/beta45-withoutaddons/e10s_experiment.ipynb beta 45 experiment]
| rvitillo
| 1216 ms (median)
| style="background-color: PaleGreen;" | 1229 ms (median)
| style="background-color: PaleGreen;" | no change
| style="background-color: SkyBlue;" | TODO
|}
|}


* Startup times may regress up to 10% and shutdown time up to 25% compared to non-e10s.
* [[Electrolysis/Release Criteria/Slow Script|More information on slow script analysis]]
* In telemetry performance reports, search for payload/simpleMeasurements/firstPaint, payload/simpleMeasurements/sessionRestored, and payload/simpleMeasurements/shutdownDuration.
* Slow script analysis meta {{bug|1251545}}
 
== Tests ==
 
All tests disabled for e10s must be triaged and:


== Memory Usage ==
* reenabled (and fixed, if necessary)
* or annotated that the test is irrelevant for e10s?
* or deleted?
 
We want each test directory to be signed-off by the feature area's owner.


{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
|  
| Blake, Felipe
|-
|-
! Accountable
! Accountable
|  
| jimm
|-
|-
! Supporting
! Supporting
|  
| dolske (coordinating people fixing front-end tests)
|-
|-
! Consulted
! Consulted
Line 661: Line 701:
| cpeterson, elan, release management
| cpeterson, elan, release management
|}
|}
* e10s test meta {{bug|e10s-tests}}
* [https://treeherder.mozilla.org/perf.html#/e10s Perfherder e10s dashboard]


{| class="wikitable"
{| class="wikitable"
|-
|-
! Metric
! Task
! Analysis
! Owner
! Analysis By
! non-e10s value
! e10s value
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
|-
| tp5o Main_RSS
| [https://wiki.mozilla.org/QA/E10s_Test_Plan Write test plan]
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=Main_RSS Perfherder]
| RyanVM
|
| style="background-color: LightYellow;" | In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan.
| 384117275.55 (OS X)
| style="background-color: PaleGreen;" | 365409824.16 (OS X)
| style="background-color: PaleGreen;" | 5%–35% better
| style="background-color: SkyBlue;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tp5o Private Bytes
| [https://wiki.mozilla.org/QA/E10s_Test_Plan Execute test plan]
| style="background-color: LightYellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=Private%20Bytes Perfherder]
| RyanVM
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | In progress
| 636896373.57 (Linux)
| style="background-color: Tomato;" | 932390353.63 (Linux)
| style="background-color: Tomato;" | 20% to 50% worse ({{bug|1253984}})
| style="background-color: SkyBlue;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| MEMORY_TOTAL ±X% (browser + content process)
| Re-enable [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/ disabled e10s tests]
| style="background-color: LightYellow;" | TBD
| Felipe, Blake
| style="background-color: LightYellow;" | chutten
| style="background-color: LightYellow;" | In progress
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: SkyBlue;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


* We expect e10s MEMORY_TOTAL to regress compared to non-e10s because we are using more processes.
* Notes from jimm: This looks good. I've removed XP from the operating system lists.


== Release Blocking Bugs ==
== Release Blocking Bugs ==


All M8 and M9 bugs must be fixed before we release.
All M9 bugs must be fixed before we release.


{| class="wikitable"
{| class="wikitable"
Line 722: Line 753:
* Bugzilla query for e10s blocking bugs: '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=e10s-release-blockers&sharer_id=430528 is.gd/e10s_blockers]'''
* Bugzilla query for e10s blocking bugs: '''[https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=e10s-release-blockers&sharer_id=430528 is.gd/e10s_blockers]'''


=== M8/M9 bugs ===
=== M9 bugs ===


<bugzilla>
<bugzilla>
Line 728: Line 759:
     "f1": "cf_tracking_e10s",
     "f1": "cf_tracking_e10s",
     "o1": "anywords",
     "o1": "anywords",
     "v1": "m8+,m9+",
     "v1": "m9+,m8+,m7+,m6+,m5+,m4+,m3+,m2+,",
     "resolution": "---",
     "resolution": "---",
     "include_fields": "id, summary, whiteboard, keywords, assigned_to, cf_tracking_e10s"
     "include_fields": "id, summary, whiteboard, keywords, assigned_to, cf_tracking_e10s"
Line 734: Line 765:
</bugzilla>
</bugzilla>


=== Release Criteria meta bugs' blockers ===
=== Release Criteria bugs ===


<bugzilla>
<bugzilla>
{
{
     "blocks": "1251376, 1251377, 1251388, 1251545, 1251547, 1251549, 1251551, 1251555",
     "blocks": "1198187,1251376, 1251377, 1251388, 1251545, 1251547, 1251549, 1251551, 1251555",
     "resolution": "---",
     "resolution": "---",
     "include_fields": "id, summary, whiteboard, keywords, assigned_to"
     "include_fields": "id, summary, whiteboard, keywords, assigned_to"
Line 745: Line 776:


=== APZ Bugs ===
=== APZ Bugs ===
* APZ does '''not''' block e10s release. This bug list is purely informational.


<bugzilla>
<bugzilla>
Line 756: Line 785:
</bugzilla>
</bugzilla>


== Tests ==
== Accessibility ==


All tests disabled for e10s must be triaged and:
* Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.


* reenabled (and fixed, if necessary)
== Add-ons ==
* or annotated that the test is irrelevant for e10s?
* or deleted?


{| class="wikitable"
* Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
! Responsible
| Blake, Felipe
|-
! Accountable
| blassey?
|-
! Supporting
|
|-
! Consulted
| jgriffiths
|-
! Informed
| cpeterson, elan, release management
|}
 
* [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/ List of disabled e10s tests]
* e10s test meta {{bug|e10s-tests}}
* [https://treeherder.mozilla.org/perf.html#/e10s Perfherder e10s dashboard]
 
= Detail Pages =
 
{| class="wikitable"
|-
! Release Criteria !! Responsible !! Status !! Tracking Bug
|-
| P1: [[Electrolysis/Release Criteria/Stability|Stability]] || poiru || {{mrisk}} || {{bug|1251376}}
|-
| P1: [[Electrolysis/Release Criteria/Jank|Jank]] || chutten || TBD || {{bug|1251377}}
|-
| P1: [[Electrolysis/Release Criteria/Scrolling|Scrolling]] || kats || {{mrisk}} || {{bug|1251388}}
|-
| P1: [[Electrolysis/Release Criteria/Slow Script|Slow Script]] || azhang || TBD || {{bug|1251545}}
|-
| P1: no blocker bugs || blassey || {{mok}} || [https://is.gd/1a6onf m9 bug query]
|-
| P1: automated tests running and passing || mrbkap? || {{mrisk}} || {{bug|e10s-tests}}
|-
| P2: [[Electrolysis/Release Criteria/UI Smoothness|UI Smoothness]] || mconley || {{mrisk}} || {{bug|1251547}}
|-
| P2: [[Electrolysis/Release Criteria/Pageload|Page Load Time]] || jimm || {{mok}} || {{bug|1251549}}
|-
| P2: [[Electrolysis/Release Criteria/Plugin Jank|Plugin Jank]] || chutten || TBD || {{bug|1251551}}
|-
| P2: [[Electrolysis/Release Criteria/Graphics|Graphics]] || Milan || {{mok}} || {{bug|1251555}}
|}

Latest revision as of 01:38, 15 July 2016

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Rollout Criteria

We plan to ship e10s in Firefox 48 using a staged rollout. See elan's e10s release plan.

Release Criteria

Stability

Responsible bsmedberg
Accountable bsmedberg
Supporting Kairo, rvitillo
Consulted Barbara, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
chrome + content crash rate (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours "M+(c-s)" on 48 Beta jimm 5.34 crashes/khour 5.20 crashes/khour YES Barbara (2016-06-29), bsmedberg (2016-07-01)
plugin process crash rate plugin crashes per 1000 use hours "PLUGIN" on 48 Beta jimm 9.53 plugin crashes/khour 12.59 plugin crashes/khour NO, but blassey says plugin crashes do not block e10s release. N/A

Crash rates exclude XP and deprecated OS X operating systems

User Engagement

Responsible rvitillo
Accountable TBD
Supporting measurement/data teams
Consulted Barbara, release management
Informed cpeterson, elan
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Engagement Ratio (DAU/MAU) Engagement Ratio graphs (bug 1251259) rvitillo ~0.22 ~0.35 Yes rvitillo, bsmedberg (2016-05-27)

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Event loop lag INPUT_EVENT_RESPONSE_MS beta 46 experiment

Nightly 48 telemetry

chutten e10s is different but not worse chutten and jgriffiths (2016-04-08)
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder YES: e10s is 39%–88% faster chutten and jgriffiths (2016-04-08)

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o summary ±5% Perfherder 248.32 (Linux)

230.35 (Win8)

246.28 (Linux)

213.51 (Win8)

YES: -0.82% to -7.31% improvement jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

FX_PAGE_LOAD_MS ±5% beta 46 experiment (bug 1228020) rvitillo e10s is much better. YES: e10s is much better. jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

Startup/Shutdown Time

Responsible rvitillo
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder 863 (Win7) 433 (Win7) YES: 43–55% better (bug 1174767) jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10%

("simpleMeasurements/firstPaint")

beta 46 experiment rvitillo 4468 ms 4391 ms YES: e10s is slightly better jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25%

("simpleMeasurements/shutdownDuration")

beta 46 experiment rvitillo 1156 ms 1190 ms YES: e10s is slightly worse, but within ±25% tolerance. jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

  • In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS beta 46 experiment avih YES: e10s is faster kats and jgriffiths (2016-04-08)
tp5o_scroll Perfherder kats YES: e10s is 6% – 53% better kats and jgriffiths (2016-04-08)
tscrollx Perfherder kats YES: e10s is 7% – 58% better kats and jgriffiths (2016-04-08)

Plugin Jank

Responsible chutten
Accountable aklotz
Supporting bsmedberg
Consulted Barbara
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
Plugin module initialization jank time BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is better.

Java: YES. e10s is better.

Silverlight: YES. e10s is better.

Unity: YES. e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% Beta 46 experiment chutten Flash: MAYBE? e10s is slightly worse.

Java: YES. e10s is slightly better.

Silverlight: MAYBE? e10s is slightly worse.

Unity: YES! e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is different, but not worse.

Java: MAYBE? e10s is slightly worse.

Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)

Unity: YES. e10s is different, but not worse.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Memory Usage

Responsible erahm
Accountable blassey
Supporting jmaher (Talos)
Consulted Barbara
Informed cpeterson, elan, release management
Metric Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder 401623141.33 (OS X) 331418381.64 (OS X) e10s is 17%–35% better erahm (2016-04-15), Barbara (2016-04-18)
MEMORY_TOTAL ±25% (browser + content process) beta 46 experiment and erahm's blog post e10s is 10–20% worse, but within our ±25% tolerance erahm (2016-04-15), Barbara (2016-04-18)
VSIZE_MAX_CONTIGUOUS (browser + content process) beta 46 experiment e10s is slightly different, but not worse? erahm (2016-04-15), Barbara (2016-04-18)

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted Barbara
Informed cpeterson, elan, release management

Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.

Metric Metric Owner Analysis Measurements Bugs Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder

e10s / APZ
xp = 4.4% combined
win7 = 1.8% / 9%
win8 = -0.6% / 10% (improvement)
osx = 6% / 3.5%
linux = 0.5% / 16%

e10s: bug 1251699
apz:bug 1253412

Borderline mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
tpaint ±5% (first paint for a new window) mconley Perfherder

xp = 0%
win7 = 9.3%
win8 = 3.4%
osx = -11% (improvement)
linux = 7%

bug 1174770 Borderline mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
tresize ±5% (window resize responsiveness) jimm Perfherder

e10s / APZ
xp = 1.0% combined
win7 = 0.0%
win8 = -22% (improvement overall)
osx = 1.0% combined
linux = 7% / 2.0%

bug 1252947 Yes jgriffiths (2016-03-17, bug 1252947#c23), mconley (2016-04-21)
tps ±5% (tab switch speed) mconley Perfherder

xp = -27%
win7 = -28%
win8 = -34%
osx = -30%
linux = -19%

e10s: bug 1186585
apz: bug 1216924

Yes mconley (2016-04-19), Barbara (2016-04-20)
FX_TAB_SWITCH_TOTAL_MS Gabor beta 46 experiment 52.66 ms (median) < 82.61 ms (median) bug 1252031 Yes gabor (2016-03-23, bug 1252031#c2), Barbara (2016-04-21)
CART ±5% (Customize Animation Regression Test) mconley Perfherder

e10s / APZ
xp = improvement
win7 = 4.0% / 2.5%
win8 = 2.0% / 2.5%
osx = 3.7% / 2.7%
linux = 6.8% / 6.8%

e10s: bug 1252768
apz: bug 1253380

Borderline avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)

Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted Barbara, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder 18 19 Mixed results from 3% worse to 66% better Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5% Perfherder 6444 6301 Mixed results from 2% worse to 3% better Barbara (2016-05-06), Milan (2016-05-06)

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted Barbara, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT beta 47 experiment azhang 0.00117087669174 warnings per use hour (aka 854 hours/warning) 0.00158655016424 warnings per use hour (aka 630 hours/warning)

(bug 1260769)

Barbara and Jim are OK with this regression because display rates are extremely low in both modes. Barbara, jimm, billm (2016-05-25)

Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?

We want each test directory to be signed-off by the feature area's owner.

Responsible Blake, Felipe
Accountable jimm
Supporting dolske (coordinating people fixing front-end tests)
Consulted jgriffiths
Informed cpeterson, elan, release management
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan RyanVM In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan. TODO
Execute test plan RyanVM In progress TODO
Re-enable disabled e10s tests Felipe, Blake In progress TODO
  • Notes from jimm: This looks good. I've removed XP from the operating system lists.

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1063169 [meta] tracking performance issues with e10s [webvr] meta
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


APZ Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Accessibility

  • Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

  • Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Release_Criteria&oldid=1139838"