Revision as of 16:20, 30 March 2017

Page to collect and track Electrolysis Multi Process release criteria.

Rollout Criteria

We plan to ship e10s-multi in Firefox 54/55 using a staged rollout.
See elan's e10s-multi release plan.

Stability dashboards:
- e10s-multi
  - Firefox 54 Release <- Need link to telemetry report
  - Firefox 54 Beta <- Need link to telemetry report
  - "M+(c-s)" is the metric that matters: main (chrome) process crash rate + content process crash rate - content process shutdown crash rate.
- e10s
  - Firefox 48 Release
  - Firefox 48 Beta

Engagement Ratio dashboards: (DAU/MAU)
- e10s-multi
- Firefox 54 Release <- Need link to telemetry report
- Firefox 54 Beta <- Need link to telemetry report

- e10s
- Firefox 48 Release
- Firefox 48 Beta

chutten's telemetry:
- e10s-multi
  - Need link to telemetry report

- e10s
  - report for Firefox 48 Beta. (FIXME: fix URL to point to Firefox 48 GA report when available!)

Release Criteria

Stability

Responsible	e10s - bsmedberg e10s-multi - TBD
Accountable	e10s - bsmedberg e10s-multi - TBD
Supporting	e10s - Kairo, rvitillo e10s-multi - TBD
Consulted	e10s - Barbara, release management e10s-multi - TBD
Informed	e10s - cpeterson, elan e10s-multi - TBD

Description	Metric	e10s Analysis	Analysis By	e10s value	e10s-multi value	Criteria Met?	Criteria Met Signed-off By
chrome + content crash rate	(chrome + (content - content shutdown termination crash reports) ) / 1000 use hours	"M+(c-s)" on 48 Beta Update Link here	TBD	5.20 crashes/khour	TBD	TBD	TBD

Crash rates exclude XP and deprecated OS X operating systems

More information on Stability analysis

User Engagement

Is this needed for e10-multi?

Responsible	rvitillo
Accountable	TBD
Supporting	measurement/data teams
Consulted	Barbara, release management
Informed	cpeterson, elan

Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
Engagement Ratio (DAU/MAU)	Engagement Ratio graphs (bug 1251259)	rvitillo	~0.22	~0.35	Yes	rvitillo, bsmedberg (2016-05-27)

Jank

Responsible	e10s - chutten e10s-multi - TBD
Accountable	e10s - bsmedberg e10s-multi - TBD
Supporting	e10s - data team, RyanVM, rvitillo, avih, Softvision e10s-multi - TBD
Consulted	e10s - jgriffiths, billm e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management, e10s-multi - TBD

Description	Metric	e10s Analysis	e10s-multi Analysis	Analysis By	Criteria Met?	Criteria Met Signed-off By
GC pauses	GC_MAX_PAUSE_MS (parent, children)	beta 46 experiment	TBD	TBD	TBD	TBD
Cycle Collector pauses	CYCLE_COLLECTOR_MAX_PAUSE (parent, children)	beta 46 experiment	TBD	TBD	TBD	TBD
Event loop lag	INPUT_EVENT_RESPONSE_MS	beta 46 experiment Nightly 48 telemetry	TBD	TBD	TBD	TBD
Browser responsiveness during page load	Talos tp5o_responsiveness	Perfherder	TBD	TBD	TBD	TBD

More information on Jank analysis
Jank analysis meta bug 1251377

Page Load

Responsible	e10s - jimm e10s-multi - TBD
Accountable	e10s - RyanVM e10s-multi - TBD
Supporting	e10s - jmaher e10s-multi - TBD
Consulted	e10s - jgriffiths, rvitillo e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

Metric	Analysis	Analysis By	e10s value	e10s-multi value	Criteria Met?	Criteria Met Signed-off By
tp5o summary ±5%	Perfherder Need updated link	TBD	246.28 (Linux) 213.51 (Win8)	TBD	TBD	TBD
FX_PAGE_LOAD_MS ±5%	beta 46 experiment (bug 1228020) Need updated link	TBD	e10s is much better.	TBD	TBD	TBD

More information on Page Load analysis
Page Load analysis meta bug 1251549

Startup/Shutdown Time

Responsible	e10s - rvitillo e10s-multi - TBD
Accountable	e10s - e10s-multi - TBD
Supporting	e10s - e10s-multi - TBD
Consulted	e10s - jgriffiths, jmaher e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

Release Criteria	Metric	Analysis	Analysis By	e10s value	e10s-multi value	Criteria Met?	Criteria Met Signed-off By
Talos tests	ts_paint ±10%	Perfherder Need updated link	TBD	433 (Win7)	TBD	TBD	TBD
Time to first paint	SIMPLE_MEASURES_FIRSTPAINT ±10% ("simpleMeasurements/firstPaint")	beta 46 experiment Need updated link	TBD	4391 ms	TBD	TBD	TBD
Shutdown time	SIMPLE_MEASURES_SHUTDOWNDURATION ±25% ("simpleMeasurements/shutdownDuration")	beta 46 experiment Need updated link	TBD	1190 ms	TBD	TBD	TBD

In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible	e10s - kats e10s-multi - TBD
Accountable	e10s - RyanVM e10s-multi - TBD
Supporting	e10s - avih, jmaher, graphics team (mstange) e10s-multi - TBD
Consulted	e10s - jgriffiths, jmaher e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

Metric	Analysis	Analysis By	Criteria Met?	Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS	beta 46 experiment Need updated link	TBD	TBD	TBD
tp5o_scroll	Perfherder Need updated link	TBD	TBD	TBD
tscrollx	Perfherder Need updated link	TBD	TBD	TBD

More information on Scrolling analysis
Scrolling analysis meta bug 1251388

Plugin Jank

Is this needed for e10-multi?

Responsible	chutten
Accountable	aklotz
Supporting	bsmedberg
Consulted	Barbara
Informed	cpeterson, elan, release management

Description	Metric	Analysis	Analysis By	Criteria Met?	Criteria Met Signed-off By
Plugin module initialization jank time	BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5%	Beta 46 experiment	chutten	Flash: YES. e10s is better. Java: YES. e10s is better. Silverlight: YES. e10s is better. Unity: YES. e10s is better.	chutten (2016-04-08, bug 1261362) aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)
Plugin initialization jank time	BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%	Beta 46 experiment	chutten	Flash: MAYBE? e10s is slightly worse. Java: YES. e10s is slightly better. Silverlight: MAYBE? e10s is slightly worse. Unity: YES! e10s is better.	chutten (2016-04-08, bug 1261362) aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)
Plugin destruction jank time	BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%	Beta 46 experiment	chutten	Flash: YES. e10s is different, but not worse. Java: MAYBE? e10s is slightly worse. Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms) Unity: YES. e10s is different, but not worse.	chutten (2016-04-08, bug 1261362) aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

More information on Plugin Jank analysis
Plugin jank analysis meta bug 1251551

Memory Usage

Responsible	e10s - erahm e10s-multi - TBD
Accountable	e10s - blassey e10s-multi - TBD
Supporting	e10s jmaher (Talos) e10s-multi - TBD
Consulted	e10s - Barbara e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

Metric	Analysis	e10s value	e10s-multi value	Criteria Met?	Criteria Met Signed-off By
tp5o Main_RSS	Perfherder Need updated link	331418381.64 (OS X)	TBD	TBD	TBD
MEMORY_TOTAL ±25% (browser + content process)	beta 46 experiment and erahm's blog post Need updated link(s)		TBD	TBD	TBD
VSIZE_MAX_CONTIGUOUS (browser + content process)	beta 46 experiment Need updated link		TBD	TBD	TBD
Additional Metric(s) Needed? - (process count 1, 2, 4 and 8 data) - AWSY?

We expect e10s-multi MEMORY_TOTAL to regress compared to e10s because we are using more processes.
erahm's blog post: Memory Usage of Firefox with e10s Enabled

UI Smoothness

Responsible	e10s - mconley e10s-multi - TBD
Accountable	e10s - RyanVM e10s-multi - TBD
Supporting	e10s - jmaher, avih, gabor e10s-multi - TBD
Consulted	e10s - Barbara e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.

Metric	Metric Owner	Analysis	e10s Data	e10s-multi Data	Criteria Met?	Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test)	TBD	Perfherder Need updated link	e10s / APZ xp = 4.4% combined win7 = 1.8% / 9% win8 = -0.6% / 10% (improvement) osx = 6% / 3.5% linux = 0.5% / 16%	TBD	TBD	TBD
tpaint ±5% (first paint for a new window)	TBD	Perfherder Need updated link	xp = 0% win7 = 9.3% win8 = 3.4% osx = -11% (improvement) linux = 7%	TBD	TBD	TBD
tresize ±5% (window resize responsiveness)	TBD	Perfherder Need updated link	e10s / APZ xp = 1.0% combined win7 = 0.0% win8 = -22% (improvement overall) osx = 1.0% combined linux = 7% / 2.0%	TBD	TBD	TBD
tps ±5% (tab switch speed)	TBD	Perfherder Need updated link	xp = -27% win7 = -28% win8 = -34% osx = -30% linux = -19%	TBD	TBD	TBD
FX_TAB_SWITCH_TOTAL_MS	TBD	beta 46 experiment Need updated link	52.66 ms (median) < 82.61 ms (median)	TBD	TBD	TBD
CART ±5% (Customize Animation Regression Test)	TBD	Perfherder Need updated link	e10s / APZ xp = improvement win7 = 4.0% / 2.5% win8 = 2.0% / 2.5% osx = 3.7% / 2.7% linux = 6.8% / 6.8%	TBD	TBD	TBD

More information on UI Smoothness analysis
UI smoothness analysis meta bug 1251547

Graphics Performance

Is this needed for e10-multi?

Responsible	Milan
Accountable	RyanVM
Supporting	gw280
Consulted	Barbara, jmaher, graphics team
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
glterrain ±5%	jgilbert	Perfherder	18	19	Mixed results from 3% worse to 66% better	Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5%		Perfherder	6444	6301	Mixed results from 2% worse to 3% better	Barbara (2016-05-06), Milan (2016-05-06)

More information on Graphics Performance analysis
Graphics analysis meta bug 1251555

Slow Scripts

Responsible	e10s - azhang e10s-multi - TBD
Accountable	e10s - bsmedberg e10s-multi - TBD
Supporting	e10s - e10s-multi - TBD
Consulted	e10s - Barbara, billm e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

Description	Metric	Analysis	Analysis By	e10s value	e10s-multi value	Criteria Met?	Criteria Met Signed-off By
Slow script warnings	SLOW_SCRIPT_PAGE_COUNT	beta 47 experiment Need updated link	TBD	0.00158655016424 warnings per use hour (aka 630 hours/warning) (bug 1260769)	TBD	TBD	TBD

More information on slow script analysis
Slow script analysis meta bug 1251545

Tests

All unit tests disabled for e10s-multi must be triaged and:

re-enabled (and fixed, if necessary)
or annotated that the test is irrelevant for e10s-multi
or deleted

We want each test directory to be signed-off by the feature area's owner.

Responsible	e10s - Blake, Felipe e10s-multi - TBD
Accountable	e10s - jimm e10s-multi - TBD
Supporting	e10s - dolske (coordinating people fixing front-end tests) e10s-multi - TBD
Consulted	e10s - jgriffiths e10s-multi - TBD
Informed	e10s - cpeterson, elan, release management e10s-multi - TBD

e10s-multi test meta bug 1315042
Perfherder e10s dashboard <- Need updated link

Task	Owner	Criteria Met?	Criteria Met Signed-off By
Write test plan	Tracy	In progress: almost done (based on this doc triage)	TBD
Execute test plan	Tracy, SV	In progress	TBD
Re-enable disabled e10s tests	Gabor, Blake	In progress	TBD

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible	jimm
Accountable	blassey
Supporting	release management
Consulted	jgriffiths
Informed	cpeterson, elan

Bugzilla query for e10s blocking bugs: is.gd/e10s_blockers

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Release Criteria bugs

Need updated bug list

Bugzilla query error

Query options must be valid JSON.1

APZ Bugs

Need updated bug list

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Accessibility

Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.

Service Workers

???

@@ Line 448: / Line 448: @@
 | style="background-color: Yellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=Main_RSS Perfherder] '''<span style="color:#9932cc"> Need updated link</span>'''
 | style="background-color: PaleGreen;" | 331418381.64 (OS X)
-|
+| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
 | style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
 | style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
@@ Line 455: / Line 455: @@
 | style="background-color: Yellow;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] and [http://www.erahm.org/2016/02/11/memory-usage-of-firefox-with-e10s-enabled/ erahm's blog post] '''<span style="color:#9932cc"> Need updated link(s)</span>'''
 |
-|
+| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
 | style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
 | style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
@@ Line 462: / Line 462: @@
 | style="background-color: Yellow;" | [https://github.com/vitillo/e10s_analyses/blob/master/beta/46/noapz/e10s_experiment.ipynb beta 46 experiment] '''<span style="color:#9932cc"> Need updated link</span>'''
 |
-|
+| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
 | style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
 | style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''

Electrolysis/Multi Release Criteria: Difference between revisions

Revision as of 16:20, 30 March 2017

Contents

Rollout Criteria

Release Criteria

Stability

User Engagement

Jank

Page Load

Startup/Shutdown Time

Scrolling

Plugin Jank

Memory Usage

UI Smoothness

Graphics Performance

Slow Scripts

Tests

Release Blocking Bugs

M9 bugs

Release Criteria bugs

Bugzilla query error

APZ Bugs

Accessibility

Add-ons

Service Workers

Navigation menu

Electrolysis/Multi Release Criteria: Difference between revisions

Revision as of 16:20, 30 March 2017

Rollout Criteria

Release Criteria

Stability

User Engagement

Jank

Page Load

Startup/Shutdown Time

Scrolling

Plugin Jank

Memory Usage

UI Smoothness

Graphics Performance

Slow Scripts

Tests

Release Blocking Bugs

M9 bugs

Release Criteria bugs

Bugzilla query error

APZ Bugs

Accessibility

Add-ons

Service Workers

Navigation menu

Search