Revision as of 16:22, 14 March 2016

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

APZ Regressions

Talos numbers listed on this page are pulled from the Perfherder e10s dashboard. These results include the affects of enabling APZ on mozilla central. The component of individual regressions and improvements associated with APZ is listed below and is based on a Perfherder comparison between a try run with APZ disabled and mozilla-central. Test that showed no change are not listed.

Talos Metric	Change	Related bugs
CART	3%-5%	bug 1253380
DAMP	5%-7%
TART	10%-13%	bug 1253412
tp5o %proc time	5%-7% (windows)
tp5o main_RSS	3%-15%	bug 1254273 (OS X)
tp5o	mixed	bug 1254273 (OS X)
tp5o_scroll	16%-40%	bug 1179735, bug 1253860
tpaint	2% (Win7)
tps	9%-11%	bug 1216924
tresize	4% (WinXP)
ts_paint	3% (WinXP)
tscrollx	18%-42%
tsvgr_opacity	3% (Linux)
tsvgx	3%-4%

Release Criteria

User Engagement

Responsible	rvitillo
Accountable	TBD
Supporting	measurement/data teams
Consulted	jgriffiths, release management
Informed	cpeterson, elan

Description	Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
Firefox Usage	TBD (bug 1249665)	TBD	glind?	TBD	TBD	TBD	TODO
User engagement (DAU/MAU)	e10s/non-10s engagement ratio	bug 1251259	rvitillo	TBD	TBD	TBD	TODO

Stability

Responsible	bsmedberg
Accountable	bsmedberg
Supporting	Kairo, rvitillo
Consulted	jgriffiths, release management
Informed	cpeterson, elan

Description	Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
chrome+content process crash rate	chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength	IPython notebook	rvitillo	84,390 (Beta 45)	109,348 (Beta 45)	NO	TODO
plugin process crash rate	SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength	IPython notebook	rvitillo	41,897 (Beta 45)	55,809 (Beta 45)	NO	TODO

More information on Stability analysis
Stability analysis meta bug 1251376

Jank

Responsible	chutten
Accountable	bsmedberg
Supporting	data team, RyanVM, rvitillo, avih, Softvision
Consulted	jgriffiths, billm
Informed	cpeterson, elan, release management

Description	Metric	Analysis	Analysis By	Criteria Met?	Criteria Met Signed-off By
GC pauses	GC_MAX_PAUSE_MS (parent, children)	beta 45 experiment	rvitillo	e10s is slightly better	TODO
Cycle Collector pauses	CYCLE_COLLECTOR_MAX_PAUSE (parent, children)	beta 45 experiment	rvitillo	e10s is slightly better	TODO
Event loop lag	INPUT_EVENT_RESPONSE_MS	beta 45 experiment	chutten	about the same	TODO
Browser responsiveness during page load	Talos tp5o_responsiveness	Perfherder		80%-90% faster!	TODO

More information on Jank analysis
Jank analysis meta bug 1251377

Scrolling

Responsible	kats
Accountable	RyanVM
Supporting	avih, jmaher, graphics team (mstange)
Consulted	jgriffiths, jmaher
Informed	cpeterson, elan, release management

Metric	Analysis	Analysis By	Criteria Met?	Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS	TBD	avih	regressed	TODO
tp5o_scroll	Perfherder	kats	results are mixed, but Linux is 46% better (bug 1253860 fixes this)	TODO
tscrollx	Perfherder	kats	Up to 60% better on non-XP, but 10% worse on XP. (bug 1253860 fixes this) e10s is better on other platforms.	TODO

More information on Scrolling analysis
Scrolling analysis meta bug 1251388

Slow Scripts

Responsible	azhang
Accountable	bsmedberg
Supporting
Consulted	jgriffiths, billm
Informed	cpeterson, elan, release management

Description	Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
Slow script warnings	SLOW_SCRIPT_PAGE_COUNT	TBD (waiting for Beta 46 experiment)	azhang	TBD (waiting for Beta 46 experiment)	TBD (waiting for Beta 46 experiment)	TBD	TODO

More information on slow script analysis
Slow script analysis meta bug 1251545

UI Smoothness

Responsible	mconley
Accountable	RyanVM
Supporting	jmaher, avih, gabor
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test)	mconley	Perfherder	TBD	3.96 (Linux)	4.55 (Linux)	7%–15% (bug 1251699)	TODO
tpaint ±5% (first paint for a new window)	mconley	Perfherder	TBD	197 (Linux)	286 (Linux	29%–45% (bug 1174770)	TODO
tresize ±5% (window resize responsiveness)	jimm?	Perfherder	TBD	9.92 (XP)	11.21 (XP)	13% (XP) (bug 1252947)	TODO
tps ±5% (tab switch speed)	mconley	Perfherder	TBD	43.27 (Linux)	72.27 (Linux)	45%-70% (bug 1186585)	TODO
FX_TAB_SWITCH_TOTAL_MS	mconley?	46 experiment	TBD	TBD	TBD	TBD	TODO
CART ±5% (Customize Animation Regression Test)	mconley	Perfherder	TBD	20.76 (Linux)	23.53 (Linux)	5%–13% (bug 1252768)	TODO

More information on UI Smoothness analysis
UI smoothness analysis meta bug 1251547

Page Load

Responsible	jimm
Accountable	RyanVM
Supporting	jmaher
Consulted	jgriffiths, rvitillo
Informed	cpeterson, elan, release management

Description	Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
Talos page load time	tp5o ±5%	Perfherder	TBD	TBD	TBD	no change	TODO
Page load time telemetry	FX_PAGE_LOAD_MS ±5%	beta 45 experiment	rvitillo	TBD	TBD	improvement	TODO

More information on Page Load analysis
Scrolling analysis meta bug 1251549

Plugin Jank

Responsible	chutten
Accountable	bsmedberg
Supporting	aklotz
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Description	Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
Plugin initialization jank time	BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%	TBD	TBD	TBD	TBD	TBD	TODO
Plugin destruction jank time	BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%	TBD	TBD	TBD	TBD	TBD	TODO

More information on Plugin Jank analysis
Plugin jank analysis meta bug 1251551

Graphics Performance

Responsible	Milan
Accountable	RyanVM
Supporting	gw280
Consulted	jgriffiths, jmaher, graphics team
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
glterrain ±5%	jgilbert	Perfherder	18	19	Mixed results from 5% worse to 64% better	TODO
tcanvasmark ±5%		Perfherder	6444	6301	Mixed results from 2% worse to 3% better	TODO
tsvgx ±5%	mstange	Perfherder	122	92	25%–51% better	TODO
tsvg_opacity ±5%	avih says no real owner. mattwoodrow? seth?	Perfherder	536	865	about 65% slower on 32-bit Windows (bug 1255936)	TODO

More information on Graphics Performance analysis
Scrolling analysis meta bug 1251555

Startup/Shutdown Time

Responsible	rvitillo
Accountable
Supporting
Consulted	jgriffiths, jmaher
Informed	cpeterson, elan, release management

Release Criteria	Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
Talos tests	ts_paint ±10%	Perfherder		863 (Win7)	433 (Win7)	43–55% better	TODO
Time to first paint	SIMPLE_MEASURES_FIRSTPAINT	beta 45 experiment	rvitillo	4636 ms (median)	4620 ms (median)	no change	TODO
Shutdown time	SIMPLE_MEASURES_SHUTDOWNDURATION	beta 45 experiment	rvitillo	1216 ms (median)	1229 ms (median)	no change	TODO

Startup times may regress up to 10% and shutdown time up to 25% compared to non-e10s.
In telemetry performance reports, search for payload/simpleMeasurements/firstPaint, payload/simpleMeasurements/sessionRestored, and payload/simpleMeasurements/shutdownDuration.

Memory Usage

Responsible
Accountable
Supporting
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Metric	Analysis	Analysis By	non-e10s value	e10s value	Criteria Met?	Criteria Met Signed-off By
tp5o Main_RSS	Perfherder		384117275.55 (OS X)	365409824.16 (OS X)	5%–35% better	TODO
tp5o Private Bytes	Perfherder	TBD	636896373.57 (Linux)	932390353.63 (Linux)	20% to 50% worse (bug 1253984)	TODO
MEMORY_TOTAL ±X% (browser + content process)	TBD	chutten	TBD	TBD	TBD	TODO

We expect e10s MEMORY_TOTAL to regress compared to non-e10s because we are using more processes.

Release Blocking Bugs

All M8 and M9 bugs must be fixed before we release.

Responsible	jimm
Accountable	blassey
Supporting	release management
Consulted	jgriffiths
Informed	cpeterson, elan

Bugzilla query for e10s blocking bugs: is.gd/e10s_blockers

M8/M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Release Criteria meta bugs' blockers

Full Query

ID	Summary	Whiteboard	Keywords	Assigned to
1250350	Talos svg results for e10s seem invalid
1260769	[e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s		regression

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);

APZ Bugs

APZ does not block e10s release. This bug list is purely informational.

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Tests

All tests disabled for e10s must be triaged and:

reenabled (and fixed, if necessary)
or annotated that the test is irrelevant for e10s?
or deleted?

Responsible	Blake, Felipe
Accountable	blassey?
Supporting
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Detail Pages

Release Criteria	Responsible	Status	Tracking Bug
P1: Stability	poiru	[AT RISK]	bug 1251376
P1: Jank	chutten	TBD	bug 1251377
P1: Scrolling	kats	[AT RISK]	bug 1251388
P1: Slow Script	azhang	TBD	bug 1251545
P1: no blocker bugs	blassey	[ON TRACK]	m9 bug query
P1: automated tests running and passing	mrbkap?	[AT RISK]	bug e10s-tests
P2: UI Smoothness	mconley	[AT RISK]	bug 1251547
P2: Page Load Time	jimm	[ON TRACK]	bug 1251549
P2: Plugin Jank	chutten	TBD	bug 1251551
P2: Graphics	Milan	[ON TRACK]	bug 1251555

@@ Line 647: / Line 647: @@
 {| class="wikitable"
 ! Responsible
-| chutten
+|
 |-
 ! Accountable
-| bsmedberg
+|
 |-
 ! Supporting

Electrolysis/Release Criteria: Difference between revisions

Revision as of 16:22, 14 March 2016

Contents

APZ Regressions

Release Criteria

User Engagement

Stability

Jank

Scrolling

Slow Scripts

UI Smoothness

Page Load

Plugin Jank

Graphics Performance

Startup/Shutdown Time

Memory Usage

Release Blocking Bugs

M8/M9 bugs

Release Criteria meta bugs' blockers

APZ Bugs

Tests

Detail Pages

Navigation menu

Electrolysis/Release Criteria: Difference between revisions

Revision as of 16:22, 14 March 2016

APZ Regressions

Release Criteria

User Engagement

Stability

Jank

Scrolling

Slow Scripts

UI Smoothness

Page Load

Plugin Jank

Graphics Performance

Startup/Shutdown Time

Memory Usage

Release Blocking Bugs

M8/M9 bugs

Release Criteria meta bugs' blockers

APZ Bugs

Tests

Detail Pages

Navigation menu

Search