Electrolysis/Release Criteria

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Release Criteria

User Engagement

Responsible	glind? dzeber? hulmer?
Accountable	TBD
Supporting	measurement/data teams
Consulted	jgriffiths, release management
Informed	cpeterson, elan

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
User Retention	Experiment user drop-off rate (?)	bug 1249665	glind?	TBD	TBD	NO ONE
User Retention	e10s/non-10s engagement ratio	bug 1251259	dzeber, hulmer?	TBD	TBD	NO ONE

Stability

Responsible	chutten
Accountable	bsmedberg
Supporting	Kairo, measurement/data teams
Consulted	jgriffiths, release management
Informed	cpeterson, elan

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
chrome+content process crash rate	chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength	IPython notebook	rvitillo	TBD	NO	NO ONE
plugin process crash rate	SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength	IPython notebook	rvitillo	TBD	NO	NO ONE

• More information on Stability analysis
• Stability analysis meta bug 1251376

Jank

Responsible	chutten
Accountable	bsmedberg
Supporting	data team, RyanVM, rvitillo, avih, Softvision
Consulted	jgriffiths, billm
Informed	cpeterson, elan, release management

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
GC pauses	GC_MAX_PAUSE_MS	beta 45 experiment	TBD	TBD	TBD	TBD
Cycle Collector pauses	CYCLE_COLLECTOR_MAX_PAUSE	beta 45 experiment	TBD	TBD	TBD	TBD
Event loop lag	INPUT_EVENT_RESPONSE_MS	TBD	TBD	TBD	TBD	TBD
Browser responsiveness during page load	Talos tp5o_responsiveness	Perfherder	TBD	TBD	About -87% faster!	TBD
Frame delay	FX_REFRESH_DRIVER_{CHROME,CONTENT}_FRAME_DELAY_MS	beta 45 experiment	TBD	TBD	TBD	TBD
BHR/chrome hangs	TBD	TBD	TBD	TBD	TBD	TBD

• More information on Jank analysis
• Jank analysis meta bug 1251377

Scrolling

Responsible	kats
Accountable	RyanVM
Supporting	avih, jmaher, graphics team (mstange)
Consulted	jgriffiths, jmaher
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS	kats?	bug 1228147?	kats?	TBD	TBD	TBD
tp5o_scroll	avih? mstange?	Perfherder (bug 1179735)	kats?	TBD	Up to 105% worse!	TBD
tp5r	kats?	TBD	kats?	TBD	TBD	TBD
tscrollx	avih? mstange?	Perfherder (bug 1174772? bug 1249976?)	kats?	TBD	Up to 32% worse!	TBD

• More information on Scrolling analysis
• Scrolling analysis meta bug 1251388

Slow Scripts

Responsible	azhang
Accountable	bsmedberg
Supporting
Consulted	jgriffiths, billm
Informed	cpeterson, elan, release management

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
Slow script warnings	SLOW_SCRIPT_PAGE_COUNT (bug 1251667)	TBD	TBD	TBD	TBD	TBD

• More information on slow script analysis
• Slow script analysis meta bug 1251545

UI Smoothness

Responsible	mconley
Accountable	RyanVM
Supporting	jmaher, avih, gabor
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test)	mconley	Perfherder	TBD	TBD	Up to 17% worse	TBD
tpaint ±5% (first paint for a new window)	jimm	Perfherder	TBD	TBD	Up to 46% worse	TBD
tresize ±5% (window resize responsiveness)	jimm	Perfherder	TBD	TBD	Mixed results: up to 5.44% worse	TBD
tps ±5% (tab switch speed)	mconley	Perfherder	TBD	TBD	Up to 107% worse!	TBD
FX_TAB_SWITCH_TOTAL_MS	mconley?	TBD	TBD	TBD	TBD	TBD
CART ±5% (Customize Animation Regression Test)	mconley	Perfherder	TBD	TBD	Up to 14% worse	TBD

• More information on UI Smoothness analysis
• UI smoothness analysis meta bug 1251547

Page Load

Responsible	jimm
Accountable	RyanVM
Supporting	jmaher
Consulted	jgriffiths, rvitillo
Informed	cpeterson, elan, release management

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
Talos tests for page load time	tp5o ±5%	Perfherder	no change	TBD	TBD	TBD
Page load time telemetry	FX_PAGE_LOAD_MS ±5%	beta 45 experiment	rvitillo	TBD	TBD	NO ONE

• More information on Page Load analysis
• Scrolling analysis meta bug 1251549

Plugin Jank

Responsible	chutten
Accountable	bsmedberg
Supporting	aklotz
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
Plugin initialization jank time	BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5%	TBD	TBD	TBD	TBD	TBD
Plugin destruction jank time	BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5%	TBD	TBD	TBD	TBD	TBD

• More information on Plugin Jank analysis
• Scrolling analysis meta bug 1251551

Graphics Performance

Responsible	Milan
Accountable	RyanVM
Supporting	gw280
Consulted	jgriffiths, jmaher, graphics team
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
glterrain ±5%	jgilbert	Perfherder	TBD	TBD	Mixed results: from 4% worse to 63% better	TBD
tcanvasmark ±5%	TBD	Perfherder	TBD	TBD	From 2% worse to 3% better	TBD
tsvgx ±5%	mstange?	Perfherder	TBD	TBD	Up to 51% better	TBD
tsvg_opacity ±5%	avih says no real owner. mattwoodrow? seth?	Perfherder	TBD	TBD	Up to -79% better	TBD

• More information on Graphics Performance analysis
• Scrolling analysis meta bug 1251555

Startup/Shutdown Time

Responsible	rvitillo?
Accountable
Supporting
Consulted	jgriffiths, jmaher
Informed	cpeterson, elan, release management

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
Talos tests	ts_paint ±10%	Perfherder	TBD	TBD	Up to 55% better	TBD
Time to first paint	SIMPLE_MEASURES_FIRSTPAINT ±10%	beta 45 experiment	bad probability #'s	TBD	TBD	TBD
Session restore time	SIMPLE_MEASURES_SESSIONRESTORED ±10%	beta 45 experiment	no data?	TBD	TBD	TBD
Shutdown time	SIMPLE_MEASURES_SHUTDOWNDURATION ±25%	beta 45 experiment	no data?	TBD	TBD	TBD

• Startup times may regress up to 10% and shutdown time up to 25% compared to non-e10s.

Memory Usage

Responsible	chutten
Accountable	bsmedberg
Supporting
Consulted	jgriffiths
Informed	cpeterson, elan, release management

Metric	Metric Owner	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
tp5o Main_RSS		Perfherder	TBD	TBD	4% to 35% better	TBD
tp5o Private Bytes		Perfherder	TBD	TBD	20% to 50% worse	TBD

Release Criteria	Metric	Analysis	Analysis By	Analysis Signed-off By	Criteria Met?	Criteria Met Signed-off By
Swap Rate (TBD)	bug 1198211	TBD	chutten	TBD	TBD	TBD
Overall memory usage (browser + content)	bug 1198209	TBD	chutten	TBD	TBD	TBD

Release Blocking Bugs

All M8 and M9 bugs must be fixed before we release.

Responsible	jimm?
Accountable	blassey?
Supporting	release management
Consulted	jgriffiths
Informed	cpeterson, elan

M8/M9 bugs:

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Release Criteria meta bugs' blockers:

Full Query

ID	Summary	Whiteboard	Keywords	Assigned to
1250350	Talos svg results for e10s seem invalid
1260769	[e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s		regression

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);

• Bugzilla query for e10s-release-blockers

Tests

All tests disabled for e10s must be triaged and:

• reenabled (and fixed, if necessary)
• or annotated that the test is irrelevant for e10s?
• or deleted?

Responsible	Blake, Felipe
Accountable	blassey?
Supporting
Consulted	jgriffiths
Informed	cpeterson, elan, release management

• List of disabled e10s tests
• e10s test meta bug e10s-tests
• Perfherder e10s dashboard

Detail Pages

Release Criteria	Responsible	Status	Tracking Bug
P1: Stability	poiru	[AT RISK]	bug 1251376
P1: Jank	chutten	TBD	bug 1251377
P1: Scrolling	kats	[AT RISK]	bug 1251388
P1: Slow Script	azhang	TBD	bug 1251545
P1: no blocker bugs	blassey	[ON TRACK]	m9 bug query
P1: automated tests running and passing	mrbkap?	[AT RISK]	bug e10s-tests
P2: UI Smoothness	mconley	[AT RISK]	bug 1251547
P2: Page Load Time	jimm	[ON TRACK]	bug 1251549
P2: Plugin Jank	chutten	TBD	bug 1251551
P2: Graphics	Milan	[ON TRACK]	bug 1251555