Revision as of 14:12, 19 May 2017

Page to collect and track Electrolysis Multi Process release criteria.

Rollout Criteria

We plan to ship e10s-multi in Firefox 54 / 55 using a staged rollout. See elan's e10s-multi release plan.

Report / Dashboards

E10s Cohort Count Graphs - number of clients running with various cohort settings.
Perfherder e10s comparison dashboard
Perfherder Compare
Beta release calendar

Rollout Status

Nightly: 4 processes
Beta: experiments running, user default is 1 content process
Release: 1 content process

Notes

1) We can't rely on Talos numbers generated in automation for beta builds. The default there is one content process. We'll have to use try runs of the beta code base to generate comparisons.

2) 32-bit / 64-bit builds for Windows - which? both?

3) Probes we might want to add:

COMPOSITOR_ANIMATION_MAX_CONTIGUOUS_DROPS_APZ	For APZ/scrolling compositor animations, the maximum number of contiguous frame drops encountered
COMPOSITOR_ANIMATION_MAX_CONTIGUOUS_DROPS_CHROME	For compositor animations of chrome-process layers, the maximum number of contiguous frame drops encountered
COMPOSITOR_ANIMATION_MAX_CONTIGUOUS_DROPS_CONTENT	For compositor animations of content-process layers, the maximum number of contiguous frame drops encountered
TIME_TO_NON_BLANK_PAINT_MS	The time between navigation start and the first non-blank paint of a foreground root content document, in milliseconds. This only records documents that were in an active docshell throughout the whole time between navigation start and non-blank paint. The non-blank paint timestamp is taken during display list building and does not include rasterization or compositing of that paint.

AWSY Notes

1) Maximum number of tabs set by automation is currently 30.

2) test source

Release Criteria

Stability

Reporting	Jim Mathies
Data Wrangler
Regression(s)	Blake Kaplan, Gabor Krizsanits

Metric	Description	Report Links	Analysis By	Criteria Met?	Criteria Met Signed-off By
Crash Rate	(chrome crashes + (content process(es) - content process(es) shutdown termination crash reports)) / 1000 use hours	beta 54 (redash)	jimm	SO FAR - multi1 / multi4 rates track	TBD

Jank

Reporting	Jim Mathies
Data Wrangler	Ben Miroglio
Regression(s)	Blake Kaplan, Gabor Krizsanits
Report Link	e10s-multi performance in Telemetry

Metric	Description	Analysis By	Criteria Met?	Criteria Met Signed-off By
GC_MAX_PAUSE_MS	Longest GC slice in a single GC cycle in milliseconds, broken down by parent and child process(es).	tracy	Win: YES OSX: limited data Linux: limited data	TBD
CYCLE_COLLECTOR_MAX_PAUSE	Longest pause for an individual slice of one cycle collection, including preparation in milliseconds, broken down by parent and child process(es).	tracy	Win: YES OSX: limited data Linux: limited data	TBD
INPUT_EVENT_RESPONSE_MS	Time in milliseconds for input event lifetime (created -> fully handled) including traversal to a child process and back.	tracy	Win: YES OSX: limited data Linux: limited data	TBD
FX_TAB_SWITCH_TOTAL_MS	Time in milliseconds a tab switch takes, including first paint of the tab.	tracy	Win: YES OSX: limited data Linux: limited data	TBD

Metric	Description	Report Links	Analysis By	Results for Child Process Cohorts		Criteria Met?	Criteria Met Signed-off By
Metric	Description	Report Links	Analysis By	1	4	Criteria Met?	Criteria Met Signed-off By
tp5o_responsiveness graph	TP5O Talos numbers, comparing current single process beta builds with try runs using the beta code base and four content processes. Targets: 64-bit opt/pgo builds	Perherder Comparison run 5/9-5/10	tracy	Win7 32bit: 6.93 Win8 64bit: 5.13 OSX: n/a Linux: 4.58	Win7 32bit: 6.27 Win8 64bit: 5.67 OSX: n/a Linux: 4.52	YES - pending investigation in bug 1362920	TBD

Page Load

Reporting	Jim Mathies
Data Wrangler	Ben Miroglio
Regression(s)	Blake Kaplan, Gabor Krizsanits
Report Link	e10s-multi performance in Telemetry

Metric	Description	Analysis By	Criteria Met?	Criteria Met Signed-off By
FX_PAGE_LOAD_MS	Time taken to load a page (ms). This includes all static contents, no dynamic content. Loading of about: pages is not counted.	tracy	Win: YES OSX: limited data Linux: limited data	TBD

UI Smoothness

Reporting	Jim Mathies
Data Wrangler	Ben Miroglio
Regression(s)	Blake Kaplan, Gabor Krizsanits

Metric	Description	Report Links	Analysis By	Results for Child Process Cohorts		Criteria Met?	Criteria Met Signed-off By
Metric	Description	Report Links	Analysis By	1	4	Criteria Met?	Criteria Met Signed-off By
TART graph	Tab animation regression test (Talos).	Perfherder Comparison run 5/9-5/10	tracy	Win7 32bit: 7.34 Win8 64bit: 6.26 OSX: 11.22 Linux: 6.29	Win7 32bit: 7.35 Win8 64bit: 6.25 OSX: 11.17 Linux: 6.27	YES	TBD
TPAINT graph	First paint for a new window (Talos).	Perfherder Comparison run 5/9-5/10	tracy	Win7 32bit: 294.32 Win8 64bit: 288.36 OSX: 325.042 Linux: 276.33	Win7 32bit: 298.68 Win8 64bit: 287.07 OSX: 320.40 Linux: 273.71	YES	TBD
TRESIZE graph	Window resize responsiveness (Talos).	Perfherder Comparison run 5/9-5/10	tracy	Win7 32bit: 11.91 Win8 64bit: 10.62 OSX: 28.69 Linux: 23.01	Win7 32bit: 11.93 Win8 64bit: 10.63 OSX: 28.82 Linux: 23.59	YES	TBD

Memory Usage

Reporting	Jim Mathies
Data Wrangler	Ben Miroglio
Regression(s)	Blake Kaplan, Gabor Krizsanits
Report Link	e10s-multi performance in Telemetry

Metric	Description	Analysis By	Criteria Met?	Criteria Met Signed-off By
MEMORY_TOTAL	Total memory across all processes (KB).	tracy	Win: YES OSX: limited data Linux: limited data	TBD
MEMORY_VSIZE_MAX_CONTIGUOUS	Maximum-sized block of contiguous virtual memory (KB).	tracy	Win: YES OSX: limited data Linux: limited data	TBD
MEMORY_DISTRIBUTION_AMONG_CONTENT	Absolute difference of each content process' USS and the mean of USS, normalized by the mean, as a percentage. Measure memory consumption distribution between content processes. We're looking for ugly numbers here while running with 4+ content processes. Does not apply to single content process configs.	tracy	Win: no data OSX: no data Linux: no data	TBD

Are We Slim Yet (AWSY)

RSS chrome process + USS content processes (Talos)

5/10/2017	Perfherder

(insert passing criteria declaration)

Windows 7 Opt 32-bit
Metric	Regression	Criteria Met?	Sign-off
Resident Memory Fresh start	-0.47%
Resident Memory Fresh start [+30s]	-0.67%
Resident Memory After tabs open	29.80%
Resident Memory After tabs open [+30s]	32.87%
Resident Memory After tabs open [+30s, forced GC]	26.59%
Resident Memory Tabs closed	-29.74%
Resident Memory Tabs closed [+30s]	-18.40%
Resident Memory Tabs closed [+30s, forced GC]	-12.14%

Linux Opt 32-bit
Metric	Regression	Criteria Met?	Sign-off
Resident Memory Fresh start	0.11%
Resident Memory Fresh start [+30s]	0.27%
Resident Memory After tabs open	19.87%
Resident Memory After tabs open [+30s]	18.98%
Resident Memory After tabs open [+30s, forced GC]	13.15%
Resident Memory Tabs closed	-20.46%
Resident Memory Tabs closed [+30s]	-10.59%
Resident Memory Tabs closed [+30s, forced GC]	-7.76%

Linux Opt 64-bit
Metric	Regression	Criteria Met?	Sign-off
Resident Memory Fresh start	-1.76%
Resident Memory Fresh start [+30s]	-1.80%
Resident Memory After tabs open	22.51%
Resident Memory After tabs open [+30s]	21.92%
Resident Memory After tabs open [+30s, forced GC]	15.06%
Resident Memory Tabs closed	-21.38%
Resident Memory Tabs closed [+30s]	-10.48%
Resident Memory Tabs closed [+30s, forced GC]	-7.23%

erahm's blog post: Memory Usage of Firefox with e10s Enabled
Currently using AWSY 'Resident Memory summary opt' values from Perfherder, which should be (RSS + (USS for children)).

Tab Switching

Reporting	Jim Mathies
Data Wrangler	Ben Miroglio
Regression(s)	Blake Kaplan, Gabor Krizsanits, Mike Conley

Metric	Description	Report Links	Analysis By	Results for Child Process Cohorts		Criteria Met?	Criteria Met Signed-off By
Metric	Description	Report Links	Analysis By	1	4	Criteria Met?	Criteria Met Signed-off By
FX_TAB_SWITCH_UPDATE_MS	Time in ms spent updating UI in response to a tab switch.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
FX_TAB_SWITCH_TOTAL_E10S_MS	Time in ms between tab selection and tab content paint.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
FX_TAB_SWITCH_SPINNER_VISIBLE_MS	Time in ms spent updating UI in response to a tab switch.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
FX_TAB_SWITCH_SPINNER_VISIBLE_LONG_MS	If the spinner interstitial displays during tab switching, records the time in ms the graphic is visible. This probe is similar to FX_TAB_SWITCH_SPINNER_VISIBLE_MS, but is for truly degenerate cases.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
FX_TAB_REMOTE_NAVIGATION_DELAY_MS	Time taken in milliseconds between the browser sending a naviagion event to content and content receiving it. This message can be either SessionStore:restoreTabContent or WebNavigation:LoadURI and these names are used as keys for this histogram. This is e10s only and recorded in the content process. Measures delays associated with child process creation.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
TPS Test graph	Tab switch timing (Talos)	Perfherder Comparison run 5/9-5/10	tracy	Win7-32bit: 26.89 Win8-64bit: 24.54 OSX: 28.20 Linux: 29.37	Win7-32bit: 27.24 Win8-64bit: 24.52 OSX: 31.11 Linux: 29.47	NO - pending investigation in bug 1362920 (even though that bug is listed as Win7, I think it may affect all Win and Mac)	TBD

Slow Scripts

Reporting	Jim Mathies
Data Wrangler	Ben Miroglio
Regression(s)	Blake Kaplan, Gabor Krizsanits

Metric	Description	Report Links	Analysis By	Results for Child Process Cohorts		Criteria Met?	Criteria Met Signed-off By
Metric	Description	Report Links	Analysis By	1	4	Criteria Met?	Criteria Met Signed-off By
SLOW_SCRIPT_NOTICE_COUNT	Total count of slow script notices displayed to user.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
SLOW_SCRIPT_PAGE_COUNT	The number of pages that trigger slow script notices.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD
SLOW_SCRIPT_NOTIFY_DELAY	The difference between the js slow script timeout for content set in prefs and the actual time we waited before displaying the notification in milliseconds.	TBD	TBD	Win: ? OSX: ? Linux: ?	Win: ? OSX: ? Linux: ?	TBD	TBD

Misc. broken?

Reporting	Tracy Walker
Data Wrangler	Tracy Walker

Metric	Description	Report Links	Analysis By	Results for Child Process Cohorts		Criteria Met?	Criteria Met Signed-off By
Metric	Description	Report Links	Analysis By	1	4	Criteria Met?	Criteria Met Signed-off By
tp5o XRes graph	These results are extremely bifurcated	Perfherder comparison	tracy	Linux: ~3830 : ~1,574,000	Linux: ~3830 : ~1,574,000	test is bimodal, results are as expected	not called out as a release criteria
tp5n main_normal_fileio graph	This test is producing a wide range of results	Perfherder comparison	tracy	Win7-32bit: 2,796,507.80	Win7-32bit: 3,085,232.18	possibly also related to bug 1362920	not called out as a release criteria
tp5n main_normal_netio graph	This test seems to produce mostly "0" value results	Perfherder comparison	tracy	Win7-32bit: 0 : 3932.80	Win7-32bit: 0 : 385,376.91	per jmaher, 0 values are mostly expected	not called out as a release criteria
tp5n main_startup_netio graph	This test seems to produce mostly "0" value results	Perfherder comparison	tracy	Win7-32bit: 0 : 40.40	Win7-32bit: 0 : 115.27	per jmaher, 0 values are mostly expected	not called out as a release criteria

Tests

Reporting	Blake Kaplan / Gabor Krizsanits
Accountable	Everyone
Sign Off	Module Owners / Jeff Griffiths

All unit tests disabled for e10s-multi must be triaged and:

re-enabled (and fixed, if necessary)
or annotated that the test is irrelevant for e10s-multi
or deleted

We want each test directory to be signed-off by the feature area's owner.

e10s-multi test meta bug 1315042

QA Test Plan

Responsible	Erin Lancaster
Accountable	SoftVision
Sign Off	SoftVision / Jeff Griffiths

Task	Owner	Criteria Met?	Criteria Met Signed-off By
Write test plan	Tracy	Done	Tracy
Execute test plan	Tracy, Ben M, SV	In progress	TBD

@@ Line 296: / Line 296: @@
 ! Regression(s)
 | Blake Kaplan, Gabor Krizsanits
+|-
+! Report Link
+|[https://metrics.mozilla.com/protected/bmiroglio/multi/e10sMulti_experiment.html e10s-multi performance in Telemetry]
 |}
 {| class="wikitable"
 |-
-!rowspan="2"|Metric
+!Metric
-!rowspan="2"|Description
+!Description
-!rowspan="2"|Report Links
+!Analysis By
-!rowspan="2"|Analysis By
+!Criteria Met?
-!rowspan="1" colspan="2"|Results for Child Process Cohorts
+!Criteria Met Signed-off By
-!rowspan="2"|Criteria Met?
-!rowspan="2"|Criteria Met Signed-off By
-|-
-! 1
-! 4
 |-
 | MEMORY_TOTAL
 | style="background-color: White;width:250px;"|  Total memory across all processes (KB).
-| style="background-color: White;" | TBD
+| style="background-color: White;" | tracy
-| style="background-color: White;" | TBD
 | style="background-color: White;" |
-Win: ?<br/>
+Win: YES<br/>
-OSX: ?<br/>
+OSX: limited data<br/>
-Linux: ?
+Linux: limited data
-| style="background-color: White;" |
-Win: ?<br/>
-OSX: ?<br/>
-Linux: ?
-| style="background-color: White;" | TBD
 | style="background-color: White;" | TBD
 |-
 | MEMORY_VSIZE_MAX_CONTIGUOUS
 | style="background-color: White;width:250px;"| Maximum-sized block of contiguous virtual memory (KB).
-| style="background-color: White;" | TBD
+| style="background-color: White;" | tracy
-| style="background-color: White;" | TBD
-| style="background-color: White;" |
-Win: ?<br/>
-OSX: ?<br/>
-Linux: ?
 | style="background-color: White;" |
-Win: ?<br/>
+Win: YES<br/>
-OSX: ?<br/>
+OSX: limited data<br/>
-Linux: ?
+Linux: limited data
-| style="background-color: White;" | TBD
 | style="background-color: White;" | TBD
 |-
 | MEMORY_DISTRIBUTION_AMONG_CONTENT
 | style="background-color: White;width:250px;"|  Absolute difference of each content process' USS and the mean of USS, normalized by the mean, as a percentage. Measure memory consumption distribution between content processes. We're looking for ugly numbers here while running with 4+ content processes. Does not apply to single content process configs.
-| style="background-color: White;" | TBD
+| style="background-color: White;" | tracy
-| style="background-color: White;" | TBD
+| style="background-color: #FFFF99;" |
-| style="background-color: White;" | N/A
+Win: no data<br/>
-| style="background-color: White;" |
+OSX: no data<br/>
-Win: ?<br/>
+Linux: no data
-OSX: ?<br/>
-Linux: ?
-| style="background-color: White;" | TBD
 | style="background-color: White;" | TBD
 |}

Electrolysis/Multi Release Criteria: Difference between revisions

Revision as of 14:12, 19 May 2017

Contents

Rollout Criteria

Report / Dashboards

Rollout Status

Notes

AWSY Notes

Release Criteria

Stability

Jank

Page Load

UI Smoothness

Memory Usage

Are We Slim Yet (AWSY)

Tab Switching

Slow Scripts

Misc. broken?

Tests

QA Test Plan

Navigation menu

Electrolysis/Multi Release Criteria: Difference between revisions

Revision as of 14:12, 19 May 2017

Rollout Criteria

Report / Dashboards

Rollout Status

Notes

AWSY Notes

Release Criteria

Stability

Jank

Page Load

UI Smoothness

Memory Usage

Are We Slim Yet (AWSY)

Tab Switching

Slow Scripts

Misc. broken?

Tests

QA Test Plan

Navigation menu

Search