Electrolysis/Release Criteria: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Scrolling: Update tscrollx and tp5o_scroll results)
(Change "Criteria Met Signed-Off By" color from red to blue (neutral-but-incomplete status))
Line 107: Line 107:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| User engagement (DAU/MAU)
| User engagement (DAU/MAU)
Line 116: Line 116:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 156: Line 156:
| style="background-color: Tomato;" | 109,348 (Beta 45)
| style="background-color: Tomato;" | 109,348 (Beta 45)
| style="background-color: Tomato;" | NO
| style="background-color: Tomato;" | NO
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| plugin process crash rate
| plugin process crash rate
Line 165: Line 165:
| style="background-color: Tomato;" | 55,809 (Beta 45)
| style="background-color: Tomato;" | 55,809 (Beta 45)
| style="background-color: Tomato;" | NO
| style="background-color: Tomato;" | NO
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 204: Line 204:
| rvitillo
| rvitillo
| style="background-color: PaleGreen;" | e10s is slightly better
| style="background-color: PaleGreen;" | e10s is slightly better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Cycle Collector pauses
| Cycle Collector pauses
Line 211: Line 211:
| rvitillo
| rvitillo
| style="background-color: PaleGreen;" | e10s is slightly better
| style="background-color: PaleGreen;" | e10s is slightly better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Event loop lag
| Event loop lag
Line 218: Line 218:
| chutten
| chutten
| style="background-color: PaleGreen;" | about the same
| style="background-color: PaleGreen;" | about the same
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Browser responsiveness during page load
| Browser responsiveness during page load
Line 225: Line 225:
|
|
| style="background-color: PaleGreen;" | 80%-90% faster!
| style="background-color: PaleGreen;" | 80%-90% faster!
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
|}
|}
Line 263: Line 263:
| avih
| avih
| style="background-color: Tomato;" | regressed
| style="background-color: Tomato;" | regressed
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tp5o_scroll
| tp5o_scroll
Line 269: Line 269:
| kats
| kats
| style="background-color: Tomato;" | results are mixed, but Linux is 46% better ({{bug|1253860}} fixes this)
| style="background-color: Tomato;" | results are mixed, but Linux is 46% better ({{bug|1253860}} fixes this)
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tscrollx
| tscrollx
Line 276: Line 276:
| style="background-color: Tomato;" | Up to 60% better on non-XP, but 10% worse on XP. ({{bug|1253860}} fixes this)
| style="background-color: Tomato;" | Up to 60% better on non-XP, but 10% worse on XP. ({{bug|1253860}} fixes this)
e10s is better on other platforms.
e10s is better on other platforms.
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 319: Line 319:
| style="background-color: LightYellow;" | TBD (waiting for Beta 46 experiment)
| style="background-color: LightYellow;" | TBD (waiting for Beta 46 experiment)
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 362: Line 362:
| style="background-color: Tomato;" | 4.55 (Linux)
| style="background-color: Tomato;" | 4.55 (Linux)
| style="background-color: Tomato;" | 7%–15% ({{bug|1251699}})
| style="background-color: Tomato;" | 7%–15% ({{bug|1251699}})
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tpaint ±5% (first paint for a new window)
| tpaint ±5% (first paint for a new window)
Line 371: Line 371:
| style="background-color: Tomato;" | 286 (Linux
| style="background-color: Tomato;" | 286 (Linux
| style="background-color: Tomato;" | 29%–45% ({{bug|1174770}})
| style="background-color: Tomato;" | 29%–45% ({{bug|1174770}})
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tresize ±5% (window resize responsiveness)
| tresize ±5% (window resize responsiveness)
Line 380: Line 380:
| style="background-color: Tomato;" | 11.21 (XP)
| style="background-color: Tomato;" | 11.21 (XP)
| style="background-color: Tomato;" | 13% (XP) ({{bug|1252947}})
| style="background-color: Tomato;" | 13% (XP) ({{bug|1252947}})
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tps ±5% (tab switch speed)
| tps ±5% (tab switch speed)
Line 389: Line 389:
| style="background-color: Tomato;" | 72.27 (Linux)
| style="background-color: Tomato;" | 72.27 (Linux)
| style="background-color: Tomato;" | 45%-70% ({{bug|1186585}})
| style="background-color: Tomato;" | 45%-70% ({{bug|1186585}})
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| FX_TAB_SWITCH_TOTAL_MS
| FX_TAB_SWITCH_TOTAL_MS
Line 398: Line 398:
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| CART ±5% (Customize Animation Regression Test)
| CART ±5% (Customize Animation Regression Test)
Line 407: Line 407:
| style="background-color: Tomato;" | 23.53 (Linux)
| style="background-color: Tomato;" | 23.53 (Linux)
| style="background-color: Tomato;" | 5%–13% ({{bug|1252768}})
| style="background-color: Tomato;" | 5%–13% ({{bug|1252768}})
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 450: Line 450:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | no change
| style="background-color: PaleGreen;" | no change
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Page load time telemetry
| Page load time telemetry
Line 459: Line 459:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | improvement
| style="background-color: PaleGreen;" | improvement
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 502: Line 502:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Plugin destruction jank time
| Plugin destruction jank time
Line 511: Line 511:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 554: Line 554:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | Mixed results: from 4% worse to 63% better
| style="background-color: PaleGreen;" | Mixed results: from 4% worse to 63% better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tcanvasmark ±5%
| tcanvasmark ±5%
Line 563: Line 563:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | From 2% worse to 3% better
| style="background-color: PaleGreen;" | From 2% worse to 3% better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tsvgx ±5%
| tsvgx ±5%
Line 572: Line 572:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | Up to 51% better
| style="background-color: PaleGreen;" | Up to 51% better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tsvg_opacity ±5%
| tsvg_opacity ±5%
Line 581: Line 581:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: PaleGreen;" | Up to -79% better
| style="background-color: PaleGreen;" | Up to -79% better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 624: Line 624:
| style="background-color: PaleGreen;" | 433 (Win7)
| style="background-color: PaleGreen;" | 433 (Win7)
| style="background-color: PaleGreen;" | 43–55% better
| style="background-color: PaleGreen;" | 43–55% better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Time to first paint
| Time to first paint
Line 633: Line 633:
| style="background-color: PaleGreen;" | 4620 ms (median)
| style="background-color: PaleGreen;" | 4620 ms (median)
| style="background-color: PaleGreen;" | no change
| style="background-color: PaleGreen;" | no change
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| Shutdown time
| Shutdown time
Line 642: Line 642:
| style="background-color: PaleGreen;" | 1229 ms (median)
| style="background-color: PaleGreen;" | 1229 ms (median)
| style="background-color: PaleGreen;" | no change
| style="background-color: PaleGreen;" | no change
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Line 683: Line 683:
| style="background-color: PaleGreen;" | 365409824.16 (OS X)
| style="background-color: PaleGreen;" | 365409824.16 (OS X)
| style="background-color: PaleGreen;" | 5%–35% better
| style="background-color: PaleGreen;" | 5%–35% better
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| tp5o Private Bytes
| tp5o Private Bytes
Line 691: Line 691:
| style="background-color: Tomato;" | 932390353.63 (Linux)
| style="background-color: Tomato;" | 932390353.63 (Linux)
| style="background-color: Tomato;" | 20% to 50% worse ({{bug|1253984}})
| style="background-color: Tomato;" | 20% to 50% worse ({{bug|1253984}})
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|-
|-
| MEMORY_TOTAL ±X% (browser + content process)
| MEMORY_TOTAL ±X% (browser + content process)
Line 699: Line 699:
| style="background-color: LightYellow;" | TBD
| style="background-color: LightYellow;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TBD
| style="background-color: Tomato;" | TODO
| style="background-color: SkyBlue;" | TODO
|}
|}


Revision as of 20:02, 11 March 2016

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

APZ Regressions

Talos numbers listed on this page are pulled from the Perfherder e10s dashboard. These results include the affects of enabling APZ on mozilla central. The component of individual regressions and improvements associated with APZ is listed below and is based on a Perfherder comparison between a try run with APZ disabled and mozilla-central. Test that showed no change are not listed.

Talos Metric Change Related bugs
CART 3%-5% bug 1253380
DAMP 5%-7%
TART 10%-13% bug 1253412
tp5o %proc time 5%-7% (windows)
tp5o main_RSS 3%-15% bug 1254273 (OS X)
tp5o mixed bug 1254273 (OS X)
tp5o_scroll 16%-40% bug 1179735, bug 1253860
tpaint 2% (Win7)
tps 9%-11% bug 1216924
tresize 4% (WinXP)
ts_paint 3% (WinXP)
tscrollx 18%-42%
tsvgr_opacity 3% (Linux)
tsvgx 3%-4%

Release Criteria

User Engagement

Responsible rvitillo
Accountable TBD
Supporting measurement/data teams
Consulted jgriffiths, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Firefox Usage TBD (bug 1249665) TBD glind? TBD TBD TBD TODO
User engagement (DAU/MAU) e10s/non-10s engagement ratio bug 1251259 rvitillo TBD TBD TBD TODO

Stability

Responsible bsmedberg
Accountable bsmedberg
Supporting Kairo, rvitillo
Consulted jgriffiths, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
chrome+content process crash rate chrome process crash ping counts + SUBPROCESS_CRASHES_WITH_DUMP['content'] per 1000 hours of subsessionLength IPython notebook rvitillo 84,390 (Beta 45) 109,348 (Beta 45) NO TODO
plugin process crash rate SUBPROCESS_CRASHES_WITH_DUMP['plugin'] per 1000 hours of subsessionLength IPython notebook rvitillo 41,897 (Beta 45) 55,809 (Beta 45) NO TODO

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 45 experiment rvitillo e10s is slightly better TODO
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 45 experiment rvitillo e10s is slightly better TODO
Event loop lag INPUT_EVENT_RESPONSE_MS beta 45 experiment chutten about the same TODO
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder 80%-90% faster! TODO

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS TBD avih regressed TODO
tp5o_scroll Perfherder kats results are mixed, but Linux is 46% better (bug 1253860 fixes this) TODO
tscrollx Perfherder kats Up to 60% better on non-XP, but 10% worse on XP. (bug 1253860 fixes this)

e10s is better on other platforms.

TODO

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT TBD (waiting for Beta 46 experiment) azhang TBD (waiting for Beta 46 experiment) TBD (waiting for Beta 46 experiment) TBD TODO

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted jgriffiths
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder TBD 3.96 (Linux) 4.55 (Linux) 7%–15% (bug 1251699) TODO
tpaint ±5% (first paint for a new window) mconley Perfherder TBD 197 (Linux) 286 (Linux 29%–45% (bug 1174770) TODO
tresize ±5% (window resize responsiveness) jimm? Perfherder TBD 9.92 (XP) 11.21 (XP) 13% (XP) (bug 1252947) TODO
tps ±5% (tab switch speed) mconley Perfherder TBD 43.27 (Linux) 72.27 (Linux) 45%-70% (bug 1186585) TODO
FX_TAB_SWITCH_TOTAL_MS mconley? 46 experiment TBD TBD TBD TBD TODO
CART ±5% (Customize Animation Regression Test) mconley Perfherder TBD 20.76 (Linux) 23.53 (Linux) 5%–13% (bug 1252768) TODO

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos page load time tp5o ±5% Perfherder TBD TBD TBD no change TODO
Page load time telemetry FX_PAGE_LOAD_MS ±5% beta 45 experiment rvitillo TBD TBD improvement TODO

Plugin Jank

Responsible chutten
Accountable bsmedberg
Supporting aklotz
Consulted jgriffiths
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% TBD TBD TBD TBD TBD TODO
Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% TBD TBD TBD TBD TBD TODO

Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted jgriffiths, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder TBD TBD TBD Mixed results: from 4% worse to 63% better TODO
tcanvasmark ±5% TBD Perfherder TBD TBD TBD From 2% worse to 3% better TODO
tsvgx ±5% mstange? Perfherder TBD TBD TBD Up to 51% better TODO
tsvg_opacity ±5% avih says no real owner. mattwoodrow? seth? Perfherder TBD TBD TBD Up to -79% better TODO

Startup/Shutdown Time

Responsible rvitillo
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder 863 (Win7) 433 (Win7) 43–55% better TODO
Time to first paint SIMPLE_MEASURES_FIRSTPAINT beta 45 experiment rvitillo 4636 ms (median) 4620 ms (median) no change TODO
Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION beta 45 experiment rvitillo 1216 ms (median) 1229 ms (median) no change TODO
  • Startup times may regress up to 10% and shutdown time up to 25% compared to non-e10s.
  • In telemetry performance reports, search for payload/simpleMeasurements/firstPaint, payload/simpleMeasurements/sessionRestored, and payload/simpleMeasurements/shutdownDuration.

Memory Usage

Responsible chutten
Accountable bsmedberg
Supporting
Consulted jgriffiths
Informed cpeterson, elan, release management
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder 384117275.55 (OS X) 365409824.16 (OS X) 5%–35% better TODO
tp5o Private Bytes Perfherder TBD 636896373.57 (Linux) 932390353.63 (Linux) 20% to 50% worse (bug 1253984) TODO
MEMORY_TOTAL ±X% (browser + content process) TBD chutten TBD TBD TBD TODO
  • We expect e10s MEMORY_TOTAL to regress compared to non-e10s because we are using more processes.

Release Blocking Bugs

All M8 and M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M8/M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria meta bugs' blockers

Full Query
ID Summary Whiteboard Keywords Assigned to
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);


APZ Bugs

  • APZ does not block e10s release. This bug list is purely informational.

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?
Responsible Blake, Felipe
Accountable blassey?
Supporting
Consulted jgriffiths
Informed cpeterson, elan, release management

Detail Pages

Release Criteria Responsible Status Tracking Bug
P1: Stability poiru [AT RISK] bug 1251376
P1: Jank chutten TBD bug 1251377
P1: Scrolling kats [AT RISK] bug 1251388
P1: Slow Script azhang TBD bug 1251545
P1: no blocker bugs blassey [ON TRACK] m9 bug query
P1: automated tests running and passing mrbkap? [AT RISK] bug e10s-tests
P2: UI Smoothness mconley [AT RISK] bug 1251547
P2: Page Load Time jimm [ON TRACK] bug 1251549
P2: Plugin Jank chutten TBD bug 1251551
P2: Graphics Milan [ON TRACK] bug 1251555
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Release_Criteria&oldid=1121680"