Electrolysis/Multi Release Criteria: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 110: Line 110:
{| class="wikitable"
{| class="wikitable"
! Responsible
! Responsible
| e10s - jimm<br> e10s-multi - '''<span style="color:#9932cc"> TBD</span>'''
| jimm
|-
|-
! Accountable
! Accountable
| e10s - RyanVM<br> e10s-multi - '''<span style="color:#9932cc"> TBD</span>'''
| '''<span style="color:#9932cc"> TBD</span>'''
|-
! Supporting
| e10s - jmaher<br> e10s-multi - '''<span style="color:#9932cc"> TBD</span>'''
|-
! Consulted
| e10s - jgriffiths, rvitillo<br> e10s-multi - '''<span style="color:#9932cc"> TBD</span>'''
|-
! Informed
| e10s - cpeterson, elan, release management<br> e10s-multi - '''<span style="color:#9932cc"> TBD</span>'''
|}
|}


Line 134: Line 125:
! Criteria Met?
! Criteria Met?
! Criteria Met Signed-off By
! Criteria Met Signed-off By
|-
| tp5o summary ±5%
| style="background-color: Yellow;" | [https://treeherder.mozilla.org/perf.html#/e10s?filter=tp5o%20summary Perfherder] '''<span style="color:#9932cc"> Need updated link</span>'''
| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
| style="background-color: PaleGreen;" | 246.28 (Linux)
213.51 (Win8)
| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
| style="background-color: #b0e0e6;" | '''<span style="color:#9932cc"> TBD</span>'''
|-
|-
| FX_PAGE_LOAD_MS ±5%
| FX_PAGE_LOAD_MS ±5%

Revision as of 17:34, 30 March 2017

Page to collect and track Electrolysis Multi Process release criteria.

Rollout Criteria

We plan to ship e10s-multi in Firefox 54/55 using a staged rollout.
See elan's e10s-multi release plan.

Release Criteria

Stability

Responsible jimm
Accountable chutten
Description Metric e10s Analysis Analysis By e10s value e10s-multi value Criteria Met? Criteria Met Signed-off By
chrome + content crash rate (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours "M+(c-s)" on 48 Beta Update Link here TBD 5.20 crashes/khour TBD TBD TBD

Crash rates exclude XP and deprecated OS X operating systems

Jank

Responsible Gabor
Accountable chutten
Description Metric e10s Analysis e10s-multi Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 46 experiment TBD TBD TBD TBD
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 46 experiment TBD TBD TBD TBD
Event loop lag INPUT_EVENT_RESPONSE_MS beta 46 experiment

Nightly 48 telemetry

TBD TBD TBD TBD
Browser responsiveness during page load Tab paint test N/A Need bug TBD TBD TBD

Page Load

Responsible jimm
Accountable TBD
Metric Analysis Analysis By e10s value e10s-multi value Criteria Met? Criteria Met Signed-off By
FX_PAGE_LOAD_MS ±5% beta 46 experiment (bug 1228020) Need updated link TBD e10s is much better. TBD TBD TBD

Startup/Shutdown Time

Responsible e10s - rvitillo
e10s-multi - TBD
Accountable e10s -
e10s-multi - TBD
Supporting e10s -
e10s-multi - TBD
Consulted e10s - jgriffiths, jmaher
e10s-multi - TBD
Informed e10s - cpeterson, elan, release management
e10s-multi - TBD
Release Criteria Metric Analysis Analysis By e10s value e10s-multi value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder Need updated link TBD 433 (Win7) TBD TBD TBD
Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10%

("simpleMeasurements/firstPaint")

beta 46 experiment Need updated link TBD 4391 ms TBD TBD TBD
Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25%

("simpleMeasurements/shutdownDuration")

beta 46 experiment Need updated link TBD 1190 ms TBD TBD TBD
  • In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible e10s - kats
e10s-multi - TBD
Accountable e10s - RyanVM
e10s-multi - TBD
Supporting e10s - avih, jmaher, graphics team (mstange)
e10s-multi - TBD
Consulted e10s - jgriffiths, jmaher
e10s-multi - TBD
Informed e10s - cpeterson, elan, release management
e10s-multi - TBD
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS beta 46 experiment Need updated link TBD TBD TBD
tp5o_scroll Perfherder Need updated link TBD TBD TBD
tscrollx Perfherder Need updated link TBD TBD TBD

Plugin Jank

Is this needed for e10-multi?

Responsible chutten
Accountable aklotz
Supporting bsmedberg
Consulted Barbara
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
Plugin module initialization jank time BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is better.

Java: YES. e10s is better.

Silverlight: YES. e10s is better.

Unity: YES. e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% Beta 46 experiment chutten Flash: MAYBE? e10s is slightly worse.

Java: YES. e10s is slightly better.

Silverlight: MAYBE? e10s is slightly worse.

Unity: YES! e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is different, but not worse.

Java: MAYBE? e10s is slightly worse.

Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)

Unity: YES. e10s is different, but not worse.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Memory Usage

Responsible e10s - erahm
e10s-multi - TBD
Accountable e10s - blassey
e10s-multi - TBD
Supporting e10s jmaher (Talos)
e10s-multi - TBD
Consulted e10s - Barbara
e10s-multi - TBD
Informed e10s - cpeterson, elan, release management
e10s-multi - TBD
Metric Analysis e10s value e10s-multi value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder Need updated link 331418381.64 (OS X) TBD TBD TBD
MEMORY_TOTAL ±25% (browser + content process) beta 46 experiment and erahm's blog post Need updated link(s) TBD TBD TBD
VSIZE_MAX_CONTIGUOUS (browser + content process) beta 46 experiment Need updated link TBD TBD TBD
Additional Metric(s) Needed?
- process count 2, 4 and 8 data comparisons?
- AWSY?

UI Smoothness

Responsible e10s - mconley
e10s-multi - TBD
Accountable e10s - RyanVM
e10s-multi - TBD
Supporting e10s - jmaher, avih, gabor
e10s-multi - TBD
Consulted e10s - Barbara
e10s-multi - TBD
Informed e10s - cpeterson, elan, release management
e10s-multi - TBD

Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.

Metric Metric Owner Analysis e10s Data e10s-multi Data Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) TBD Perfherder Need updated link

e10s / APZ
xp = 4.4% combined
win7 = 1.8% / 9%
win8 = -0.6% / 10% (improvement)
osx = 6% / 3.5%
linux = 0.5% / 16%

TBD TBD TBD
tpaint ±5% (first paint for a new window) TBD Perfherder Need updated link

xp = 0%
win7 = 9.3%
win8 = 3.4%
osx = -11% (improvement)
linux = 7%

TBD TBD TBD
tresize ±5% (window resize responsiveness) TBD Perfherder Need updated link

e10s / APZ
xp = 1.0% combined
win7 = 0.0%
win8 = -22% (improvement overall)
osx = 1.0% combined
linux = 7% / 2.0%

TBD TBD TBD
tps ±5% (tab switch speed) TBD Perfherder Need updated link

xp = -27%
win7 = -28%
win8 = -34%
osx = -30%
linux = -19%

TBD TBD TBD
FX_TAB_SWITCH_TOTAL_MS TBD beta 46 experiment Need updated link 52.66 ms (median) < 82.61 ms (median) TBD TBD TBD
CART ±5% (Customize Animation Regression Test) TBD Perfherder Need updated link

e10s / APZ
xp = improvement
win7 = 4.0% / 2.5%
win8 = 2.0% / 2.5%
osx = 3.7% / 2.7%
linux = 6.8% / 6.8%

TBD TBD TBD

Graphics Performance

Is this needed for e10-multi?

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted Barbara, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder 18 19 Mixed results from 3% worse to 66% better Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5% Perfherder 6444 6301 Mixed results from 2% worse to 3% better Barbara (2016-05-06), Milan (2016-05-06)

Slow Scripts

Responsible e10s - azhang
e10s-multi - TBD
Accountable e10s - bsmedberg
e10s-multi - TBD
Supporting e10s -
e10s-multi - TBD
Consulted e10s - Barbara, billm
e10s-multi - TBD
Informed e10s - cpeterson, elan, release management
e10s-multi - TBD
Description Metric Analysis Analysis By e10s value e10s-multi value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT beta 47 experiment Need updated link TBD 0.00158655016424 warnings per use hour (aka 630 hours/warning)

(bug 1260769)

TBD TBD TBD

Tests

All unit tests disabled for e10s-multi must be triaged and:

  • re-enabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s-multi
  • or deleted

We want each test directory to be signed-off by the feature area's owner.

Responsible e10s - Blake, Felipe
e10s-multi - TBD
Accountable e10s - jimm
e10s-multi - TBD
Supporting e10s - dolske (coordinating people fixing front-end tests)
e10s-multi - TBD
Consulted e10s - jgriffiths
e10s-multi - TBD
Informed e10s - cpeterson, elan, release management
e10s-multi - TBD
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan Tracy In progress: almost done (based on this doc triage) TBD
Execute test plan Tracy, SV In progress TBD
Re-enable disabled e10s tests Gabor, Blake In progress TBD

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria bugs

Need updated bug list

Bugzilla query error

Query options must be valid JSON.1


APZ Bugs

Need updated bug list

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Accessibility

  • Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

  • Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.

Service Workers

  • ???
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Multi_Release_Criteria&oldid=1167143"