Electrolysis/Multi Release Criteria: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(24 intermediate revisions by 2 users not shown)
Line 20: Line 20:


1) We can't rely on Talos numbers generated in automation for beta builds. The default there is one content process. We'll have to use try runs of the beta code base to generate comparisons.
1) We can't rely on Talos numbers generated in automation for beta builds. The default there is one content process. We'll have to use try runs of the beta code base to generate comparisons.
2) 32-bit / 64-bit builds for Windows - which? both?
3) Probes we might want to add:
{| class="wikitable"
! COMPOSITOR_ANIMATION_MAX_CONTIGUOUS_DROPS_APZ
|style="width:300px;"|For APZ/scrolling compositor animations, the maximum number of contiguous frame drops encountered
|-
! COMPOSITOR_ANIMATION_MAX_CONTIGUOUS_DROPS_CHROME
| For compositor animations of chrome-process layers, the maximum number of contiguous frame drops encountered
|-
! COMPOSITOR_ANIMATION_MAX_CONTIGUOUS_DROPS_CONTENT
| For compositor animations of content-process layers, the maximum number of contiguous frame drops encountered
|-
! TIME_TO_NON_BLANK_PAINT_MS
| The time between navigation start and the first non-blank paint of a foreground root content document, in milliseconds. This only records documents that were in an active docshell throughout the whole time between navigation start and non-blank paint. The non-blank paint timestamp is taken during display list building and does not include rasterization or compositing of that paint.
|}


== AWSY Notes ==
== AWSY Notes ==
Line 103: Line 85:
| GC_MAX_PAUSE_MS
| GC_MAX_PAUSE_MS
| style="background-color: White;width:250px;" | Longest GC slice in a single GC cycle in milliseconds, broken down by parent and child process(es).
| style="background-color: White;width:250px;" | Longest GC slice in a single GC cycle in milliseconds, broken down by parent and child process(es).
| style="background-color: White;" | tracy
| style="background-color: White;" | ben/tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: YES<br/>
Win: YES<br/>
Line 112: Line 94:
| CYCLE_COLLECTOR_MAX_PAUSE
| CYCLE_COLLECTOR_MAX_PAUSE
| style="background-color: White;width:250px;" | Longest pause for an individual slice of one cycle collection, including preparation in milliseconds, broken down by parent and child process(es).
| style="background-color: White;width:250px;" | Longest pause for an individual slice of one cycle collection, including preparation in milliseconds, broken down by parent and child process(es).
| style="background-color: White;" | tracy
| style="background-color: White;" | ben/tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: YES<br/>
Win: YES<br/>
Line 121: Line 103:
| INPUT_EVENT_RESPONSE_MS
| INPUT_EVENT_RESPONSE_MS
| style="background-color: White;width:250px;" | Time in milliseconds for input event lifetime (created -> fully handled) including traversal to a child process and back.  
| style="background-color: White;width:250px;" | Time in milliseconds for input event lifetime (created -> fully handled) including traversal to a child process and back.  
| style="background-color: White;" | tracy
| style="background-color: White;" | ben/tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: YES<br/>
Win: YES<br/>
Line 130: Line 112:
| FX_TAB_SWITCH_TOTAL_MS
| FX_TAB_SWITCH_TOTAL_MS
| style="background-color: White;width:250px;" | Time in milliseconds a tab switch takes, including first paint of the tab.
| style="background-color: White;width:250px;" | Time in milliseconds a tab switch takes, including first paint of the tab.
| style="background-color: White;" | tracy
| style="background-color: White;" | ben/tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: YES<br/>
Win: YES<br/>
Line 153: Line 135:
| tp5o_responsiveness [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,6be9b4d8231c4ed2bccfcc497c5af51317f67552,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| tp5o_responsiveness [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,6be9b4d8231c4ed2bccfcc497c5af51317f67552,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| style="background-color: White;width:250px;" | TP5O Talos numbers, comparing current single process beta builds with try runs using the beta code base and four content processes.<br/>Targets: 64-bit opt/pgo builds
| style="background-color: White;width:250px;" | TP5O Talos numbers, comparing current single process beta builds with try runs using the beta code base and four content processes.<br/>Targets: 64-bit opt/pgo builds
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perherder Comparison]<br/>
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=96e5124a8a1c&newProject=try&newRevision=4c1f1a121337&framework=1&showOnlyImportant=0 Comparison 20170522]
run 5/9-5/10
[https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Comparison 20170510]
| style="background-color: White;" | tracy
| style="background-color: White;" | tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 6.93<br/>
Win7 32bit: 6.11<br/>
Win8 64bit: 5.13<br/>
Win8 64bit: 4.93<br/>
OSX: n/a<br/>
OSX: n/a<br/>
Linux: [https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-beta,8cba34a5b756cc4c005381526f637533c22f2f31,1,1%5D&selected=%5Bmozilla-beta,8cba34a5b756cc4c005381526f637533c22f2f31,198607,95684037%5D 4.58]
Linux: 4.55
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 6.27<br/>
Win7 32bit: 6.25 (+2.39%)<br/>
Win8 64bit: 5.67<br/>
Win8 64bit: 4.86 (-1.40%)<br/>
OSX: n/a<br/>
OSX: n/a<br/>
Linux: 4.52
Linux: 4.52 (-0.59)
| style="background-color: #99FF99;" | YES - pending investigation in {{bug|1362920}}
| style="background-color: #99FF99;" | YES - note: {{bug|1362920}}
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|}
|}
Line 196: Line 178:
| FX_PAGE_LOAD_MS
| FX_PAGE_LOAD_MS
| style="background-color: White;width:250px;"|Time taken to load a page (ms). This includes all static contents, no dynamic content. Loading of about: pages is not counted.
| style="background-color: White;width:250px;"|Time taken to load a page (ms). This includes all static contents, no dynamic content. Loading of about: pages is not counted.
| style="background-color: White;" | tracy
| style="background-color: White;" | ben/tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: YES<br/>
Win: YES<br/>
Line 232: Line 214:
| TART [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,f36eb8fccce4dc7259b699af75ddd07d63cd914e,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| TART [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,f36eb8fccce4dc7259b699af75ddd07d63cd914e,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| Tab animation regression test (Talos).
| Tab animation regression test (Talos).
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder Comparison]<br/>
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=96e5124a8a1c&newProject=try&newRevision=4c1f1a121337&framework=1&showOnlyImportant=0 20170522]
run 5/9-5/10
<br/>[https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 20170510]
| style="background-color: White;" | tracy
| style="background-color: White;" | tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 7.34<br/>
Win7 32bit: 7.36<br/>
Win8 64bit: 6.26<br/>
Win8 64bit: 6.26<br/>
OSX: 11.22<br/>
OSX: 11.18<br/>
Linux: 6.29
Linux: 6.31
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 7.35<br/>
Win7 32bit: 7.34 (-0.27%)<br/>
Win8 64bit: 6.25<br/>
Win8 64bit: 6.21 (-0.81%)<br/>
OSX: 11.17<br/>
OSX: 11.35 (+1.54%)<br/>
Linux: 6.27
Linux: 6.32 (+0.17%)
| style="background-color: #99FF99;" | YES
| style="background-color: #99FF99;" | YES
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
Line 250: Line 232:
| TPAINT [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| TPAINT [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| First paint for a new window (Talos).
| First paint for a new window (Talos).
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder Comparison]<br/>
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=96e5124a8a1c&newProject=try&newRevision=4c1f1a121337&framework=1&showOnlyImportant=0 20170522]
run 5/9-5/10
<br/>[https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 20170510]
| style="background-color: White;" | tracy
| style="background-color: White;" | tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 294.32<br/>
Win7 32bit: 298.80<br/>
Win8 64bit: 288.36<br/>
Win8 64bit: 283.82<br/>
OSX: 325.042<br/>
OSX: 323.16<br/>
Linux: 276.33
Linux: 277.12
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 298.68<br/>
Win7 32bit: 296.06 (-0.92%)<br/>
Win8 64bit: 287.07<br/>
Win8 64bit: 287.60 (+1.33%)<br/>
OSX: 320.40<br/>
OSX: 322.43 (-0.23%)<br/>
Linux: 273.71
Linux: 277.34 (+0.08%)
| style="background-color: #99FF99;" | YES
| style="background-color: #99FF99;" | YES
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
Line 268: Line 250:
| TRESIZE [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,4ac681a39a4caefb56468c5bc86fa23b8cee4c4f,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| TRESIZE [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,4ac681a39a4caefb56468c5bc86fa23b8cee4c4f,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| Window resize responsiveness (Talos).
| Window resize responsiveness (Talos).
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder Comparison]<br/>
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=96e5124a8a1c&newProject=try&newRevision=4c1f1a121337&framework=1&showOnlyImportant=0 20170522]
run 5/9-5/10
<br/>[https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 20170510]
| style="background-color: White;" | tracy
| style="background-color: White;" | tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 11.91<br/>
Win7 32bit: 11.99<br/>
Win8 64bit: 10.62<br/>
Win8 64bit: 10.65<br/>
OSX: 28.69<br/>
OSX: 28.87<br/>
Linux: 23.01
Linux: 23.34
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7 32bit: 11.93<br/>
Win7 32bit: 11.95 (-0.33%)<br/>
Win8 64bit: 10.63<br/>
Win8 64bit: 10.61 (-0.42%)<br/>
OSX: 28.82<br/>
OSX: 28.89 (+0.07%)<br/>
Linux: 23.59
Linux: 23.90 (+2.38%)
| style="background-color: #99FF99;" | YES
| style="background-color: #99FF99;" | YES
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
Line 296: Line 278:
! Regression(s)
! Regression(s)
| Blake Kaplan, Gabor Krizsanits
| Blake Kaplan, Gabor Krizsanits
|-
! Report Link
|[https://metrics.mozilla.com/protected/bmiroglio/multi/e10sMulti_experiment.html e10s-multi performance in Telemetry]
|}
|}


{| class="wikitable"
{| class="wikitable"
|-
|-
!rowspan="2"|Metric
!Metric
!rowspan="2"|Description
!Description
!rowspan="2"|Report Links
!Analysis By
!rowspan="2"|Analysis By
!Criteria Met?
!rowspan="1" colspan="2"|Results for Child Process Cohorts
!Criteria Met Signed-off By
!rowspan="2"|Criteria Met?
!rowspan="2"|Criteria Met Signed-off By
|-
! 1
! 4
|-
|-
| MEMORY_TOTAL
| MEMORY_TOTAL
| style="background-color: White;width:250px;"|  Total memory across all processes (KB).
| style="background-color: White;width:250px;"|  Total memory across all processes (KB).
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: ?<br/>
Win: YES<br/>
OSX: ?<br/>
OSX: limited data<br/>
Linux: ?
Linux: limited data
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| MEMORY_VSIZE_MAX_CONTIGUOUS
| MEMORY_VSIZE_MAX_CONTIGUOUS
| style="background-color: White;width:250px;"| Maximum-sized block of contiguous virtual memory (KB).
| style="background-color: White;width:250px;"| Maximum-sized block of contiguous virtual memory (KB).
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: ?<br/>
Win: YES<br/>
OSX: ?<br/>
OSX: limited data<br/>
Linux: ?
Linux: limited data
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
| MEMORY_DISTRIBUTION_AMONG_CONTENT
| style="background-color: White;width:250px;"|  Absolute difference of each content process' USS and the mean of USS, normalized by the mean, as a percentage. Measure memory consumption distribution between content processes. We're looking for ugly numbers here while running with 4+ content processes. Does not apply to single content process configs.
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | N/A
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|}
|}
Line 529: Line 485:
! Regression(s)
! Regression(s)
| Blake Kaplan, Gabor Krizsanits, Mike Conley
| Blake Kaplan, Gabor Krizsanits, Mike Conley
|-
! Report Link
|[https://metrics.mozilla.com/protected/bmiroglio/multi/e10sMulti_experiment.html e10s-multi performance in Telemetry]
|}
|}


{| class="wikitable"
{| class="wikitable"
|-
|-
!rowspan="2"|Metric
!Metric
!rowspan="2"|Description
!Description
!rowspan="2"|Report Links
!Analysis By
!rowspan="2"|Analysis By
!Criteria Met?
!rowspan="1" colspan="2"|Results for Child Process Cohorts
!Criteria Met Signed-off By
!rowspan="2"|Criteria Met?
!rowspan="2"|Criteria Met Signed-off By
|-
! 1
! 4
|-
|-
| FX_TAB_SWITCH_UPDATE_MS
| FX_TAB_SWITCH_UPDATE_MS
| style="background-color: White;width:250px;"| Time in ms spent updating UI in response to a tab switch.
| style="background-color: White;width:250px;"| Time in ms spent updating UI in response to a tab switch.
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: ?<br/>
Win: YES<br/>
OSX: ?<br/>
OSX: limited data<br/>
Linux: ?
Linux: limited data
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| FX_TAB_SWITCH_TOTAL_E10S_MS
| FX_TAB_SWITCH_TOTAL_E10S_MS
| style="background-color: White;width:250px;"| Time in ms between tab selection and tab content paint.
| style="background-color: White;width:250px;"| Time in ms between tab selection and tab content paint.
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: ?<br/>
Win: YES<br/>
OSX: ?<br/>
OSX: limited data<br/>
Linux: ?
Linux: limited data
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| FX_TAB_SWITCH_SPINNER_VISIBLE_MS
| FX_TAB_SWITCH_SPINNER_VISIBLE_MS
| style="background-color: White;width:250px;"| Time in ms spent updating UI in response to a tab switch.
| style="background-color: White;width:250px;"| Time in ms spent updating UI in response to a tab switch.
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: ?<br/>
Win: YES<br/>
OSX: ?<br/>
OSX: limited data<br/>
Linux: ?
Linux: limited data
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| FX_TAB_SWITCH_SPINNER_VISIBLE_LONG_MS
| FX_TAB_SWITCH_SPINNER_VISIBLE_LONG_MS
| style="background-color: White;width:250px;"| If the spinner interstitial displays during tab switching, records the time in ms the graphic is visible. This probe is similar to FX_TAB_SWITCH_SPINNER_VISIBLE_MS, but is for truly degenerate cases.
| style="background-color: White;width:250px;"| If the spinner interstitial displays during tab switching, records the time in ms the graphic is visible. This probe is similar to FX_TAB_SWITCH_SPINNER_VISIBLE_MS, but is for truly degenerate cases.
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: White;" |  
| style="background-color: White;" |  
Win: ?<br/>
Win: YES<br/>
OSX: ?<br/>
OSX: limited data<br/>
Linux: ?
Linux: limited data
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
| FX_TAB_REMOTE_NAVIGATION_DELAY_MS<br/>
| style="background-color: White;width:250px;"| Time taken in milliseconds between the browser sending a naviagion event to content and content receiving it. This message can be either SessionStore:restoreTabContent or WebNavigation:LoadURI and these names are used as keys for this histogram. This is e10s only and recorded in the content process. Measures delays associated with child process creation. [https://telemetry.mozilla.org/new-pipeline/dist.html#!arch=x86&cumulative=0&e10s=true&end_date=2017-05-16&keys=WebNavigation%253ALoadURI!SessionStore%253ArestoreTabContent!__none__!__none__&max_channel_version=beta%252F54&measure=FX_TAB_REMOTE_NAVIGATION_DELAY_MS&min_channel_version=nightly%252F55&processType=*&product=Firefox&sanitize=0&sort_keys=submissions&start_date=2017-04-20&table=0&trim=0&use_submission_date=0 TELEMETRY LINK]
| style="background-color: White;" | ben/tracy
| style="background-color: lightgreen;" |
Win: [https://telemetry.mozilla.org/new-pipeline/dist.html#!arch=x86&cumulative=0&e10s=true&end_date=2017-05-15&keys=WebNavigation%253ALoadURI!SessionStore%253ArestoreTabContent!__none__!__none__&max_channel_version=beta%252F54&measure=FX_TAB_REMOTE_NAVIGATION_DELAY_MS&min_channel_version=beta%252F51&os=Windows_NT&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-05-04&table=0&trim=1&use_submission_date=0 YES]<br/>
OSX: [https://telemetry.mozilla.org/new-pipeline/dist.html#!arch=x86-64!x86&cumulative=0&e10s=true&end_date=2017-05-15&keys=WebNavigation%253ALoadURI!SessionStore%253ArestoreTabContent!__none__!__none__&max_channel_version=beta%252F54&measure=FX_TAB_REMOTE_NAVIGATION_DELAY_MS&min_channel_version=beta%252F51&os=Darwin&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-05-04&table=0&trim=1&use_submission_date=0 YES]<br/>
Linux: [https://telemetry.mozilla.org/new-pipeline/dist.html#!arch=x86-64!x86&cumulative=0&e10s=true&end_date=2017-05-16&keys=WebNavigation%253ALoadURI!SessionStore%253ArestoreTabContent!__none__!__none__&max_channel_version=beta%252F54&measure=FX_TAB_REMOTE_NAVIGATION_DELAY_MS&min_channel_version=beta%252F51&os=Linux&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-05-04&table=0&trim=1&use_submission_date=0 YES]
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|}
{| class="wikitable"
|-
|-
| FX_TAB_REMOTE_NAVIGATION_DELAY_MS
!rowspan="2"|Metric
| style="background-color: White;width:250px;"| Time taken in milliseconds between the browser sending a naviagion event to content and content receiving it. This message can be either SessionStore:restoreTabContent or WebNavigation:LoadURI and these names are used as keys for this histogram. This is e10s only and recorded in the content process. Measures delays associated with child process creation.
!rowspan="2"|Description
| style="background-color: White;" | TBD
!rowspan="2"|Report Links
| style="background-color: White;" | TBD
!rowspan="2"|Analysis By
| style="background-color: White;" |  
!rowspan="1" colspan="2"|Results for Child Process Cohorts
Win: ?<br/>
!rowspan="2"|Criteria Met?
OSX: ?<br/>
!rowspan="2"|Criteria Met Signed-off By
Linux: ?
|-
| style="background-color: White;" |  
! 1
Win: ?<br/>
! 4
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| TPS Test [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,d96c356354e1d01835f87142841749ee6d353184,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| TPS Test [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,d96c356354e1d01835f87142841749ee6d353184,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| Tab switch timing (Talos)
| Tab switch timing (Talos)
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder Comparison]<br/>
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=96e5124a8a1c&newProject=try&newRevision=4c1f1a121337&framework=1&showOnlyImportant=0 20170522]
run 5/9-5/10
<br/>[https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 20170510]
| style="background-color: White;" | tracy
| style="background-color: White;" | tracy
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7-32bit: 26.89<br/>
Win7-32bit: 26.87<br/>
Win8-64bit: 24.54<br/>
Win8-64bit: 24.06<br/>
OSX: 28.20<br/>
OSX: 29.39<br/>
Linux: 29.37
Linux: 29.74
| style="background-color: White;" |  
| style="background-color: White;" |  
Win7-32bit: 27.24<br/>
Win7-32bit: 28.80 (+10.92%)<br/>
Win8-64bit: 24.52<br/>
Win8-64bit: 22.83 (-5.11%)<br/>
OSX: 31.11<br/>
OSX: 25.48 (-13.30%)<br/>
Linux: 29.47
Linux: 28.66 (-3.65%)
| style="background-color: #FFFF99;" | NO - pending investigation in {{bug|1362920}} (even though that bug is listed as Win7, I think it may affect all Win and Mac)
| style="background-color: #99FF99;" | YES - note: {{bug|1362920}}
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|}
|}
Line 649: Line 587:
! Regression(s)
! Regression(s)
| Blake Kaplan, Gabor Krizsanits
| Blake Kaplan, Gabor Krizsanits
|-
! Report Link
|[https://metrics.mozilla.com/protected/bmiroglio/multi/e10sMulti_experiment.html e10s-multi performance in Telemetry]
|}
|}


{| class="wikitable"
{| class="wikitable"
|-
|-
!rowspan="2"|Metric
!Metric
!rowspan="2"|Description
!Description
!rowspan="2"|Report Links
!Analysis By
!rowspan="2"|Analysis By
!Criteria Met?
!rowspan="1" colspan="2"|Results for Child Process Cohorts
!Criteria Met Signed-off By
!rowspan="2"|Criteria Met?
!rowspan="2"|Criteria Met Signed-off By
|-
! 1
! 4
|-
|-
| SLOW_SCRIPT_NOTICE_COUNT
| SLOW_SCRIPT_NOTICE_COUNT
| style="background-color: White;width:250px;"| Total count of slow script notices displayed to user.
| style="background-color: White;width:250px;"| Total count of slow script notices displayed to user.
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: #FFFF99;" |  
| style="background-color: White;" |
Win: YES<br/>
Win: ?<br/>
OSX: limited data<br/>
OSX: ?<br/>
Linux: no data
Linux: ?
| style="background-color: White;" |  
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| SLOW_SCRIPT_PAGE_COUNT
| SLOW_SCRIPT_PAGE_COUNT
| style="background-color: White;width:250px;"| The number of pages that trigger slow script notices.
| style="background-color: White;width:250px;"| The number of pages that trigger slow script notices.
| style="background-color: White;" | TBD
| style="background-color: White;" | ben/tracy
| style="background-color: White;" | TBD
| style="background-color: #FFFF99;" |  
| style="background-color: White;" |
Win: YES<br/>
Win: ?<br/>
OSX: limited data<br/>
OSX: ?<br/>
Linux: no data
Linux: ?
| style="background-color: White;" |  
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|-
|-
| SLOW_SCRIPT_NOTIFY_DELAY
| SLOW_SCRIPT_NOTIFY_DELAY
| style="background-color: White;width:250px;"| The difference between the js slow script timeout for content set in prefs and the actual time we waited before displaying the notification in milliseconds.
| style="background-color: White;width:250px;"| The difference between the js slow script timeout for content set in prefs and the actual time we waited before displaying the notification in milliseconds.
| style="background-color: White;" | ben/tracy
| style="background-color: #FFFF99;" |
Win: YES<br/>
OSX: limited data<br/>
Linux: no data
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" |
Win: ?<br/>
OSX: ?<br/>
Linux: ?
| style="background-color: White;" | TBD
| style="background-color: White;" | TBD
|}
== Misc.  broken? ==
{| class="wikitable"
! Reporting
| Tracy Walker
|-
! Data Wrangler
| Tracy Walker
|-
|}
{| class="wikitable"
|-
!rowspan="2"|Metric
!rowspan="2"|Description
!rowspan="2"|Report Links
!rowspan="2"|Analysis By
!rowspan="1" colspan="2"|Results for Child Process Cohorts
!rowspan="2"|Criteria Met?
!rowspan="2"|Criteria Met Signed-off By
|-
! 1
! 4
|-
| tp5o XRes [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,a20264b4683ad3ae2483813370c9a211daf2a348,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| style="background-color: White;width:250px;"|These results are extremely bifurcated
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder comparison]
| style="background-color: White;" | tracy
| style="background-color: White;" |
Linux: ~3830 : ~1,574,000
| style="background-color: White;" |
Linux: ~3830 : ~1,574,000
| style="background-color: #99FF99;" | test is bimodal, results are as expected
| style="background-color: White;" | not called out as a release criteria
|-
| tp5n main_normal_fileio [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,dbdfdf5c5cf2a1138c6e6af467caf200254748a9,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| style="background-color: White;width:250px;"|This test is producing a wide range of results
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder comparison]
| style="background-color: White;" | tracy
| style="background-color: White;" |
Win7-32bit: 2,796,507.80
| style="background-color: White;" |
Win7-32bit: 3,085,232.18
| style="background-color: #FFFF99;" | possibly also related to {{bug|1362920}}
| style="background-color: White;" | not called out as a release criteria
|-
| tp5n main_normal_netio [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,596e7f02d6d5647d3a3f828c32d2a901ff70c274,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| style="background-color: White;width:250px;"|This test seems to produce mostly "0" value results
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder comparison]
| style="background-color: White;" | tracy
| style="background-color: White;" |
Win7-32bit: 0 : 3932.80
| style="background-color: White;" |
Win7-32bit: 0 : 385,376.91
| style="background-color: #99FF99;" | per jmaher, 0 values are mostly expected
| style="background-color: White;" | not called out as a release criteria
|-
| tp5n main_startup_netio [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Btry,1628e13f9987d10ef971cf2f3e92a5b47fb27906,1,1%5D&highlightedRevisions=4aeff94a7fc8&highlightedRevisions=abd843dd9c40 graph]
| style="background-color: White;width:250px;"|This test seems to produce mostly "0" value results
| style="background-color: White;" | [https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=4aeff94a7fc872e4716def28f89647fd0d0b4488&newProject=try&newRevision=abd843dd9c400aa08a06a64bf886e5fc972e407a&framework=1&filter=e10s&showOnlyImportant=0 Perfherder comparison]
| style="background-color: White;" | tracy
| style="background-color: White;" |
Win7-32bit: 0 : 40.40
| style="background-color: White;" |
Win7-32bit: 0 : 115.27
| style="background-color: #99FF99;" | per jmaher, 0 values are mostly expected
| style="background-color: White;" | not called out as a release criteria
|}
|}


Line 809: Line 658:
|-
|-
! Accountable
! Accountable
| SoftVision
| SoftVision / tracy
|-
|-
! Sign Off
! Sign Off
| SoftVision / Jeff Griffiths
| Jeff Griffiths / tracy
|}
|}


Line 825: Line 674:
| Tracy
| Tracy
| style="background-color: LightGreen;" | Done
| style="background-color: LightGreen;" | Done
| style="background-color: LightGreen;" | Tracy
| style="background-color: LightGreen;" | tracy
|-
|-
| [https://wiki.mozilla.org/QA/e10s_Multi Execute test plan]
| [https://wiki.mozilla.org/QA/e10s_Multi Execute test plan]
| Tracy, Ben M, SV
| Tracy, Ben M, SV
| style="background-color: LightYellow;" | In progress
| style="background-color: LightGreen;" | on going perf testing.  so far so good
| style="background-color: White;" | TBD
| style="background-color: white;" | tbd
|}
|}

Latest revision as of 15:01, 23 May 2017

Page to collect and track Electrolysis Multi Process release criteria.

Rollout Criteria

We plan to ship e10s-multi in Firefox 54 / 55 using a staged rollout. See elan's e10s-multi release plan.

Report / Dashboards

Rollout Status

  • Nightly: 4 processes
  • Beta: experiments running, user default is 1 content process
  • Release: 1 content process

Notes

1) We can't rely on Talos numbers generated in automation for beta builds. The default there is one content process. We'll have to use try runs of the beta code base to generate comparisons.

AWSY Notes

1) Maximum number of tabs set by automation is currently 30.

2) test source

Release Criteria

Stability

Reporting Jim Mathies
Data Wrangler
Regression(s) Blake Kaplan, Gabor Krizsanits
Metric Description Report Links Analysis By Criteria Met? Criteria Met Signed-off By
Crash Rate (chrome crashes + (content process(es) - content process(es) shutdown termination crash reports)) / 1000 use hours beta 54 (redash) jimm SO FAR - multi1 / multi4 rates track TBD

Jank

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
GC_MAX_PAUSE_MS Longest GC slice in a single GC cycle in milliseconds, broken down by parent and child process(es). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
CYCLE_COLLECTOR_MAX_PAUSE Longest pause for an individual slice of one cycle collection, including preparation in milliseconds, broken down by parent and child process(es). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
INPUT_EVENT_RESPONSE_MS Time in milliseconds for input event lifetime (created -> fully handled) including traversal to a child process and back. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_TOTAL_MS Time in milliseconds a tab switch takes, including first paint of the tab. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
Metric Description Report Links Analysis By Results for Child Process Cohorts Criteria Met? Criteria Met Signed-off By
1 4
tp5o_responsiveness graph TP5O Talos numbers, comparing current single process beta builds with try runs using the beta code base and four content processes.
Targets: 64-bit opt/pgo builds 		Comparison 20170522

Comparison 20170510

tracy

Win7 32bit: 6.11
Win8 64bit: 4.93
OSX: n/a
Linux: 4.55

Win7 32bit: 6.25 (+2.39%)
Win8 64bit: 4.86 (-1.40%)
OSX: n/a
Linux: 4.52 (-0.59)

YES - note: bug 1362920 TBD

Page Load

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
FX_PAGE_LOAD_MS Time taken to load a page (ms). This includes all static contents, no dynamic content. Loading of about: pages is not counted. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD

UI Smoothness

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Metric Description Report Links Analysis By Results for Child Process Cohorts Criteria Met? Criteria Met Signed-off By
1 4
TART graph Tab animation regression test (Talos). 20170522


20170510

tracy

Win7 32bit: 7.36
Win8 64bit: 6.26
OSX: 11.18
Linux: 6.31

Win7 32bit: 7.34 (-0.27%)
Win8 64bit: 6.21 (-0.81%)
OSX: 11.35 (+1.54%)
Linux: 6.32 (+0.17%)

YES TBD
TPAINT graph First paint for a new window (Talos). 20170522


20170510

tracy

Win7 32bit: 298.80
Win8 64bit: 283.82
OSX: 323.16
Linux: 277.12

Win7 32bit: 296.06 (-0.92%)
Win8 64bit: 287.60 (+1.33%)
OSX: 322.43 (-0.23%)
Linux: 277.34 (+0.08%)

YES TBD
TRESIZE graph Window resize responsiveness (Talos). 20170522


20170510

tracy

Win7 32bit: 11.99
Win8 64bit: 10.65
OSX: 28.87
Linux: 23.34

Win7 32bit: 11.95 (-0.33%)
Win8 64bit: 10.61 (-0.42%)
OSX: 28.89 (+0.07%)
Linux: 23.90 (+2.38%)

YES TBD

Memory Usage

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
MEMORY_TOTAL Total memory across all processes (KB). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
MEMORY_VSIZE_MAX_CONTIGUOUS Maximum-sized block of contiguous virtual memory (KB). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD

Are We Slim Yet (AWSY)

RSS chrome process + USS content processes (Talos)

5/10/2017 Perfherder

(insert passing criteria declaration)

Windows 7 Opt 32-bit
Metric Regression Criteria Met? Sign-off
Resident Memory Fresh start -0.47%
Resident Memory Fresh start [+30s] -0.67%
Resident Memory After tabs open 29.80%
Resident Memory After tabs open [+30s] 32.87%
Resident Memory After tabs open [+30s, forced GC] 26.59%
Resident Memory Tabs closed -29.74%
Resident Memory Tabs closed [+30s] -18.40%
Resident Memory Tabs closed [+30s, forced GC] -12.14%
Linux Opt 32-bit
Metric Regression Criteria Met? Sign-off
Resident Memory Fresh start 0.11%
Resident Memory Fresh start [+30s] 0.27%
Resident Memory After tabs open 19.87%
Resident Memory After tabs open [+30s] 18.98%
Resident Memory After tabs open [+30s, forced GC] 13.15%
Resident Memory Tabs closed -20.46%
Resident Memory Tabs closed [+30s] -10.59%
Resident Memory Tabs closed [+30s, forced GC] -7.76%
Linux Opt 64-bit
Metric Regression Criteria Met? Sign-off
Resident Memory Fresh start -1.76%
Resident Memory Fresh start [+30s] -1.80%
Resident Memory After tabs open 22.51%
Resident Memory After tabs open [+30s] 21.92%
Resident Memory After tabs open [+30s, forced GC] 15.06%
Resident Memory Tabs closed -21.38%
Resident Memory Tabs closed [+30s] -10.48%
Resident Memory Tabs closed [+30s, forced GC] -7.23%

Tab Switching

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits, Mike Conley
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
FX_TAB_SWITCH_UPDATE_MS Time in ms spent updating UI in response to a tab switch. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_TOTAL_E10S_MS Time in ms between tab selection and tab content paint. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_SPINNER_VISIBLE_MS Time in ms spent updating UI in response to a tab switch. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_SPINNER_VISIBLE_LONG_MS If the spinner interstitial displays during tab switching, records the time in ms the graphic is visible. This probe is similar to FX_TAB_SWITCH_SPINNER_VISIBLE_MS, but is for truly degenerate cases. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_REMOTE_NAVIGATION_DELAY_MS
 Time taken in milliseconds between the browser sending a naviagion event to content and content receiving it. This message can be either SessionStore:restoreTabContent or WebNavigation:LoadURI and these names are used as keys for this histogram. This is e10s only and recorded in the content process. Measures delays associated with child process creation. TELEMETRY LINK ben/tracy

Win: YES
OSX: YES
Linux: YES

TBD
Metric Description Report Links Analysis By Results for Child Process Cohorts Criteria Met? Criteria Met Signed-off By
1 4
TPS Test graph Tab switch timing (Talos) 20170522


20170510

tracy

Win7-32bit: 26.87
Win8-64bit: 24.06
OSX: 29.39
Linux: 29.74

Win7-32bit: 28.80 (+10.92%)
Win8-64bit: 22.83 (-5.11%)
OSX: 25.48 (-13.30%)
Linux: 28.66 (-3.65%)

YES - note: bug 1362920 TBD

Slow Scripts

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
SLOW_SCRIPT_NOTICE_COUNT Total count of slow script notices displayed to user. ben/tracy

Win: YES
OSX: limited data
Linux: no data

TBD
SLOW_SCRIPT_PAGE_COUNT The number of pages that trigger slow script notices. ben/tracy

Win: YES
OSX: limited data
Linux: no data

TBD
SLOW_SCRIPT_NOTIFY_DELAY The difference between the js slow script timeout for content set in prefs and the actual time we waited before displaying the notification in milliseconds. ben/tracy

Win: YES
OSX: limited data
Linux: no data

TBD

Tests

Reporting Blake Kaplan / Gabor Krizsanits
Accountable Everyone
Sign Off Module Owners / Jeff Griffiths

All unit tests disabled for e10s-multi must be triaged and:

  • re-enabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s-multi
  • or deleted

We want each test directory to be signed-off by the feature area's owner.

QA Test Plan

Responsible Erin Lancaster
Accountable SoftVision / tracy
Sign Off Jeff Griffiths / tracy
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan Tracy Done tracy
Execute test plan Tracy, Ben M, SV on going perf testing. so far so good tbd
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Multi_Release_Criteria&oldid=1171816"