Electrolysis/Release Criteria/Jank

e10s can't browser responsiveness

RASCI:

• Responsible: chutten
• Accountable: bsmedberg
• Supporting: data team, RyanVM, rvitillo, avih, Softvision
• Consulted:
• Informed: cpeterson, elan, release management

Technical proposal:

The following metrics will be compared with e10s and non-e10s A/B tests:

FX_REFRESH_DRIVER_*_FRAME_DELAY_MS

Please provide detailed explanation of how we are comparing/validating this. This needs to include:

• confidence that the telemetry histogram is measuring the right thing (automated or manual testing?)
• the statistical method to compare e10s and non-e10s metrics, since they are not directly comparable
• explanation of any confounding factors such as APZ

Note from billm: bug 1228147 seems invalid because it considers users outside the experiment.

Email thread was unclear about whether this needs to include REFRESH_DRIVER_TICK metrics.

BHR/Chrome hangs

We have concerns about the accuracy of the data being collected for each of these measures, see bug 1240887. But we have agreed to accept the existing analysis which says that BHR and chromehangs improved with e10s and consider this requirement PASSed.

Followup may be required if BHR data is used to validate future addon-related jank.

Please provide: links to the bugs/spark analyses.

Event loop lag

Originally proposed: EVENTLOOP_UI_ACTIVITY_EXP_MS

INPUT_EVENT_RESPONSE_MS is the better measure for e10s/nonE10s comparisons, as it is valid across more than one OS and more than one process (EVENTLOOP_UI_ACTIVITY_EXP_MS is valid on Windows only and in the chrome process only). I was using the analysis of that measure as the primary reason for closing bug 1223780 ( https://bugzilla.mozilla.org/show_bug.cgi?id=1223780 ) using analyses on beta45ex1 (prelim analysis was done here: https://gist.github.com/chutten/9b9e29df10e0f7306f99 analysis on the later data was performed, but not published, as it was largely identical) and prelim data from beta45ex2 ( https://gist.github.com/chutten/3129baf8d5e0f10ef54a )

Are we done with this, call it a PASS?

jank per minute of active usage

This is a combined metric, bug 1198650. We have made the decision that this no longer blocks e10s, because we are looking at the individual components.

Talos tp5o_responsiveness

This test is hugely better in e10s. tp5o_responsiveness on WinXP mozilla-central (results not available on Aurora).

Concerns: Is this number valid? Why is it so much better in e10s? Is it from the chrome process or content?

A RESPONSIBLE PARTY NEEDS TO BE FOUND FOR THIS TEST

GC pauses

GC_MAX_PAUSE_MS, CYCLE_COLLECTOR_MAX_PAUSE e10s does better here PASS ?