E10s/Status/April8: Difference between revisions

From MozillaWiki
< E10s‎ | Status
Jump to navigation Jump to search
(Created page with "thumbnail|left<br> =e10s Update: '''March 29'''= ==Executive Summary== * Overall, the phase of the project we currently in is best described as Converg...")
 
 
(4 intermediate revisions by one other user not shown)
Line 1: Line 1:
[[File:Yellow-sm.jpg|thumbnail|left]]<br>
[[File:Yellow-sm.jpg|thumbnail|left]]<br>


=e10s Update: '''March 29'''=
=e10s Update: '''April 8'''=
==Executive Summary==
==Executive Summary==
* Overall, the phase of the project we currently in is best described as Convergence. For Firefox 47, the most likely scenario will be that we continue to conduct A/B Experiments in our beta channel until we can [https://bugzilla.mozilla.org/show_bug.cgi?id=1259512 lower our rate of main process, content, and Plugin crashes]. Our latest A/B experiment derived data set indicates we have work to do on stability before we're ready to ship to beta. We are, however, hitting our goals in many areas of our [https://wiki.mozilla.org/Electrolysis/Release_Criteria release criteria] such as Page Load, Startup/Shutdown time, and Scrolling. We still have work to do in areas such as UI Smoothness, Plugin Jank, Memory Usage, and GFX Performance but not to the level that would necessarily block beta. We have made excellent progress in our cadence of deploying A/B experiments and quickly ascertaining the data so we have hit a place where we are tightly coordinated and people are empowered with the right data to move on issues swiftly. Another important thing to note is that we are currently using the System Add-On to deploy our A/B experiments so we are testing the mechanism we designed to ship e10s rather than using experiment code. The current experiment (add-ons + ally = no, apz= yes) will end later this week, in time for Beta 7 and we will have updated results soon thereafter.  
* The biggest news this week is that Product has signed off on APZ being enabled when e10s is enabled. User-visible scrolling is much better with APZ than without for most cases.
* For Firefox 47, the most likely scenario is still that we continue to conduct A/B Experiments in our beta channel until we can [https://bugzilla.mozilla.org/show_bug.cgi?id=1259512 lower our rate of main process, content, and Plugin crashes]. IF, the crash rate lowers thanks to the stability fixes/uplifts that have happened as of late, we may be able to extend the experiment for a longer period of time. We will have the final plan by 04/15. Our goal is to expose as many of the eligible population for as many weeks as possible in order to collect a larger data set of telemetry and crash data and to hopefully see a higher return on bug filing related to issues people see. Our latest A/B experiment derived data set indicates we have work to do on stability before we're ready to ship to beta. We are, however, hitting our goals in many areas of our [https://wiki.mozilla.org/Electrolysis/Release_Criteria release criteria] such as jank and scrolling. We still have work to do in areas such as Startup/Shutdown time, UI Smoothness, Plugin Jank, Memory Usage, and GFX Performance but not to the level that would necessarily block beta.  


* The team has shifted to burning down M9's [https://wiki.mozilla.org/E10s/Status/m9]. M9 tickets need to hit zero before we're ready to do our cohort release for GA. M9 is focused on fixing issues derived from our Telemetry A/B experiments such as top crashes, regressions, and backlog items deemed to be blockers by product. We have hit a stage where a portion of the M9's are the responsibility of teams outside of the e10s team so we will be rolling those tickets into central triage managed by RelMan to help with load balance.
With the approach of Firefox 47 merging to Beta on April 25, quality and market readiness remain our central focus.
 
*With the approach of Firefox 47 merging to Beta on April 18, quality and market readiness remain our central focus.


==Why yellow? e10s is designated 'yellow' or 'at risk' because: ==
==Why yellow? e10s is designated 'yellow' or 'at risk' because: ==
Incoming data from our latest A/B experiment indicates we have more work to do on the stability side. Given that we are at week 4 in our dev cycle leading to Firefox 47 beta, we will likely continue to do experiments in Beta 47 vs. releasing to a % of our cohort population.
Incoming data from our latest A/B experiment indicates we have more work to do on the stability side. We also need to tighten up how we are operating by increasing fidelity of our stability data and and reducing latency to measure progress as much as we can.  
 
==Highlights and Accomplishments==
*Check out a summary of the engineering happenings  last week: https://wiki.mozilla.org/Electrolysis/Meetings/2016-03-24
*We've also made great progress fixing automated tests; by early next week we will have [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit#gid=1558085802 owners for all tests]
*WIP: manual test plan. Manual testing has been ongoing, this new test plan has new layout that is easier to follow along with risk commentary, etc. More to share soon on that.


==Next Steps==
==Next Steps==
* The main and content process crashes are well-documented. Next, we need to dig into the plugin crashes
* 04/13: e10s stability task force checkpoint
* Triage the [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1198187&hide_resolved=1 RC Blocker Meta dependency tree] to identify other tasks that need to happen prior to GA (besides M9s)
* 04/15: lock on final plan for Firefox 47 beta
* Now that we have a [https://docs.google.com/spreadsheets/d/101ra3HjqyufjP79JkGa4e0RyHPLVlJXS43rXtKuoJw4/edit#gid=0 track back schedule] which articulates what we need to accomplish in preparation for beta, we need to now create a precise roll-out plan. We should have something to share the week of April 7
* We will be reaching out to module owners to sign off on their efforts to fix automated tests in the next week


==Release Criteria==
==Release Criteria==
* [https://wiki.mozilla.org/Electrolysis/Release_Criteria e10s release criteria status]
* [https://wiki.mozilla.org/Electrolysis/Release_Criteria e10s release criteria status]
** We have telemetry data from phase 1 (e10s without APZ) of Beta 46 experiment.
** We have telemetry data from phase 1 (e10s without APZ) of Beta 46 experiment.
* Signed off:
** Jank!
** Scrolling!
** Red release criteria:
** Red release criteria:
*** MEMORY_TOTAL: 80% worse, but expected only 10%–25% worse.
*** MEMORY_TOTAL: 80% worse. We expected only 10%–25% worse, but 100% regression might be acceptable. TBD.
*** SLOW_SCRIPT_PAGE_COUNT: 0.47 per hour vs 0.36 per hour
*** SLOW_SCRIPT_PAGE_COUNT: 0.47 per hour vs 0.36 per hour
** Some release criteria are all green/passing. Looking to sign-off on those criteria soon.


==Add-Ons + WebExtensions==
==Add-Ons + WebExtensions==
* Working on add-on [https://docs.google.com/document/d/1v0tsxR2OS0eKY2mzDvadrkl-WTjNN-wr9qKU8rpJLa4/edit SDK issues] mostly related to e10s
* Working on add-on [https://docs.google.com/document/d/1v0tsxR2OS0eKY2mzDvadrkl-WTjNN-wr9qKU8rpJLa4/edit SDK issues] mostly related to e10s
* Outreach to add-on devs to help with e10s compat is ongoing
* Outreach to add-on devs to help with e10s compat is ongoing
* Next steps include simplifying testing for SV, and identifying a plan for shim handling going forward
* Next steps include getting a pref for testing without a shim


==Automated tests==
==Automated tests==

Latest revision as of 20:37, 8 April 2016

Yellow-sm.jpg


e10s Update: April 8

Executive Summary

  • The biggest news this week is that Product has signed off on APZ being enabled when e10s is enabled. User-visible scrolling is much better with APZ than without for most cases.
  • For Firefox 47, the most likely scenario is still that we continue to conduct A/B Experiments in our beta channel until we can lower our rate of main process, content, and Plugin crashes. IF, the crash rate lowers thanks to the stability fixes/uplifts that have happened as of late, we may be able to extend the experiment for a longer period of time. We will have the final plan by 04/15. Our goal is to expose as many of the eligible population for as many weeks as possible in order to collect a larger data set of telemetry and crash data and to hopefully see a higher return on bug filing related to issues people see. Our latest A/B experiment derived data set indicates we have work to do on stability before we're ready to ship to beta. We are, however, hitting our goals in many areas of our release criteria such as jank and scrolling. We still have work to do in areas such as Startup/Shutdown time, UI Smoothness, Plugin Jank, Memory Usage, and GFX Performance but not to the level that would necessarily block beta.

With the approach of Firefox 47 merging to Beta on April 25, quality and market readiness remain our central focus.

Why yellow? e10s is designated 'yellow' or 'at risk' because:

Incoming data from our latest A/B experiment indicates we have more work to do on the stability side. We also need to tighten up how we are operating by increasing fidelity of our stability data and and reducing latency to measure progress as much as we can.

Next Steps

  • 04/13: e10s stability task force checkpoint
  • 04/15: lock on final plan for Firefox 47 beta

Release Criteria

  • e10s release criteria status
    • We have telemetry data from phase 1 (e10s without APZ) of Beta 46 experiment.
  • Signed off:
    • Jank!
    • Scrolling!
    • Red release criteria:
      • MEMORY_TOTAL: 80% worse. We expected only 10%–25% worse, but 100% regression might be acceptable. TBD.
      • SLOW_SCRIPT_PAGE_COUNT: 0.47 per hour vs 0.36 per hour

Add-Ons + WebExtensions

  • Working on add-on SDK issues mostly related to e10s
  • Outreach to add-on devs to help with e10s compat is ongoing
  • Next steps include getting a pref for testing without a shim

Automated tests

  • Everything we can have enabled in production is enabled, everything else is enabled on a separate branch called 'Ash'.
  • Goal to have all tests be owned by 03/25 (we're close)
  • We would like each directory signed-off by test owners week of 04/07
  • We're tracking all the tests and who owns them, here
  • We are tracking tests that are running and passing on some platforms, but are disabled on others.From a feature coverage point of view, it's in pretty great shape. There's very little that is disabled both on opt *and* debug on a same platform (meaning that that test is not running on this platform). Look for the "Fully disabled in a platform" entries in this spreadsheet.

A11y

  • Proposed technical path to encourage a11y clients to communicate directly with the content process bug 1258839. dbolter is getting some first impressions from clients to see if this is a viable approach.
  • We're also looking into Windows touchscreen/a11y compat, see more information as the plan is taking shape, here.

Milestones

We are at the point where we are setting milestones weekly; given the complexity of deliverables, we are using a Trackback Schedule to keep everything organized.

Release Schedule

Date Trunk Aurora Beta Release
3-07 48 default 47 default 46 A/B Tests 45 off
4-18 49 default 48 default 47 A/B Tests, Possible Rollout (add-ons = ally = no) 46 off
6-07 50 default 49 default 48 Possible Rollout (add-ons + ally = no) 47 off
8-02 51 default 50 default 49 Possible Rollout (add-ons + ally = no) 48 Possible Rollout (add-ons + ally = no)
9-13 52 default 51 default 50 Possible Rollout (add-ons + ally = no) 49 Possible Rollout (add-ons + ally = no)
Retrieved from "https://wiki.mozilla.org/index.php?title=E10s/Status/April8&oldid=1126749"