E10s/Status/March25: Difference between revisions

From MozillaWiki
< E10s‎ | Status
Jump to navigation Jump to search
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[File:Yellow-sm.jpg|thumbnail|left]]<br>
[[File:Yellow-sm.jpg|thumbnail|left]]<br>


=e10s Weekly Update: '''March 25'''=
=e10s Update: '''March 29'''=
==Executive Summary==
==Executive Summary==
* Our current Plan of record to enable e10s as of Firefox . This initial GA release will be a cohort release targeting 100% of users '''without''' Add-Ons, users '''without''' a11y features enabled, and, the non-RTL language versions of Firefox. Talk about release criteria here.  
* Our current Plan of record is to continue to conduct A/B Experiments in our beta channel until we can [https://bugzilla.mozilla.org/show_bug.cgi?id=1259512 lower our rate of main process, content, and Plugin crashes]. Our latest A/B experiment derived data set indicates we have work to do on stability before we're ready to ship to beta. We are, however, hitting our goals in many areas of our [https://wiki.mozilla.org/Electrolysis/Release_Criteria release criteria] such as Page Load, Startup/Shutdown time, and Scrolling. We still have work to do in areas such as UI Smoothness, Plugin Jank, Memory Usage, and GFX Performance but not to the level that would necessarily block beta. We have made excellent progress in our cadence of deploying A/B experiments and quickly ascertaining the data so we have hit a place where we are tightly coordinated and people are empowered with the right data to move on issues swiftly. Another important thing to note is that we are currently using the System Add-On to deploy our A/B experiments so we are testing the mechanism we designed to ship e10s rather than using experiment code.


* The has shifted to burning down M9's [https://wiki.mozilla.org/E10s/Status/m9]. M9 tickets block riding the trains for the above populations for GA. M9 is focused on fixing issues derived from our Telemetry A/B experiments such as top crashes, regressions, and backlog items deemed to be blockers by product.  
* The team has shifted to burning down M9's [https://wiki.mozilla.org/E10s/Status/m9]. M9 tickets need to hit zero before we're ready to do our cohort release for GA. M9 is focused on fixing issues derived from our Telemetry A/B experiments such as top crashes, regressions, and backlog items deemed to be blockers by product. We have hit a stage where a portion of the M9's are the responsibility of teams outside of the e10s team so we will be rolling those tickets into central triage managed by RelMan to help with load balance.  


*We have hid stride with Telemetry data from the Firefox 45 Beta experiment to help us evaluate readiness for ; initial data indicates we are in good shape in terms of jank {{Bug|1223780}}. Though, we need to continue to confirm this data is valid, especially during the second phase of Beta 45 A/B testing. We also are getting help from platform product to audit our current set of [https://docs.google.com/document/d/1kdYHycZN1ETRcw3B3xH4fnNnAs4dzRh3loUX9x8UhZQ/edit performance criteria] and help us get a plan in place to manually validate this data in order to confirm that positive data is equivalent to a positive user experience in the real world.
*With the approach of Firefox 47 merging to Beta on April 18, market readiness remains our central focus. Overall, the phase of the project we current is best described as Convergence.
 
*With the approach of Firefox 46 merging to Beta on March 7, market readiness becomes a central focus. We're kicking off an e10s Cross-Functional weekly meeting as of Friday, Feb 19.


==Why yellow? e10s is designated 'yellow' or 'at risk' because: ==
==Why yellow? e10s is designated 'yellow' or 'at risk' because: ==
Although our release criteria has been set since December, we're finding there are enough questions on part of key teams such as release management and product management on the platform side so we're taking the time to fine-tune to be certain everyone is very confident in the POR. Additionally, although the A/B Experiment logistics and incoming data thus far is better than it has been, it's still really new. We need to be 100% certain it's valid. Additionally, we want to ensure that the improvements we may be seeing in data holds weight with users so we are spinning up efforts to make this happen (more details next week).  
Incoming data from our latest A/B experiment indicates we have more work to do on the stability side. Given that we are at week 4 in our dev cycle leading to Firefox 47 beta, we will likely continue to do experiments in Beta 47 vs. releasing to a % of our  cohort population.


Also, we need to flesh our a plan for our add-ons population. Although, accomodating users with add-ons is not a blocker for the initial release, it's important we have a plan because we'd like to have code land in 47, 48+ so we need to be ready. We hare getting closer, efforts on add-on compat have been steady for awhile, but we don't have a final plan of record for the client/UX part of the equation.  
==Highlights and Accomplishments==
*Check out a summary of the engineering happenings  last week: https://wiki.mozilla.org/Electrolysis/Meetings/2016-03-24


==Highlights and Accomplishments==
*We've also made great progress fixing automated tests; by early next week we will have [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit#gid=1558085802 owners for all tests]
We recently landed [https://wiki.mozilla.org/E10s/Status/m8#Landed_in_47_but_not_uplifted_to_46 the following M8's in Fx47]. Work is complete to deploy the second phase of the Telemetry A/B Experiments testing. We have some great analysis on incoming stability data from the Beta 45 experiment so far {{bug|1246180}}. Check out a summary of the engineering happenings this last week, [https://wiki.mozilla.org/Electrolysis/Meetings/2016-02-11 here].


With the help of the quality team/SV, we have been conducting a backlog scrub of all the [https://bugzilla.mozilla.org/buglist.cgi?priority=--&v4=needinfo&keywords=meta%2C%20qawanted&keywords_type=nowords&f1=cf_tracking_e10s&o3=nowordssubstr&v3=1013364%2C%20984139%2C%20905436%2C%201063169%2C%20993639%2C%201004745%2C%201029143%2C%20875871%2C%20915546%2C%201207306%2C%201058354&o1=equals&resolution=---&o2=nowordssubstr&f4=flagtypes.name&query_format=advanced&f3=blocked&f2=short_desc&o4=notsubstring&v1=%2B&v2=intermittent%2C%20test%2C%20addon%2C%20add-on&list_id=12851479] issues being tracked for e10s. We've made quick progress to be sure there are not blockers or a combination small things that we would consider a negative trend and block to be found. We also wanted to be sure that we could close as many open items as possible. [https://docs.google.com/spreadsheets/d/1fEqqT05Gi3dAKbwsotmpXPO9nR19Ra4ecmCPoHqyAMg/edit#gid=995994685] See the report from SV for details.
*WIP: manual test plan. Manual testing has been ongoing, this new test plan has new layout that is easier to follow along with risk commentary, etc. More to share soon on that.


==Next Steps==
==Next Steps==
As we continue to converge towards an eye for Firefox 46 beta, we'll need to start closely tracking M9's. As previously mentioned, focusing on our add-ons population and also being sure that we validate our measurements with real world proof is important. More details on how that will happen will be available soon. Solidifying that we have consensus and confidence in our roll-out plan is also an important next step and will be a major topic in the cross-functional kicking off next week.
* The main and content process crashes are well-documented. Next, we need to dig into the plugin crashes
 
* Triage the [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1198187&hide_resolved=1 RC Blocker Meta dependency tree] to identify the the precise tasks that need to happen prior to GA (besides M9s)
==Validation for Stability and Performance==
* Now that we have a track back schedule, and we tightly coordinated
*Phase 2 of the Beta 45 experiment is ready for deployment, see details, here http://pastebin.com/cNKAnW2d.
** Track back schedule captures transitioning to beta.  
* We have solid data on stability and are taking action, see more [https://wiki.mozilla.org/Electrolysis/Meetings/2016-02-11#poiru_.28e10s_stability.29] here.
** Roll-Out plan (transitioning from beta to release)
* Jank results are positive thus far though we need to be sure to validate in the real world. See prelim details, [https://wiki.mozilla.org/Electrolysis/Meetings/2016-02-11#chutten_.28e10s_responsiveness.29] here.
* Information about add-ons which are causing jank is also incoming and we need to continue to ensure it's being validated. More on this, soon.


*Upcoming Milestones:
==Release Criteria==
**'''Feb 12''': Phase 2 starts
*  
**'''Feb 25''': Testing concludes for 45
''Look here for more milestones tracking actions around incoming data/any adjustments we need to make''
 
== Memory Usage==
* See report on Memory Usage which is mostly positive, [http://www.erahm.org/?p=22 here].


==Add-Ons + WebExtensions==
==Add-Ons + WebExtensions==
*Devs are looking for open bugs in webextensions API
*Devs are looking for open bugs in webextensions API
* Working to converge SDK issues: https://docs.google.com/document/d/1v0tsxR2OS0eKY2mzDvadrkl-WTjNN-wr9qKU8rpJLa4/edit
* Working to converge SDK issues: https://docs.google.com/document/d/1v0tsxR2OS0eKY2mzDvadrkl-WTjNN-wr9qKU8rpJLa4/edit
*(Carry Over)We have a meeting next week with Product to review Use cases then UX/dev priorities/realities for roll-out selection capabilities for e10s with regards to add-ons (allow/deny lists).
 
*(Carry Over)We need to make progress on creating a white/black/grey list for incompatible add-ons bug 1233455. The first phase of the Telemetry A/B experiment launching on the 27th will be focused on curating data to help create this black list.
*Source of truth for add-ons for priority and status on compatibility can be found, here.


==Automated tests==
==Automated tests==
* Module owners need to be sure their tests are e10s compliant.
* Everything we can have enabled in production is enabled, everything else is enabled on a separate branch called 'Ash'.
*Browser Chrome + and Mochi Test Plan (blocks). Need to get to 90%.
* Goal to have all tests be owned by 03/25 (we're close)
*Need to make sure that M-C and M-A are in lock step
* We would like each directory signed-off by test owners week of 04/07
*Goal to get all tests signed-off in M/C by Feb 29th
*We're tracking all the tests and who owns them, [https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit#gid=1777180571 here]
*More specific status available first week of Feb
*We are tracking tests that are running and passing on some platforms, but are disabled on others.From a feature coverage point of view, it's in pretty great shape. There's very little that is disabled both on opt *and* debug on a same platform (meaning that that test is not running on this platform). Look for the "Fully disabled in a platform" entries in this [https://docs.google.com/spreadsheets/d/1oBnRct8FuTnAoVfbE08m10UHi1_5kOFaOPglJ6ittWc/edit#gid=0 spreadsheet].
*Here is a good test-by-test reference:
 
https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit#gid=1777180571


==A11y==
==A11y==
Line 58: Line 46:
*Currently focusing on test remoting of a11y calls from main process.
*Currently focusing on test remoting of a11y calls from main process.
*Good progress is being made on a11y work in general; I will be tracking more closely once we're done with M8.
*Good progress is being made on a11y work in general; I will be tracking more closely once we're done with M8.
==Milestones==
We are at the point where we are setting milestones weekly; given the complexity of deliverables, we are using a [https://docs.google.com/spreadsheets/d/101ra3HjqyufjP79JkGa4e0RyHPLVlJXS43rXtKuoJw4/edit#gid=0%20 Trackback Schedule] to keep everything organized.


==Release Schedule==
==Release Schedule==
Line 66: Line 57:
! Date !! Trunk !! Aurora !! Beta !! Release
! Date !! Trunk !! Aurora !! Beta !! Release
|-
|-
| 3-30 || 40 default (working on m5) || 39 off || 38 off || 37 off
| 3-07 || 48 default (release criteria driven) || 47 default || 46 A/B || 45 off
|-
|-
| 5-11 || 41 default (working on m6) || 40 prompt || 39 off || 38 off
| 4-18 || 49 default (release criteria driven || 48 default || 47 (addons & a11y = off) || 46 off
|-
| 6-29 || 42 default (working on m7/m8) || 41 prompt|| 40 off || 39 off
|-
| 8-10 || 43 default (working on m8) || 42 default || 41 off || 40 off
|-
| 9-21 || 44 default (release criteria driven) || 43 default || 42 off || 41 off
|-
| 11-02 || 45 default (release criteria driven) || 44 default || 43 off || 42 off
|-
| 12-14 || 46 default (release criteria driven) || 45 default || 44 A/B || 43 off
|-
| 1-25 || 47 default (release criteria driven) || 46 default || 45 A/B || 44 off
|-
| 3-07 || 48 default (release criteria driven) || 47 default || 46 50-50 (addons & a11y = off) || 45 off
|-
| 4-18 || 49 default (release criteria driven || 48 default || 47 50-50 (addons & a11y = off) || 46 on (addons & a11y = off)
|}
|}
==Milestones==
We are at the point where we are setting milestones weekly; given the complexity of deliverables, we are using a [https://docs.google.com/spreadsheets/d/101ra3HjqyufjP79JkGa4e0RyHPLVlJXS43rXtKuoJw4/edit#gid=0%20 Trackback Schedule] to keep everything organized.

Latest revision as of 18:19, 29 March 2016

Yellow-sm.jpg


e10s Update: March 29

Executive Summary

  • Our current Plan of record is to continue to conduct A/B Experiments in our beta channel until we can lower our rate of main process, content, and Plugin crashes. Our latest A/B experiment derived data set indicates we have work to do on stability before we're ready to ship to beta. We are, however, hitting our goals in many areas of our release criteria such as Page Load, Startup/Shutdown time, and Scrolling. We still have work to do in areas such as UI Smoothness, Plugin Jank, Memory Usage, and GFX Performance but not to the level that would necessarily block beta. We have made excellent progress in our cadence of deploying A/B experiments and quickly ascertaining the data so we have hit a place where we are tightly coordinated and people are empowered with the right data to move on issues swiftly. Another important thing to note is that we are currently using the System Add-On to deploy our A/B experiments so we are testing the mechanism we designed to ship e10s rather than using experiment code.
  • The team has shifted to burning down M9's [1]. M9 tickets need to hit zero before we're ready to do our cohort release for GA. M9 is focused on fixing issues derived from our Telemetry A/B experiments such as top crashes, regressions, and backlog items deemed to be blockers by product. We have hit a stage where a portion of the M9's are the responsibility of teams outside of the e10s team so we will be rolling those tickets into central triage managed by RelMan to help with load balance.
  • With the approach of Firefox 47 merging to Beta on April 18, market readiness remains our central focus. Overall, the phase of the project we current is best described as Convergence.

Why yellow? e10s is designated 'yellow' or 'at risk' because:

Incoming data from our latest A/B experiment indicates we have more work to do on the stability side. Given that we are at week 4 in our dev cycle leading to Firefox 47 beta, we will likely continue to do experiments in Beta 47 vs. releasing to a % of our cohort population.

Highlights and Accomplishments

  • We've also made great progress fixing automated tests; by early next week we will have owners for all tests
  • WIP: manual test plan. Manual testing has been ongoing, this new test plan has new layout that is easier to follow along with risk commentary, etc. More to share soon on that.

Next Steps

  • The main and content process crashes are well-documented. Next, we need to dig into the plugin crashes
  • Triage the RC Blocker Meta dependency tree to identify the the precise tasks that need to happen prior to GA (besides M9s)
  • Now that we have a track back schedule, and we tightly coordinated
    • Track back schedule captures transitioning to beta.
    • Roll-Out plan (transitioning from beta to release)

Release Criteria

Add-Ons + WebExtensions


Automated tests

  • Everything we can have enabled in production is enabled, everything else is enabled on a separate branch called 'Ash'.
  • Goal to have all tests be owned by 03/25 (we're close)
  • We would like each directory signed-off by test owners week of 04/07
  • We're tracking all the tests and who owns them, here
  • We are tracking tests that are running and passing on some platforms, but are disabled on others.From a feature coverage point of view, it's in pretty great shape. There's very little that is disabled both on opt *and* debug on a same platform (meaning that that test is not running on this platform). Look for the "Fully disabled in a platform" entries in this spreadsheet.


A11y

  • Wiki to track work is, here.
  • Currently focusing on test remoting of a11y calls from main process.
  • Good progress is being made on a11y work in general; I will be tracking more closely once we're done with M8.

Milestones

We are at the point where we are setting milestones weekly; given the complexity of deliverables, we are using a Trackback Schedule to keep everything organized.

Release Schedule

Current Plan of Record is to Target 100% of the population who do not have add-ons installed or a11y features enabled for Firefox 46.

Date Trunk Aurora Beta Release
3-07 48 default (release criteria driven) 47 default 46 A/B 45 off
4-18 49 default (release criteria driven 48 default 47 (addons & a11y = off) 46 off
Retrieved from "https://wiki.mozilla.org/index.php?title=E10s/Status/March25&oldid=1124743"