E10s/Status/Jan15: Difference between revisions

From MozillaWiki
< E10s‎ | Status
Jump to navigation Jump to search
 
(26 intermediate revisions by the same user not shown)
Line 5: Line 5:
==Why yellow? e10s is designated 'yellow' or 'at risk' because: ==
==Why yellow? e10s is designated 'yellow' or 'at risk' because: ==


#Our plan of record is solid at this point in terms of release criteria. The Product Brief is still in flight but doesn't pose risk to the project; we don't anticipate any major characteristics in terms of value proposition, etc. to shift from current understanding.  
#Our plan of record is solid at this point in terms of release criteria. The Product Brief is still in flight but doesn't pose risk to the project; we don't anticipate any major characteristics in terms of value proposition to shift from current understanding. At this point, that doc serves as an informational reference for cross-functional teams as they communicate "the why". The release criteria is locked down and we will be flexible in order to put quality as #1.
#Core team continues to working at a high velocity, we are not at zero M8's and we need to be by the first 2 weeks of Aurora which is Dec 28th. We don't want to take uplifts after that. There are enough M8's in flight to put this goal at risk but the team is super focused and we still have a fighting chance. [https://wiki.mozilla.org/E10s/Status/m8 See M8 Tracker].
#Core team continues to working at a high velocity, we are down to [https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&f1=cf_tracking_e10s&o1=equals&resolution=---&query_based_on=e10s-m8&o2=substring&query_format=advanced&f2=status_whiteboard&v1=m8%2B&v2=e10s-m8&known_name=e10s-m8&list_id=12803370  15 open M8's]. [https://wiki.mozilla.org/E10s/Status/m8 See M8 Tracker] to get granular details on what recent things have landed in Firefox 46, what's in review, and what we've chosen is safe enough to uplift for Firefox 45.


'''Here are the documents being worked on for the POR:'''
'''Decisions Made'''
* Over the holiday's the team met to go over risk vs. reward for 45 and we deemed Fx46 to be the next GA target for 50% of the population (who do not have add-ons or a11y installed). See the email announcing this, [http://pastebin.com/LbngSjd0 here].


'''POR Docs:'''
* [https://docs.google.com/document/d/1kdYHycZN1ETRcw3B3xH4fnNnAs4dzRh3loUX9x8UhZQ/edit Final Release Criteria]
* [https://docs.google.com/document/d/1kdYHycZN1ETRcw3B3xH4fnNnAs4dzRh3loUX9x8UhZQ/edit Final Release Criteria]
* [https://docs.google.com/document/d/1keO0SLZNaIyYVUX_g9cBBWjPBBf3d6nX8nbL5kB_EiA/edit Product Brief Draft]
* [https://docs.google.com/document/d/1keO0SLZNaIyYVUX_g9cBBWjPBBf3d6nX8nbL5kB_EiA/edit Product Brief Draft]
* Scroll below for Schedule
* Schedule


'''Decisions Made'''
==Validation for Stability and Performance==
*
*'''Beta 44 A/B Telemetry Experiment''' ended on Jan 3. We found that the population included in the experiment included users with Add-Ons and people with a11y enabled. Both of which we originally intended to be omitted from the experiment. Here are some conclusions as posted to Dev-Planning: http://pastebin.com/SUT7WPkq
** We need the BHR responsiveness measurement from the e10s content process in order to accurately compare e10s with non-e10s. {{Bug|1228437}}. 
**We also have recently uplifted a patch to fix the fact that we're over counting crash reports for the content process {{Bug|1227312}}.
**At this point, we are working on the design for the Beta 45 experiment and being certain to curate the aforementioned uplifts (long with with others) that fix telemetry measures before we deploy to 45.
**We are tracking various information, below:
** Telemetry E10S Measurement Meta {{Bug|1222849}}
** Performance Tracker Meta {{Bug|1198187}}
**Plan for measuring e10s performance + stability, [https://docs.google.com/document/d/1TyE0BehzYhii3qfmcrfjXlRJL64CcJk0B4Voup4Q0Pg/edit#, here].


==Validation for Stability and Performance==
*'''Beta 44 A/B Telemetry Experiment''' ended on Jan 11th. Currently validating data.
*'''Beta 45 A/B Telemetry Experiment''' goal is to deploy as of Firefox 45 Beta 1 on January 27th
*'''Beta 45 A/B Telemetry Experiment''' goal is to deploy as of Firefox 45 Beta 1 on January 27th
**'''Jan 19''': Experiment Design Review - send to Product for Approval
**'''Jan 19''': Experiment Design Review - send to Product for Approval
**'''Jan 21''': Hand off to QE for validation  
**'''Jan 21''': Hand off to QE for validation  
**''Jan 26'': QE Sign-Off
**'''Jan 26''': QE Sign-Off
**''Jan 27'': Deploy Experiment with Beta 1
**'''Jan 27''': Deploy Experiment with Beta 1
**''Feb 1'': Sanity check incoming data
**'''Feb 1''': Sanity check incoming data
**''Feb 5'': Run existing data set  
**'''Feb 5''': Run existing data set  
''From there, we will identify a weekly milestone pertaining to results to ensure results are as visible and actionable as possible''
''From there, we will identify a weekly milestone pertaining to results to ensure results are as visible and actionable as possible''
** Dec 12th e10s Telemetry probe triage: [https://docs.google.com/spreadsheets/d/1FIaGNv1q750mo9S7cJJY_8erbiF7j9snyX8Yd0diAbA/edit#gid=307269420 spreadsheet]
** {{bug|1182637}}: the BHR/event-loop-lag responsiveness regression in e10s has been confirmed first-hand with an extension that visualizes the BHR jank measurements: https://github.com/chutten/statuser
*** BHR reporting still needs to be improved to narrow down the origins: {{bug|1224374}}
** There also appears to be a stability regression https://bugzilla.mozilla.org/show_bug.cgi?id=1229104#c19
** There is also a known regression in e10s page-load speed in Talos & Telemetry data
** Telemetry E10S Measurement Meta {{Bug|1222849}}
** Performance Tracker Meta {{Bug|1198187}}


==Add-Ons + WebExtensions==
==Add-Ons + WebExtensions==
*Add-Ons 'Office Hours' to help developers with real-time support is ongoing: http://atsay.github.io/e10s_office_hours/
*Add-Ons 'Office Hours' to help developers with real-time support is ongoing and was quiet over the holidays: http://atsay.github.io/e10s_office_hours/.
*We need to make progress on creating a black list for incompatible add-ons {{bug|1233455}}. Next steps include working with add-ons team to identify a DRI + connecting them telemetry data to get started.
*We have a meeting next week with Product to review Use cases then UX/dev priorities/realities for roll-out selection capabilities for e10s with regards to add-ons (allow/deny lists).
** Source truth for add-ons for priority and status on compatibility can be found, [https://docs.google.com/spreadsheets/d/1KoJoMD4DMZ2zODnxFtzhrPHVpAH3GxTIr0fKN65dlCE/edit#gid=0 here].
*We need to make progress on creating a black list for incompatible add-ons bug 1233455. The first phase of the Telemetry A/B experiment launching on the 27th will be focused on curating data to help create this black list.
**'''We will NOT block on slow add-on watcher''' for any GA plans.
**Source of truth for add-ons for priority and status on compatibility can be found, here.
**'''We will NOT block on WebExtensions'' for any GA plans
 
**'''We DO need to figure out what the product requirements are''', if any for UX relating to Add-ons in an e10s world in general. This also coincides with the Shield project. UX, and Perf are engaged and we're starting conversations about user interaction and API readiness.  
==Automated tests==
==Automated tests==
*We're primarily missing 3 suites on Windows: mochitest-plain, reftest, and crashtest. Work needs to be done to green up each of these suites, so capacity issues are not currently the blocker.
*We're primarily missing 3 suites on Windows: mochitest-plain, reftest, and crashtest. Work needs to be done to green up each of these suites, so capacity issues are not currently the blocker.
* A-Team is currently working on disabling enough tests on mochitest-plain to get them enabled, but there is a shutdown leak on debug which is difficult to trace to a particular test. We may get those running by the end of the year, leaving only crashtests and reftests.
* Making good progress on Win7 opt. He will update the tests status wiki page. Seeing a lot of issues on Win7 debug
*Felipe is jumping in to help Blake, jgriffin, and Jim green up tests:
* Next request is to get win7 tests running on aurora
https://public.etherpad-mozilla.org/p/e10stests-taskforce
* RyanVM will look at doing manual testing of the password manager since those tests are currently disabled and may be hard to get running.
*Here is a good test-by-test reference:
https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit#gid=1777180571


==A11y==
==A11y==
** Wiki to track work is, [https://wiki.mozilla.org/Accessibility/Electrolysis2 here].
*Wiki to track work is, [https://wiki.mozilla.org/Accessibility/Electrolysis2 here].
** '''Decision made to not block going to GA''' on A11y. We are ramping up to help be sure we do everything we can to get a good level of parity.
*Currently focusing on test remoting of a11y calls from main process.
** We are planning on getting e10s team + Firefox functional teams together in Orlando to continue the envangelize pruning and greening up of tests on part of module owners.
*Good progress is being made on a11y work in general; I will be tracking more closely once we're done with M8.
 
==Release Schedule==
==Release Schedule==
Currently Proposed Schedule is to enable e10s by default for Users without Add-Ons for 45.0 and target a larger population for 46.0. As mentioned above, we'll do a go/no-go on the 28th of Dec.
Current Plan of Record is to Target 50% of the population who do not have add-ons installed or a11y features enabled for Firefox 46.
 


{| class="wikitable"
{| class="wikitable"
Line 88: Line 88:
| {{done|}} || '''Nov 9''' || '''Aurora 43.0 Telemetry Experiment Results'''|| Performance Team
| {{done|}} || '''Nov 9''' || '''Aurora 43.0 Telemetry Experiment Results'''|| Performance Team
|-
|-
| {{mrisk}} || '''Nov 17''' || '''Beta 43.0 Telemetry Experiment Launches''' || Performance Team
| {{Done|}} || '''Dec 4''' || '''Final Plan of Record Approved'''|| Product/Engineering
|-
|-
| {{mrisk}}  || '''Nov 30 || '''GTB Beta 7 Telemetry Experiment Final Uplifts/Testing'''|| Performance Team/RelMan
| {{Done|}} || '''Dec 14'''|| '''Merge Day - 44.0 B1 GTB''' || RelMan
|-
| {{mok}} || '''Dec 4''' || '''Final Plan of Record Approved'''|| Product/Engineering
|-
| {{mok}} || '''Dec 14'''|| '''Merge Day - 44.0 B1 GTB''' || RelMan
|-
|-
| {{Done|}} || '''Dec 15''' || '''43.0 GA - e10s Disabled''' || RelMan
| {{Done|}} || '''Dec 15''' || '''43.0 GA - e10s Disabled''' || RelMan
Line 102: Line 98:
| {{Done|}} || '''Dec 28'''|| '''Firefox 45 Go/No-Go''' || Decision Makers
| {{Done|}} || '''Dec 28'''|| '''Firefox 45 Go/No-Go''' || Decision Makers
|-
|-
| {{mok}} || '''Jan 11''' || '''44.0 Beta - e10s A/B Testing disabled'''|| Performance/RelMan
| {{Done|}} || '''Jan 3''' || '''44.0 Beta - e10s A/B Testing disabled'''|| Performance/RelMan
|-
| {{mok}} || '''Jan 27''' || '''45.0 Beta - e10s A/B Test Deployed''' || Performance/RelMan
|-
|-
|}
|}

Latest revision as of 20:14, 29 January 2016

Yellow-sm.jpg


e10s Weekly Update: Jan 15

Why yellow? e10s is designated 'yellow' or 'at risk' because:

  1. Our plan of record is solid at this point in terms of release criteria. The Product Brief is still in flight but doesn't pose risk to the project; we don't anticipate any major characteristics in terms of value proposition to shift from current understanding. At this point, that doc serves as an informational reference for cross-functional teams as they communicate "the why". The release criteria is locked down and we will be flexible in order to put quality as #1.
  2. Core team continues to working at a high velocity, we are down to 15 open M8's. See M8 Tracker to get granular details on what recent things have landed in Firefox 46, what's in review, and what we've chosen is safe enough to uplift for Firefox 45.

Decisions Made

  • Over the holiday's the team met to go over risk vs. reward for 45 and we deemed Fx46 to be the next GA target for 50% of the population (who do not have add-ons or a11y installed). See the email announcing this, here.

POR Docs:

Validation for Stability and Performance

  • Beta 44 A/B Telemetry Experiment ended on Jan 3. We found that the population included in the experiment included users with Add-Ons and people with a11y enabled. Both of which we originally intended to be omitted from the experiment. Here are some conclusions as posted to Dev-Planning: http://pastebin.com/SUT7WPkq
    • We need the BHR responsiveness measurement from the e10s content process in order to accurately compare e10s with non-e10s. bug 1228437.
    • We also have recently uplifted a patch to fix the fact that we're over counting crash reports for the content process bug 1227312.
    • At this point, we are working on the design for the Beta 45 experiment and being certain to curate the aforementioned uplifts (long with with others) that fix telemetry measures before we deploy to 45.
    • We are tracking various information, below:
    • Telemetry E10S Measurement Meta bug 1222849
    • Performance Tracker Meta bug 1198187
    • Plan for measuring e10s performance + stability, here.
  • Beta 45 A/B Telemetry Experiment goal is to deploy as of Firefox 45 Beta 1 on January 27th
    • Jan 19: Experiment Design Review - send to Product for Approval
    • Jan 21: Hand off to QE for validation
    • Jan 26: QE Sign-Off
    • Jan 27: Deploy Experiment with Beta 1
    • Feb 1: Sanity check incoming data
    • Feb 5: Run existing data set

From there, we will identify a weekly milestone pertaining to results to ensure results are as visible and actionable as possible

Add-Ons + WebExtensions

  • Add-Ons 'Office Hours' to help developers with real-time support is ongoing and was quiet over the holidays: http://atsay.github.io/e10s_office_hours/.
  • We have a meeting next week with Product to review Use cases then UX/dev priorities/realities for roll-out selection capabilities for e10s with regards to add-ons (allow/deny lists).
  • We need to make progress on creating a black list for incompatible add-ons bug 1233455. The first phase of the Telemetry A/B experiment launching on the 27th will be focused on curating data to help create this black list.
    • Source of truth for add-ons for priority and status on compatibility can be found, here.

Automated tests

  • We're primarily missing 3 suites on Windows: mochitest-plain, reftest, and crashtest. Work needs to be done to green up each of these suites, so capacity issues are not currently the blocker.
  • Making good progress on Win7 opt. He will update the tests status wiki page. Seeing a lot of issues on Win7 debug
  • Next request is to get win7 tests running on aurora
  • RyanVM will look at doing manual testing of the password manager since those tests are currently disabled and may be hard to get running.
  • Here is a good test-by-test reference:

https://docs.google.com/spreadsheets/d/10UeyRoiWV2HjkWwAU51HXyXAV7YLi4BjDm55mr5Xv6c/edit#gid=1777180571

A11y

  • Wiki to track work is, here.
  • Currently focusing on test remoting of a11y calls from main process.
  • Good progress is being made on a11y work in general; I will be tracking more closely once we're done with M8.

Release Schedule

Current Plan of Record is to Target 50% of the population who do not have add-ons installed or a11y features enabled for Firefox 46.

Date Trunk Aurora Beta Release
3-30 40 default (working on m5) 39 off 38 off 37 off
5-11 41 default (working on m6) 40 prompt 39 off 38 off
6-29 42 default (working on m7/m8) 41 prompt 40 off 39 off
8-10 43 default (working on m8) 42 default 41 off 40 off
9-21 44 default (release criteria driven) 43 default 42 TBD 41 off
11-02 45 default (release criteria driven) 44 default 43 A/B 42 off
12-14 46 default (release criteria driven) 45 default 44 TBD 43 off
1-25 47 default (release criteria driven) 46 default 45 TBD 44 off
3-07 48 default (release criteria driven) 47 default 46 TBD 45 (tbd)

Milestones

Currently working on interim Milestones to track Telemetry Experiments

Status Date Milestone DRI
[DONE] Nov 9 Aurora 43.0 Telemetry Experiment Results Performance Team
[DONE] Dec 4 Final Plan of Record Approved Product/Engineering
[DONE] Dec 14 Merge Day - 44.0 B1 GTB RelMan
[DONE] Dec 15 43.0 GA - e10s Disabled RelMan
[DONE] Dec 20 44.0 Beta 1 - e10s A/B Testing enabled Performance Team/RelMan
[DONE] Dec 28 Firefox 45 Go/No-Go Decision Makers
[DONE] Jan 3 44.0 Beta - e10s A/B Testing disabled Performance/RelMan
[ON TRACK] Jan 27 45.0 Beta - e10s A/B Test Deployed Performance/RelMan
Retrieved from "https://wiki.mozilla.org/index.php?title=E10s/Status/Jan15&oldid=1114970"