Post-mortem for the Firefox and Fennec 57 release Release Coordination vidyo room Tuesday, Dec 5th, 2017
Attendees: Sheila, Erin, Kevin, Marcia, Julien, Callek, RyanVM, TomGrab, Nicole, Ritu
List issues (good and bad) What went well? What didn't? What could we do better?
(syl) First run page super slow on old systems (3 fps) - bug 1417888
- Wasn't tested on slow systems
- not just old systems, the bug report says 100% of the linux population + about 20% of the windows population
- Erin/Nicole can help follow-up, what kind of QA support do we need here? Can we add testing on slow systems to the existing test plan?
- This was tested by SV as part of the onboarding flow: https://public.etherpad-mozilla.org/p/Onboarding_Experience
- would need to add additional testing needed.
- We might incorporate an update plan into the 58/59 time frame
- (Ritu) Too many uplifts during Beta57 cycle - 576
- 56 had 365 uplifts
- 55 had 320
- Great release quality despite the huge code churn
- Was justified due to the scope of Quantum release!
- Good to see more self-awareness
- (Ritu) Nightly and Beta milestones helped teams plan better
- Soft code freeze milestone was appreciated by eng teams
- Add a WNP/first run related milestone
- (Ritu) Each RC uplift was reviewed for second opinion by eng managers/component owners
- may not be a scaleable process
- (Ritu) More aggressive tracking via blocking flag
- Good support from all on reviewing, investigating, fixing blockers
- (Ritu) Amazing efforts to keep untriaged bug backlog within acceptable limits
- may not be a scalable process beyond 57
- Emma's dashboard will help us monitor this going forward
- (Ritu) QA team's feature doc which contained list of features, status, blockers was very useful
- (Ritu) Fennec triage could be improved
- It's hard to make quick progress on blocking bugs
- Difficult to help get second opinion on RC uplifts
- Hard to get developer time to fix issues on the frontend
- Romania, Taipei timezone cycle can be a bit slow, needing nudging from PST owners
- need a resource that is a decision maker for bugs filed (crashes, blocking, etc.)
- Fennec team could benefit from aligning better with Firefox processes
- Fennec triage/component ownership needs to be shared with Firefox team
- awareness of who knows what? resource list.
- for now, ping snorp for platform, nevin for front end. dont know? defer to snorp
- (Ritu) Awesome effort in planning and getting WNP ready during RC week
- In the past this was a huge challenge
- Hoping to get this new process become a repeatable thing for 58/59
- (Ritu) Releng team's infra code freeze helped keep things smooth and predictable
- Callek offered to ping catlee/jlund on whether this needs to happen going forward
- This CF should have been better communicated out to release-drivers mailing list
- (marcia) Engineering lead for the release worked well. Jim Mathies was very responsive and worked to keep the Wednesday triage under control. We should keep a lead in place for each future release.
- Do we need a Fennec release eng lead as well?
- (marcia) Some challenges around tracking Fennec features since they don't use the same process as Desktop - https://docs.google.com/spreadsheets/d/1Rn-F3Kg_1_VznIxxXkAGGL8mVMSAdamZZI4f1O2r8HA/edit#gid=152116571.
- (kbrosnan) laser focus on 57 caused a defocus on 58/59 maybe there should have been dedicated people who were focused on those releases (outside release management)
- (marcia) To add to that, we need to be careful we don't carry nightly regressions into beta. In Fennec we had https://bugzilla.mozilla.org/show_bug.cgi?id=1413500 carried from Nightly into beta
- (Ryan) Part of that issue was late landings due to not communicating out the milestones widely enough
- More EPM support on future release planning and tracking
- Cross functional milestones needed from relman team for future releases and keep the focus going
- (sheila) - Dot release process - decision on dot release and timing really needs to be a joint decision from Product and Engineering. Fine for Release Management to make a recommendation. Document summarizing issues and status was great.
- Relman team should continue reviewing dot release plan, fixes, user impact with senior management
- (sheila) - More formal review process post release. Communication of non-decisions.
- Relman team can build more visibility on issues that were reviewed before we decide unthrottling
- (Ryan) Early rollout to DevEdition went well++
- This was done to speed up the beta staged rollout
- This helps weed out issues sooner and ship desktop beta builds faster
- (kbrosnan) a lot of confusion about how Google Play does throttled releases
- new installs are not guaranteed to get the most recent version. they have the same % chance to get the new version as previous installs
- [julien] should we push the release candidate out earlier at low rollout %age?++
- Fennec staged rollout at 10% on launch day was deemed too slow
- Product recommended going at 25%. This helps us mimic the Desktop release throttling
- Before the dot release is pushed out, the previous release should be at 100% staged rollout.
- [elan] Shield
- We can cover this in my 57 retro but putting ideas here are welcome:
- Need a single dashboard!