Notes for Firefox 68 release post mortem
10am PT August 20, 2019 Zoom Meeting ID: 266 532 573 IRC: #release-drivers / slack: #release-coordination
Julien, Philipp, Overholt, Ritu, PLawless, lizzard, RyanVM, dveditz, Tom, Thomas, Yan, mikedeboer,tania
- nightly: March 18 to May 20 (9 weeks)
- 1 extra week due to postponing 67.0 after armagadd-on
- beta: May 20 to July 8 (7 weeks)
- Uplifts to beta: 333 (https://bugzilla.mozilla.org/buglist.cgi?list_id=14803565&chfield=flagtypes.name&chfieldfrom=2019-05-20&chfieldvalue=approval-mozilla-beta%2B&chfieldto=2019-07-01&query_format=advanced )
- Uplifts to RC: 22 (https://bugzilla.mozilla.org/buglist.cgi?chfield=flagtypes.name&chfieldfrom=2019-07-01&chfieldto=2019-07-08&f1=flagtypes.name&o1=substring&v1=approval-mozilla-release%2B )
- Fixed in 68: patches from 5353 bugs (4416 in 67)
- PI Requests received on time (prior to deadline) : 19 (Desktop)
- Late PI Request: 9, Accepted: 5, Rejected: 4, 1 request went through Exception approval process (Desktop QA)
- 9 QA requests were cancelled/moved to Future Release to prioritize Trailhead work
- % of PI requests (Desktop) with initial documentation received after deadline : 19.05% (slightly better than 67)
- % of Desktop Features with Late Code Delivery: 31.82% (slightly better than 67)
- This means features that landed after FC
- Total number of bugs found by QA:
- Nightly :75
- Beta :48
- some testing was canceled to prioritize trailhead testing and dot releases
New release process
- no mandatory testing in nightly
- exception process for missed deadlines or feature uplifts
- [julien] I heard some concern about the time it takes to get an exception approved adding to an already-delayed feature
- most features asked for Nightly testing
- [julien] I have an action item to make the exception process doc more discoverable from release process docs
- link to it from the cross functional
- We should also make the exception requests sit in the same folder
- https://docs.google.com/document/d/1jOTngDV_WWJGelTz4205ux4i5sAWfYNILHYmLLmHYeQ/edit -> Exception request document
- Relman team to figure out how to educate more feature owners on how and when to file an exception request
- List of exceptionr requests for 68 -
- https://docs.google.com/document/d/1UiGS3HQaDnszP_Vh840_fYiK8aA3khp4yvh-OwKsuZM/edit - Add-ons exception request document for Fx68
- Release process exception request from Secure Proxy (Firefox 68)
- Release Process Exception request (Pin Firefox shortcut on taskbar for new Windows 10 installs)
- AI: jcristau to find the number of exception requests we got (e.g. for late code delivery)
What went well
- fennec on esr repo
- Moving to JIRA seems successful as there was no email request received for Fx68 (desktop qa) requests.
- Response from Eng team in reviewing QA artefacts was quick, feedback received on time for most of the features, apart from Desktop this is applicable for Add-ons QA and Mobile QA
- Introduction of the Technical documentation guideline seems to be helpful as in some cases we were able to identify design gaps and report them beforehand.
- much confusion about the status of the feature going into beta
- Priority” values of the PI tickets - Most (or all?) of the PI requests were of medium priority caused more work while prioritizing them during Trailhead
- involving product management in prioritization could help
- majority of the features went through Nightly Testing
- is it slightly weird that above we're talking about the number of uplifts to beta being high but we're trying to only test in beta? Yes
- number of beta uplifts wasn't unusually high
- we clarified that we aren't necessarily trying to reduce Nightly testing
- Still difficult to plan resources each cycle for new features based on the data initially available for them/ minimal information added to PI ticket.
- Communication with Engineering didn't go well on Installer related features, scope changes and unclear answers to the clarifications resulted some misunderstandings.
- Mobile QA - Three features were moved to next Fx release due to the fact that the implementation wasn't done on time and QA didn't have enough time to test.
- Add-Ons QA - Patches have been uplifted late, causing the testing process to fall behind the planned timeline required for extensive testing.
- For the Quantum bar, although there was a reasonable amount of documentation, for the address bar in general, search, telemetry, db structure, etc. that documentation is lacking: even on the info that exists, you need the right person to point you to the right place.
- stability issues in the beginning of beta were fixed
- disabled in rc2 for perf concern (Bug 1562942)
- Do we need better Go/NoGo criteria for this feature?
- do we have telemetry for that? could the issue have been caught earlier?
- we are working on automated tests for "existing profiles" and "profiles with addons installed" so it'll be better in the future
- a number of last minute uplifts (https://mzl.la/2KI6zG1)
- [M DeBoer] Late uplifts in RC were driven by user feedback and not something easily identifable early on.
- [M DeBoer] Documentation concerns stem from the age and complexity of this project. Will ensure that Bridget and/or Michaels contact info are obvious and attached to all reporting sources and requests. PI request, Trello, tech docs, etc.
- should we have said no?
- should it have been deferred?
- the late uplifts weren't necessarily blockers, but they broke some power users' workflows; we don't know how many
- bridget and mikedeboer are points of contact on search
- a number of last minute uplifts (https://mzl.la/2KI6zG1)
- WebRender rollout
- confusion right before launch (bug 1562966)
- should features that plan a gradual rollout on release get the same treatment on beta, so we make sure prefs are in the expected state and we can beta-test the rollout itself?
- very old bug (Bug 1558299)
- incident over July 4th weekend
- ended up fixing in 68 rc3
- needed a regression fix in 68.0.2
- new ESR branch
- delayed build1 gtb
- patches to disable features not ready in time
- tooling issues (shipit)
- AMO issue on July 4 (https://github.com/mozilla/addons-server/issues/11793 / https://bugzilla.mozilla.org/show_bug.cgi?id=1563651)
- couldn't sign langpacks for 68.0 rc2
- delayed QA signoffs
- Mission Control didn't detect macOS spike https://bugzilla.mozilla.org/show_bug.cgi?id=1571833
- Some questions raised about reports not submitting and https://bugzilla.mozilla.org/show_bug.cgi?id=1566855 was filed, but some of the SME's were on PTO