Firefox 53 Post-mortem

Notes for a post-mortem for 53.0

Action items - Liz to follow up on WNP - Erin, Liz? to follow up with Fennec team follow up (triage, QE followthrough, planning to ship new features) - Release process checklist for dot releases

• Both Desktop and Fennec releases skipped staged rollout and went unthrottled
• For desktop, since we manage the update channel, we were able to go down from 100% to 25% update rate. (liz's fault from the email)
• For Fennec, once staged rollout is set to 100%, there is no going back. (Problem with infrastructure for pushing the apk, now fixed)

• With both 52 and 53, hours after Fennec go-live, we found release blocking issues. What can do we better?

• Continuing confusion over in-product links that go to SUMO, not localized, 404 to users following links from Firefox.

• Serious perf issues because of certificate transparency from 52 unrecognized till a week before 53 release. https://bugzilla.mozilla.org/show_bug.cgi?id=1353216

• partner build issues at last minute. (this will drive a dot release)

• Bug 1352681 - SharedArrayBuffers - don't ship it (consensus achieved a week before release. ) Are we going to be more certain for 54? How do we know?

We are not shipping SAB for 54, this is pushed to 55 to align with Chrome 60

• last minute (post beta-to-release merge) realization we had to disable TLS 1.3 https://bugzilla.mozilla.org/show_bug.cgi?id=1355144

We had a lot of major last minute fixes

• Fennec triage did not escalate bug 1354038
• Another fennec issue, not marked with affected versions, no tracking request, no way for anyone to tell the fix should be uplifted. "tracking-fennec" flag only. https://bugzilla.mozilla.org/show_bug.cgi?id=1332731
  • [RyanVM] Added tracking-fennec flag to the burndown queries tracked by release bosses to catch this in the future

• Bug 1329294 - Windows content temp dir not in LocalLow for parent on new profile (Do we know any better that this is fixed for 54?

• Bug 1346392 - server that tries ntlm over h2 needs an h1 fallback

• Bug 1342433 - onclick changes shouldn't recreate the tree (backout for bug 1349847) An issue that caused many last minute regressions

• Bug 1355534 - Revert to longer insecure login field contextual warning string Users are confused about new warning and think they have a virus. LANDED for rc2

• This isn't the sort of uplift request we want to see a week before release https://bugzilla.mozilla.org/show_bug.cgi?id=1347439

Bug 1356292 - Installing add-ons from non-whitelisted sites no longer gives the option to allow the install if the addon load is redirected Another last minute blocker. Bug 1354294 - Crashes (null Deref) in ScriptedProxyHandler::construct with lastpass addon Bug 1354736 - Setup WNP for Firefox 53.0 release A few days before release! Unclear if testing is set up, or done, or issues resolved. locales? This did not pass QE testing *on release day* and we shipped without it activated. The project owner for this should have been working with QE and the www team as well as releng, to make sure this was ready. It was not ready and instead, relman/www/releng were scrambling around on Wednesday morning before the release. WNP was called out in the last post-mortem for causing last minute confusion for the release. (bug 1341100)

Releng postmortem on release day https://github.com/mozilla/releasewarrior/blob/master/releases/POSTMORTEMS/2017-04-19.md

• Release notes were not ready for Fennec 53.0.1 (until 3 days later), we need ship-it v2
  • we tend to miss steps for dot releases in general. Need better checklists/use of them. Releases happen fast and several at once (usually at least 3, sometimes more)
  • https://wiki.mozilla.org/Release_Management/Release_Days#Dot_releases_and_chemspills
  • checklist for what happened for 53.0.2, https://docs.google.com/spreadsheets/d/1aJCVx5xjqZ_gu7TEOf8ZUsF81mLPnUHlTfCf1iVrbtE/edit?userstoinvite=sledru@mozilla.com&ts=590cd2bb#gid=0