Firefox 40 Post-mortem

(lmandel) beta5 build failed and we failed to ship this beta
(lmandel) beta6 build was hung, cpearce noticed
(lmandel) beta4 mobile failed to update on the Play store, required poking Google for help
- (ritu) This may happen from time to time.
  - (sylvestre) Do we know what happened? Our update gets queued in their review process and their standard turn around time is ~24 hrs.
- (sylvestre) I will have some releng permissions to retrigger jobs

(lmandel) Difficult to manage the release as Sylvestre was on PTO and Liz was out sick. Should have better relman coverage once Ritu is fully up-to-speed.
- (sylvestre) Thanks again! (lizzard) Yes thanks!!

(bhearsum) Consistency in release urgency. If we're burning the midnight oil to get releases built, we shouldn't be sitting on them.
- Eg: we sat on 39.0.3/38.1.1 chemspills for 2-3h before shipping (waiting on advisories, I think)
  - (lizzard) Yes - that was partly that dan and al were traveling, partly that I overestimated how much time we would take to qualify the release -- we finished earlier than I predicted, and the sec advisories weren't ready.

   todo:   document how to publish the sec advisories. (pmac, jgmize, www channel)

- - Advisory could be managed by others
  - (hwine) might have a new request for moc notification of actual release
  - (sylvestre) Release management should communicate more when the situation is complex
- Having a schedule before or right after a release starts may help set expectations more clearly
(bhearsum) How to know if we're doing a what's new page? We were told no in https://bugzilla.mozilla.org/show_bug.cgi?id=1181182#c14, then yes, but it turned out we were just doing a firstrun (no RelEng work needed)
- Related, verification of WNP (what's new page). Verification for ESR didn't happen until Tuesday morning, and it delayed ship (we had an issue on Mac)

TO BE TALKED ABOUT ON THURSDAY:

(bhearsum) should we hold off publishing security advisories until after we unthrottle? Not all users are offered updates when we first ship (because of the 25% rate), and if we have an immediate point release it can sometimes be a week before many users get an update even though we've posted advisories
(bhearsum) tiny reminder: "snippets" don't exist anymore!
- (sylvestre) doc updated ( https://wiki.mozilla.org/Release_Management/Release_Day )

(hwine - developer edition) some issues when 42a2 went live with download page links not updating. With increasing complexities of coordinating multiple artifacts, maybe time to add a "are we really shipped yet" checklist?
- this specific instance was we pushed updaters live on a Friday Aug 14, in #releng I got flagged that the "install" links still showed 41a2 at https://www.mozilla.org/en-US/firefox/developer/ -- iirc everyone checked the updates before leaving, no one checked the installer.
- (lmandel) +1

(Sylvestre) 38.2.1 ESR - patches landed in the thunderbird relbranch (GECKO3820_2015081310_RELBRANCH)
- We should probably rename it to GECKO3820_2015081310_RELBRANCH_TB to avoid confusion in the future
- This branch had some other fixes like https://hg.mozilla.org/releases/mozilla-esr38/rev/b4dfee2e3da8 https://hg.mozilla.org/releases/mozilla-esr38/rev/326a41e65147 which does not match the ESR criteria
  - (ritu) Was not approved by RelMan. Does that matter before landing to moz-esr38?
- we used GECKO3820esr_2015080613_RELBRANCH instead

we have now updated some of the docs for this.

(bhearsum) - notify moc@mozilla.com of unusual releases (ie, point releases)
- (sylvestre) we should cc them when doing a dot release.

(lizzard) socorro (crash-stats) didn't add 40.0.3 to its dropdown menu for tracking crash rates in the graphs (lars is investigating)

(kglazko) https://drive.google.com/file/d/0B5BraW6Ez5yvSnVvT0o2TDNQSzg/view Survey results, better planning, more discussion about backporting refactoring/fixes, good collaboration between devs and testing this time, quick uplifts.

Firefox/Channels/Postmortem/40