Firefox/Channels/Postmortem/58

Firefox 58 release post mortem

============

Attendees: Selena, DVeditz, TomGrab, Liz, Marcia, RyanVM, Julien, Ritu, Arynx, Philipp.

Stability

• [marcia] Service worker crashes were high volume during 58 and the fixes landed very late in the cycle. At its peak we were hitting over 1000 crashes per day
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1416879 was filed on 11-13 well before the break and All Hands

• (Ritu) Github repo syncs remain a challenge
  • Discussions underway on how to mitigate this with Myk Melez, Julien etc.
  • activity stream, screenshots, anything with external libraries, Test Pilot,

• (Ritu) 58 has worsening of OOM crashes (even noticeable without retain display lists)
  • [selena] can we get the rates to compare?
  • (Ritu) Add Selena to the email thread
  • we may be switching to one content process for low memory devices/platforms to mitigate this (selena is following up)

• fennec slow startup (safe browsing related?)
  • apparently an issue in 57, then people woke up towards the end of 58 rc
  • we ended up fixing it in 58.0.1
  • How is the triage situation?
    • is anyone watching (who?) the fennec-tracking+ flag? (Yes: chris karlof, mike comella, snorp, susheel's team are triaging fennec issues regularly)
    • Selena mentioned that some Fennec issues are landing into Core::General.

• (Ritu) Chemspill process for 58.0.1
  • we are working to better document the chemspill process, from wennie, ritu, and dan
  • (liz) we went pretty fast - less than 24 hours between landing on m-r and releasing.
  • Daily stand-ups really helped (thanks Wennie)! ++ really helpful!
  • Secure slack channel helped as well, and the planning document + checklist.
  • QA coverage on the weekend is still TBD

• Thanks to Gerry for smooth, unexpected, last minute hand off of the release.