Firefox/Channels/Postmortem/60

Notes for 60 post mortem

10:30am PDT Tuesday May 29

Vidyo channel: release coordination

IRC: #release-drivers

Some stats

• around 360 uplifts to beta
• uplifts during beta: https://mzl.la/2H0OqhE
• uplifts during RC week: https://mzl.la/2IXhw3b
• much higher than 59 (~ +50%), partially driven by 60 being ESR
  • (Ritu) This summary is great!

What's new page

• overall less of a struggle than it was in the past
• confusion about the "id" locale this time around
• wnp wasn't live on www.mozilla.org until release week
  • Possible reasons:
    • Content wasn't finalized
    • github pull request didn't happen in time, potential mitigation: talk to m.o folks for help in future?

Late uplifts

• yet another attempt to fix fennec crashes on S8 (ultimately unsuccessful) (https://bugzilla.mozilla.org/1455662 )
  • Consistent reproduction has been an issue, making it difficult to verify the effectiveness of the patch prior to shipping
  • Bug 1460989 open to track current efforts to improve the crash rate
• proxyConfig webextensions API (https://bugzilla.mozilla.org/1455755 and https://bugzilla.mozilla.org/1455705 )
• FxA de-sync on android (https://bugzilla.mozilla.org/show_bug.cgi?id=1456487 )
• broken preferences in ja-JP-mac repack (https://bugzilla.mozilla.org/1450656)
  • Potential mitigation: Can we detect such changes and alert for additional testing? Ritu to follow up with Flod.
  • Tom G wondering whether (and how - we don't have many international people in QA ) QA can help here.
  • Kevin mentioned some interest in dropping this locale. Follow up with Henri.

60.0.1 drivers

• driven by Cycle Collector regression from nightly 60 (https://bugzilla.mozilla.org/1449033 )
  • bug 1449033 was filed in March, had STR, but wasn't properly triaged (mccr8)
  • jonco said he thinks it could have been caught had it been in the right component (not that it wasn't at least partly his fault for not triaging it (he feels bad))
  • jonco also notes that a quantification of the regression would have been nice (like, "before this patch, CC times didn't spike beyond x and now they're hitting 2x" or something)
  • manual monitoring of telemetry was probably required here ...
  • Potential mitigation: Planning to add those two telemetry probes (CYCLE_COLLECTOR_MAX_PAUSE, CYCLE_COLLECTOR_FULL) to PI monitoring and Mission Control dashboard
  • went unnoticed until reported more widely post release
  • our telemetry alerting story seems to be lacking though
• extra fun with Kaspersky Software Updater
  • https://bugzilla.mozilla.org/1436662: untranslated uninstaller in 59
    • caught during 59 beta, not prioritized at the time
    • fix landed in nightly 61
    • conversation ongoing with Kaspersky to better understand what they're doing

60.0.1 notable ride alongs

• https://bugzilla.mozilla.org/1459362: disable WebVR on mac
  • broken in nightly 60 in February and in a 59 beta 7 uplift (https://bugzilla.mozilla.org/show_bug.cgi?id=1424709#c50)
  • reported at https://bugzilla.mozilla.org/1454204 mid April
• https://bugzilla.mozilla.org/1458906: pocket sponsored content not disabled immediately when opting out from settings
  • Impact was limited to 0.01% of release population
• https://bugzilla.mozilla.org/1461590: language packs broken for locales with a country code
  • lack of ci for langpacks. we rely on community to report issues
  • Was there adequate testing done on this? Let's take this as feedback to releng and see if there's low-cost-high-returns mitigations that can be put in place.
  • Maybe do more targeted testing of l10n work on test days when we have more global testers