Schedule / Location / Call Information

• Thursday, 2010-11-11 @ 1:00 pm PST
• In Very Good Very Mighty (3rd floor)
• 650-903-0800 x92 Conf# 8605 (US/INTL)
• 1-800-707-2533 (pin 369) Conf# 8605 (US)
• join irc.mozilla.org #post-mortem for back channel

Overview

• 3.6.12 and 3.5.15 were chemspills to address an active exploit

Things that went right

• Quick response on off-hours
• Worked well with AV vendors and Google to mitigate the attack
• RelEng had IT start staging for mirrors differently, speeding up the process
• Virtually no mechanics hiccups/problems during build, qualification, and release
• < 48 hours to release

Things that went wrong

• A patch reviewers were away at Facebook, delaying the review and subsequent final patch
• Christian pushed the build to mirrors before QA had signed off
  • He made the decsision the trade-offs were worth it
  • He didn't discuss this with QA

Suggested Improvements

• Christian will discuss all process changes over the release-drivers mailing list
• Christian will create a process workflow diagram with feedback from assorted teams
• QA will get new hardware to support update testing multiple releases in parallel
  • Should give ~50% speed up in update qualification