Releases/Firefox 3.5.2/Post Mortem

This is the agenda and notes from the post mortem for Firefox 3.0.13 and 3.5.2 that took place on August 6 at 1pm PDT.

Agenda

Please add your issues to the relevant section below.

Topics:

Development/Security

1. MFSAs online ahead of time? Put them on mozilla.com in the release notes as well?
2. It would have been nice if we could have started builds prior to Wednesday afternoon. How can we keep from having embargoed bugs in the future?

QA

1. 3.0.13 was ready for beta early Sat morning. Decision did not come until later.
   1. waited 10pm friday until 4pm saturday
2. Really great collaboration with some of the developers in bug verification

Build

1. Tripped on l10n makefile change. needed to redo l10n repacks/signing/updates.
2. New signing worked nicely; needed it for all the re-signing
3. This time, we pulled the 3.0.12 -> 3.5.1 major update before shipping at ss' request. Why? Should we continue to do that?
4. rationale could be shared pre or post fact in release-drivers [juanb]
5. did xulrunner builds at same time
6. didnt do partner builds automatically

Web work

1. Make sure both product & marketing teams are clear (and agree) on content objectives for what's new page.

IT

1. Denial of service on ocsp.globalsign.net caused by everyone trying to validate our EV cert on addons.mozilla.org (bug 508408)

Other

1. Several anti-virus products detected the Windows 3.0.13 and 3.5.2 installers as infected (false positive) (bug 508012)

Notes

• in general, development went okay
• need to work on advisories getting out earlier / on time
  • some confusion because of the expediency of this release
  • get bsterne mozilla.org check-in access (bsterne; ss will vouch)
• starting builds because we were under embargo hurt us
  • in this case, closed source won because they could check in without anyone knowing
  • need to work with security researchers to make this not an option on the table (dveditz/lucas)

• lots of great help from all over QA including the community
  • a set of trusted QA community helped test this release so we could ship fast
  • all around good QA work
  • developers also helped a bunch with verifications
• need better idea of driver hand off
  • QA was ready to go to bed with 3.0.13 on Friday night
  • no decision until Saturday afternoon
  • wasn't clear which driver to ping to get it done (ss)

• l10n makefile change messed up repacks
  • need to get such changes better tested before they land
  • automated tests for such changes should be required (ss)
• faster signing worked great
  • had to resign because of the l10n issues
  • went *really* fast, which was great
• pulling major update snippets at the last minute wasn't good
  • manual task, prone to possible mistakes
  • wasn't clear why it was happening
  • needs to get communicated better what's happening and why via r-d
  • wasn't communicated post-release either
  • a follow-up discussion should occur on this (ss)
• XULRunner builds happened at the same time
• partner builds didn't happen at the same time
  • automation for partner builds not ready yet
  • hoping to have ready soon (joduinn)

• pages we pushed this time were more-or-less fine
  • some work being done in the future to improve the messaging (jslater, beltzner)
  • be sure to get sign off for these after changes are made
• kubla was slow, like usual
  • loading the page with so many locales takes too long
  • follow up bugs to file and push on
  • need to get bugs fixed and pushed on (ss)

• DoS against EV provider caused AMO to "go down" for Firefox users only
  • DoS likely caused by us and our users...
  • working with GlobalSign to ensure they're ready for our users
  • clearly wasn't the case this time, but we're following up (mrz)

• several antivirus vendors claimed we had a virus (false positive)
  • caused by a change in compression for installers
  • part of new signing process
  • Tomcat followed up with vendors to get fixes out
  • worked okay this time, but need to notify them next time
  • maybe not take these changes during firedrill releases