Releases/Firefox 13/Test Plan/Post-mortem

From MozillaWiki
Jump to: navigation, search

Firefox 13 QA Post-mortem

When: Tuesday, June 12, 2012 @ 08:00am Pacific
Dial-in: x262
Who: ashughes, tracy, Softvision 

Notes: [1]

Areas of Success

  • Weekend coverage enabled us to ship RC/ESR on time in spite of extra Beta and smaller team
  • Nearly 100% of reproducible bug fixes verified
  • Reducing unnecessary feature testing allowed for more regression testing and fix verifications
  • Great coordination of status/work between RelMan, QA, and Softvision in spite of time difference
  • Prompt and active coordination of qa-wanted/regressionwindow-wanted bugs

Areas for Improvement

  • team re-org resulted in not giving Web QA enough notice about Web Content testing requirements on release day
  • Fixing late-breaking plug-in top-crashes resulted in a domino effect of regressions
  • Third-party releases close to Firefox release resulted in potential chemspill scenario (Hotmail, Skype, Avast, Adobe Flash, Trusteer)
  • 3.6 EOL was untested on Mac OSX 10.4 due to platform accessibility and awareness of support commitments
  • UNCONFIRMED bug triage was not initiated until the end of the Beta cycle

Remaining Questions

  • How can we all work together to get in front of top-crashes?
  • How can QA be better informed of upcoming third-party releases? A calendar was suggested
  • How can QA assist & inform third-party QA processes?
  • How do we determine cost:benefit of QA supporting the unsupported?
  • Can QA automate EOL verifications (installation, startup, and updates) and is it worth it?

Action Items

Test Strategy

  • Feature sign-off once in Beta, verify fixes as landed, re-sign-off if churn
  • Maintain a high level of coordination on bugs between RelMan, QA, and Softvision
  • Continue to reduce smoketesting to give more time for regression testing and bug triage
  • Include an "unsupported platform" spotcheck into the release sign-off process
  • Include a "pre-release software" spotcheck into the release sign-off process
  • Maintain an EOL platform pool that we can dust-off as needed
  • Institute a weekly Beta triage meeting with Softvision
  • Develop a bug triage strategy to drive focus and priorities
  • Work with community triagers to get them focused and build communities around them
  • Create a weekly query for IRC pop-ins to work from (feed into testday where appropriate)
  • Ensure every Aurora gets at least one "features" testday before Beta merge


  • Week before release, ensure weekend coverage is lined up
  • Monday before release, ensure everyone knows to be online 7:30am the next day
  • Ensure QA representation at Vendor Engagement meetings