Firefox 13 QA Post-mortem

When: Tuesday, June 12, 2012 @ 08:00am Pacific
Where: irc.mozilla.org#qa
Dial-in: x262
Who: ashughes, tracy, Softvision

Notes: [1]

Areas of Success

Weekend coverage enabled us to ship RC/ESR on time in spite of extra Beta and smaller team
Nearly 100% of reproducible bug fixes verified
Reducing unnecessary feature testing allowed for more regression testing and fix verifications
Great coordination of status/work between RelMan, QA, and Softvision in spite of time difference
Prompt and active coordination of qa-wanted/regressionwindow-wanted bugs

Mozilla.org team re-org resulted in not giving Web QA enough notice about Web Content testing requirements on release day
Fixing late-breaking plug-in top-crashes resulted in a domino effect of regressions
Third-party releases close to Firefox release resulted in potential chemspill scenario (Hotmail, Skype, Avast, Adobe Flash, Trusteer)
3.6 EOL was untested on Mac OSX 10.4 due to platform accessibility and awareness of support commitments
UNCONFIRMED bug triage was not initiated until the end of the Beta cycle

How can we all work together to get in front of top-crashes?
How can QA be better informed of upcoming third-party releases? A calendar was suggested
How can QA assist & inform third-party QA processes?
How do we determine cost:benefit of QA supporting the unsupported?
Can QA automate EOL verifications (installation, startup, and updates) and is it worth it?

Test Strategy

Feature sign-off once in Beta, verify fixes as landed, re-sign-off if churn
Maintain a high level of coordination on bugs between RelMan, QA, and Softvision
Continue to reduce smoketesting to give more time for regression testing and bug triage
Include an "unsupported platform" spotcheck into the release sign-off process
Include a "pre-release software" spotcheck into the release sign-off process
Maintain an EOL platform pool that we can dust-off as needed
Institute a weekly Beta triage meeting with Softvision
Develop a bug triage strategy to drive focus and priorities
Work with community triagers to get them focused and build communities around them
Create a weekly query for IRC pop-ins to work from (feed into testday where appropriate)
Ensure every Aurora gets at least one "features" testday before Beta merge

Staffing