Rapid Betas

The following documents the changes made to the above processes for adopting more frequent Beta builds. As of 2013-06-25 (Firefox 23) we are adopting twice-weekly Betas.

Timeline

Sign-off Criteria

Beta 1

• Features signed off by feature owners as "beta ready"
• Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
• >80% Critical bug fixes verified (as identified by QA lead)
• Manual regression tests in Moztrap uncovers 0 blocking regressions
  • Tests: Breakpad, Mac OSX 10.8 code-signing, NVidia 3D Vision, and Rel-noted features smoketests
  • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
• Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
  • betatest and releasetest channels tested before sign-off
  • beta channel tested after Release Engineering pushes the snippets
• Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters

Beta 2-9

• Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
• Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
  • betatest and releasetest channels tested before sign-off
  • beta channel tested after Release Engineering pushes the snippets
• Manual testing to occur daily outside of the sign-off process:
  • Week 2
    • Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests
    • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.7, Ubuntu Linux 64-bit
  • Week 3
    • Tests: Breakpad, Mac OSX 32-bit mode, and Web compatibility exploratory smoketests
    • Platforms: Windows Vista, Windows 7 32-bit, Mac OSX 10.6, Ubuntu Linux 32-bit
  • Week 4
    • Tests: Breakpad, Software update for root-owned files, and Add-on compatibility exploratory smoketests
    • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Week 5
    • Tests: Breakpad, What's New page for software update, and Plug-in compatibility exploratory smoketests
    • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.7, Ubuntu Linux 32-bit
• Bug fix verifications to occur daily outside of the sign-off process, supported by triage on Wednesdays & Fridays
• Testdays should be organized during Week 2 and Week 5 incorporating bug triage, regression testing, and feature testing
• Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) consulted weekly to uncover any red flags from early adopters

Beta 10

• Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
• >90% critical bug fixes verified (as identified by QA lead)
• Manual regression tests in Moztrap uncovers 0 blocking regressions
  • Tests: Breakpad, NVidia 3D Vision, Mac OSX 10.5 & Windows 2000 de-support, Web compatibility exploratory smoketests
  • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.6, Ubuntu Linux 64-bit
• Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
  • betatest and releasetest channels tested before sign-off
  • beta channel tested after Release Engineering pushes the snippets
• Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters

Release Candidate

• Features signed off by feature owners as "release ready"
• Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
• All critical bug fixes verified (as identified by QA lead)
• Manual regression tests in Moztrap uncovers 0 blocking regressions
  • Tests: Breakpad, Partner repack update, and Windows share compatibility smoketests
  • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
• Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
  • betatest tested before conditional sign-off
  • releasetest tested aftet Release Engineering pushes snippets to mirrors
  • release channel tested after Release Engineering pushes snippets to release
• Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters
• Important: Due to an intermittent build-time stability issue related to Radeon drivers we need to do twin builds
  • Once build1 is available run functional and betatest update automation
  • Once automation is complete, send an email to Release Engineering to generate build2
  • Once build2 is available run functional and betatest update automation on it
  • build1 will be pushed to releasetest on Monday if it's not found to have the stability issue
  • build2 will be pushed to releasetest on Monday if build1 is is found to have the stability
  • All manual testing and bug fix verifications should still be tested against build1