QA/Desktop Firefox/Walkthroughs/Rapid Betas

From MozillaWiki
Jump to navigation Jump to search

Rapid Betas

The following documents the changes made to the above processes for adopting more frequent Beta builds. As of 2013-06-25 (Firefox 23) we are adopting twice-weekly Betas.

Timeline

Week Monday Tuesday Wednesday Thursday Friday
1 Beta migration Beta 1 built Beta 1 QA sign-off
2 Beta 2 built Beta 2 QA sign-off Beta 3 built Beta 3 QA sign-off
3 Beta 4 built Beta 4 QA sign-off Beta 5 built Beta 5 QA sign-off
4 Beta 6 built Beta 6 QA sign-off Beta 7 built Beta 7 QA sign-off
5 Beta 8 built Beta 8 QA sign-off Beta 9 built Beta 9 QA sign-off
6 Beta 10 built Beta 10 QA sign-off
Release migration
RC go-to-build
7 RC QA pre-sign-off
Mirror push 		RC QA sign-off
Live push
Live QA sign-off

Sign-off Criteria

Beta 1

  • Features signed off by feature owners as "beta ready"
  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • >80% Critical bug fixes verified (as identified by QA lead)
  • Manual regression tests in Moztrap uncovers 0 blocking regressions
    • Tests: Breakpad, Mac OSX 10.8 code-signing, NVidia 3D Vision, and Rel-noted features smoketests
    • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest and releasetest channels tested before sign-off
    • beta channel tested after Release Engineering pushes the snippets
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters

Beta 2-9

  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest and releasetest channels tested before sign-off
    • beta channel tested after Release Engineering pushes the snippets
  • Manual testing to occur daily outside of the sign-off process:
    • Week 2
      • Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests
      • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.7, Ubuntu Linux 64-bit
    • Week 3
      • Tests: Breakpad, Mac OSX 32-bit mode, and Web compatibility exploratory smoketests
      • Platforms: Windows Vista, Windows 7 32-bit, Mac OSX 10.6, Ubuntu Linux 32-bit
    • Week 4
      • Tests: Breakpad, Software update for root-owned files, and Add-on compatibility exploratory smoketests
      • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
    • Week 5
      • Tests: Breakpad, What's New page for software update, and Plug-in compatibility exploratory smoketests
      • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.7, Ubuntu Linux 32-bit
  • Bug fix verifications to occur daily outside of the sign-off process, supported by triage on Wednesdays & Fridays
  • Testdays should be organized during Week 2 and Week 5 incorporating bug triage, regression testing, and feature testing
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) consulted weekly to uncover any red flags from early adopters

Beta 10

  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • >90% critical bug fixes verified (as identified by QA lead)
  • Manual regression tests in Moztrap uncovers 0 blocking regressions
    • Tests: Breakpad, NVidia 3D Vision, Mac OSX 10.5 & Windows 2000 de-support, Web compatibility exploratory smoketests
    • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.6, Ubuntu Linux 64-bit
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest and releasetest channels tested before sign-off
    • beta channel tested after Release Engineering pushes the snippets
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters

Release Candidate

  • Features signed off by feature owners as "release ready"
  • Functional automation uncovers 0 regressions, include 10 locales spread across all platforms
  • All critical bug fixes verified (as identified by QA lead)
  • Manual regression tests in Moztrap uncovers 0 blocking regressions
    • Tests: Breakpad, Partner repack update, and Windows share compatibility smoketests
    • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Update automation uncovers 0 failures, include 4 previous versions and 10 locales spread across all platforms
    • betatest tested before conditional sign-off
    • releasetest tested aftet Release Engineering pushes snippets to mirrors
    • release channel tested after Release Engineering pushes snippets to release
  • Stability (Marcia/KaiRo) and Support (Matt_G/Tyler) uncovers no red flags from early adopters
  • Important: Due to an intermittent build-time stability issue related to Radeon drivers we need to do twin builds
    • Once build1 is available run functional and betatest update automation
    • Once automation is complete, send an email to Release Engineering to generate build2
    • Once build2 is available run functional and betatest update automation on it
    • build1 will be pushed to releasetest on Monday if it's not found to have the stability issue
    • build2 will be pushed to releasetest on Monday if build1 is is found to have the stability
    • All manual testing and bug fix verifications should still be tested against build1
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Desktop_Firefox/Walkthroughs/Rapid_Betas&oldid=965218"