Releases/Firefox 33/Test Plan

From MozillaWiki
Jump to navigation Jump to search

Schedule

Please consult the Rapid Release Calendar for more information.

  • 2014-06-09: Firefox 33 arrives on Nightly
  • 2014-07-21: Firefox 33 merges to Aurora
  • 2014-07-25: Firefox 33 Aurora updates enabled
  • 2014-09-02: Firefox 33 merges to Beta
  • 2014-09-04: Firefox 33 Beta 1 released

Meetings

Meeting Purpose When Where
Crashkill Raise stability concerns with the Crashkill team 10:00 PST on Monday Vidyo: Stability, IRC: #crashkill
Channel Status Raise quality concerns with the Release Management team 10:00 PST on Tuesday & Thursday Vidyo: ReleaseCoordination, IRC: #planning

Checklist

Priority Check Description Frequency Last Checked
1 Stability Review the stability reports to identify new/rising crash signatures Daily
2 Pushlog Review the pushlog to identify fixed bugs needing verification Daily
3 QA Needed Bugs Review and assign qawanted bugs for investigation At least twice a week
4 Unconfirmed Bugs Triage and test unconfirmed bug reports At least once a week
5 Fixed Bugs Flag fixed bugs for verification At least once a week
6 Verification Needed Bugs Test verifyme bugs to confirm fixed bugs are fixed At least once a week
7 Reported Bugs Document and escalated reported bugs as necessary Every Monday
8 Features Ensure features are owned and on track for release Once a week
9 Automation Ensure automation bugs are owned and on track for resolution Once a week

Features

Please consult the feature testing walkthrough for more information.

Search Provider Top Suggestions(Petruta Rasa)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Places (Andrei Vaida)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


OpenH264 media plugin (Alexandra Lucinet)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


OMTC (Alexandra Lucinet)

https://wiki.mozilla.org/Platform/GFX/OffMainThreadCompositing

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Home/NewTab Search Suggestions (Petruta Rasa)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

  • Explosive Crashes - ensure new and explosive crashes have actionable bug reports which are in progress
  • Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
  • Untriaged Bugs - ensure bugs are actionable and in the correct component

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Bugs

QA Wanted Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Unconfirmed Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Fixed Bugs

Review the pushlog and place each Desktop Firefox bug in one of the following categories:

  • A) Tested by the QA Team --> add the verifyme keyword to the bug and set the status-firefox33 flag to fixed
  • B) Tested by a new volunteer --> add the [good first verify] whiteboard tag
  • C) Tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
  • D) Not tested at all --> add the [qa-] whiteboard tag

Tips:

  • If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
  • Be sure to set the status-firefox33:fixed flag
  • If the bug includes tests from a developer you can set in-testsuite+
  • Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures

Reported Bugs

Ensure these bugs remain on track to resolve and are verified when fixed.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Bug Fix Verifications

  • Review the bugs below to identify a bug you want to test
  • Make sure you understand the bug before you begin testing it
  • Test on a previously known broken build to make sure you can reproduce the bug
  • Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
  • Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
  • Reopen the bug if it still reproduces

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Regression Testing

  • Moztrap testrun to cover past regressions, at-risk components, and feature integration tests
  • WebRTC regression test
  • PDF.js and printing
  • Migration on first-run

Milestones

Aurora Migration

Pre-sign-off Checks:

  • {{|}} Automation - document all Mozmill test failures and memory usage regressions
  • {{|}} Features - all scoped features are signed off as ready for Aurora by owners
  • {{|}} Bugs - document and review any reported bugs and issues, nominate them for tracking if serious
  • {{|}} Stability - review crash stats before sign-off to raise any red flags
  • {{|}} L10n - contact :Pike to see if there are any known localization issues

Post-sign-off Checks:

  • {{|}} Updates - previous versions of Aurora update to the new version (check after updates live)
  • {{|}} MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel

Automation

Testrun Reports
Report Test Failures
Add-ons
Functional
Remote
L10n
Updates
Endurance
Memory Usage Report
Platform Min Memory Usage Max Memory Usage Avg Memory Usage Net Compared to Firefox 32
Windows 32-bit
Mac OS X
Linux 32-bit
Linux 64-bit

Memory regression being tracked/investigated in: <bug number>

Features

Feature Owner Aurora Sign-off Status Potential Blockers
OMTC Alexandra Lucinet
Web RTC - bundle in SDP handling

Updates

Platform Source Build Target Build Result
Windows
Mac OS X
Linux


Reported Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Betas

Known Mozmill Failures

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


All open Mozmill test case fail bugs

Beta 1

B1 Build Information

B1 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • New features pass basic smoketests
  • 0 beta blocking regression, stability, or support issues discovered

B1 Testing

  • Manual
    • [DONE] Smoketest new features
      • [DONE] Verify Bug 1059798 - activate Loop and have 1:1 calls with friends (basic dogfooding)
    • [DONE] Verify no HSB builds were generated - PASS
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [DONE] Updates on releasetest channel (report) - PASS
    • [DONE] Updates on beta channel (report) - PASS

B1 Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 2

B2 Build Information

B2 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B2 Testing

B2 Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 3

B3 Build Information

B3 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B3 Testing

B3 Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 4

B4 Build Information

B4 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B4 Testing

B4 Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 5

B5 Build Information

B5 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B5 Testing

  • Manual
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on beta-cdntest channel (report) - PASS
    • [DONE] Updates on beta channel (report) - PASS

B5 Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 6

B6 Build Information

B6 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B6 Testing

  • Manual
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on beta-cdntest channel (report) - PASS
    • [DONE] Updates on beta channel (report) - PASS

B6 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 7

B7 Build Information

B7 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B7 Testing

  • Manual
    • [DONE] Manual smoketesting: Web compatibility, Add-on compatibility, Secure browsing, Common media format playback, Sync & Persona sign-on.
    • [DONE] Verification of Bug 1069405
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on beta-cdntest channel (report) - PASS
    • [DONE] Updates on beta channel (report) - PASS

B7 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 8

B8 Build Information

B8 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B8 Testing

B8 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 9

B9 Build Information

B9 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

B9 Testing

  • Manual
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on beta-cdntest channel (report) - PASS
    • [DONE] Updates on beta channel (report) - PASS

B9 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


RC Build 1

RC Build 1 Information

RC Build 1 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

RC Build 1 Testing

  • Manual
    • [DONE] Verification for disabling of MacOSX disable screensaver feature - PASS
    • [DONE] Exploratory testing covering Web Compatibility, Audio/Video, Secure Browsing - FAIL (bugs 905902, 1079848)
      • Neither of those crashes reported would be considered top crashers, thus not tracked nor blocking release.
    • [DONE] Bug verification with focus on most recent fixes - PASS
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on beta-cdntest channel (report) -
    • [DONE] Updates on beta channel (report) - PASS

RC Build 1 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


RC Build 2

RC Build 2 Information

RC Build 2 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

RC Build 2 Testing

  • Manual
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on releasetest channel (report) - PASS
    • [DONE] Updates on release channel (report) - PASS

RC Build 2 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


33.0.1 Build 2

note: build 1 was scrapped before it went to release.

33.0.1 Build 2 Information

33.0.1 Build 2 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

33.0.1 Build 2 Testing

  • Manual
    • [DONE] Exploratory testing around: Web Compatibility, WebGL, WebRTC, Add-ons. etherpad - PASS
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [DONE] Updates on releasetest channel (report) - PASS
    • [DONE] Updates on release channel (report) - PASS

33.0.1 Build 2 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


33.0.2

note: build 1 was scrapped before it went to release.

33.0.2 Build 1 Information

33.0.2 Build 1 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

33.0.2 Build 1 Testing

  • Manual
    • [DONE] Exploratory testing around: Web Compatibility, WebGL. etherpad - PASS
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [ON TRACK] Updates on releasetest channel (report) -
    • [ON TRACK] Updates on release channel (report) -

33.0.2 Build 1 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


33.0.2 Build 2 Information

33.0.2 Build 2 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

33.0.2 Build 2 Testing

  • Manual
    • [DONE] Exploratory testing around: Web Compatibility, WebGL. etherpad - PASS
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [DONE] Updates on releasetest channel (report) - PASS
    • [DONE] Updates on release channel (report) - PASS

33.0.2 Build 2 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


33.0.3

note: build 1 was scrapped before it went to release.

33.0.3 Build 2 Information

33.0.3 Build 2 Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • 0 beta blocking regression, stability, or support issues discovered

33.0.2 Build 2 Testing

  • Manual
    • [DONE] Exploratory testing around: Web Compatibility, WebGL. etherpad - PASS
  • Automated (configs)
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [ON TRACK] Updates on releasetest channel (report) -
    • [ON TRACK] Updates on release channel (report) -

33.0.3 Build 2 Bugs Reported

Note: This section will be updated once QA has finished all testing activities.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_33/Test_Plan&oldid=1031312"