Releases/Firefox 32/Test Plan

From MozillaWiki
Jump to navigation Jump to search

Schedule

Please consult the Rapid Release Calendar for more information.

  • 2014-04-28: Firefox 32 arrives on Nightly
  • 2014-06-09: Firefox 32 merges to Aurora
  • 2014-06-13: Firefox 32 Aurora updates enabled
  • 2014-07-21: Firefox 32 merges to Beta
  • 2014-07-24: Firefox 32 Beta 1 released

Meetings

Meeting Purpose When Where
Crashkill Raise stability concerns with the Crashkill team 10:00 PST on Monday Vidyo: Stability, IRC: #crashkill
Channel Status Raise quality concerns with the Release Management team 10:00 PST on Tuesday & Thursday Vidyo: ReleaseCoordination, IRC: #planning

Checklist

Priority Check Description Frequency Last Checked
1 Stability Review the stability reports to identify new/rising crash signatures Daily
2 Pushlog Review the pushlog to identify fixed bugs needing verification Daily
3 QA Needed Bugs Review and assign qawanted bugs for investigation At least twice a week
4 Unconfirmed Bugs Triage and test unconfirmed bug reports At least once a week
5 Fixed Bugs Flag fixed bugs for verification At least once a week
6 Verification Needed Bugs Test verifyme bugs to confirm fixed bugs are fixed At least once a week
7 Reported Bugs Document and escalated reported bugs as necessary Every Monday
8 Features Ensure features are owned and on track for release Once a week
9 Automation Ensure automation bugs are owned and on track for resolution Once a week

Features

Please consult the feature testing walkthrough for more information.

Plugin crashes and hangs (Paul Silaghi)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Firefox Health Report (Petruta Rasa)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Translation (Bogdan Maris)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Certificate Pinning (Matt Wobensmith)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Mix-Blend-Mode (Catalin Varga)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


HTTP cache v2 (Cornel Ionce)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Combined Navigation in Context Menu (Cornel Ionce)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Login metadata in password manager UI (Andrei Vaida)

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Iteration Development

Ensure these bugs have QA status as per Iterative Development guidelines.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

  • Explosive Crashes - ensure new and explosive crashes have actionable bug reports which are in progress
  • Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
  • Untriaged Bugs - ensure bugs are actionable and in the correct component

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Bugs

QA Wanted Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Unconfirmed Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Fixed Bugs

Review the pushlog and place each Desktop Firefox bug in one of the following categories:

  • A) Tested by the QA Team --> add the verifyme keyword to the bug and set the status-firefox32 flag to fixed
  • B) Tested by a new volunteer --> add the [good first verify] whiteboard tag
  • C) Tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
  • D) Not tested at all --> add the [qa-] whiteboard tag

Tips:

  • If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
  • Be sure to set the status-firefox32:fixed flag
  • If the bug includes tests from a developer you can set in-testsuite+
  • Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures

Reported Bugs

Ensure these bugs remain on track to resolve and are verified when fixed.

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Bug Fix Verifications

  • Review the bugs below to identify a bug you want to test
  • Make sure you understand the bug before you begin testing it
  • Test on a previously known broken build to make sure you can reproduce the bug
  • Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
  • Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
  • Reopen the bug if it still reproduces

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Regression Testing

  • Moztrap testrun to cover past regressions, at-risk components, and feature integration tests
  • WebRTC regression test
  • PDF.js and printing
  • Migration on first-run

Milestones

Aurora Migration

Pre-sign-off Checks:

  • [DONE] Automation - document all Mozmill test failures and memory usage regressions
    • One failure related to master password. In one locale. Investigating. Not blocking.
  • [DONE] Features - all scoped features are signed off as ready for Aurora by owners
  • [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious
  • [DONE] Stability - review crash stats before sign-off to raise any red flags
    • As of 6/13 we only had data from the nightly audience going into the merge and that looked reasonable, if somewhat higher than usual due to instability caused by OMTC on Windows.
  • [DONE] L10n - contact :Pike to see if there are any known localization issues
    • No issues here.

Post-sign-off Checks:

  • {{|}} Updates - previous versions of Aurora update to the new version (check after updates live)
  • {{|}} MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel

Automation

Testrun Reports
Report Test Failures
Add-ons 0 failing tests; most tests skipped on Mac OS.
Functional 1 failing test; master password in one locale.
Remote A handful intermittent failures.
L10n tn not built
Updates 0 failing tests
Endurance 0 failing tests
Memory Usage Report
Platform Min Memory Usage Max Memory Usage Avg Memory Usage Net Compared to Firefox 31
Windows 32-bit
Mac OS X
Linux 32-bit
Linux 64-bit

Memory regression being tracked/investigated in: <bug number>

Features

Feature Owner Aurora Sign-off Status Potential Blockers
Plugin crashes and hangs Paul Silaghi
Firefox Health Report Petruta Rasa
Search Provider Top Suggestions Petruta Rasa
Modal/non-Modal Dialogs Camelia Badau
Places Andrei Vaida
Translation Infobar Bogdan Maris DONE
Certificate Pinning Matt Wobensmith
Mix-Blend-Mode Catalin Varga DONE
HTTP cache v2 Cornel Ionce
Combined Navigation in Context Menu Cornel Ionce DONE
Login metadata in password manager UI Andrei Vaida DONE
In-content preferences Camelia Badau
HTTPS for in-product URLs Camelia Badau

Updates

Platform Source Build Target Build Result
Windows
Mac OS X
Linux


Reported Bugs

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • New features pass basic smoketests
  • 0 beta blocking regression, stability, or support issues discovered

Testing

  • [DONE] Mozmill Functional Tests: report (config)
  • [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
  • [DONE] Smoketest new features - PASS
  • [DONE] Exploratory testing - Combined Navigation in Context Menu - PASS
  • [DONE] Fix Verifications, focusing on the most recent ones.

Bugs Reported

Most outstanding bugs reported during the Pre-Beta Sign-off

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 2

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 3

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 4

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 5

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 6

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 7

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

  • [DONE] Mozmill Functional Tests: report (config)
  • [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
  • [DONE] Regression testing - Spot checks across platforms with common user functionality.
  • [ON TRACK] Fix Verifications, focusing on the most recent ones.

Bugs Reported

Bugzilla query error

Query options must be valid JSON.1


Beta 8

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

  • {{|}} Mozmill Functional Tests: report (config)
  • {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
  • [DONE] Exploratory testing - General spot checks across platforms with common user functionality.
  • [DONE] Fix Verifications, focusing on the most recent ones.

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


Beta 9

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

  • {{|}} Mozmill Functional Tests: report (config)
  • {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
  • [DONE] Exploratory testing - General spot checks (SocialAPI, Alexa top sites)
  • [DONE] Fix Verifications, focusing on the most recent ones.

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


RC Build 1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Exploratory and focused testing reveal no major problems
  • 0 beta blocking regression, stability, or support issues discovered

Testing

Bugs Reported

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 429 [1] => Unknown Error ) ) 1


32.0.1 build1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Testing and verification, where possible, of
    • Drivers:
      • Bug 1062452 Firefox 32.0 crashes frequently in aticfx32.dll with switchable Intel+ATI graphics (~5% in total)
      • Bug 1063048 Firefox 32 startup crash in _VEC_memzero | _VEC_memzero (>3%, #2) - bjacob has a fix that we expect will resolve these to bugs on bug 1063048. Verifying the fix will be difficult on other channels and will take time. bsmedberg suggested that we push the fix to 32 users.
    • Chemspill ride alongs:
      • Bug 1046645 Mixed content favicon is shown when loading a SSL site right after opening a new tab
      • Bug 1063971 setRemoteDescription call without success-callback freezes PeerConnection queue (FF32 regression)- This is a WebRTC regression that we took in Fennec 32.0.1. As such, it's already landed on the release branch.
      • Bug 933733 Page doesn't load/render if the mouse is not moving - Bad story here. Ready the bug. The short story is that we need a backout for bug 896896.
      • Bug 1063052 Firefox 32 startup crash in nsFrame::BoxReflow - We don't have a fix but we do have a patch that should make it easier to diagnose the cause of this crash. Mats requested that we take the patch and Jet recommended that we don't. I poked in the bug to see if I can get some help determining whether this one should ride along.
  • 0 blocking regression, stability, or support issues discovered

Testing

  • {{|}} Mozmill Functional Tests: report (config)
  • {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
  • [DONE] Testing around and verification of drivers and ride alongs. Detailed results in this etherpad.
    • Bug 1062452 - Firefox 32.0 crashes frequently in aticfx32.dll with switchable Intel+ATI graphics - could not be reproduce
    • Bug 1063048 - Firefox 32 startup crash in _VEC_memzero | _VEC_memzero - could not be reproduced
    • Bug 1046645 - Mixed content favicon is shown when loading a SSL site right after opening a new tab - verified fixed
    • Bug 1063971 - setRemoteDescription call without success-callback freezes PeerConnection queue (FF32 regression) - verified fixed
    • Bug 933733 - Page doesn't load/render if the mouse is not moving - could not be reproduced


32.0.1 build2

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Testing and verification, where possible, of drivers and ride alongs.

Testing

  • Manual
    • [DONE] Manual smoketesting (Web compatibility, Add-on compatibility, Secured browsing, Common media format playback).
    • [DONE] Spotcheck of fixes initially added in build1.
      • [DONE] Verification of the new fix from build2, Bug 1058131 - Avoid getting a crashy hook from Avast 10 Beta - verified fixed.
      • Detailed test results in this etherpad.
  • Automation
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [DONE] Updates on releasetest channel (report) - PASS
    • [DONE] Updates on release channel (report) - PASS

32.0.2 build1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Testing and verification, where possible, of drivers and ride alongs.
    • Fixed in build1: Bug 1063052 - Firefox 32+ startup crash in nsFrame::BoxReflow.

Testing

  • Manual
    • [DONE] Updates testing (from Fx20, Fx27, Fx31, Fx32 and Fx32.0.1 to Fx32.0.2).
      • Targeted update methods: menu button → About Firefox, Pave update, Automatic update - PASS
    • [DONE] Manual smoketesting (Web compatibility, Add-on compatibility, Secured browsing, Common media format playback) - PASS
    • [DONE] Bug verification:
      • Bug 1063052 - Firefox 32+ startup crash in nsFrame::BoxReflow - Verified fixed
      • Bug 1058131 - crash in mozilla::dom::Element::SetAttr(int, nsIAtom*, nsAString_internal const&, bool) - Verified fixed
  • Automation

32.0.3 build1

Build Information

Sign-off Criteria

  • Mozmill functional automation ran without unexpected failures
  • Mozmill update automation ran without unexpected failures
  • Testing and verification, where possible, of drivers and ride alongs.

Testing

  • Manual
    • [DONE] Manual smoketesting: Web compatibility, Add-on compatibility, Secure browsing, Common media format playback, Sync & Persona sign-on.
    • [DONE] Verification of Bug 1069405
  • Automation
    • [DONE] Automation Functional tests (report) - PASS
    • [DONE] Updates on betatest channel (report) - PASS
    • [DONE] Updates on releasetest channel (report) - PASS
    • [DONE] Updates on release channel (report) - PASS