Releases/Firefox 32/Test Plan
From MozillaWiki
< Releases
Contents
- 1 Schedule
- 2 Meetings
- 3 Checklist
- 4 Features
- 4.1 Plugin crashes and hangs (Paul Silaghi)
- 4.2 Firefox Health Report (Petruta Rasa)
- 4.3 Translation (Bogdan Maris)
- 4.4 Certificate Pinning (Matt Wobensmith)
- 4.5 Mix-Blend-Mode (Catalin Varga)
- 4.6 HTTP cache v2 (Cornel Ionce)
- 4.7 Combined Navigation in Context Menu (Cornel Ionce)
- 4.8 Login metadata in password manager UI (Andrei Vaida)
- 5 Iteration Development
- 6 Stability
- 7 Bugs
- 8 Bug Fix Verifications
- 9 Regression Testing
- 10 Milestones
Schedule
Please consult the Rapid Release Calendar for more information.
- 2014-04-28: Firefox 32 arrives on Nightly
- 2014-06-09: Firefox 32 merges to Aurora
- 2014-06-13: Firefox 32 Aurora updates enabled
- 2014-07-21: Firefox 32 merges to Beta
- 2014-07-24: Firefox 32 Beta 1 released
Meetings
| Meeting | Purpose | When | Where |
| Crashkill | Raise stability concerns with the Crashkill team | 10:00 PST on Monday | Vidyo: Stability, IRC: #crashkill |
| Channel Status | Raise quality concerns with the Release Management team | 10:00 PST on Tuesday & Thursday | Vidyo: ReleaseCoordination, IRC: #planning |
Checklist
| Priority | Check | Description | Frequency | Last Checked |
| 1 | Stability | Review the stability reports to identify new/rising crash signatures | Daily | |
| 2 | Pushlog | Review the pushlog to identify fixed bugs needing verification | Daily | |
| 3 | QA Needed Bugs | Review and assign qawanted bugs for investigation | At least twice a week | |
| 4 | Unconfirmed Bugs | Triage and test unconfirmed bug reports | At least once a week | |
| 5 | Fixed Bugs | Flag fixed bugs for verification | At least once a week | |
| 6 | Verification Needed Bugs | Test verifyme bugs to confirm fixed bugs are fixed | At least once a week | |
| 7 | Reported Bugs | Document and escalated reported bugs as necessary | Every Monday | |
| 8 | Features | Ensure features are owned and on track for release | Once a week | |
| 9 | Automation | Ensure automation bugs are owned and on track for resolution | Once a week |
Features
Please consult the feature testing walkthrough for more information.
Plugin crashes and hangs (Paul Silaghi)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 983313 | Write crash events for plugin crashes and hangs | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Firefox Health Report (Petruta Rasa)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 893098 | FHR: Record more application update data | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Translation (Bogdan Maris)
120 Total; 13 Open (10.83%); 36 Resolved (30%); 71 Verified (59.17%);
Certificate Pinning (Matt Wobensmith)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 744204 | Allow Certificate (key) pinning for sites | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Mix-Blend-Mode (Catalin Varga)
86 Total; 19 Open (22.09%); 61 Resolved (70.93%); 6 Verified (6.98%);
HTTP cache v2 (Cornel Ionce)
82 Total; 1 Open (1.22%); 73 Resolved (89.02%); 8 Verified (9.76%);
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1000513 | Combined navigation items in the context menu | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Login metadata in password manager UI (Andrei Vaida)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 998081 | Show login metadata in password manager UI | RESOLVED | FIXED |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Iteration Development
Ensure these bugs have QA status as per Iterative Development guidelines.
120 Total; 0 Open (0%); 0 Resolved (0%); 120 Verified (100%);
Stability
This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.
- Explosive Crashes - ensure new and explosive crashes have actionable bug reports which are in progress
- Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
- Untriaged Bugs - ensure bugs are actionable and in the correct component
9 Total; 9 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Bugs
QA Wanted Bugs
| ID | Summary | Status | Qa contact | Component |
|---|---|---|---|---|
| 945192 | The updater.exe loads the bcrypt.dll and other dll's from the working and binary directory when not using the service (Application Update) | VERIFIED | Kamil Jozwiak [:kjozwiak] | Application Update |
| 1020041 | nsStandardURL::SetHost called net_ToLowerCase with a bogus pointer, #3 | VERIFIED | Networking | |
| 1029942 | allow activation from about:home | RESOLVED | SocialAPI |
3 Total; 0 Open (0%); 1 Resolved (33.33%); 2 Verified (66.67%);
Unconfirmed Bugs
89 Total; 89 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Fixed Bugs
Review the pushlog and place each Desktop Firefox bug in one of the following categories:
- A) Tested by the QA Team --> add the verifyme keyword to the bug and set the status-firefox32 flag to fixed
- B) Tested by a new volunteer --> add the [good first verify] whiteboard tag
- C) Tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
- D) Not tested at all --> add the [qa-] whiteboard tag
Tips:
- If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
- Be sure to set the status-firefox32:fixed flag
- If the bug includes tests from a developer you can set in-testsuite+
- Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures
Reported Bugs
Ensure these bugs remain on track to resolve and are verified when fixed.
Bugzilla query error
error, http-bad-status, Array
Bug Fix Verifications
- Review the bugs below to identify a bug you want to test
- Make sure you understand the bug before you begin testing it
- Test on a previously known broken build to make sure you can reproduce the bug
- Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
- Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
- Reopen the bug if it still reproduces
| ID | Summary | Component | Severity | Op sys |
|---|---|---|---|---|
| 1007389 | Implement plugin whitelist, round 2 | Plug-ins | normal | All |
| 1015636 | pdfjs privilege escalation round 2 | PDF Viewer | normal | Linux |
| 1026919 | IonMonkey: (ARM) Simplify the assembler buffer with constant pools. | JavaScript Engine: JIT | normal | All |
| 1045886 | Remove Cache directory from Android profiles | Networking | major | All |
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Regression Testing
- Moztrap testrun to cover past regressions, at-risk components, and feature integration tests
- WebRTC regression test
- PDF.js and printing
- Migration on first-run
Milestones
Aurora Migration
Pre-sign-off Checks:
- [DONE] Automation - document all Mozmill test failures and memory usage regressions
- One failure related to master password. In one locale. Investigating. Not blocking.
- [DONE] Features - all scoped features are signed off as ready for Aurora by owners
- [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious
- [DONE] Stability - review crash stats before sign-off to raise any red flags
- As of 6/13 we only had data from the nightly audience going into the merge and that looked reasonable, if somewhat higher than usual due to instability caused by OMTC on Windows.
- [DONE] L10n - contact :Pike to see if there are any known localization issues
- No issues here.
Post-sign-off Checks:
- {{|}} Updates - previous versions of Aurora update to the new version (check after updates live)
- {{|}} MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
Automation
- Testrun Reports
| Report | Test Failures |
| Add-ons | 0 failing tests; most tests skipped on Mac OS. |
| Functional | 1 failing test; master password in one locale. |
| Remote | A handful intermittent failures. |
| L10n | tn not built |
| Updates | 0 failing tests |
| Endurance | 0 failing tests |
- Memory Usage Report
| Platform | Min Memory Usage | Max Memory Usage | Avg Memory Usage | Net Compared to Firefox 31 |
| Windows 32-bit | ||||
| Mac OS X | ||||
| Linux 32-bit | ||||
| Linux 64-bit |
Memory regression being tracked/investigated in: <bug number>
Features
| Feature | Owner | Aurora Sign-off Status | Potential Blockers |
| Plugin crashes and hangs | Paul Silaghi | ||
| Firefox Health Report | Petruta Rasa | ||
| Search Provider Top Suggestions | Petruta Rasa | ||
| Modal/non-Modal Dialogs | Camelia Badau | ||
| Places | Andrei Vaida | ||
| Translation Infobar | Bogdan Maris | DONE | |
| Certificate Pinning | Matt Wobensmith | ||
| Mix-Blend-Mode | Catalin Varga | DONE | |
| HTTP cache v2 | Cornel Ionce | ||
| Combined Navigation in Context Menu | Cornel Ionce | DONE | |
| Login metadata in password manager UI | Andrei Vaida | DONE | |
| In-content preferences | Camelia Badau | ||
| HTTPS for in-product URLs | Camelia Badau |
Updates
| Platform | Source Build | Target Build | Result |
| Windows | |||
| Mac OS X | |||
| Linux | |||
Reported Bugs
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Beta 1
Build Information
- build2: 76dd853f227b (ID: 20140722195627)
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- New features pass basic smoketests
- 0 beta blocking regression, stability, or support issues discovered
Testing
- [DONE] Mozmill Functional Tests: report (config)
- [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Smoketest new features - PASS
- [DONE] Exploratory testing - Combined Navigation in Context Menu - PASS
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
Most outstanding bugs reported during the Pre-Beta Sign-off
| ID | Summary | Resolution |
|---|---|---|
| 777680 | "Bookmark This Page" in context menu should be highlighted | FIXED |
| 1008983 | "Assertion failure: 0 (Unsupported blend mode!)" with mix-blend-mode, layers.force-active | WORKSFORME |
2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 2
Build Information
- build1: 6a87b4799b21 (ID: 20140728123914)
- Changes between beta 1 and beta 2
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- [DONE] Mozmill Functional Tests: report (config)
- [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Exploratory testing for Games and Apps - PASS (some issues encountered)
- [DONE] Regression testing - PASS
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
| ID | Summary | Resolution |
|---|---|---|
| 1008178 | Intermittently, mouse-over on personas doesn't preview the themes | INACTIVE |
| 1045621 | Firefox prompts to install Flash even though it is already installed | INCOMPLETE |
2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 3
Build Information
- build1: 33eea1df6465 (ID: TBD)
- Changes between beta 2 and beta 3
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- {{|}} Mozmill Functional Tests: report (config)
- {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Regression testing - PASS
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 4
Build Information
- build1: 54fe10b3558d (ID: 20140804164216)
- Changes between beta 3 and Beta 4
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- [DONE] Mozmill Functional Tests: report (config)
- [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Regression testing - PASS
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
| ID | Summary | Resolution |
|---|---|---|
| 947079 | Passive mixed content indicator (grey triangle) shows up on full HTTPS sites that have no mixed content | FIXED |
1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 5
Build Information
- build1: 599c7756380c (ID: 20140807212602)
- Changes between beta 4 and Beta 5
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- {{|}} Mozmill Functional Tests: report (config)
- {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Regression testing - PASS
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 6
Build Information
- build1: 8bf5700cb82e (ID: 20140811180644)
- Changes between beta 5 and Beta 6
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- [DONE] Mozmill Functional Tests: report (config)
- [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Regression testing - PASS
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
7 Total; 7 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 7
Build Information
- build1: 51f60a862089 (ID: 20140814150857)
- Changes between beta 6 and beta 7
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- [DONE] Mozmill Functional Tests: report (config)
- [DONE] Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Regression testing - Spot checks across platforms with common user functionality.
- [ON TRACK] Fix Verifications, focusing on the most recent ones.
Bugs Reported
Bugzilla query error
Beta 8
Build Information
- build1: 45953c4613d2 (ID: 20140818191513)
- Changes between beta 7 and beta 8
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- {{|}} Mozmill Functional Tests: report (config)
- {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Exploratory testing - General spot checks across platforms with common user functionality.
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 9
Build Information
- build1: 58eb677e55f3 (ID: 20140822024446)
- Changes between beta 8 and beta 9
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- {{|}} Mozmill Functional Tests: report (config)
- {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Exploratory testing - General spot checks (SocialAPI, Alexa top sites)
- [DONE] Fix Verifications, focusing on the most recent ones.
Bugs Reported
5 Total; 5 Open (100%); 0 Resolved (0%); 0 Verified (0%);
RC Build 1
Build Information
- build1: 44234f451065 (ID: 20140825202822)
- Changes between beta 9 and RC
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Exploratory and focused testing reveal no major problems
- 0 beta blocking regression, stability, or support issues discovered
Testing
- {{|}} Mozmill Functional Tests: report (config)
- {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Exploratory testing - General spot checks, Downloads
- [ON TRACK] Fix Verifications, focusing on the most recent ones.
Bugs Reported
13 Total; 13 Open (100%); 0 Resolved (0%); 0 Verified (0%);
32.0.1 build1
Build Information
- build1: 227d1d0bf16b (ID: 20140910203853)
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Testing and verification, where possible, of
- Drivers:
- Bug 1062452 Firefox 32.0 crashes frequently in aticfx32.dll with switchable Intel+ATI graphics (~5% in total)
- Bug 1063048 Firefox 32 startup crash in _VEC_memzero | _VEC_memzero (>3%, #2) - bjacob has a fix that we expect will resolve these to bugs on bug 1063048. Verifying the fix will be difficult on other channels and will take time. bsmedberg suggested that we push the fix to 32 users.
- Chemspill ride alongs:
- Bug 1046645 Mixed content favicon is shown when loading a SSL site right after opening a new tab
- Bug 1063971 setRemoteDescription call without success-callback freezes PeerConnection queue (FF32 regression)- This is a WebRTC regression that we took in Fennec 32.0.1. As such, it's already landed on the release branch.
- Bug 933733 Page doesn't load/render if the mouse is not moving - Bad story here. Ready the bug. The short story is that we need a backout for bug 896896.
- Bug 1063052 Firefox 32 startup crash in nsFrame::BoxReflow - We don't have a fix but we do have a patch that should make it easier to diagnose the cause of this crash. Mats requested that we take the patch and Jet recommended that we don't. I poked in the bug to see if I can get some help determining whether this one should ride along.
- Drivers:
- 0 blocking regression, stability, or support issues discovered
Testing
- {{|}} Mozmill Functional Tests: report (config)
- {{|}} Mozmill Update Tests: betatest, releasetest, beta (config)
- [DONE] Testing around and verification of drivers and ride alongs. Detailed results in this etherpad.
- Bug 1062452 - Firefox 32.0 crashes frequently in aticfx32.dll with switchable Intel+ATI graphics - could not be reproduce
- Bug 1063048 - Firefox 32 startup crash in _VEC_memzero | _VEC_memzero - could not be reproduced
- Bug 1046645 - Mixed content favicon is shown when loading a SSL site right after opening a new tab - verified fixed
- Bug 1063971 - setRemoteDescription call without success-callback freezes PeerConnection queue (FF32 regression) - verified fixed
- Bug 933733 - Page doesn't load/render if the mouse is not moving - could not be reproduced
32.0.1 build2
Build Information
- build2: 01a075f70172 (ID: 20140911151253)
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Testing and verification, where possible, of drivers and ride alongs.
- Fixed in
build1andbuild2: Bug 1062452, Bug 1063048, Bug 1046645, Bug 1063971, Bug 933733. - Fixed just in
build2: Bug 1058131 - Avoid getting a crashy hook from Avast 10 Beta
- Fixed in
Testing
- Manual
- [DONE] Manual smoketesting (Web compatibility, Add-on compatibility, Secured browsing, Common media format playback).
- [DONE] Spotcheck of fixes initially added in
build1.- [DONE] Verification of the new fix from
build2, Bug 1058131 - Avoid getting a crashy hook from Avast 10 Beta - verified fixed. - Detailed test results in this etherpad.
- [DONE] Verification of the new fix from
- Automation
32.0.2 build1
Build Information
- build1: be350b081d47 (ID: 20140917194002)
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Testing and verification, where possible, of drivers and ride alongs.
- Fixed in
build1: Bug 1063052 - Firefox 32+ startup crash in nsFrame::BoxReflow.
- Fixed in
Testing
- Manual
- [DONE] Updates testing (from
Fx20,Fx27,Fx31,Fx32andFx32.0.1toFx32.0.2).- Targeted update methods:
menu button → About Firefox,Pave update,Automatic update- PASS
- Targeted update methods:
- [DONE] Manual smoketesting (Web compatibility, Add-on compatibility, Secured browsing, Common media format playback) - PASS
- Detailed test results are available in this etherpad.
- [DONE] Bug verification:
- Bug 1063052 - Firefox 32+ startup crash in nsFrame::BoxReflow - Verified fixed
- Bug 1058131 - crash in mozilla::dom::Element::SetAttr(int, nsIAtom*, nsAString_internal const&, bool) - Verified fixed
- [DONE] Updates testing (from
- Automation
32.0.3 build1
Build Information
- build1: f9ccc64ca17d (ID: 20140923175406)
Sign-off Criteria
- Mozmill functional automation ran without unexpected failures
- Mozmill update automation ran without unexpected failures
- Testing and verification, where possible, of drivers and ride alongs.
- Fixed in
build1: Bug 1064636
- Fixed in
Testing
- Manual
- [DONE] Manual smoketesting: Web compatibility, Add-on compatibility, Secure browsing, Common media format playback, Sync & Persona sign-on.
- Detailed test results are available in this etherpad.
- [DONE] Verification of Bug 1069405
- [DONE] Manual smoketesting: Web compatibility, Add-on compatibility, Secure browsing, Common media format playback, Sync & Persona sign-on.
