Releases/Firefox 11/Test Plan
« Firefox 10 | Firefox 11 | Firefox 12 »
Firefox 11 Test Plan
Summary
| Milestone | Firefox 11 |
| Status | Currently in Beta -- Release on March 13, 2012 |
| Lead | Anthony Hughes (irc: ashughes) |
| 2nd | Juan Becerra (irc: juanb) |
| Builds | FTP, HTTP |
The following is the test plan for Firefox 11 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 11. After the release this document will be used as an archive of what was done to validate this release.
If you have some free time, please pick a task below and get in touch with one of the leads.
Details
The following is a description of the tasks which need to be completed throughout this release.
- 2011-11-10: [DONE] Nightly Merge
- 2011-12-22: [DONE] Aurora Merge
- 2012-02-03: [DONE] Beta 1
- 2012-02-07: [ON TRACK] Beta 2
- 2012-02-14: {{|}} Beta 3
- 2012-02:21: {{|}} Beta 4
- 2012-02-28: {{|}} Beta 5
- 2012-03-06: {{|}} Beta 6
- 2012-03-09: {{|}} Release Candidate
- 2012-03-13: {{|}} Final
Features
See the Feature Release Tracking wiki for a general overview
- [DONE]
- Chrome Migration (Paul Silaghi): SIGNED OFF on 2011-12-20
- SPDY (Ioana Budnar): SIGNED OFF on 2011-12-20
- Page Inspector Tilt (Alin Trif): SIGNED OFF 2012-01-25
- Add-on Sync (Tracy Walker): SIGNED OFF 2012-01-25
- NOTE: will not sync across application versions, add-ons outside of AMO, plugins, personas, and search plugins
- [ON TRACK]
- Style Editor (Alex Lakatos): 2012-01-31
- Pending uplift of some bug fixes bug 720468 (Outstanding bugs)
- [AT RISK]
- [MISSED]
- Home Tab (Vlad Ghetiu): delayed to Firefox 12
- Media Controls Refresh (Mihaela Velimiroviciu): delayed to Firefox 12
- Improve Awesomebar Results (Simona Bidau): delayed to Firefox 12
- UAC Service (Simona Bidau): delayed to Firefox 12
- New Tab Page (Virgil Dicu): delayed to Firefox 12
- TLS Telemetry (unowned): delayed to Firefox 13
- In-content UI Unification (Ioana Budnar): unscoped, no milestone
- Final Sign-off
- Litmus testrun is being built (to be run in week leading up to Beta merge
Endurance Automation
Endurance automation is run every day during Nightly. The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.
- Owner: Mihaela Velimiroviciu
- Status: SIGNED OFF
- Reports: dashboard, spreadsheet
- Issues: none found
Functional Automation
Functional automation is run every day on Nightly and Aurora. The purpose of this section is to track any potential functional regressions via results reported in the dashboard.
- Owner: Vlad Maniac (2nd - Alex Lakatos)
- Status: on track
- Reports: dashboard
- Issues: none found
Bug Triage
This section tracks bug triage work. The purpose is to triage UNCONFIRMED incoming bugs during Nightly and Aurora. Once fixed, they will be triaged and verified during Beta.
- Owner: Vlad Ghetiu
- Status: on track
- Queries: UNCONFIRMED, REOPENED, NEW
- Issues: none found
Aurora Merge
Functional Automation
| Platform | Result: report |
| Windows 2000 | PASS - 2011-12-22 |
| Windows XP | PASS - 2011-12-22 |
| Windows Vista | PASS - 2011-12-22 |
| Windows 7 32-bit | PASS - 2011-12-22 |
| Windows 7 64-bit | PASS - 2011-12-22 |
| Mac OS X 10.6 | PASS - 2011-12-22 |
| Mac OS X 10.7 | PASS - 2011-12-22 |
| Ubuntu 11.04 32-bit | PASS - 2011-12-22 |
| Ubuntu 11.04 64-bit | PASS - 2011-12-22 |
Updates
| Platform | Result |
| Auroratest | |
| Win32 | Mozilla/5.0 (Windows NT 5.1; rv:10.0a2) Gecko/20111211 Firefox/10.0a2 Mozilla/5.0 (Windows NT 5.1; rv:11.0a2) Gecko/20111222 Firefox/11.0a2 |
| Mac | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111215 Firefox/10.0a2 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a2) Gecko/20111222 Firefox/11.0a2 |
| Linux | Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0a2) Gecko/20111125 Firefox/10.0a2 Mozilla/5.0 (X11; Linux i686 on x86_64; rv:11.0a2) Gecko/20111222 Firefox/11.0a2 |
| Aurora | |
| Win32 | Mozilla/5.0 (Windows NT 5.1; rv:10.0a2) Gecko/20111211 Firefox/10.0a2 Mozilla/5.0 (Windows NT 5.1; rv:11.0a2) Gecko/20111222 Firefox/11.0a2 |
| Mac | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111215 Firefox/10.0a2 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a2) Gecko/20111222 Firefox/11.0a2 |
| Linux | Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0a2) Gecko/20111125 Firefox/10.0a2 Mozilla/5.0 (X11; Linux i686 on x86_64; rv:11.0a2) Gecko/20111222 Firefox/11.0a2 |
Beta 1
Builds: FTP, HTTP
Litmus: testrun, report
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [DONE] Manual tests
- [DONE] Bug fix verifications (10%)
- [DONE] Check with Support for top issues from previous Beta/Release
- [DONE] Check with Marcia for top-crashers spiking
- [DONE] Update tests on releasetest
- [DONE] Get final sign-off from Bob Moss
- [DONE] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
| Platform | Assignee | Result: report |
| Windows 2000 | ashughes | PASS - 2012-02-01 |
| Windows XP | ashughes | PASS - 2012-02-01 |
| Windows Vista | ashughes | PASS - 2012-02-01 |
| Windows 7 32-bit | ashughes | PASS - 2012-02-01 |
| Windows 7 64-bit | ashughes | PASS - 2012-02-01 |
| Mac OS X 10.6 | ashughes | PASS - 2012-02-01 |
| Mac OS X 10.7 | ashughes | PASS - 2012-02-01 |
| Ubuntu 11.10 32-bit | ashughes | PASS - 2012-02-01 |
| Ubuntu 11.10 64-bit | ashughes | PASS - 2012-02-01 |
- Test Failures
Manual Tests
The following are the detailed results of the manual testing.
- Smoketests
Run the manual smoketests and report results below.
| Platform | Assignee | Result |
| Windows 2000 | Simona | PASS - 2012-02-02 |
| Windows XP | Ioana | PASS - 2012-02-02 |
| Windows Vista | Simona | PASS - 2012-02-02 |
| Windows 7 32-bit | Mihaela | PASS - 2012-02-02 |
| Windows 7 64-bit | Paul | PASS - 2012-02-02 |
| Mac OS X 10.6.8 | VladG | PASS - 2012-02-02 |
| Mac OS X 10.7.2 | VladG | PASS - 2012-02-02 |
| Ubuntu 11.10 32-bit | Ioana | PASS - 2012-02-02 |
| Ubuntu 11.10 64-bit | Virgil | PASS - 2012-02-02 |
Results in Litmus
- Focus Testing
The following are the detailed results of the focused feature testing. Provide much detail about your testing: test cases, steps, bugs filed, URLs, add-on/plugin versions, platforms tested, etc
- [DONE] Rich Profile Updates Etherpad
- [DONE] Add-on Compatibility (including default compatible) Etherpad
- [DONE] Performance (typing and scrolling on intense pages -- like Facebook timeline) Etherpad
- [DONE] Web Compatibility (bank sites, social networking, video and audio streaming, etc) Etherpad
- [DONE] Anti-virus Compatibility (checking that AV software does not interfere, degrade Firefox performance) Etherpad
- [DONE] Plugin Stability (games & videos, flash, java, silverlight, etc) Etherpad
- [DONE] Run the Software Update and Silent Update FFTs
- Feature Sign-off Testing
The following are the results of feature testing for those features which are shipping. Use the Firefox 11.0 Feature Sign-off tests in the Litmus testrun to qualify the features. Additionally, if time permits, do some exploratory testing around these features.
| Feature | Owner | Win32 | Mac | Linux |
| Chrome Migration | Paul Silaghi | PASS - 2012-02-02 | PASS - 2012-02-02 | PASS - 2012-02-02 |
| SPDY | Ioana Budnar | PASS - 2012-02-02 | PASS - 2012-02-02 | PASS - 2012-02-02 |
| Page Inspector Tilt | Alin Trif | PASS - 2012-02-02 | PASS - 2012-02-02 | PASS - 2012-02-02 |
| Add-on Sync | Tracy Walker | PASS - 2012-02-02 | PASS - 2012-02-02 | PASS - 2012-02-02 |
Update Tests
The following are details of update verifications
| Platform | Releasetest: report | Beta: report |
| Windows 2000 | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Windows XP | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Windows Vista | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Windows 7 32-bit | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Windos 7 64-bit | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Mac OS X 10.6 | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Mac OS 10.7 | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Ubuntu 11.10 32-bit | PASS - 2012-02-02 | PASS - 2012-02-03 |
| Ubuntu 11.10 64-bit | PASS - 2012-02-02 | PASS - 2012-02-03 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox11 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
- Queries
- QA+ - used by testers to triage and verify fixes
- UNTRIAGED: 102, VERIFIED: 12
- Softvision verifications etherpad
- QA+ SG - used by the Security team to triage and verify security fixes
- UNTRIAGED: 30, VERIFIED: 0
- status-firefox11:fixed - used by QA lead to triage for potential QA+ bugs
- QA?: 13, QA+: 52, QA-: 26
Issues Found
| Bug | Status |
| bug 723444 - Angry Birds is running jerky | Need help investigating, should not block 11b1 |
| bug 723615 - Fix Mozilla UA sniffing | results "update to Firefox 10" what's new page |
| bug 723920 - Firefox 10beta does not auto-update | INVALID: updates download in the background, prompt on completion |
Known Issues
| Bug | Status |
| bug 681922 In google docs Upload -> Files has no highlighting | Not a Firefox bug, regressed due to Google back-out |
| bug 697696 Highlight issue in amazon's password box | Needs reduced testcase for debugging, no functional deficiency |
| bug 692446 Firefox prompts to install WMP plug-in though it's installed | Still investigating, might be issue with Microsoft Installer |
| bug 702045 Make Firefox able to survive a Windows System Restore | Development work still ongoing |
| bug 278860 confusing "profile in use"/"already running" error when profile is missing (not found) | NEW |
Feedback
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
Builds can be found here: FTP, HTTP
You can post your feedback below or in this etherpad.
Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.
Beta 2
Builds: FTP, HTTP
Litmus: testrun, report
- Checklist
- [DONE] Functional automated tests, synced to Litmus
- [ON TRACK] Manual tests
- [ON TRACK] Bug fix verifications (20%)
- [ON TRACK] Check with Support for top issues from previous Beta/Release
- [ON TRACK] Check with Marcia for top-crashers spiking
- [ON TRACK] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
| Platform | Assignee | Result: report |
| Windows 2000 | ashughes | PASS - 2012-02-08 |
| Windows XP | ashughes | PASS - 2012-02-08 |
| Windows Vista | ashughes | PASS - 2012-02-08 |
| Windows 7 32-bit | ashughes | PASS - 2012-02-08 |
| Windows 7 64-bit | ashughes | PASS - 2012-02-08 |
| Mac OS X 10.6 | ashughes | PASS - 2012-02-08 |
| Mac OS X 10.7 | ashughes | PASS - 2012-02-08 |
| Ubuntu 11.10 32-bit | ashughes | PASS - 2012-02-08 |
| Ubuntu 11.10 64-bit | ashughes | PASS - 2012-02-08 |
- Test Failures
- testSecurity/testSubmitUnencryptedInfoWarning: report (bug 725486)
Manual Tests
The following are the detailed results of the manual testing.
- Smoketests
Run the manual smoketests and report results below.
| Platform | Assignee | Result |
| Windows 2000 | Simona | result - YYYY-MM-DD |
| Windows XP | Remus | result - YYYY-MM-DD |
| Windows Vista | Simona | result - YYYY-MM-DD |
| Windows 7 32-bit | AlinT | result - YYYY-MM-DD |
| Windows 7 64-bit | Paul | result - YYYY-MM-DD |
| Mac OS X 10.6.8 | Mihaela | result - YYYY-MM-DD |
| Mac OS X 10.7.2 | Virgil | result - YYYY-MM-DD |
| Ubuntu 11.10 32-bit | Ioana | result - YYYY-MM-DD |
| Ubuntu 11.10 64-bit | Virgil | result - YYYY-MM-DD |
Results in Litmus
- Focus Testing
The following are the detailed results of the focused feature testing. Provide much detail about your testing: test cases, steps, bugs filed, URLs, add-on/plugin versions, platforms tested, etc
- [ON TRACK] Rich Profile Updates
- [ON TRACK] Add-on Compatibility (including default compatible)
- [ON TRACK] Performance (typing and scrolling on intense pages -- like Facebook timeline)
- [ON TRACK] Web Compatibility (bank sites, social networking, video and audio streaming, etc)
- [ON TRACK] Anti-virus Compatibility (checking that AV software does not interfere, degrade Firefox performance)
- [ON TRACK] Plugin Stability (games & videos, flash, java, silverlight, etc)
- [ON TRACK] Run the Software Update and Silent Update FFTs
Update Tests
The following are details of update verifications
| Platform | Releasetest: [ report] | Beta: [ report] |
| Windows 2000 | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows XP | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows Vista | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windows 7 32-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Windos 7 64-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Mac OS X 10.6 | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Mac OS 10.7 | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Ubuntu 11.10 32-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
| Ubuntu 11.10 64-bit | result - YYYY-MM-DD | result - YYYY-MM-DD |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox11 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
- Queries
- QA+ - used by testers to triage and verify fixes
- UNTRIAGED: ?, VERIFIED: 0
- Softvision verifications etherpad
- QA+ SG - used by the Security team to triage and verify security fixes
- UNTRIAGED: ?, VERIFIED: 0
- status-firefox11:fixed - used by QA lead to triage for potential QA+ bugs
- QA?: 0, QA+: 0, QA-: 0
- P1 Verifications
- bug 718939 - Java applet causes text entry fields to become semi-unresponsive
- See comment 23 and 25 for steps to reproduce
- RESULTS: result (testers)
- bug 724284 - Use after free in nsXBLDocumentInfo::ReadPrototypeBindings
- RESULTS: result (testers)
- RESULTS: result (testers)
Issues Found
| Bug | Status |
| bug 723444 - Angry Birds is running jerky | Need help investigating, should not block 11b1 |
| bug 723615 - Fix Mozilla UA sniffing | results "update to Firefox 10" what's new page |
| bug 723920 - Firefox 10beta does not auto-update | INVALID: updates download in the background, prompt on completion |
Known Issues
| Bug | Status |
| bug 681922 In google docs Upload -> Files has no highlighting | Not a Firefox bug, regressed due to Google back-out |
| bug 697696 Highlight issue in amazon's password box | Needs reduced testcase for debugging, no functional deficiency |
| bug 692446 Firefox prompts to install WMP plug-in though it's installed | Still investigating, might be issue with Microsoft Installer |
| bug 702045 Make Firefox able to survive a Windows System Restore | Development work still ongoing |
| bug 278860 confusing "profile in use"/"already running" error when profile is missing (not found) | NEW |
Feedback
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
Builds can be found here: FTP, HTTP
You can post your feedback below or in this etherpad.
Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.