Releases/Firefox 27/Test Plan
« Firefox 26 | Firefox 27 | Firefox 28 »
Contents
Firefox 27 Test Plan
Summary
Milestone | Firefox 27 |
Status | Currently in Beta |
Lead | Tracy Walker (irc: tracy) |
2nd | Anthony Hughes (irc: ashughes) |
Builds | [ftp] [http] [web] |
The following is the test plan for Firefox 27 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 27. After the release this document will be used as an archive of what was done to validate this release.
If you have some free time, please pick a task below and get in touch with one of the leads.
Schedule
- 2013.09.16: [DONE] Nightly
- 2013.10.29: [DONE] Aurora
- 2013.12.05: [DONE] Pre-Beta feature sign-offs
- 2013.12.12: [DONE] Beta 1
- 2013.12.17: [DONE] Beta 2
- skip: Beta 3
- 2014.01.07: [DONE] Beta 4
- 2014.01.10: [DONE] Beta 5
- 2014.01.14: [DONE] Beta 6
- 2014.01.17: [DONE] Beta 7
- 2014.01.21: [DONE] Beta 8
- 2014.01.24: [DONE] Beta 9
- 2014.01.28: [DONE] Release Candidate
- 2014.02.04: [DONE] Final
- 2014.02.13: [DONE] 27.0.1
Features
- Sign-off Criteria
- All user stories covered by manual or automated tests
- Manual testing complete with no blocking regressions
- Automated testing complete with no blocking regressions
- 0 untriaged unconfirmed bugs
- 0 untriaged unverified fixes
- Status
Feature | Owner | Documentation | Tracked Bugs | Aurora Merge Sign-off | Pre-beta Sign-off |
Firefox Metro | Juan Becerra | Bug Testing | [DONE] | Disabled | |
HTTP Backend Cache | Mihaela Velimiroviciu, Catalin Varga | bugs | Disabled | NA | |
New First Run experience | Catalin Varga | 1 bug | [DONE] | Disabled | |
Add a simple tool to display reflows | Petruta Rasa | MDN doc | 1 bug | [DONE] | [DONE] |
SocialAPI: Allow multiple workers | Petruta Rasa | 1 bug | NA | [DONE] | |
Enable history swipe animations by default | Juan Becerra | 1 bug | NA | ||
Interface for predictive actions | No need for QA owner - covered automatically | 1 bug | NA | NA | |
WebGL Shader Editor | No need for QA owner - covered automatically | 1 bug | NA | NA | |
CSS Inheritance changes | No need for QA owner - covered automatically | 2 bugs | NA | NA |
Beta
Known Issues
ID | Summary | Priority | Status |
---|---|---|---|
607586 | Realplayer Plugin does not work | -- | RESOLVED |
739043 | Can't fill fillable PDF forms with PDF Viewer | -- | RESOLVED |
810636 | Poor copy & paste behavior with pdf.js | P2 | VERIFIED |
906057 | Delay on WebRTC calls | -- | RESOLVED |
922051 | Image freezes on caller side after a short period of time | -- | RESOLVED |
922132 | Image freezes on callee side after a while | -- | RESOLVED |
928379 | QuickTime controls bar is missing | -- | RESOLVED |
936034 | Sound is distorted on one side on 1:1 call | P5 | RESOLVED |
943511 | Use fallible allocation in nsSegmentedBuffer::AppendNewSegment | -- | VERIFIED |
943885 | Investigate when "+N other downloads" should be shown instead of "Show all downloads" | -- | NEW |
945703 | Pasting a download link into the download manager sometimes fails. | -- | RESOLVED |
948944 | The download indicator is missing from the applications dock | -- | RESOLVED |
12 Total; 1 Open (8.33%); 9 Resolved (75%); 2 Verified (16.67%);
Regression Testing
Beginning with Fx 27, we've completely decoupled the regression testing from each beta. Instead, coverage of regression testing will take place throughout the beta cycle. The goal for Fx27 will be to finish all the regression testing prior to beta8. Breakpad will only be retested for changes landed in breakpad and/or significant changes in Socorro.
Under Test | Win 7 | Win 8.1 | Mac 10.8 | Linux |
Breakpad | [DONE] 27.0b1 | [DONE] 27.0b1 | [DONE] 27.0b1 | [DONE] 27.0b1 |
Mac OSX code-signing | N/A | N/A | [DONE] 27.0b1 | N/A |
Nvidia 3d vision | [MISSED] tracy | [MISSED] tracy | [MISSED] tracy | [MISSED] tracy |
Downloads | [DONE] 27.0b1 | [DONE] 27.0b1 | [DONE] 27.0b1 | [DONE] 27.0b1 |
Session Restore | [DONE] 27.0b1 | [DONE] 27.0b1 | [DONE] 27.0b1 | [DONE] 27.0b1 |
Plug-in compatibility | [DONE] 27.0b4 | [DONE] 27.0b4 | [DONE] 27.0b4 | [DONE] 27.0b4 |
Mac OSX 32-bit mode | N/A | N/A | [DONE] 27.0b4 | N/A |
Web compatibility | [DONE] 27.0b2 | [DONE] 27.0b2 | [DONE] 27.0b2 | [DONE] 27.0b2 |
Software update for root-owned files | [DONE] 27.0b4 | [DONE] 27.0b4 | [DONE] 27.0b4 | [DONE] 27.0b4 |
Add-on compatibility | [DONE] 27.0b2 | [DONE] 27.0b2 | [DONE] 27.0b2 | [DONE] 27.0b2 |
Plug-in compatibility | [DONE] 27.0b9 | [DONE] 27.0b9 | [DONE] 27.0b9 | [DONE] 27.0b9 |
What's New page | [DONE] 27.0b5 | [DONE] 27.0b5 | [DONE] 27.0b5 | [DONE] 27.0b5 |
- Guidelines
- Breakpad End-to-End functionality: use this MozTrap test
- Mac OSX code-signing: use this test
- Downloads: bug 845408, bug 906620, bug 908256, bug 910236, bug 923186, bug 913110
- Session Restore: spotcheck restoring dirty and light sessions
- Additional Testing
- For Fx 27.0b5 the Delicious service was tested on the following environments: Windows 7 x64 , Ubuntu 12.10 x86 and OS X 10.8. The focus of the testing was to assure proper functionality of the browser while Delicious service and sidebar are enabled.
Issues Found
List any new issues found or old issues reopened. These issues will be triaged and escalated by the release leads.
11 Total; 1 Open (9.09%); 10 Resolved (90.91%); 0 Verified (0%);
27.0.1
Build Info
- Builds: build1
- Build ID: 20140212131424
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Regression testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on release channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Smoketest (Windows 7 64bit, Windows 8.1 64bit, Mac OS X 10.8, Ubuntu 12.04 32bit)
- Bugs for verification:
- [DONE] VERIFIED FIXED bug 934509 - crash in nsGlobalWindow::RunTimeout(nsTimeout*)
- [DONE] VERIFIED FIXED bug 945912 - Crash [@ MOZ_PNG_do_expand_plte] with broken png file
- [DROPPED] [qa-] bug 941381 - IonMonkey: Incorrect fround(asin())
- If time allows, please check crash reporter.
- Included in the smoke (report).
Release Candidate / Final
Build Info
- Builds: build1
- Build ID: 20140127194636
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Regression testing (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on beta channel (report)
- [DONE] Updates on release channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Nvidia 3d vision, Breakpad end-to-end functionality
- Platforms: Windows 8.1 64bit, Windows XP 32bit, Mac OSX 10.8, Ubuntu 13.10 32bit
- Instructions
- Run these test
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
Beta 9
Build Info
- Builds: build1
- Build ID: 20140123185438
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Regression testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Plug-in compatibility
- Platforms: Windows Vista 32bit, Windows 7 64bit, Mac OSX 10.6, Ubuntu 13.04 32bit
- Results: https://etherpad.mozilla.org/firefox27b9-testing
- Instructions
- Run this test
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
Beta 8
Build Info
- Builds: build1
- Build ID: 20140120132616
- Changes: Changelog
- Mozmill Configs (mciconf)
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Regression testing (see below)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing:
- Functionality: Download Manager (including file permissions)
- Platforms: Windows 7, Mac OSX 10.9, Ubuntu 13.10, Windows XP
- Results: https://etherpad.mozilla.org/Fx27b8-Downloads
Bugs Reported
6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);
Beta 7
Build Info
- Builds: build1
- Build ID: 20140116125114
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Regression testing (see below)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Focus areas: WebRTC
- Instructions
- Use the Moztrap test as a guideline.
- Have only one browser open on each machine at any given time
- Have only one call running at any given time on a particular machine
- Make sure the caller and callee are always on different machines.
- When finding regressions, report a bug and CC Randell Jesup and Maire Reavy (they can help track down if the regressions are real or not).
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment.
Caller | Callee | Result |
Firefox 27.0b7 on Windows 7 | Firefox 27.0b7 on Windows 7 | PASS |
Firefox 23.0 on Windows 7 | Firefox 27.0b7 on Windows XP | [FAIL] - bug 906057 |
Chrome Stable on Windows 7 | Firefox 27.0b7 on Windows 8 | PASS |
Firefox 27.0b7 on Windows 7 | Firefox 22.0 on Mac OSX 10.8 | [FAIL] - bug 922051 |
Firefox 27.0b7 on Windows 7 | Chrome Stable on Ubuntu 12.04 | PASS |
Firefox 25.0 on Windows XP | Firefox 27.0b7 on Windows 7 | [FAIL] - bug 906057 |
Firefox 27.0b7 on Windows XP | Firefox 24.0 on Windows XP | [FAIL] - bug 906057 |
Firefox 27.0b7 on Windows XP | Chrome Stable on Windows 8.1 | PASS |
Chrome Stable on Windows XP | Firefox 27.0b7 on Mac OSX 10.7 | [FAIL] - bug 936034 |
Firefox 23.0 on Windows XP | Firefox 27.0b7 on Ubuntu 13.04 | [FAIL] - bug 906057 |
Chrome Stable on Windows 8.1 | Firefox 27.0b7 on Windows 7 | PASS |
Firefox 27.0b7 on Windows 8 | Chrome Stable on Windows XP | PASS |
Firefox 22.0 on Windows 8 | Firefox 27.0b7 on Windows 8.1 | PASS |
Firefox 27.0b7 on Windows 8.1 | Firefox 26.0 on Windows 8.1 | PASS |
Firefox 26.0 on Windows 8 | Firefox 27.0b7 on Mac OSX 10.8 | [FAIL] - bug 936034 |
Chrome Stable on Windows 8.1 | Firefox 27.0b7 on Ubuntu 13.04 | PASS |
Firefox 27.0b7 on Mac OSX 10.9 | Firefox 22.0 on Windows 7 | '[FAIL] - bug 922132 |
Chrome Stable on Mac OSX 10.8 | Firefox 27.0b7 on Windows XP | PASS |
Firefox 22.0 on Mac OSX 10.6 | Firefox 27.0b7 on Windows 8 | PASS |
Firefox 27.0b7 on Mac OSX 10.7 | Chrome Stable on Mac OSX 10.8 | [FAIL] - bug 936034 |
Firefox 27.0b7 on Mac OSX 10.8 | Firefox 26.0 on Ubuntu 13.04 | PASS |
Firefox 27.0b7 on Ubuntu 12.04 | Chrome Stable on Windows 7 | PASS |
Firefox 22.0 on Ubuntu 12.04 | Firefox 27.0b7 on Windows XP | [FAIL] - bug 922051 |
Chrome Stable on Ubuntu 12.04 | Firefox 27.0b7 on Windows 8 | PASS |
Firefox 27.0b7 on Ubuntu 12.04 | Firefox 22.0 on Mac OSX 10.7 | PASS |
Firefox 27.0b7 on Ubuntu 12.04 | Chrome Stable on Ubuntu 13.04 | PASS |
Firefox 27.0b7 on Windows Vista | Chrome Stable on Ubuntu 12.04 | PASS |
Firefox 27.0b7 on Mac OSX 10.9 | Firefox 26.0 on Windows Vista | PASS |
Firefox 27.0b7 on Windows Vista | Firefox 22.0 on Windows 7 | [FAIL] - bug 922051 |
Beta 6
Build Info
- Builds: build1
- Build ID: 20140113161826
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Regression testing (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: PDF Viewer, Find toolbar, App Manager
- Platforms: Windows XP 32bit, Windows 7 64bit, Mac OSX 10.7, Ubuntu 13.10 64bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
Beta 5
Build Info
- Builds: build1
- Build ID: 20140109165205
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
- [DONE] Regression testing (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: What's New page, WebGL, Web Audio
- Platforms: Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.8.5, Ubuntu 13.10 32bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
Beta 4
Build Info
- Builds: build1
- Build ID: 20140106141415
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
- [DONE] Regression testing (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Plugin Compatibility, Update with root owned files, Mac OSX 32-bit Mode
- Platforms: Windows 8 64bit, Windows 7 64bit, Mac OSX 10.9, Ubuntu 12.04 32bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
Beta 2
Build Info
- Builds: build1
- Build ID: 20131216183647
- Changes: Changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
- [DONE] Regression testing (report)
Regression Testing
The following areas have been identified for regression testing and will be covered via Moztrap smoketests.
- Coverage
- Tests: Secure Browsing, Add-on Compatibility, Social API
- Platforms: Windows 8 64bit, Windows 7 64bit, Mac OSX 10.9, Ubuntu 12.04 32bit
- Instructions
- Run these tests
- Select one of the platforms
- Follow the steps in each of the tests and report your results within Moztrap
- Details related to "Browsing Secure Sites" test scenario can be seen in this etherpad
- Make sure to also do some exploratory testing where permitted
- If you encounter a bug, report it and add it to the list below using the bug number template
Beta 1
Build Info
- Builds: build1
- Build ID: 20131209204824
- Changes: changelog
- Mozmill Configs
Check List
- [DONE] Automation Functional tests (report)
- [DONE] Updates on betatest channel (report)
- [DONE] Updates on releasetest channel (report)
- [DONE] Updates on beta channel (report)
Aurora
Branch Migration
Pre Sign-off Tasks
- [DONE] Firefox branch migrated (Release Management)
- [DONE] Mozmill Tests branch migrated (Automation Development)
- [DONE] Updates throttled on aurora channel (Release Engineering)
- [DONE] Updates verified throttled on aurora channel (QA)
Sign-off Tasks
- [DONE] Mozmill Daily functional testruns pass with no failures (report)
- [DONE] Mozmill Daily auroratest update testruns pass with no failures (report)
- [DONE] Scope features tested and signed off for Aurora uplift
- minimal feature work for 27
Post Sign-off Tasks
- [DONE] Aurora sign-off email sent (QA)
- [DONE] Updates unthrottled on aurora channel (Release Engineering)
- [DONE] Updates verified unthrottled on aurora channel (QA)
Nightly
WebRTC
Using the Moztrap test as a guideline, verify that Firefox 27.0a1 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:
- Have only one browser open on each machine at any given time
- Have only one call running at any given time on a particular machine
- Make sure the caller and callee are always on *different* machines.
- Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
- If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
- Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller | Callee | Result |
Firefox 27.0a1 on Windows 7 | Firefox 27.0a1 on Windows 7 | PASS |
Firefox 22.0 on Windows 7 | Firefox 27.0a1 on Windows XP | FAIL (1*) |
Chrome Stable on Windows 7 | Firefox 27.0a1 on Windows 8 | PASS |
Firefox 27.0a1 on Windows 7 | Firefox 22.0 on Mac OSX 10.8 | FAIL (3*) |
Firefox 27.0a1 on Windows 7 | Chrome Stable on Ubuntu 12.04 | FAIL (6*) |
Firefox 22.0 on Windows XP | Firefox 27.0a1 on Windows 7 | FAIL (1*) |
Firefox 27.0a1 on Windows XP | Firefox 22.0 on Windows XP | FAIL (3*) |
Firefox 27.0a1 on Windows XP | Chrome Stable on Windows 8 | PASS (4*) |
Chrome Stable on Windows XP | Firefox 27.0a1 on Mac OSX 10.8 | PASS |
Firefox 22.0 on Windows XP | Firefox 27.0a1 on Ubuntu 13.04 | FAIL (1*) |
Chrome Stable on Windows 8 | Firefox 27.0a1 on Windows 7 | PASS |
Firefox 27.0a1 on Windows 8 | Chrome Stable on Windows XP | PASS |
Firefox 22.0 on Windows 8 | Firefox 27.0a1 on Windows 8 | PASS (5*) |
Firefox 27.0a1 on Windows 8 | Firefox 22.0 on Windows 8 | PASS (4*) |
Firefox 22.0 on Windows 8 | Firefox 27.0a1 on Mac OSX 10.8 | PASS (2*) |
Chrome Stable on Windows 8 | Firefox 27.0a1 on Ubuntu 13.04 | PASS |
Firefox 27.0a1 on Mac OSX 10.8 | Firefox 22.0 on Windows 7 | PASS |
Chrome Stable on Mac OSX 10.8 | Firefox 27.0a1 on Windows XP | PASS (2*) |
Firefox 22.0 on Mac OSX 10.8 | Firefox 27.0a1 on Windows 8 | PASS |
Firefox 27.0a1 on Mac OSX 10.8 | Chrome Stable on Mac OSX 10.8 | PASS |
Firefox 27.0a1 on Mac OSX 10.8 | Firefox 22.0 on Ubuntu 13.04 | PASS |
Firefox 27.0a1 on Ubuntu 12.04 | Chrome Stable on Windows 7 | PASS |
Firefox 22.0 on Ubuntu 12.04 | Firefox 27.0a1 on Windows XP | PASS |
Chrome Stable on Ubuntu 12.04 | Firefox 27.0a1 on Windows 8 | FAIL (6*) |
Firefox 27.0a1 on Ubuntu 12.04 | Firefox 22.0 on Mac OSX 10.8 | FAIL (3*) |
Firefox 27.0a1 on Ubuntu 12.04 | Chrome Stable on Ubuntu 13.04 | PASS |
Detailed Results:
- (1*) Image freezes after 2 minutes or less on callee side (the caller video freezes and the audio is working)
- not reproducible with Firefox 22 and 25.0a2
- bug 922132
- (2*) callee hears delayed sound (~5seconds)
- reproducible with Firefox 22.0
- (3*) image freezes on caller side
- not reproducible with Firefox 22.0, 25beta2
- reproducible with Firefox 26.0a2
- bug 922051
- (4*) Caller hears delayed sound
- reproducible with Firefox 22.0
- (5*) Callee hears delayed sound
- reproducible with Firefox 22.0
- (6*) Caller and callee remain in "Connecting..." state
- reproducible with Firefox 22 and 26.0a2
Bug Triage
Query | Description | Remaining |
QAWANTED | Bugs requiring immediate investigative assistance | 1 |
VERIFYME | Bugs with fixes requiring immediate verification | |
FIXED | Bugs with fixes which should be verified as soon as possible | |
REGRESSION | Bugs requiring identification of the first affected build | |
UNCONFIRMED | Bugs requiring confirmation if they are reproducible in a current build |