Releases/Firefox 31/Test Plan
Contents
- 1 Schedule
- 2 Meetings
- 3 Checklist
- 4 Features
- 4.1 Generational Garbage Collection (lizzard)
- 4.2 HTTPS for Yahoo Searches (Camelia Badau)
- 4.3 In-content Preferences (Camelia Badau)
- 4.4 Mozilla PKIX (Matt Wobensmith)
- 4.5 New Tab Page Search (Petruta Rasa)
- 4.6 New Tab Page Tiles (Cornel Ionce)
- 4.7 Plugin Whitelisting (Bogdan Maris)
- 4.8 WebVTT (Alexandra Lucinet)
- 5 Iteration Development
- 6 Stability
- 7 Bugs
- 8 Bug Fix Verifications
- 9 Milestones
Schedule
Please consult the Rapid Release Calendar for more information.
- 2014-03-17: Firefox 31 arrives on Nightly
- 2014-04-28: Firefox 31 merges to Aurora
- 2014-05-02: Firefox 31 Aurora updates enabled
- 2014-06-09: Firefox 31 merges to Beta
- 2014-06-12: Firefox 31 Beta 1 released
Meetings
| Meeting | Purpose | When | Where |
| Crashkill | Raise stability concerns with the Crashkill team | 10:00 PST on Monday | Vidyo: Stability, IRC: #crashkill |
| Channel Status | Raise quality concerns with the Release Management team | 10:00 PST on Tuesday & Thursday | Vidyo: ReleaseCoordination, IRC: #planning |
Checklist
| Priority | Check | Description | Frequency | Last Checked |
| 1 | Stability | Review the stability reports to identify new/rising crash signatures | Daily | 2014-06-18 |
| 2 | Pushlog | Review the pushlog to identify fixed bugs needing verification | Daily | 2014-06-19 |
| 3 | QA Needed Bugs | Review and assign qawanted bugs for investigation | At least twice a week | 2014-06-03 |
| 4 | Unconfirmed Bugs | Triage and test unconfirmed bug reports | At least once a week | 2014-06-09 |
| 5 | Verification Needed Bugs | Test verifyme bugs to confirm fixed bugs are fixed | At least once a week | 2014-05-05 |
| 6 | Reported Bugs | Document and escalated reported bugs as necessary | Every Monday | 2014-06-03 |
| 7 | Features | Ensure features are owned and on track for release | Once a week | 2014-05-21 |
| 8 | Automation | Ensure automation bugs are owned and on track for resolution | Once a week | 2014-06-03 |
Features
Please consult the feature testing walkthrough for more information.
Generational Garbage Collection (lizzard)
53 Total; 0 Open (0%); 50 Resolved (94.34%); 3 Verified (5.66%);
HTTPS for Yahoo Searches (Camelia Badau)
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 997215 | [ja, ja-JP-mac] Update Yahoo to use SSL, implement suggestions and add resultdomain | RESOLVED | FIXED |
| 1007159 | Add Yahoo to search.order | RESOLVED | FIXED |
| 1008206 | [rw, tn, zu] (Destkop) Restore region.properties to en-US defaults | RESOLVED | FIXED |
| 1008223 | [uk] Missing webcal handler in mobile region.properties | RESOLVED | FIXED |
4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);
In-content Preferences (Camelia Badau)
43 Total; 0 Open (0%); 19 Resolved (44.19%); 24 Verified (55.81%);
Mozilla PKIX (Matt Wobensmith)
241 Total; 32 Open (13.28%); 199 Resolved (82.57%); 10 Verified (4.15%);
New Tab Page Search (Petruta Rasa)
19 Total; 1 Open (5.26%); 6 Resolved (31.58%); 12 Verified (63.16%);
New Tab Page Tiles (Cornel Ionce)
11 Total; 0 Open (0%); 4 Resolved (36.36%); 7 Verified (63.64%);
Plugin Whitelisting (Bogdan Maris)
67 Total; 0 Open (0%); 47 Resolved (70.15%); 20 Verified (29.85%);
WebVTT (Alexandra Lucinet)
170 Total; 32 Open (18.82%); 124 Resolved (72.94%); 14 Verified (8.24%);
Iteration Development
Ensure these bugs have QA status as per Iterative Development guidelines.
120 Total; 0 Open (0%); 0 Resolved (0%); 120 Verified (100%);
Stability
This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.
- Explosive Crashes - ensure new and explosive crashes have actionable bug reports which are in progress
- Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
- Untriaged Bugs - ensure bugs are actionable and in the correct component
- Top crashers by Bug Number
| ID | Summary | Component | Op sys | Status firefox31 |
|---|---|---|---|---|
| 883134 | crash in nsPluginHost::StopPluginInstance mainly with Adicional de Seguranca CAIXA add-on | Plug-ins | Windows 7 | affected |
| 963962 | crash in sse2_composite_src_x888_8888 | Graphics: CanvasWebGL | Windows 8.1 | fixed |
| 974656 | Linux crash in gfxContext::PushClipsToDT(mozilla::gfx::DrawTarget*) | Graphics | Linux | affected |
3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Bugs
QA Wanted Bugs
| ID | Summary | Status | Qa contact | Component |
|---|---|---|---|---|
| 945192 | The updater.exe loads the bcrypt.dll and other dll's from the working and binary directory when not using the service (Application Update) | VERIFIED | Kamil Jozwiak [:kjozwiak] | Application Update |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Unconfirmed Bugs
219 Total; 219 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Fixed Bugs
Review the pushlog and place each Desktop Firefox bug in one of the following categories:
- A) Tested by the QA Team --> add the verifyme keyword to the bug and set the status-firefox31 flag to fixed
- B) Tested by a new volunteer --> add the [good first verify] whiteboard tag
- C) Tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
- D) Not tested at all --> add the [qa-] whiteboard tag
Tips:
- If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
- Be sure to set the status-firefox31:fixed flag
- If the bug includes tests from a developer you can set in-testsuite+
- Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures
- Untriaged Bugs
- Product fixes landed in Firefox 31 (2 bugs remaining as of 2014-07-11)
- Platform fixes landed in Firefox 31 (759 bugs remaining as of 2014-07-11)
- Product fixes uplifted in Firefox 31 (1 bugs remaining as of 2014-07-16)
- Platform fixes uplifted in Firefox 31 (59 bugs remaining as of 2014-07-16)
Reported Bugs
Ensure these bugs remain on track to resolve and are verified when fixed.
22 Total; 2 Open (9.09%); 11 Resolved (50%); 9 Verified (40.91%);
Bug Fix Verifications
- Review the bugs below to identify a bug you want to test
- Make sure you understand the bug before you begin testing it
- Test on a previously known broken build to make sure you can reproduce the bug
- Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
- Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
- Reopen the bug if it still reproduces
| ID | Summary | Component | Severity | Op sys |
|---|---|---|---|---|
| 989549 | Call signOut() in FxAccountsClient.jsm from signOut() in FxAccounts.jsm | Sync | normal | All |
| 991251 | Heap-buffer-overflow in mozilla::AudioBlockCopyChannelWithScale triggered with ChannelMergerNode | Web Audio | critical | All |
| 1007389 | Implement plugin whitelist, round 2 | Plug-ins | normal | All |
| 1015636 | pdfjs privilege escalation round 2 | PDF Viewer | normal | Linux |
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Milestones
Aurora Migration
Pre-sign-off Checks:
- [DONE] Automation - document all Mozmill test failures and memory usage regressions
- [DONE] Features - all scoped features are signed off as ready for Aurora by owners
- [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious
- [DONE] Stability - review crash stats before sign-off to raise any red flags
- [DONE] L10n - contact :Pike to see if there are any known localization issues
Post-sign-off Checks:
- [DONE] Updates - previous versions of Aurora update to the new version (check after updates live)
- [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
Automation
- Testrun Reports
| Report | Test Failures |
| Add-ons | 0 failing tests |
| Functional | bug 999357: /testSessionStore/testUndoTab.js::testUndoTabFromContextMenu bug 866599: /testPreferences/testPasswordSavedAndDeleted.js::testSaveAndDeletePassword bug 904610: /restartTests/testAddons_changeTheme/test3.js::testChangedThemeToDefault |
| Remote | bug 1003083 /testSecurity/testSecurityNotification.js::testSecNotification bug /restartTests/testAddons_InstallAddonWithoutEULA/test2.js::testInstallAddonWithoutEULA bug 992187 /restartTests/testAddons_InstallAddonWithoutEULA/test1.js::testInstallAddonWithEULA bug 952034 /restartTests/testAddons_installFromFTP/test2.js::testAddonInstalled |
| L10n | bug 1004827 /testAccessKeys/test1.js::testPrefWindowAccessKeys |
| Updates | 0 failing tests |
| Endurance | 0 failing tests |
- Memory Usage Report
| Platform | Min Memory Usage | Max Memory Usage | Avg Memory Usage | Net Compared to Firefox 30 |
| Windows 32-bit | 78 MB | 99 MB | 95 MB | +14 MB [17%] |
| Mac OS X | 133 MB | 140 MB | 135 MB | +17 MB [14%] |
| Linux 32-bit | 90 MB | 91 MB | 90 MB | +11 MB [14%] |
| Linux 64-bit | 124 MB | 125 MB | 125 MB | +15 MB [14%] |
Memory regression being tracked/investigated in bug 1004608
Features
| Feature | Owner | Aurora Sign-off Status | Potential Blockers |
| Newtab page Tile Grid | Cornel Ionce | Ready for Aurora | N/A |
| HTTPS for Yahoo Searches | Camelia Badau | Ready for Aurora | N/A |
| In-content preferences | Camelia Badau | Ready for Aurora | N/A |
| Search field in Newtab page | Petruta Rasa | Ready for Aurora | N/A |
| WebVTT | Alexandra Lucinet | Ready for Aurora | N/A |
| Plugin whitelist | Bogdan Maris | ON TRACK | N/A |
Updates
| Platform | Source Build | Target Build | Result |
| Windows | Firefox 30.0a2 en-US 2014-04-06 | Firefox 31.0a2 en-US 2014-05-02 | PASS |
| Firefox 30.0a2 eu 2014-04-28 | Firefox 31.0a2 eu 2014-05-02 | PASS | |
| Mac OS X | Firefox 30.0a2 en-US 2014-04-10 | Firefox 31.0a2 en-US 2014-05-02 | PASS |
| Firefox 30.0a2 as 2014-04-25 | Firefox 31.0a2 as 2014-05-02 | PASS | |
| Linux | Firefox 30.0a2 en-US 2014-04-01 | Firefox 31.0a2 en-US 2014-05-02 | PASS |
| Firefox 31.0a2 zh-TW 2014-04-29 | Firefox 31.0a2 zh-TW 2014-05-02 | PASS with issues [1] |
1. some locales are not updating passed Firefox 31.0a2 2014-05-01
Reported Bugs
| ID | Summary | Resolution |
|---|---|---|
| 1003155 | Firefox 31 memory usage increase since April 27th | FIXED |
| 1005394 | Make sure all Aurora users are updated to Firefox 31.0a2 latest on Monday May 5th, 2014 | FIXED |
2 Total; 2 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 1
Read the Beta Coordination Guide for more info
- Build1 (20140610163407)
- Automated Testing
| ID | Summary | Priority | Status |
|---|---|---|---|
| 1024141 | Firefox shutdown crash in testHomeButton.js | P1 | RESOLVED |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
- Manual Testing
- [DONE] Smoketest new features
- [DONE] Verify no HSB builds were generated
- [DONE] Verify notice is shown to beta users when we turn telemetry on by default (bug 981689)
5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);
Beta 2
Read the Beta Coordination Guide for more info
- Build1 (20140616143923)
- Automated Testing
- Manual Testing
- [DONE] Verify that now, the HSB builds were generated - PASS
- [DONE] Regression testing
- Update testing on releasetest due to bug 1026738 (source)
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1026582 | Test failure in testSessionStore.testUndoTab.testUndoTabViaShortcut for [hsb] locale | RESOLVED | FIXED |
| 1026738 | Firefox 31.0b2 update tests are failing on releasetest | RESOLVED | FIXED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Beta 3 [SKIPPED DUE TO BUG 1027983]
Read the Beta Coordination Guide for more info
- Build1 (20140619131915)
- Automated Testing
- Manual Testing
- [DONE] Regression testing - PASS
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1027983 | bm-2008-sm-00xx fail to link xul.dll when build is PGO enabled | RESOLVED | WONTFIX |
| 1028035 | Intermittent Win PGO timeout running expandlibs_exec.py for xul.dll ("command timed out: 7200 seconds without output, attempting to kill") | RESOLVED | DUPLICATE |
| 1028183 | [OSX10.6] Audio controls issues on mp4 files | RESOLVED | INCOMPLETE |
| 1028230 | [OS X] Tooltip for "Print" button in panel menu should not end with an ellipsis | NEW | |
| 1028237 | Video controls cut and flickering when fading away | VERIFIED | FIXED |
5 Total; 1 Open (20%); 3 Resolved (60%); 1 Verified (20%);
Beta 4
Read the Beta Coordination Guide for more info
- Build1 (20140623175014)
- Automated Testing
- Functional Tests (top fails)
- Update Tests on betatest
- Update Tests on releasetest
- Update Tests on beta
- Manual Testing
- [DONE] Regression testing - PASS
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1027983 | bm-2008-sm-00xx fail to link xul.dll when build is PGO enabled | RESOLVED | WONTFIX |
| 1028035 | Intermittent Win PGO timeout running expandlibs_exec.py for xul.dll ("command timed out: 7200 seconds without output, attempting to kill") | RESOLVED | DUPLICATE |
| 1028230 | [OS X] Tooltip for "Print" button in panel menu should not end with an ellipsis | NEW | |
| 1028237 | Video controls cut and flickering when fading away | VERIFIED | FIXED |
| 1029480 | Ubuntu menu bar and dock are not always hidden in fullscreen mode | RESOLVED | WORKSFORME |
5 Total; 1 Open (20%); 3 Resolved (60%); 1 Verified (20%);
Beta 5
Read the Beta Coordination Guide for more info
- Build1 (20140626181429)
- Automated Testing
- Functional Tests (top fails)
- Update Tests on betatest
- Update Tests on releasetest
- Update Tests on beta
- Manual Testing
- [DONE] Regression testing - PASS
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1027103 | Google Maps Street View crashes in gfxContext::PushClipsToDT(mozilla::gfx::DrawTarget*) | VERIFIED | FIXED |
1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);
Beta 6
Read the Beta Coordination Guide for more info
- Build2 (20140630185627)
- Automated Testing
- Functional Tests (top fails)
- Update Tests on betatest
- Update Tests on releasetest
- Update Tests on beta
- Manual Testing
- [ON TRACK] Regression testing - PASS
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1032770 | Updating older beta FF versions through About Firefox page, updates the build to FF 29 instead of FF 31.0b6. | RESOLVED | DUPLICATE |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Beta 7
Read the Beta Coordination Guide for more info
- Build1 (20140703154127)
- Automated Testing
- Manual Testing
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1034592 | Social service cloud icon still on screen after closing a large number of chat panels | RESOLVED | WONTFIX |
1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);
Beta 8
Read the Beta Coordination Guide for more info
- Build1 (20140707160635)
- Automated Testing
- Manual Testing
- Social Panel Regression Tests - run these on Windows XP, Windows 8.1, Mac OSX 10.9, and Ubuntu 13.10
- WebGL Demos - MDN, WebGL.com - run these on Windows 7, Windows 8.1, Mac OSX 10.9, and Ubuntu 13.10 with at least one AMD, one NVidia, and one Intel GPU
- Firefox 31 regression tests
- Bugs
| ID | Summary | Status | Resolution |
|---|---|---|---|
| 1035776 | fix link text in demo provider to have clear explanations | RESOLVED | INVALID |
| 1035803 | Expanding the toolbar by pressing ">>" button is not working the first time if demo social provider is enabled. | RESOLVED | DUPLICATE |
| 1035835 | panel resizing seems to stop working after opening several windows | RESOLVED | WONTFIX |
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Beta 9
Read the Beta Coordination Guide for more info
- Build1 (20140710141843)
- Automated Testing
- Manual Testing
- Exploratory testing for stability issues - PASS
- Bugs
Bugzilla query error
error, http-bad-status, Array
Release Candidate 1
Read the Beta Coordination Guide for more info
- Build1 (20140714151536)
- Automated Testing
Note: keep an eye on addon hotfix bugs to verify them asap: https://bugzilla.mozilla.org/buglist.cgi?f1=blocked&list_id=10737327&o1=equals&resolution=---&resolution=FIXED&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&v1=928173
Sign-off for Beta
- Verifications
- bug 1038282 Remove sw during beta-release migration for Firefox 31, keep hsb - PASS
Sign-off for Release
- Verifications
- bug 1028972 crash in xul.dll@0x8e13cd | xul.dll@0x1ab44c | xul.dll@0x126047 | xul.dll@0x19effe | xul.dll@0x1c8403 | _MD_CURRENT_THREAD | PR_Unlock | xul.dll@0x11d2d4 | xul.dll@0x1c844d | xul.dll@0x34a70d
- bug 988549 startup crash in _VEC_memzero | _VEC_memzero - N/A (latest change is NOT in Firefox 31 RC build 1)
Fixes landed on RC build (still need verification)
- https://bugzilla.mozilla.org/show_bug.cgi?id=812695 - [Read comment #414] [D2D] Text Rendering Issues on Windows 7 with Platform Update KB2670838 (MSIE 10 Prerequisite) or on Windows 8.1 ==> this is a rather old rendering issue affecting some Radeon cards. Going through the bug I could not find any Radeon model that we have available. There is only one machine with an AMD Radeon HD 3000, which may have been affected, but we haven't tried it yet.
- https://bugzilla.mozilla.org/show_bug.cgi?id=1034327 - Memory leak with TURN relay ==> according to Nils: "This was caught in the WebRTC QA lab and can be verified with that as well. Additionally manual verification has been done and can be done as needed."
Release Candidate 2
Read the Beta Coordination Guide for more info
- Build1 (20140716183446)
- Automated Testing
- Manual Testing
- Conduct testing of the New Tab page to ensure nothing regressed due to bug 1038997 - FAIL (test results)
Noteable Issues
- bug 1015341, bug 1035534, bug 1035537 and bug 1035529 all related to Movie Toolbar (bandoo) and combine to be our #7 topcrash
- Bandoo has been contacted and we'll be tracking this in Release (They are working on a fix)
- bug 984033 is an OOM that is now #4, correlated with Crossrider framework add-ons
- 0 crashes in Nightly since the patch landed in mozilla-central on July 17
- too late for Release but it should be uplifted to 32 and might be good to consider as a ridealong if we do a .point release
- bug 1012924 (experiments shutddown crash) is #2/#3 (depending on specific report) after OOM|small, competing with processMarkStackTop
- this should be gone when we release (just like Fx30) but raises concerns about running experiments through the entire Beta cycle (perhaps disable it in the last Beta until stability settles down)
- New tab page issues (etherpad) -- do not block as they are also reproducible in Fx 30 (ie. not regressions from backing out New Tab Page Tiles)
