Releases/Firefox 19/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 18 | Firefox 19 | Firefox 20 »

Contents

Firefox 19 Test Plan

Summary

Milestone Firefox 19
Status Currently in Beta -- merging to Release on February 19th, 2013
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)

The following is the test plan for Firefox 19 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 19. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-10-08: [DONE] Nightly Merge
  • 2012-11-23: [DONE] Aurora Merge
  • 2013-01-07: [DONE] Pre-Beta feature sign-offs
  • 2013-01-10: [DONE] Beta 1
  • 2013-01-17: [ON TRACK] Beta 2
  • 2013-01-24: {{|}} Beta 3
  • 2013-01-31: {{|}} Beta 4
  • 2013-02-07: {{|}} Beta 5
  • 2013-02-14: {{|}} Beta 6
  • 2013-02-15: {{|}} Release Candidate
  • 2013-02-19: {{|}} Final

Nightly

Merge

Functional Automation - result
Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Panel-based Download Manager Simona Badau Delayed to Firefox 20
Tab Strip Visual Redesign Virgil Dicu Delayed
Windows Plugin Hang UI Manuela Muntean Delayed
Show PDF inline Mihaela Velimiroviciu On Track
Awesomebar Search Provider Reset On Track

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated - Nov 1st, 2012
ID Summary Priority Status
807338 Memory increase discovered by mozmill endurance tests - Mac 10.7&10.8 since Oct25 -- RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated

Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (SoftVision) - Last Updated

Result Spreadsheet

Aurora Merge

Builds nightlies
Changeset tip (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-11-23
Windows Vista PASS - 2012-11-23
Windows 7 32-bit PASS - 2012-11-23
Windows 7 64-bit PASS - 2012-11-23
Windows 8 32-bit PASS - 2012-11-23
Windows 8 64-bit PASS - 2012-11-23
Mac OS X 10.6 PASS - 2012-11-23
Mac OS X 10.7 PASS - 2012-11-23
Mac OS X 10.8 PASS - 2012-11-23
Ubuntu 12.04 32-bit PASS - 2012-11-23
Ubuntu 12.04 64-bit PASS - 2012-11-23
Test Failures (known bugs)

Update Testing

Spotcheck updates from a previous Aurora to Aurora 19.

Platform Aurora channel
Windows 7 64-bit PASS - 2012-11-26 Firefox 17.0a2 2012-10-11 -> Firefox 19.0a2 2012-11-26
Windows 8 64-bit PASS - 2012-11-26 Firefox 18.0a2 2012-11-15 -> Firefox 19.0a2 2012-11-26
Mac OS X 10.7 PASS - 2012-11-26 Firefox 18.0a2 2012-11-19 -> Firefox 19.0a2 2012-11-26
Ubuntu 12.04 64-bit PASS - 2012-11-26 Firefox 17.0a2 2012-10-01 -> Firefox 19.0a2 2012-11-26
Issues

Bug Triage

  1. VERIFYME: verify the fix and do some exploratory testing around the issue
  2. FIXED: triage these bugs and test to ensure the fix is effective
  3. QAWANTED: triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED: triage these bugs for potential new regressions

QAWANTED

Bugs requiring QA's assistance in debugging (steps, regression range, exploratory testing, verification, etc)

Status as of 2012-12-20:

  • bug 820805 crash in js::ion::IonBailoutIterator::IonBailoutIterator or js::ion::OperatorIn
    • WORKSFORME - Unclear if qawanted is still neeeded (marcia)
  • bug 818626 crash in nsStyleContext::~nsStyleContext
    • ASSIGNED - Untracked due to failed attemtps to reproduce, no more leads
  • bug 823119 crash in mozilla::gfx::DrawTargetD2D::CopySurface with PDF Viewer
    • NEW - Reproduced many stability issues but not the specific signature (ashughes)
  • bug 668583 crash @ js::gc::ScanRope
    • NEW - Pending feedback from Engineering about leads for QA to test (ashughes)
  • bug 698296 Firefox Crash Report @ je_free | js::gc::Arena::finalize
    • NEW - So far unreproducible, any other leads? (ashughes)

Pre-beta Feature Sign-off

Firefox 19.0a2 Feature Status

Feature Owner Status
In-line PDF Viewer MihaelaV (SV) Ready for Beta
Awesomebar Search Provider Reset (bug 718088) Paul Silaghi (SV), Manuela Muntean (SV) Will be tested via Testday
Mac OSX 10.8 Notification Center Support (bug 728106, bug 777409) unassigned Moved to Firefox 20
Panel-based Download Manager Simona Bidau (SV) Moved to Firefox 20
Windows Plugin Hang UI Manuela Muntean (SV) Moved to Firefox 20
Tab Strip Visual Redesign Virgil Dicu (SV) Moved to Firefox 20

Beta Checklist

The following checklist should be completed before signing off for Release.

Beta 1
  • [DONE] Breakpad End-to-End functionality
  • [DONE] Mac OSX 10.8 Signing
  • [DONE] Add-on compatibility (top AMO, top third-party, toolbars, yandex, avg, norton, yahoo, bandoo, conduit, etc)
  • [DONE] Exploratory testing and bug triage against the flagship feature
  • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.8, Ubuntu 12.04 32-bit
Beta 2
  • [DONE] Breakpad End-to-End functionality
  • [DONE] Stop accepting MD5 as a hash algorithm in signatures
  • [DONE] Plug-in compatibility (Flash, Java, Silverlight, Realplayer, Quicktime, Shockwave, Unity [Mac])
  • [DONE] Firefox Beta testday stressing flagship features and bug triage
  • [DONE] Exploratory testing and bug triage against the flagship feature
  • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
Beta 3
  • [DONE] Breakpad End-to-End functionality
  • [DONE] Mac OSX 32-bit mode
  • [DONE] Web compatibility (search, webmail, banking, video, audio, gaming, news, cms, etc.)
  • [DONE] Exploratory testing and bug triage against the flagship feature
  • Platforms: Windows 7 32-bit, Windows Vista, Mac OSX 10.6, Ubuntu 12.04 32-bit
Beta 4
  • [DONE] Breakpad End-to-End functionality
  • [DONE] Verify update applies with root owned files
  • [DONE] Add-on compatibility (top AMO, top third-party, toolbars, yandex, avg, norton, yahoo, bandoo, conduit, etc)
  • [DONE] Exploratory testing and bug triage against the flagship feature
  • [DONE] Firefox Beta testday stressing flagship features and bug triage
  • Platforms: Windows 7 64-bit, Windows XP, Mac OSX 10.8, Ubuntu 12.10 32-bit
Beta 5
  • [DONE] Breakpad End-to-End functionality
  • [DONE] What's New Page on Firefox update
  • [DONE] Plug-in compatibility (Flash, Java, Silverlight, Realplayer, Quicktime, Shockwave, Unity [Mac])
  • [DONE] Exploratory testing and bug triage against the flagship feature
  • Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 32-bit
Beta 6
  • {{ |}} Breakpad End-to-End functionality
  • {{ |}} Mac OSX 10.5 and Windows 2000 de-support (install and update)
  • {{ |}} Web compatibility (search, webmail, banking, video, audio, gaming, news, cms, etc.)
  • {{ |}} Exploratory testing and bug triage against the flagship feature
  • Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.6, Ubuntu 12.04 64-bit, Mac OSX 10.5, Windows 2000

Beta 1

Builds FTP, HTTP
ID 20130109111322
Changeset 222e6877be4b (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Triage
  • [DONE] Update Automation on betatest channel
  • [DONE] Update Automation on releasetest channel
  • [DONE] Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP [DONE] 0 Fails [DONE] 0 Fails
Windows Vista [DONE] 0 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 64-bit [DONE] 0 Fails [DONE] 0 Fails
Test Failures (known bugs)

Manual Testing

Report any bugs found during exploratory testing in Bugzilla and add them to the Issues Found section below.

Focus Area Windows 7 Windows 8 Mac OSX 10.7 Mac OSX 10.8 Ubuntu 12.04
Regression Tests:
Run the Firefox 19.0b1 Regression Tests in MozTrap. Do some exploratory testing around the testcases to try to exercise some edge-cases.
[DONE]
Manuela
1 Bugs
[DONE]
Simona
0 Bugs
[DONE]
Bogdan
0 Bugs
[DONE]
Paul
0 Bugs
[DONE]
Alexandra
0 Bugs
PDF.js:
Test the new built-in PDF reader to check for issues with viewing and printing PDFs. Check some of the behaviour with other PDF readers side-loaded (ie. Adobe Reader, Foxit Reader, etc).
[DONE]
Manuela
0 Bugs
[DONE]
Simona
1 Bugs
[DONE]
Bogdan
0 Bugs
[DONE]
Paul
0 Bugs
[DONE]
Cornel
0 Bugs
Add-on Compatibility:
Test the top add-ons for stability issues and functional issues. Test featured add-ons from AMO and Third-party toolbars like Yahoo, Yandex, AVG, Amazon, Norton, etc.
[DONE]
Manuela
1 Bugs
[DONE]
Simona
1 Bugs
[DONE]
Bogdan
0 Bugs
[DONE]
Paul
0 Bugs
[DONE]
Cornel
0 Bugs
Plug-in Compatibility:
Test the latest plug-ins for stability issues and visual glitches. Test plugins like Flash, Reader, Java, Quicktime, Realplayer, Windows Media Player, Silverlight, etc.
[DONE]
Manuela
0 Bugs
[DONE]
Simona
0 Bugs
[DONE]
Bogdan
0 Bugs
[DONE]
Paul
0 Bugs
[DONE]
Cornel
0 Bugs
Web Compatibility:
Test some of the top visited websites to look for stability issues and UI glitches. Some suggestions include WebMail (including built in messengers), Social Networks, Video sites, Media streaming websites, News websites, Content Management systems, Web banking, HTML5 video, WebGL/MapsGL, User forums
[DONE]
Manuela
0 Bugs
[DONE]
Simona
0 Bugs
[DONE]
Bogdan
0 Bugs
[DONE]
Paul
0 Bugs
[DONE]
Cornel
0 Bugs

Update Testing

Results of running the Mozmill update automation.
18.0b6 should get a partial update, <=18.0b5 should get a complete update.

Platform Betatest (2013-01-10) Releasetest (2013-01-10) Beta (2013-01-10)
Windows XP [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows Vista [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.6 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.7 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.8 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails

Bug Triage

QAWANTED

Bugs: Test to find steps to reproduce, regression range, or testcases Status
bug 820506 Problem selecting image node in SVG when used with clip-path in certain condition TESTING (ashughes)
bug 668583 Crash @ js::gc::ScanRope NEEDINFO: Need more information from developer
bug 698296 Firefox Crash Report @ je_free js::gc::Arena::finalize NEEDINFO: Need more information from developer
bug 814200 Xt plugins cannot be resized larger in FF17 VERIFYME: To be verified in 19b2
bug 815542 Crash in IncrementalCollectSlice DROPPED: Nothing left for QA to do

NORMAL FIXES

Bugs: Verify these fixes in the latest Beta candidate Status
bug 828507 Firefox 19b1 Beta updates fail to apply TESTING (ashughes)
bug 802476 Printing crashes with a local file VERIFIED FIXED (Cornel)
bug 789933 Crash in mozilla::gfx::DrawTargetCairo::Stroke VERIFIED FIXED (Bogdan)
bug 818276 Scrollbar on popoups disappeared after updating VERIFIED FIXED (Paul)
bug 812828 "Open All in Tabs" fails if Library window is open with no background windows VERIFIED FIXED (Ioana)
bug 810586 Collapsible table not rendered correctly NEEDINFO: Bug still reproduces under certain conditions (Paul)
bug 810531 Invalidation artifacts when scrolling w3.org working draft pages VERIFIED FIXED (Manuela)
bug 799900 Wrong position of East Asian input method candidate box in HiDPI mode

SECURITY FIXES

Bugs: Verify these fixes in the latest Beta candidate (requires privileged access) Status
bug 814839 WebGL conformance test hang [@mozilla::WebGLContext::Clear] VERIFIED FIXED (mwobensmith)
bug 756581 JS OOM Testing: Assertion failure DROPPED: Needs custom test script, reporter to verify (mwobensmith)
bug 812341 Opt-only Crash [@ SuppressDeletedPropertyHelper] DROPPED: Needs custom shell, reporter to verify (mwobensmith)
bug 814407 WebGL conformance crash with Mesa 8.0 after calling texImage2D DROPPED: Needs custom hardware, reporter to verify (mwobensmith)
bug 814716 WebGL conformance crash (NVIDIA Noveau Linux driver) DROPPED: Needs custom hardware, reporter to verify (mwobensmith)
bug 780770 Content can prevent chrome buttons from working VERIFIED FIXED (mnoorenberghe)

TOPCRASH FIXES

Verify these signatures are no longer present on crashstats in the current Beta

  • N/A for Beta 1 due to lack of crash data

UNCONFIRMED

Bugs: Triage unconfirmed bugs for potential incoming regressions. Status
bug 806137 Private browsing wipes regular browsing session CONFIRMED (Ioana)
bug 803840 keyboard shortcuts are different on each platform CONFIRMED (Ioana)
bug 800032 Twitter website F5 key not working NEEDINFO (Ioana)
bug 823722 "Flash Video Downloader - Youtube Downloader" stopped working with update RESOLVED WORKSFORME (Ioana)
bug 810131 Status bar appears when switching loading tab to new tab NEEDINFO (Ioana)
bug 812871 Highlighted context menu items font rendering issues on Windows XP NEEDINFO (Ioana)
bug 809805 GameDesire Pool doesn't open in Nightly NEEDINFO (Ioana)
bug 818863 Pinned tabs do not work with "don't load tabs until selected" feature RESOLVED DUPE of bug 713190 (Ioana)
bug 812888 Issue with viewing YouTube videos. RESOLVED WORKSFORME (Ioana)
bug 805426 Navigator error pops up when clicked on picture RESOLVED INVALID (Ioana)

Issues Found

Bugs: New issues found during testing the Beta candidate Status
bug 828372 Yandex Elements toolbar buttons are missplaced RESOLVED INVALID: Yandex Elements bug
bug 828473 Yahoo Toolbar changes position after Yandex bar is installed RESOLVED INVALID: Yahoo Toolbar bug
bug 828507 Firefox 19b1 Beta updates fail to apply RESOLVED FIXED (Simona)
bug 828898 Firefox fails to display all PDF handling options RESOLVED INVALID: Foxit Reader bug

Items in bold are BLOCKER bugs

Beta 2

Builds FTP, HTTP
ID 20130116072953
Changeset 8b833c1150b5 (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Triage
  • [DONE] Update Automation on betatest channel
  • [DONE] Update Automation on releasetest channel
  • [DONE] Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP [DONE] 0 Fails [DONE] 0 Fails
Windows Vista [DONE] 0 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 64-bit [DONE] 0 Fails [DONE] 0 Fails
Test Failures (known bugs)

Manual Testing

Focus testing around bug fixes landed between Beta 1 and Beta 2 (see Bug Triage section below).

Update Testing

Results of running the Mozmill update automation.
19.0b1 should get a partial update, <=18.0b6 should get a complete update.

Platform Betatest (2013-01-16) Releasetest (2013-01-17) Beta 2013-01-17)
Windows XP [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows Vista [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.6 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.7 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.8 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails

Bug Triage

NORMAL FIXES

Verify these fixes in the latest Beta candidate.

  • [DONE] VERIFIED bug 828202 System proxy settings issue with Firefox 18 [1] (Manuela)
  • [DONE] VERIFIED bug 814200 Xt plugins cannot be resized larger in FF17 (Virgil)
  • [DONE] VERIFIED bug 828632 System Proxy Configuration Causes Memory Leak, syslogd load [2] (Simona)
  • [DONE] VERIFIED bug 828236 issue with NTLM and some SSL Proxies [3] (reporter)
  • [DONE] VERIFIED bug 829646 Proxy Auto Config calling myIPAddress() in the top level fails (Alexandra)
  • [DONE] VERIFIED bug 818468 [Linux] Langpacks bundled in distribution/extensions are registered but disabled (reporter)
  • [DONE] VERIFIED bug 814434 [hidpi/retina] url suggestion box pops up on wrong screen (mwobensmith)
  • [DONE] VERIFIED bug 830303 Fix plugin-container spawning on BSD after bug 753046 (reporter)
  • [MISSED] NEEDS CRASHSTATS bug 828184 startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll
  • [MISSED] UNVERIFIABLE bug 828205 crash in js::DebugScopeProxy::has with Firebug
    • Relies on having Ubuntu-provided Firefox 19.0b2 builds for Ubuntu 10.04 which do not exist

These will require setting up a proxy and having Firefox connect to the internet through that proxy. 1. "This only impacts system proxy pac file settings on all platforms. file:// uris are the only type known to be effected. uris pointing to non existant files, using strange syntax, etc..."

2. "This is an OSX specific issue and the patch is OSX specific at compile time, so don't waste time with other platforms. Configure a system proxy using a file:// pac file and you should see the logging entries syslog (using 18.0.0) and verify they are gone with 18.0.1 while verifying that the pac file is honored. The fix generally impacts the pac file handling for os x so some general testing could be redone."

3. "Setup squid to do integrated windows authentication (i.e. NTLM) to access the proxy and then configure proxy use for http and https traffic. Honestly, that's a giant pain in the butt to do but that's what the interop problem was - specifically with accessing https through the proxy. Manual configuration is fine in this case."

4. "This bug is pretty self descriptive and the patch for it includes an automated regression test. The issue involves using the myIPAddress() function in PAC file outside the scope of the FindProxyForURL() function - the test in the patch illustrates such a case."

SECURITY FIXES

Verify these fixes in the latest Beta candidate (requires privileged access)

  • [DONE] VERIFIED bug 775228 (CVE-2012-5830) use-after-free when loading html file on osx (mwobensmith)
  • [DONE] VERIFIED bug 812341 Opt-only Crash [@ SuppressDeletedPropertyHelper] (mwobensmith)
  • [DONE] VERIFIED bug 785734 (CVE-2012-5833) Mesa crashes on certain texImage2D calls (reporter)

TOPCRASH FIXES

Verify these signatures are no longer present on crashstats in the current Beta

QAWANTED

Test the following bugs to find steps to reproduce, regression range, or testcases.

  • [AT RISK] NEEDS ENGR bug 668583 crash @ js::gc::ScanRope (ashughes)
  • [AT RISK] NEEDS ENGR bug 698296 crash in js::SweepBackgroundThings @ js::gc::Arena::finalize
  • [ON TRACK] VERIFYME bug 818468 Langpacks bundled in distribution/extensions are registered but disabled even if shown enabled
  • [ON TRACK] VERIFYME bug 828184 startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll (Websense Endpoint)
  • UNASSIGNED bug 783369 Norton Confidential 2012/2013 crashes (couictlr.dll, coffplgn.dll)

UNCONFIRMED

Bugs: Triage unconfirmed bugs for potential incoming regressions. Status

Issues Found

Bugs: New issues found during testing the Beta candidate Status

Items in bold are BLOCKER bugs

Known Issues

ID Summary Priority Status
747909 Youtube.com In Full Screen mode - copy actions are not allowed on HTML5 videos -- RESOLVED
797116 [Win8] Flash hang in F_1152915508 while playing Knight Hunters P3 NEW

2 Total; 1 Open (50%); 1 Resolved (50%); 0 Verified (0%);

Beta 3

Builds FTP, HTTP
ID 20130123083802
Changeset 19124ba6cd20 (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Triage
  • [DONE] Update Automation on releasetest channel
  • [ON TRACK] Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP PASS PASS
Windows Vista PASS PASS
Windows 7 32-bit PASS PASS
Windows 7 64-bit PASS PASS
Windows 8 32-bit PASS PASS
Windows 8 64-bit PASS PASS
Mac OS X 10.6 PASS PASS
Mac OS X 10.7 PASS PASS
Mac OS X 10.8 PASS PASS
Ubuntu 12.04 32-bit PASS PASS
Ubuntu 12.04 64-bit PASS PASS
Test Failures (known bugs)

Manual Testing

Focus testing around bug fixes landed between Beta 2 and Beta 3 (see Bug Triage section below).

Update Testing

Results of running the Mozmill update automation.
19.0bx should get a partial update, <=xx.x should get a complete update.

Platform [link Betatest] (2013-01-24) [link Releasetest] (2013-01-24) [link Beta] (2013-01-24)
Windows XP Skipped http://mozmill-ondemand.blargon7.com/#/update/reports?branch=19.0&platform=All&from=2013-01-24&to=2013-01-25 status: number of fails
Windows Vista Skipped http://mozmill-ondemand.blargon7.com/#/update/reports?branch=19.0&platform=All&from=2013-01-24&to=2013-01-25 status: number of fails
Windows 7 32-bit Skipped PASS: results status: number of fails
Windows 7 64-bit Skipped PASS: results status: number of fails
Windows 8 32-bit Skipped PASS: results status: number of fails
Windows 8 64-bit Skipped PASS: results status: number of fails
Mac OS X 10.6 Skipped PASS: results status: number of fails
Mac OS X 10.7 Skipped PASS: results status: number of fails
Mac OS X 10.8 Skipped PASS: results status: number of fails
Ubuntu 12.04 32-bit Skipped PASS: results status: number of fails
Ubuntu 12.04 64-bit Skipped PASS: results status: number of fails

Bug Triage

NORMAL FIXES

Verify these fixes in the latest Beta candidate. (change link above for beta 3)

  • [DONE] VERIFIED bug 828286 center of svg animation is incorrect on industrial demo at zamek.pmmf.hu, ever since SVG display lists became enabled (Ioana)
  • [DONE] VERIFIED bug 829921 click events not delivered with trivial CSS transform, margin, overflow: hidden (Ioana)
  • [DONE] VERIFIED bug 827217 webgl.can-lose-context-in-foreground=false crashes in desktop Firefox (Ioana)
  • [DONE] NEEDINFO bug 826115 warnings about rollups too noisy (Manuela)
  • [DONE] VERIFIED bug 818023 IonMonkey: function.caller does not always work (and in interpreter and JM) (Paul)
  • [DONE] [QA-] bug 829858 Null dereference in nsHttpChannel::InitOfflineCacheEntry() (Paul)
  • [DONE] NEEDINFO bug 789975 Extremely long pause while collecting telemetry information on the main thread (Paul)
  • [DONE] VERIFIED bug 830066 Drag & drop links into toolkit downloads manager to download files fails (TypeError: aInitiatingDocument is undefined) (Cornel)
  • [DONE] DROPPED bug 804605 Kerberos authentication does not work with CNAME (Cornel)
    • Asked the reporter to verify.
  • [DONE] NEEDINFO bug 812647 Startup error: Could not open any libc. (osfile_unix_allthreads.jsm) (Virgil)
  • [DONE] VERIFIED bug 827225 "Assertion failure: obj->isTypedArray()" with WebGL readPixels (Virgil)
  • [DONE] VERIFIED bug 825544 Clicking on a duckduckgo result link while the page is loading prevents the result page from being recorded (Alexandra)
  • [DONE] DROPPED bug 821329 Input widgets don't accept character input with MacOS's build-in handwriting feature (Bogdan)
    • Asked the reporter to verify.
  • [DONE] DROPPED bug 825620 Regression: Flash is triggering the discrete GPU (Bogdan)
    • Asked the reporter to verify.
  • [DONE] VERIFIED bug 831780 scrollWidth, scrollHeight of DIV containing IMG reported as 40x40px until the image's actual size is known, regardless of CSS sizing of the IMG (Simona)
  • [DONE] REOPENED bug 830214 crash when printing selection of a PDF with many pages in pdf.js (Simona)
  • [DONE] VERIFIED bug 829435 Japanese characters appear completely garbled in PDF viewer (Mihaela)

SECURITY FIXES

Verify these fixes in the latest Beta candidate (requires privileged access) (change link above for beta 3)

  • [DONE] VERIFIED bug 831095 Use-After-Free crash @xul!nsImageLoadingContent::OnStopContainer (mwobensmith)
  • [DONE] VERIFIED bug 827070 Heap-buffer-overflow WRITE in nsSaveAsCharset::DoCharsetConversion (mwobensmith)
  • [DONE] DROPPED bug 690970 cycle collector can be made to Unlink live objects (by using weak maps or watchpoints) (mwobensmith)
    • Verification of fix not possible and developer feels issue is theoretical

TOPCRASH FIXES

Verify these signatures are no longer present on crashstats in the current Beta (change link above for beta 3)

  • [DONE] VERIFIED bug 819888 crash in nsWindow::DealWithPopups (Paul)
  • [DONE] DROPPED bug 803568 Nightly crashed when see pdf file in fullscreen (presentation mode) or zooming (Cornel)
  • [DONE] VERIFIED bug 806206 crash in JSObject::global (Ioana)
  • [DONE] VERIFIED bug 810618 crash in mozilla::dom::EventHandlerNonNull::`.* deleting destructor' (Alexandra)
  • [DONE] VERIFIED bug 803125 crash in imgStatusTrackerObserver::OnStartDecode (Alexandra)

QAWANTED

Test the following bugs to find steps to reproduce, regression range, or testcases.

  • {{|}} VERIFIED bug bugID Bug description (verifier)

UNCONFIRMED

Bugs: Triage unconfirmed bugs for potential incoming regressions. (change link above for beta 3) Status
  • [DONE] UNCONFIRMED bug 834014 Edit menu greyed when ("select all" or "copy" is available) (Manuela - changed the component)
  • [DONE] UNCONFIRMED bug 834237 pdf.js: Chapter headlines in some PDFs are ugly because one headline has an overlap with another. (Manuela - changed the component)
  • [DONE] RESOLVED INCOMPLETE bug 829540 Firefox would not load any pages in normal mode. Had to restrst in safe mode with extensions and add-ons disabled (Manuela)
  • [DONE] UNCONFIRMED bug 807991 FireFTP causes manu js sinc reflows (Simona - changed the component)
  • [DONE] RESOLVED DUPLICATE bug 829414 Clear List button broken in Beta 19 (Virgil)

Issues Found

Bugs: New issues found during testing the Beta candidate Status
bug 834256 Firefox crashes while viewing PDF when many PDF documents are opened NEW

Items in bold are BLOCKER bugs

Beta 4

Builds FTP, HTTP
ID 20130130080006
Changeset 74d5479be250 (pushlog)
Checklist
  • [DONE] Functional Automation
  • [ON TRACK] Manual Testing
  • [ON TRACK] Bug Triage
  • [ON TRACK] Update Automation on betatest channel
  • [ON TRACK] Update Automation on releasetest channel
  • [ON TRACK] Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP [DONE] 0 Fails [DONE] 0 Fails
Windows Vista [DONE] 0 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 0 Fails [DONE] 1 Fails
Ubuntu 12.04 64-bit [DONE] 0 Fails [DONE] 1 Fails
Test Failures (known bugs)

Manual Testing

Report any bugs found during exploratory testing in Bugzilla and add them to the Issues Found section below.

Focus Area Windows 7 Windows 8 Mac OSX 10.7 Mac OSX 10.8 Ubuntu 12.04
Regression Tests:
Run the Firefox 19.0b4 Regression Tests in MozTrap. Do some exploratory testing around the testcases to try to exercise some edge-cases.
[DONE]
Alexandra
0 Bugs
[DONE]
Bogdan
0 Bugs
[DONE]
Manuela
0 Bugs
[DONE]
Virgil
0 Bugs
[DONE]
Simona
0 Bugs
PDF.js:
Test the new built-in PDF reader to check for issues with viewing and printing PDFs. Check some of the behaviour with other PDF readers side-loaded (ie. Adobe Reader, Foxit Reader, etc). Several bugs were marked for tracking in 19 for this feature. We may disable it. Pay close attention to this.
[DONE]
Alexandra
1 Bugs
[DONE]
Bogdan
1 Bugs
[DONE]
Manuela
1 Bugs
[DONE]
Virgil
0 Bugs
[DONE]
Simona
0 Bugs

Update Testing

Results of running the Mozmill update automation.
19.0b3 should get a partial update, <= 19.0b2 should get a complete update.

Platform Betatest (2013-01-30) Releasetest (2013-01-30) Beta (2013-01-30)
Windows XP [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Windows Vista [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Windows 7 32-bit [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Windows 7 64-bit [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Windows 8 32-bit [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Windows 8 64-bit [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Mac OS X 10.6 [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Mac OS X 10.7 [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Mac OS X 10.8 [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Ubuntu 12.04 32-bit [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails
Ubuntu 12.04 64-bit [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails [ON TRACK] : 0 Fails

Bug Triage

QAWANTED

Bugs: Test to find steps to reproduce, regression range, or testcases Status

NORMAL FIXES

Bugs: Verify these fixes in the latest Beta candidate Status
  • [DONE] VERIFIED bug 832998 web page is MUCH slower in FF than in Chrome (Ioana)
  • [DONE] VERIFIED bug 828205 crash in js::DebugScopeProxy::has with Firebug (Ioana)
  • [DONE] DROPPED bug 835090 Where possible xpcshell tests should run on xulrunner (Ioana)
    • Automated tests changes to run on XULrunner.
  • [DONE] VERIFIED bug 832576 WebGL uniform setters got 3-4x slower due to refactoring (Ioana)
  • [DONE] VERIFIED bug 809846 Click to Play doesn't play music on jango.com or other sites using the SoundManager2 library (Paul)
  • [DONE] VERIFIED bug 820497 click-to-play: asynchronously adding different plugins won't update the popup notification (Paul)
  • [DONE] VERIFIED bug 744745 Click to play overlay not appearing on vimeo.com (Paul)
  • [DONE] VERIFIED bug 819871 Nightly: can not login at Mail.com (Paul)
  • [DONE] VERIFIED bug 813550 Location bar's placeholder in an RTL chrome should be aligned on the right (Paul)
  • [DONE] VERIFIED bug 830086 NullPointerException in my webapp in Firefox 18 (Virgil)

SECURITY FIXES

  • [DONE] VERIFIED bug 761448 "ASSERTION: We should never be reusing a shared inner window" with pagehide event (mwobensmith)
  • [DONE] VERIFIED bug 812893 Heap-use-after-free in nsOverflowContinuationTracker::Finish, with -moz-columns (mwobensmith)
  • [DONE] DROPPED bug 814716 WebGL conformance crash (NVIDIA Noveau Linux driver) (mwobensmith)
    • Needs custom hardware, reporter to verify

TOPCRASH FIXES

Verify these signatures are no longer present on crashstats in the current Beta

  • [DONE] UPDATED bug 812776 crash in nsCSSRendering::PrepareBackgroundLayer @ nsIFrame::GetStyleVisibility (Ioana)
  • Recent crashes available in Socorro.
  • [DONE] UPDATED bug 813489 crash in nsHttpTransaction::SetSecurityCallbacks (Ioana)
  • Recent crashes available in Socorro.
  • [DONE] UPDATED bug 812203 crash in nsHttpConnection::SetSecurityCallbacks (Ioana)
  • Recent crashes available in Socorro.
  • [DONE] VERIFIED bug 814953 crash in nsWindow::ProcessMessage (Ioana)
  • [DONE] VERIFIED bug 803568 Nightly crashed when see pdf file in fullscreen (presentation mode) or zooming (Ioana)

UNCONFIRMED

Bugs: Triage unconfirmed bugs for potential incoming regressions. Status

Issues Found

Bugs: New issues found during testing the Beta candidate Status

Items in bold are BLOCKER bugs PDF Viewer

  • [DONE] NEW bug 836732 Buttons in pdf.js remain in hover state after selection (Virgil)
  • [DONE] NEW bug 836736 Selected text font changes to default when firefox is left in the background in pdf.js (Alexandra)
  • [DONE] NEW bug 836748 Loading stuck on a very large PDF file (Bogdan)

Beta 5

Builds FTP, HTTP
ID 20130206083616
Changeset 3e382bb14817 (pushlog)
Checklist
  • {{ |}} Functional Automation
  • {{ |}} Manual Testing
  • {{ |}} Bug Triage
  • {{ |}} Update Automation on betatest channel
  • {{ |}} Update Automation on releasetest channel
  • {{ |}} Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP [DONE] 0 Fails [DONE] 0 Fails
Windows Vista [DONE] 0 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 64-bit [DONE] 0 Fails [DONE] 0 Fails
Test Failures (known bugs)

Manual Testing

Report any bugs found during exploratory testing in Bugzilla and add them to the Issues Found section below.

Regression Tests

Run the following regression tests from the above linked testrun. Do some exploratory testing and try to exercise some edge-cases around these scenarios.

  • Breakpad End-to-End functionality
    • [DONE] Windows 7 (Bogdan)
    • [DONE] Mac OSX 10.8 (Alexandra)
    • [DONE] Ubuntu 12.04 (Manuela)
  • What's New Page on Firefox update
    • [DONE] Windows 7 (Bogdan)
    • [DONE] Mac OSX 10.8 (Alexandra)
    • [DONE] Ubuntu 12.04 (Manuela)
PDF.js (if time permits)

Test the new built-in PDF reader to check for issues with viewing and printing PDFs. Check some of the behaviour with other PDF readers side-loaded (ie. Adobe Reader, Foxit Reader, etc). Several bugs were marked for tracking in 19 for this feature. We may disable it. Pay close attention to this.

  • [DONE] Windows 7 (Bogdan)
  • [DONE] Mac OSX 10.8 (Alexandra)
  • [DONE] Ubuntu 12.04 (Manuela)

PDF Viewer Blocker Triage

The following bugs need triaging for potentially blocking PDF.js from shipping in Firefox 19. Try to reproduce these bugs in Firefox 19, set the tracking-firefox19:? flag if it reproduces and you believe it should block. Add your justification in the bug and in this wiki below.

Add a bullet-item below for any bugs you think are potential feature blockers. Be sure to include a reason you believe it should block. Easily reproducible crashes, hangs, or inability to load or download a PDF during a common user scenario would be examples of potential blockers.

Sign-off Status

QA is okay with a conditional sign-off for Firefox 19 considering we act on the following initiatives:

  • show the infobar more aggressively (maybe even always) and 3)
  • relnote that some PDF documents may not be displayed correctly but people can use an external reader
  • have a SUMO article ready
  • have a plan to uplift P1 and Critical fixes to Firefox 20
  • be prepared to hotfix disable PDF.js in Firefox 19 if negative feedback is explosive (bug 839239)
  • PDF.js Blocker Shortlist
    • bug 830278 Printing selections in pdf.js don't work and can cause OOM errors in large PDFs
    • bug 738967 pdf.js doesn't work for embedded PDFs
    • bug 839096 Searching for consecutive words is broken in pdf.js
    • bug 838549 PDF renders incorrectly if HWA turned off while document loaded
    • bug 839548 Excessive memory while loading PDF thumbnail view

Update Testing

Results of running the Mozmill update automation.
19.0b3 should get a partial update, <= 19.0b2 should get a complete update.

Platform Betatest Releasetest Beta
Windows XP [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows Vista [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.6 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.7 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.8 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails

Bug Fix Verifications

The following fixes require verification before sign-off.

  • [DONE] COMMENTED bug 824162 Strange focus switching with chat windows (Virgil)
    • QA: Unable to verify fixed because fixes need to be implemented on Facebook's end
    • RelMan: did not find any regressions so further testing does not block
  • [AT RISK] REOPENED bug 830278 Printing selections in pdf.js don't work and can cause OOM errors in large PDFs (Simona)
    • QA: Still crashing in Firefox 19.0b5
    • RelMan: retest with Beta, Aurora, and Nightly; reopen and update flags as appropriate
    • QA: not fixed, all branches still crash with empty signature.
  • [DONE] VERIFIED bug 834075 PDF url in the address bar becomes wrong in a certain situation (Alexandra)
  • [DONE] VERIFIEDbug 836887 Zoom-In button using wrong icon on HiDPI screens (mwobensmith, juanb)
  • [DONE] VERIFIED bug 797288 Pdf.js download behavior inconsistent (Manuela)
  • [DONE] COMMENTED bug 837371 Crash in nsContainerFrame::BuildDisplayListForNonBlockChildren (AMD Radeon HD 6xxx) (Bogdan)
    • QA: Unable to reproduce these crashes but were able to crash with null signatures
    • RelMan: doesn't seem related so doesn't block ship, will need to watch crashstats post-release
  • [DONE] VERIFIED bug 825035 Blocklisted click-to-play notification only fades in/out once per window per session (Paul)
  • [DONE] VERIFIED bug 836463 PDF.js Broken Forms (Bogdan)
  • [DONE] VERIFIED bug 835814 Heap-use-after-free in nsAsyncDOMEvent::Run (ashughes)
  • [DONE] VERIFIED bug 837845 Multi-GB memory spike when using regular expressions (Virgil)
  • [DONE] VERIFIED bug 838001 Input Method can't be opened to type to input fields in a panel on Firefox 19+ (pzhang)
  • [DONE] VERIFIED bug 799900 Wrong position of East Asian input method candidate box in HiDPI mode (haozhun)

Issues Found

The following are new issues found while testing the Beta candidate. Items in bold are BLOCKER bugs.

  • [DONE] NEW bug 839074 With pdf.js enabled, selected text doesn't stay selected after zoom (Manuela)
  • [DONE] INVALID bug 839092 Clicking on titles from Show Document Outline does not always bring up on top of the page the title selected (Bogdan)
  • [DONE] FIXED bug 839089 Moving with arrow keys left and right changes page on a pdf zoomed page (Bogdan)
  • [DONE] NEW bug 839096 Searching for consecutive words is broken in pdf.js (Alexandra)

Beta 6

Builds FTP, HTTP
ID 20130212082553
Changeset 258ce028c3be (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Triage
  • [DONE] Update Automation on betatest channel
  • [DONE] Update Automation on releasetest channel
  • [DONE] Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP [DONE] 2 Fails [DONE] 0 Fails
Windows Vista [DONE] 1 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 1 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 1 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 0 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 1 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 1 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 64-bit [DONE] 0 Fails [DONE] 0 Fails
Test Failures (known bugs)
  • /testToolbar/testStopReloadButtons.js -> bug 832180
  • /testSecurity/testSafeBrowsingNotificationBar.js -> bug 840810
  • /testToolbar/testBackForwardButtons.js -> bug 840022

Manual Testing

Report any bugs found during exploratory testing in Bugzilla and add them to the Issues Found section below.

Regression Tests

Run the following regression tests from the above linked testrun. Do some exploratory testing and try to exercise some edge-cases around these scenarios.

Breakpad End-to-End functionality
  • [DONE] PASS Windows 7 64-bit (Paul)
  • [DONE] PASS Windows 8 32-bit (Alexandra)
  • [DONE] PASS Mac OSX 10.6 (Manuela)
  • [DONE] PASS Ubuntu 12.04 64-bit (Virgil)
PDF.js

Test the new integrated PDF viewer. Pay particular attention to PDFs served by financial institutions (bank statements, bill statements, etc).

  • [DONE] PASS Windows 7 64-bit (Paul)
  • [DONE] PASS Windows 8 32-bit (Alexandra)
  • [DONE] PASS Mac OSX 10.6 (Manuela)
  • [DONE] PASS Ubuntu 12.04 64-bit (Virgil)
Proxy regressions from Fx18

See bug 828202#c25, bug 828632, bug 829646 and double check these issues remain resolved.

  • [DONE] PASS Windows 7 64-bit (Paul)
  • [DONE] PASS Windows 8 32-bit (Alexandra)
  • [DONE] PASS Mac OSX 10.6 (Manuela)
  • [DONE] PASS Ubuntu 12.04 64-bit (Virgil)
Mozmill-fails Regression Checking

Spotcheck verify the Stop, Reload, Back, and Forward buttons work under various conditions. Spotcheck verify the safe browsing notification bar works as expected.

  • [DONE] PASS Windows 7 64-bit (Paul)
  • [DONE] PASS Windows 8 32-bit (Alexandra)
  • [DONE] PASS Mac OSX 10.6 (Manuela)
  • [DONE] PASS Ubuntu 12.04 64-bit (Virgil)
Web compatibility (search, webmail, banking, video, audio, gaming, news, cms, etc.)
  • [DONE] PASS Windows 7 64-bit (Paul)
  • [DONE] PASS Windows 8 32-bit (Alexandra)
  • [DONE] PASS Mac OSX 10.6 (Manuela)
  • [DONE] PASS Ubuntu 12.04 64-bit (Virgil)
Mac OSX 10.5 and Windows 2000 de-support (install and update)
  • [DONE] PASS Windows 2000 (Simona)
  • [DONE] PASS Mac OSX 10.5 (Simona)

PDF Viewer Blocker Triage

The following bugs need triaging for potentially blocking PDF.js from shipping in Firefox 19. Try to reproduce these bugs in Firefox 19, set the tracking-firefox19:? flag if it reproduces and you believe it should block. Add your justification in the bug and in this wiki below.

Add a bullet-item below for any bugs you think are potential feature blockers. Be sure to include a reason you believe it should block. Easily reproducible crashes, hangs, or inability to load or download a PDF during a common user scenario would be examples of potential blockers.

  • bug 738967 pdf.js doesn't work for embedded PDFs
  • bug 838549 PDF renders incorrectly if HWA turned off while document loaded
  • bug 839548 Excessive memory while loading PDF thumbnail view
  • bug 829954 OOM crash in mozilla::gfx::AlphaBoxBlur::Blur
  • bug 799695 Background too dark
  • Nominated by community for UX:
    • bug 738952 "Save as..." File menu entry or Ctrl+S produces unexpected results
    • bug 761539 pdf.js doesn't display and print some characters (e.g. German umlauts)

QA is okay with a conditional sign-off for Firefox 19 considering we act on the following initiatives:

  • show the infobar more aggressively (maybe even always)
  • relnote that some PDF documents may not be displayed correctly but people can use an external reader
  • have a SUMO article ready
  • have a plan to uplift P1 and Critical fixes to Firefox 20
  • be prepared to hotfix disable PDF.js in Firefox 19 if negative feedback is explosive (bug 839239)

Update Testing

Results of running the Mozmill update automation.

  • Partial: 19.0b5
  • Complete: 19.0b4 and earlier
Platform Betatest Releasetest Beta
Windows XP [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows Vista [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 7 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Windows 8 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.6 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.7 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Mac OS X 10.8 [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 32-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails
Ubuntu 12.04 64-bit [DONE] : 0 Fails [DONE] : 0 Fails [DONE] : 0 Fails

Bug Fix Verifications

The following fixes require verification before sign-off.

  • [DONE] VERIFIED bug 830278 Printing selections in pdf.js cause OOM errors in large PDFs (simona)
  • [DONE] VERIFIED bug 837461 Black squares appear in some PDFs using PDF.js (contributor)
  • [DONE] VERIFIED bug 838864 keyword.URL reset interferes poorly with some partner add-ons (Bogdan)
  • [DONE] CONDITIONAL bug 839270 crash in nsStyleContext::nsStyleContext mainly with AMD Radeon HD 6250/6310/6320 (juanb)
    • Was hard to reproduce, will depend on post-release crashstats to verify
  • [DONE] VERIFIED STAGING bug 840161 Re-block AMD Radeon HD 6290/6300/6310/6320 for Direct3D 9/DIRECT2D (juanb)

Issues Found

The following are new issues found while testing the Beta candidate. Items in bold are BLOCKER bugs.

Social API

  • bug 841007 - Unreadable text on notification Facebook panel

Session Restore

  • NEW bug 791195 Session is not restored after Firefox crash

Release Candidate 1

Builds FTP, HTTP
ID 20130215130331
Changeset 20238b786063 (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Fix Verifications
  • [DONE] Updates qualified on betatest channel
  • [DONE] Conditional sign-off for push to mirrors
  • [DONE] Updates qualified on releasetest channel
  • [DONE] Final sign-off for push to release
  • [DONE] Updates qualified on release channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional Results Restart Results
Windows XP [DONE] 1 Fails [DONE] 0 Fails
Windows Vista [DONE] 1 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 1 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 1 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 1 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 1 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 1 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 1 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 1 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 1 Fails [DONE] 0 Fails
Ubuntu 12.04 64-bit [DONE] 1 Fails [DONE] 0 Fails
Test Failures (known bugs)
  • /testSecurity/testEncryptedPageWarning.js (bug 810820)

Manual Testing

Report any bugs found during exploratory testing in Bugzilla and add them to the Issues Found section below.

Regression Tests

Run the following regression tests from the above linked testrun. Do some exploratory testing and try to exercise some edge-cases around these scenarios.

Breakpad End-to-End functionality
  • [DONE] PASS Windows 7 64-bit (Manuela)
  • [DONE] PASS Windows 8 32-bit (Virgil)
  • [DONE] PASS Mac OSX 10.6 (Simona)
  • [DONE] PASS Ubuntu 12.04 64-bit (Mihaela)
PDF.js

Test the new integrated PDF viewer. Pay particular attention to PDFs served by financial institutions (bank statements, bill statements, etc).

  • [DONE] PASS Windows 7 64-bit (Manuela)
  • [DONE] PASS Windows 8 32-bit (Virgil)
  • [DONE] PASS Mac OSX 10.6 (Simona)
  • [DONE] PASS Ubuntu 12.04 64-bit (Mihaela)

Update Testing

Results of running the Mozmill update automation.

  • Partial: 18.0.2, 17.0.1, 16.0.2
  • Complete: 18.0.1, 18.0, 17.0, 16.0.1, <=16.0
Platform betatest report releasetest: report release: report
Windows XP [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Windows Vista [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Windows 7 32-bit [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Windows 7 64-bit [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Windows 8 32-bit [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Windows 8 64-bit [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.6 [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.7 [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Mac OS X 10.8 [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 32-bit [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails
Ubuntu 12.04 64-bit [DONE] 0 Fails [DONE] 0 Fails [DONE] 0 Fails

Bug Fix Verifications

The following fixes require verification before sign-off.

  • [DONE] VERIFIED bug 817790 Stretched Facebook icon briefly shows over entire toolbar during window startup (Simona)
  • [DONE] VERIFIED bug 815359 Bing Maps v7 Ajax Pushpin Incompatibility (Manuela)
  • [DONE] VERIFIED bug 800363 (CVE-2012-5837) XSS in Web Developer Toolbar's chrome privilege page (Paul)
  • [DONE] VERIFIED bug 827979 Add Growl support back (Cornel)
  • [DONE] COMMENTED bug 829710 Firefox "slants" on Mac 10.6 (Simona)
  • [DONE] COMMENTED bug 834325 SVG graphic rendering broken when embedded (Bogdan)
  • [DONE] COMMENTED bug 822620 disable Notification Center support (Cornel)
  • [DONE] COMMENTED bug 821577 DMD hang on Mac at start-up (Virgil)
  • [DONE] VERIFIED bug 816603 background-color does not print in certain site (Ioana)
  • [DONE] VERIFIED bug 813851 Canvas gets confused about its path/transform state (Manuela)
  • TRACKING bug 839270 crash in nsStyleContext::nsStyleContext mainly with AMD Radeon HD 6250/6310/6320
    • We think the crash explosiveness has been mitigated but have failed to verify 100% due to lack of user feedback

Issues Found

The following are new issues found while testing the Beta candidate. Items in bold are BLOCKER bugs.

19.0.1

The following release is Windows 8 only
Builds FTP, HTTP
ID 20130226172142
Changeset 668544a286c1 (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Fix Verifications
  • [DONE] Conditional sign-off for push to mirrors
  • [DONE] Updates qualified on releasetest channel (19.0.1: Win8-only, 19.0: all else)
  • [DONE] Final sign-off for push to release
  • [DONE] Updates qualified on release channel (19.0.1: Win8-only, 19.0: all else)
  • {{ |}} Confirm download links on download.mozilla.org (19.0.1: Win8-only, 19.0: all else)

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform (report) Functional Results Restart Results
Windows XP 0 Failures 0 Failures
Windows Vista 0 Failures 0 Failures
Windows 7 32-bit 0 Failures 0 Failures
Windows 7 64-bit 0 Failures 0 Failures
Windows 8 32-bit 0 Failures 0 Failures
Windows 8 64-bit 0 Failures 0 Failures
Mac OS X 10.6 0 Failures 0 Failures
Mac OS X 10.7 0 Failures 0 Failures
Mac OS X 10.8 0 Failures 0 Failures
Ubuntu 12.04 32-bit 0 Failures 0 Failures
Ubuntu 12.04 64-bit 0 Failures 0 Failures
Test Failures (known bugs)

Manual Testing

Report any bugs found during exploratory testing in Bugzilla Run the following regression tests and do some exploratory testing to exercise some possible edge-cases. Report any bugs found in Bugzilla and list them in the Issues Found section below.

Breakpad End-to-End functionality (testcase)
  • [DONE] Windows 8 32-bit (Bogdan)
  • [DONE] Windows 8 64-bit (Mihaela)
Confirm uninstallable on other Windows

At 7:30 a.m Pacific Time, updates were not available on betatest channel for any of the Windows versions (Win XP, Win 7, Win 8, Win Vista)

  • FAIL Windows 7 32-bit (Mihaela)
    • can install
  • FAIL Windows Vista (Bogdan)
    • can install
  • FAIL Windows XP (Mihaela)
    • can install
Exploratory testing on AMD Radeon hardware

Do some exploratory testing on various Radeon GPUs and Catalyst driver versions to try and crash Firefox. Common areas of instability are Flash, Video, Facebook, WebGL, and Canvas.

  • PASS Mihaela, Windows 8 x64, AMD Radeon HD 7700 Series, Catalyst 12.10
    • Sites and scenarios:
      • facebook: timeline navigation, view photos, status update, play games
      • gmail: compose and send email, read and navigate through emails
      • yahoo mail: read/navigate through emails
      • youtube: search, watch video, view videos from playlist, interact with video
      • google maps WebGL enabled: search, zoom, street view
      • vimeo: play video, interact with video
      • http://www.ebizmba.com/articles/best-flash-sites: interact with some of the websites
  • FAIL Mihaela, Windows 8 x64, AMD Radeon HD 7700 Series, Catalyst 12.10
    • Scenario:
      • Have Catalyst 12.10
      • Open some pages in firefox (mail, facebook, youtube, flash games)
      • Update Catalyst to 13.1
      • Result: Firefox crashes while update is installing (crash report)
  • PASS Mihaela, Windows 8 x64, AMD Radeon HD 7700 Series, Catalyst 13.1
    • Sites and scenarios:
      • facebook: timeline navigation, view photos, status update, play games
      • gmail: compose and send email, read and navigate through emails
      • yahoo mail: read/navigate through emails
      • youtube: search, watch video, view videos from playlist, interact with video
      • google maps WebGL enabled: search, zoom, street view
      • vimeo: play video, interact with video
      • http://www.ebizmba.com/articles/best-flash-sites: interact with some of the websites
  • PASS Bogdan, Windows 8 x32, AMD Radeon HD 6450, Catalyst 13.1
    • Sites and scenarios:
  • PASS Bogdan, Windows 8 x32, AMD Radeon HD 6450, Catalyst 13.1
  • PASS Bogdan, Windows 8 x32, AMD Radeon HD 6450, Catalyst 13.1
  • PASS Bogdan, Windows 8 x32, AMD Radeon HD 6450, Catalyst 11.12
    • Update driver test:
      • Updated the GPU driver from version 11.12 to 13.1 while videos were playing and Canvas test was running.
  • result Tester, Windows Version, GPU, Catalyst version
    • list of sites, scenarios tested
QAWANTED Bugs
  • bug 806071 crash in js::analyze::ScriptAnalysis::analyzeLifetimes
    • See if it can be reproduced with AMD Radeon HD 6250/6290/6310/6320/6400M on Windows 7 in Firefox 19.0 & 19.0.1

Update Testing

Platform Target releasetest (reports: 19.0.1, 19.0) release (reports: 19.0.1, 19.0)
Windows XP Firefox 19.0 0 Failures 0 Failures
Windows Vista Firefox 19.0 0 Failures 0 Failures
Windows 7 32-bit Firefox 19.0 0 Failures 0 Failures
Windows 7 64-bit Firefox 19.0 0 Failures 0 Failures
Windows 8 32-bit Firefox 19.0.1 0 Failures 0 Failures
Windows 8 64-bit Firefox 19.0.1 0 Failures 0 Failures
Mac OS X 10.6 Firefox 19.0 0 Failures 0 Failures
Mac OS X 10.7 Firefox 19.0 0 Failures 0 Failures
Mac OS X 10.8 Firefox 19.0 0 Failures 0 Failures
Ubuntu 12.04 32-bit Firefox 19.0 0 Failures 0 Failures
Ubuntu 12.04 64-bit Firefox 19.0 0 Failures 0 Failures
Results provided via Mozmill automation
Bugs Found
  • [DONE] VERIFIED FIXED bug 845947 Firefox <=19.0 updates on betatest do not go passed Firefox 19.0

Unthrottling via bug 845447

Test results for unthrottling 18.0.2 updates on Windows 8; 19.0 updates on all other platforms.

Platform Source Target Result
Mac OSX 10.7 Firefox 18.0.2 en-US Firefox 19.0 en-US PASS
Firefox 18.0 de Firefox 19.0 de PASS
Ubuntu 12.04 Firefox 17.0 en-US Firefox 19.0 en-US PASS
Firefox 18.0.1 fr Firefox 19.0 fr PASS
Windows 7 Firefox 18.0 en-US Firefox 19.0 en-US PASS
Firefox 17.0.1 it Firefox 19.0 it PASS
Windows 8 Firefox 18.0.2 en-US no update PASS
Firefox 18.0.2 es-ES no update PASS
Firefox 18.0 en-US Firefox 18.0.2 en-US PASS
Firefox 17.0 zh-TW Firefox 18.0.2 zh-TW PASS

Bug Fix Verifications

Bugs QA Contact Status
bug 844053 Update only win8 users to 19.0.1 Anthony Hughes [DONE] VERIFIED FIXED
bug 830531 [Win8] crash in XPC_WN_Helper_NewResolve Anthony Hughes [DONE] Verified crashes are mitigated
bug 845447 Please unthrottle updates for non-Win8 users Anthony Hughes [DONE] VERIFIED FIXED

Issues Found

The following are new issues found while testing the Beta candidate. Items in bold are BLOCKER bugs.

Bug Triage

QAWANTED

These bugs require help finding steps to reproduce, regression ranges, and/or test cases.

Bug QA Contact Status
bug 806071 crash in js::analyze::ScriptAnalysis::analyzeLifetimes
bug 829954 OOM crash in mozilla::gfx::AlphaBoxBlur::Blur Juan Becerra [DROPPED] Unable to find steps to reproduce
bug 839239 Prepare a hotfix to disabled PDF.js in Firefox 19 Anthony Hughes [DROPPED] RESOLVED INVALID
bug 836236 Crash in _ZThn184_N22nsComboboxControlFrame15GetRollupWidgetEv Juan Becerra [DROPPED] RESOLVED WONTFIX
bug 838549 Loaded document disappears in some PDFs Anthony Hughes [DROPPED] RESOLVED WONTFIX
bug 820379 crash in mozilla::dom::CanvasRenderingContext2D::GetImageDataArray Anthony Hughes [DROPPED] RESOLVED WONTFIX
bug 839232 Move pre-release CTP blocks to Firefox 20 and up Anthony Hughes [DONE] VERIFIED FIXED
bug 797288 Pdf.js download behavior inconsistent Manuela (Softvision) [DONE] VERIFIED FIXED

CRITICAL FIXES

These bugs need their fixes test and verified. These bugs should be tested before any other fixed bugs. Add a bullet item for any bugs you test.

  • [DONE] VERIFIED bug 823119 crash in mozilla::gfx::DrawTargetD2D::CopySurface with PDF Viewer (ioana)
  • [DONE] UPDATED bug 828184 startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll (Websense Endpoint) (ioana)
    • recent crashes available in Socorro.

NORMAL FIXES

These bugs need their fixes tested and verified. Add a bullet item for any bugs you test.

  • [DONE] VERIFIED bug 808466 Changing selection sometimes doesn't repaint immediately (Cornel)
  • [DONE] VERIFIED bug 685202 Using a Windows non-admin user account, applying an automatic Firefox update causes the program to not clean up after itself, leaving the Apply Update button incorrectly enabled and Firefox unable to recieve futher automatic updates. (Cornel)
  • [DONE] VERIFIED bug 770252 HTML Video / Audio Playback does not playback through Headphones (Cornel)
  • [DONE] VERIFIED bug 816458 Removing an inline css transform on a table applies twice the resulting computed transform (Cornel)
  • [DONE] VERIFIED bug 800799 http-on-modify-request now called after JS that started the load is no longer on the stack (Bogdan)
  • [DONE] COMMENTED bug 834325 SVG graphic rendering broken when embedded (Bogdan)
  • [DONE] VERIFIED bug 815593 large image within div with overflow-y: scroll disappears for a few seconds when scrolling (Bogdan)
  • [DONE] VERIFIED bug 795701 WebGL Renderer : no information in about:support (Manuela)
  • [DONE] VERIFIED bug 809178 bad drawing on a page with SVG embedded in iframes (Manuela)

SECURITY FIXES

These bugs need their fixes tested and verified. This will require elevated Bugzilla privileges. Add a bullet item for any bugs you test.

TOPCRASH FIXES

These bugs need to be tested to confirm the crash no longer reproduces. You will need to confirm you can reproduce the crash in a known broken build before verifying it's fixed in the latest build. If the crash is not reproducible for you in a known broken build, check crashstats for instances of the crash signature in the latest Firefox versions. Add a bullet item for any bugs you test.

UNCONFIRMED

These bugs need confirmation that they reproduce and need to be moved into an appropriate Bugzilla component. Add a bullet item for any bugs you test.

Known Issues

ID Summary Priority Status
791195 Session is not restored after Firefox crash -- RESOLVED
803456 Highlighting from Find remains after closing Find bar P2 NEW
803468 Highlighting doesn't follow rotation P3 RESOLVED
828991 In a page with wide content (triggering shrink-to-fit), "print selection" prints very tiny output, even if the wide stuff isn't selected -- NEW

4 Total; 2 Open (50%); 2 Resolved (50%); 0 Verified (0%);


19.0.2

Builds FTP, HTTP
ID 20130307023931
Changeset 6ffe3e9da8a8 (pushlog)
Checklist
  • [DONE] Functional Automation
  • [DONE] Manual Testing
  • [DONE] Bug Fix Verifications
  • [DONE] Conditional sign-off for push to mirrors
  • [DONE] Updates qualified on releasetest channel
  • [DONE] Final sign-off for push to release
  • [DONE] Updates qualified on release channel
  • [DONE] Confirm download links on download.mozilla.org

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Functional [report] Remote [report] Addon [report]
Windows XP 0 Failures 0 Failures 0 Failures
Windows Vista 0 Failures 0 Failures 0 Failures
Windows 7 32-bit 0 Failures 0 Failures 0 Failures
Windows 7 64-bit 0 Failures 0 Failures 0 Failures
Windows 8 32-bit 0 Failures 0 Failures 0 Failures
Windows 8 64-bit 0 Failures 0 Failures 0 Failures
Mac OS X 10.6 0 Failures 0 Failures 0 Failures
Mac OS X 10.7 0 Failures 0 Failures 0 Failures
Mac OS X 10.8 0 Failures 0 Failures 0 Failures
Ubuntu 12.04 32-bit 0 Failures 0 Failures 0 Failures
Ubuntu 12.04 64-bit 0 Failures 0 Failures 0 Failures
Test Failures (known bugs)

Manual Testing

Report any bugs found during exploratory testing in Bugzilla Run the following regression tests and do some exploratory testing to exercise some possible edge-cases. Report any bugs found in Bugzilla and list them in the Issues Found section below.

Breakpad End-to-End functionality (testcase)
  • [DONE] PASS Windows 8 32-bit (Manuela)
  • [DONE] PASS Windows 7 64-bit (Simona)
  • [DONE] PASS Ubuntu 12.10 32-bit (Mihaela)
  • [DONE] PASS Mac OSX 10.8 (Paul)
Smoke Testing (basic functionality and web compatibility, particularly websites with built in text editors)
  • [DONE] PASS Windows 8 32-bit (Manuela)
  • [DONE] PASS Windows 7 64-bit (Simona)
  • [DONE] PASS Ubuntu 12.10 32-bit (Mihaela)
  • [DONE] PASS Mac OSX 10.8 (Paul)

Detailed results in Etherpad

Update Testing

Platform Target betatest [report] releasetest [report] release [report]
Windows XP Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Windows Vista Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Windows 7 32-bit Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Windows 7 64-bit Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Windows 8 32-bit Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Windows 8 64-bit Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Mac OS X 10.6 Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Mac OS X 10.7 Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Mac OS X 10.8 Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Ubuntu 12.04 32-bit Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Ubuntu 12.04 64-bit Firefox 19.0.2 0 Failures 0 Failures 0 Failures
Partial: 16.0.2, 17.0.1, 18.0.2, 19.0, 19.0.1
Complete: <=16.0.1, 17.0, 18.0, 18.0.1
Locales: en-US, ...
Bugs Found

Bug Fix Verifications

  • [DONE] bug 848644 ZDI CanSecWest 2013 bug (softvision)
  • [DONE] bug 830531 [Win8] crash in XPC_WN_Helper_NewResolve with AMD Radeon (ashughes)
  • [DONE] bug 839270 crash in nsStyleContext::nsStyleContext with AMD Radeon (bsmedberg via VNC)
  • [DONE] bug 846391 back out bug 846388 as part of next desktop firefox release (blocks Win8 release updates)

Issues Found

The following are new issues found while testing the Beta candidate. Items in bold are BLOCKER bugs.

Post-mortem

Successes

  • The beta checklist developed while Firefox 19 was in beta has reduced the time spent on superfluous testing.
    • Great, is it enough freed up time to start moving our strategy in a proactive direction?
    • Definitely. A first step might be creating beta test runs as part of the community test run proposal.

Failures

  • PDF.js UX concerns
    • Could we have done more to put pressure on the team to fix or disable?
    • Initial negative feedback has died down significantly, suspected early detractors
    • How do we feel about using our Release early adopter audience as betatesters?
    • Should we be doing more to contribute to Beta audience growth?
  • AMD concerns
    • Could we have done more to test various hardware before release?
    • Should we consider out/crowdsource testing a lot of hardware once before release?
  • l10n blind-spot
    • Aurora l10n win32 builds were missing and QA didn't catch it in time
    • Propose adding l10n to Aurora/Beta/Release sign-off checklist, to consist of:
      • functional testing via mozmill-ci (currently en-US & fr via pulse)
      • update testing via mozmill-ondemand (Beta/Release), manual spotcheck (Aurora)
      • verify download links via manual spotcheck
      • minimum of one random locale per platform
      • more proactive triaging of dashboards (daily, weekly?)
      • longterm: look at ways to improve cross-team awareness and automation coverage
    • Auto-Dev:
      • Can we expand locale coverage for mozmill-ci functional? update?
      • Can we expand locale coverage for mozmill-ondemand update?
      • Can we expand locale coverage for mozmill-ci update?
      • What will it take to bring online more nodes?
      • What would it take to minimally qualify all locales? (time to execute, number of nodes)
      • Should we add auroratest to the mozmill-ci update jobs?
    • Web QA
      • Do they have tests to verify the download links?
      • If not, lets get some in place
      • If so, are they auto or manual trigger? where do the results go?
    • L10n Team
      • Do they have verification mechanisms in place?
      • If not, should they bear some verification responsibility?
      • If so, how can QA be better informed of the l10n state before sign-off?
    • Build Team
      • Do they have scripts to verify build/update availability?
      • If not, should they bear some verification responsibility?
      • If so, what can be done to make QA more aware?
    • Dev/Product ownership Team
      • Feature pages are not kept up-to-date, ergo we can't track down which features are targeted for which release. This means QA is starting the work on most features much later that necessary.
      • Some developers never answer feature-related emails, or they reply very late.
    • QA Team
      • The bugs with no tracking flags for 19, that have been fixed on Firefox 19 while it was in nightly, should have been included in the bug verifications query
      • Crash bugs are verified up to some level by the team monitoring crash stats. How much value did our verifying them add?

Action Items

  • [DONE] [ioana] Create a community oriented testrun to spur participation in releases
  • [DONE] [ashughes] Work with auto-dev to expand automation coverage to en-US, fr, de, and it
  • [DONE] [juanb] Work with Asa to improve clarity of feature development
  • [DONE] [ioana] Escalate instances of unresponsive developers to Juan and Anthony
  • [ON TRACK] [ashughes] Talk with development team leads to find more effective forms of communication
  • [DONE] [ashughes] Update bug queries to include Firefox version reported and milestone, not just tracking/status flags
  • [ON TRACK] [ashughes] Talk to stability team to identify value add of crash verifications, improve processes