Personal tools

Releases/Firefox 13/Test Plan

From MozillaWiki

Jump to: navigation, search

« Firefox 12 | Firefox 13 | Firefox 14 »

Contents

Firefox 13 Test Plan

Summary

Milestone Firefox 13
Status Currently in Beta -- releasing on June 5, 2012
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 13 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 13. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-01-31: [DONE] Nightly Merge
  • 2012-03-16: [DONE] Aurora Merge
  • 2012-04-24: [DONE] Pre-Beta feature sign-offs
  • 2012-04-26: [DONE] Beta 1
  • 2012-05-03: [DONE] Beta 2
  • 2012-05-10: [DONE] Beta 3
  • 2012-05-17: [DONE] Beta 4
  • 2012-05-24: [DONE] Beta 5
  • 2012-05-28: [DONE] Beta 6
  • 2012-05-31: [DONE] Beta 7
  • 2012-06-01: {{|}} Release Candidate
  • 2012-06-05: {{|}} Final

Nightly

Merge

Functional Automation - PASS
Update Spotchecks - PASS

Features

See the Feature Release Tracking wiki for a general overview

[DONE]
[ON TRACK]
[AT RISK]
[MISSED]

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated 2012-02-01
  • spreadsheet
  • bug 724553 Memory usage spike detected in Mozmill Endurance tests for Mac OS X 10.6.8 x86_64
    • Caused by memory leak in Private Browsing mode: bug 724677
  • bug 725701 Memory usage spike detected in Mozmill Endurance tests for Win 5.1.2600 x86
  • bug 728871 - High memory usage spike detected in Mozmill Endurance tests on Feb 19 caused by opening new tabs
  • bug 734857 - Memory leak increase with about 10% in open new tabs tests (Win XP)

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated 2012-02-01
  • No issues found

Unconfirmed Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:

P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible
  1. Check this link for UNCONFIRMED bugs found between 4/24/2012 and 6/5/2012
  2. Read each bug and determine what product and component they belong under in bugzilla
  3. Move the bug to the right product and component
  4. CC any applicable team members that may be interested in the bug
P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
P3 Task - Confirm UNCONFIRMED Bugs from Other Releases
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated 2012-02-01

Result Spreadsheet

Aurora Merge

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows 2000 Result
Windows XP Result
Windows Vista Result
Windows 7 32-bit Result
Windows 7 64-bit Result
Mac OS X 10.6 Result
Mac OS X 10.7 Result
Ubuntu 11.04 32-bit Result
Ubuntu 11.04 64-bit Result
Issues Found

Updates

The following are details of update verifications for Firefox 12 merge to Aurora

Auroratest
  • Windows XP: PASS (2012-03-16)
    • Mozilla/5.0 (Windows NT 5.1; rv:12.0a2) Gecko/20120312 Firefox/12.0a2
    • Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20120316 Firefox/13.0a2
  • Windows 7: PASS (2012-03-16)
    • Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0a2) Gecko/20120309 Firefox/12.0a2
    • Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120316 Firefox/13.0a2
  • Mac OS X 10.6:
    • Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:12.0a2) Gecko/20120309 Firefox/12.0a2
    • Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:13.0) Gecko/20120316 Firefox/13.0a2
  • Ubuntu 11.10: PASS (2012-03-16)
    • Mozilla/5.0 (X11; Linux x86_64; rv:12.0a2) Gecko/20120310 Firefox/12.0a2
    • Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120316 Firefox/13.0a2
Aurora
  • [ON TRACK] [ashughes] Windows XP:
  • [ON TRACK] [ashughes] Windows 7:
  • [ON TRACK] [ashughes] Mac OS X 10.6:
  • [ON TRACK] [ashughes] Ubuntu 11.04:
Issues Found

Manual Spot Checks

1. New Home Page exploratory (bug 711157, bug 731546)

2. UAC Service exploratory

Aurora Feature Sign-off

The tests for the features with Litmus testcases can be run here.

QAWANTED

Bugs: query

Bug Status
bug 715149 Assertion failure: [infer failure] Missing type pushed 0: void Trouble tracking regression (2012-03-26)
bug 738392 Plugin doesn't render inside a CSS transform on Mac Not reproducible (2012-03-26)

Beta 1

Builds: FTP, HTTP
Litmus: testrun, report

Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications
  • {{ |}} Check with Marcia for top-crashers spiking
  • [DONE] Update tests on releasetest
  • {{ |}} Get final sign-off from Bob Moss
  • {{ |}} Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-04-25
Windows Vista PASS - 2012-04-25
Windows 7 32-bit PASS - 2012-04-25
Windows 7 64-bit PASS - 2012-04-25
Mac OS X 10.6 PASS - 2012-04-25
Mac OS X 10.7 PASS - 2012-04-25
Ubuntu 11.10 32-bit PASS - 2012-04-25
Ubuntu 11.10 64-bit PASS - 2012-04-25
Test Failures

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-04-26
Windows Vista Simona PASS - 2012-04-26
Windows 7 32-bit Simona PASS - 2012-04-26
Windows 7 64-bit Virgil PASS - 2012-04-26
Mac OS X 10.6.8 Mihaela PASS - 2012-04-26
Mac OS X 10.7.3 Mihaela PASS - 2012-04-26
Ubuntu 11.10 32-bit Virgil PASS - 2012-04-26
Ubuntu 11.10 64-bit Ioana PASS - 2012-04-26

Results in Litmus

Regression Testing

Run the Firefox 13.0 Regression Tests and Firefox 13.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit Virgil PASS - 2012-04-26
Mac OS X 10.6.8 Mihaela PASS - 2012-04-26
Ubuntu 11.10 32-bit Paul PASS - 2012-04-26
Windows XP Ioana PASS - 2012-04-26

Feature Testing

Run the Firefox 13.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
Mandatory ASLR on Windows for binary components Virgil Dicu PASS - 2012-04-26 n/a n/a
New Tab Page Virgil Dicu PASS - 2012-04-26 PASS - 2012-04-26 PASS - 2012-04-26
Home Tab Vlad Ghetiu PASS - 2012-04-26 PASS - 2012-04-26 PASS - 2012-04-26
Clean up user profile Ioana Budnar PASS - 2012-04-26 PASS - 2012-04-26 PASS - 2012-04-26
Enable Smooth Scrolling Mihaela Velimiroviciu PASS - 2012-04-26 PASS - 2012-04-26 PASS - 2012-04-26
Automatic Session Restore with Tabs on Demand Mihaela Velimiroviciu PASS - 2012-04-26 PASS - 2012-04-26 PASS - 2012-04-26

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-04-26 PASS - 2012-04-26
Windows Vista PASS - 2012-04-26 PASS - 2012-04-26
Windows 7 32-bit PASS - 2012-04-26 PASS - 2012-04-26
Windows 7 64-bit PASS - 2012-04-26 PASS - 2012-04-26
Mac OS X 10.6 PASS - 2012-04-26 PASS - 2012-04-26
Mac OS 10.7 PASS - 2012-04-26 PASS - 2012-04-26
Ubuntu 11.10 32-bit PASS - 2012-04-26 PASS - 2012-04-26
Ubuntu 11.10 64-bit PASS - 2012-04-26 PASS - 2012-04-26
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining
QA+: for testers to verify functional fixes 7 69
QA+ SG: for the Security team to verify security fixes 0 23
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 201 0

Softvision verifications etherpad

QAWANTED

Bug (query) Status
bug 747847 crash in nsJSContext::EvaluateString with Skype extension 5.11 Investigating reproducible testcase for Firefox 12
bug 744063 'Fullscreen playback' NVIDIA stereo 3d for youtube not working in latest Aurora v13 Investigating regression range in Firefox 13 Nightly (2012-04-26)
bug 733337 Picture being rendered outside intended region RESOLVED FIXED (2012-04-26)

Issues Found

ID Summary Priority Status
749112 "Live Bookmarks Loading" message is displayed at the bottom of the RSS feeds -- NEW
749115 "Live Bookmark loading..." message does not disappear after RSS list is refreshed -- RESOLVED

Open; Resolved; Total (50% complete)


Known Issues

ID Summary Priority Status

Open; Resolved; Total (0% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 2

Builds FTP, HTTP
ID 20120501201020
Changeset 168f3814b32e
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [ON TRACK] Manual tests
  • [ON TRACK] Bug fix verifications
  • [DONE] Update tests on releasetest
  • [DONE] Get final sign-off from Bob Moss
  • {{ |}} Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-05-02
Windows Vista PASS - 2012-05-02
Windows 7 32-bit PASS - 2012-05-02
Windows 7 64-bit PASS - 2012-05-02
Mac OS X 10.6 PASS - 2012-05-02
Mac OS X 10.7 PASS - 2012-05-02
Ubuntu 11.10 32-bit PASS - 2012-05-02
Ubuntu 11.10 64-bit PASS - 2012-05-02
Test Failures (known bugs)
  • /testAddons/testManagerKeyboardShortcut.js
  • /testAddons/testPluginDisableAffectsAboutPlugins.js (bug 739935)
  • /testAwesomeBar/testAccessLocationBar.js

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-05-02
Windows Vista Simona PASS - 2012-05-03
Windows 7 32-bit Mihaela PASS - 2012-05-02
Windows 7 64-bit Virgil PASS - 2012-05-02
Mac OS X 10.6.8 Mihaela PASS - 2012-05-02
Mac OS X 10.7.3 Ioana PASS - 2012-05-02
Ubuntu 11.10 32-bit Ioana PASS - 2012-05-02
Ubuntu 11.10 64-bit Virgil PASS - 2012-05-02

Results in Litmus

Regression Testing

Run the Firefox 13.0 Regression Tests and Firefox 13.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit Paul PASS - 2012-05-03
Mac OS X 10.6.8 Mihaela PASS - 2012-05-03
Ubuntu 11.10 32-bit Virgil PASS - 2012-05-03
Windows XP Ioana PASS - 2012-05-03

Feature Testing

Run the Firefox 13.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
Mandatory ASLR on Windows for binary components Virgil Dicu PASS - 2012-05-04 n/a n/a
New Tab Page Virgil Dicu PASS - 2012-05-03 PASS - 2012-05-03 PASS - 2012-05-03
Home Tab Vlad Ghetiu PASS - 2012-05-03 PASS - 2012-05-03 PASS - 2012-05-03
Clean up user profile Ioana Budnar PASS - 2012-05-03 PASS - 2012-05-03 PASS - 2012-05-03
Enable Smooth Scrolling Mihaela Velimiroviciu result - yyyy-mm-dd PASS - 2012-05-03 result - yyyy-mm-dd
Automatic Session Restore with Tabs on Demand Mihaela Velimiroviciu PASS - 2012-05-03 PASS - 2012-05-03 result - yyyy-mm-dd

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-05-02 PASS - 2012-05-03
Windows Vista PASS - 2012-05-02 PASS - 2012-05-03
Windows 7 32-bit PASS - 2012-05-02 PASS - 2012-05-03
Windows 7 64-bit PASS - 2012-05-02 PASS - 2012-05-03
Mac OS X 10.6 PASS - 2012-05-02 PASS - 2012-05-03
Mac OS 10.7 PASS - 2012-05-02 PASS - 2012-05-03
Ubuntu 11.10 32-bit PASS - 2012-05-02 PASS - 2012-05-03
Ubuntu 11.10 64-bit PASS - 2012-05-02 PASS - 2012-05-03
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining
QA+: for testers to verify functional fixes 5 69
QA+ SG: for the Security team to verify security fixes 0 23
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 201 0

Softvision verifications etherpad

QAWANTED

Bug (query) Status
bug 700288 Crash in mozilla::gfx::BaseRect::UnionEdges with AMD Radeon HD 6xxx series NEW - Feedback Needed - Unreproducible (2012-05-08)
bug 729507 Crash in JSObject::finish NEW - Feedback Needed - Unreproducible (2012-05-08)
bug 750625 Crash in nsUrlClassifierPrefixSet::GetPrefixes NEW - Feedback needed - Where to Begin (2012-05-08)
bug 736152 "Good Primes" web worker test hangs browser and is slow to update NEW - Need to investigate (2012-05-08)
bug 744063 'Fullscreen playback' NVIDIA stereo 3d for youtube not working in latest Aurora v13 NEW - jsmith investigating (2012-05-08)
bug 751585 XPCOM component loading rejected due to ASLR not set FIXED - ashughes testing fix (2012-05-08)
bug 751753 Firefox 12 aggressively caching pages NEW - juanb investigating (2012-05-08)

Issues Found

ID Summary Priority Status

Open; Resolved; Total (0% complete)


Known Issues

ID Summary Priority Status
725296 Browser is unresponsive while Silverlight content is loading -- NEW
725614 Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content -- NEW
709029 The RealPlayer plugin crashes when trying to play a RealPlayer video -- RESOLVED

Open; Resolved; Total (33.33% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 3

Builds FTP, HTTP
ID 20120508180640
Changeset f40f8e40bae8
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [MISSED] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [DONE] Get final sign-off from Bob Moss
  • {{ |}} Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-05-09
Windows Vista PASS - 2012-05-09
Windows 7 32-bit PASS - 2012-05-09
Windows 7 64-bit PASS - 2012-05-09
Mac OS X 10.6 PASS - 2012-05-09
Mac OS X 10.7 PASS - 2012-05-09
Ubuntu 11.10 32-bit PASS - 2012-05-09
Ubuntu 11.10 64-bit PASS - 2012-05-09
Test Failures (known bugs)
  • /testPasswordManager/testPasswordNotificationBar.js (bug 749156)
  • /testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
  • /testPrivateBrowsing/testCloseWindow.js (bug 573461)
  • /testPrivateBrowsing/testDownloadManagerClosed.js (bug 573461)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-05-10
Windows Vista Simona PASS - 2012-05-10
Windows 7 32-bit Mihaela PASS - 2012-05-10
Windows 7 64-bit Paul PASS - 2012-05-10
Mac OS X 10.6.8 VladG PASS - 2012-05-10
Mac OS X 10.7.3 VladG PASS - 2012-05-10
Ubuntu 11.10 32-bit Ioana PASS - 2012-05-09
Ubuntu 11.10 64-bit Virgil PASS - 2012-05-10

Results in Litmus

Regression Testing

Run the Firefox 13.0 Regression Tests and Firefox 13.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit Mihaela PASS - 2012-05-10
Mac OS X 10.6.8 VladG PASS - 2012-05-10
Ubuntu 11.10 32-bit Ioana PASS - 2012-05-09
Windows XP Ioana PASS - 2012-05-10
  1. [DONE] Exploratory testing Etherpad

Feature Testing

Run the Firefox 13.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
Mandatory ASLR on Windows for binary components Virgil Dicu PASS - 2012-05-10 n/a n/a
New Tab Page Virgil Dicu PASS - 2012-05-10 PASS - 2012-05-09 PASS - 2012-05-09
Home Tab Vlad Ghetiu PASS - 2012-05-10 PASS - 2012-05-09 PASS - 2012-05-09
Clean up user profile Ioana Budnar PASS - 2012-05-10 PASS - 2012-05-09 PASS - 2012-05-09
Enable Smooth Scrolling Mihaela Velimiroviciu PASS - 2012-05-10 PASS - 2012-05-09 PASS - 2012-05-09
Automatic Session Restore with Tabs on Demand Mihaela Velimiroviciu PASS - 2012-05-10 PASS - 2012-05-09 PASS - 2012-05-09

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-05-10 PASS - 2012-05-11
Windows Vista PASS - 2012-05-10 PASS - 2012-05-11
Windows 7 32-bit PASS - 2012-05-10 PASS - 2012-05-11
Windows 7 64-bit PASS - 2012-05-10 PASS - 2012-05-11
Mac OS X 10.6 PASS - 2012-05-10 PASS - 2012-05-11
Mac OS 10.7 PASS - 2012-05-10 PASS - 2012-05-11
Ubuntu 11.10 32-bit PASS - 2012-05-10 PASS - 2012-05-11
Ubuntu 11.10 64-bit PASS - 2012-05-10 PASS - 2012-05-11
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 42 43 49%
QA+ SG: for the Security team to verify security fixes 0 30 0%
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • bug 736731 Hotmail no longer auto-updates (due to "globalStorage" removal)
    • Confirmed regression in Firefox 13.0b3
  • bug 736152 "Good Primes" web worker test hangs browser and is slow to update
    • Slow Script warning in Fx10, 14, and 15
    • UI responsiveness issue in Fx10, 11, 12
    • Counter is "choppy" in Fx13, starts out choppy in Fx14 and 15 but improves
    • Conditions are 100% reproducible
  • bug 744063 'Fullscreen playback' NVIDIA stereo 3d for youtube not working in latest Aurora v13
    • Hardware set up, confirmed "fullscreen side-by-side" started in Fx13, working on regression range
    • Side-by-side effect occurs as early as Fx12 for the progress bar hover -- is there a bug on this?
  • bug 751585 XPCOM component loading rejected due to ASLR not set
    • Using CFF Explorer, firefox.exe appears to be using ASLR, shlwapi.dll is not -- need further instructions
  • bug 751753 Firefox 12 aggressively caching pages
    • Fixed Nightly needs testing by reporters -- not sure if there's much QA can do to help further

Issues Found

ID Summary Priority Status
753684 Removing an add-on installed twice doesn't actually remove it -- NEW
753770 Printing progress bar is not properly displayed -- NEW
753889 stage.mo -> releases.mo breaks staging beta builds for update testing -- VERIFIED
753680 Cannot enable all personas from Add-ons Manager list -- VERIFIED

Open; Resolved; Total (0% complete)


Known Issues

ID Summary Priority Status
585971 autocomplete matches unrelated hosts when searching for www.example.com -- NEW
725296 Browser is unresponsive while Silverlight content is loading -- NEW
725614 Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content -- NEW
749112 "Live Bookmarks Loading" message is displayed at the bottom of the RSS feeds -- NEW
730829 Restore "Open all in tabs" and "open <site>" to the bottom of livemarks popups (was: "Open all in tabs" missing after bug 613588) -- ASSIGNED
709029 The RealPlayer plugin crashes when trying to play a RealPlayer video -- RESOLVED
726943 Fullscreen button in HTML video makes the browser go fullscreen but not the video -- VERIFIED
727702 Undoing applying a Persona reverts back to default theme instead of old Persona -- VERIFIED

Open; Resolved; Total (12.5% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 4

Builds FTP, HTTP
ID 20120516113045
Changeset eb5336a48ab0
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [DONE] Get final sign-off from Bob Moss
  • {{ |}} Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-05-16
Windows Vista PASS - 2012-05-16
Windows 7 32-bit PASS - 2012-05-16
Windows 7 64-bit PASS - 2012-05-16
Mac OS X 10.6 PASS - 2012-05-16
Mac OS X 10.7 PASS - 2012-05-16
Ubuntu 11.10 32-bit PASS - 2012-05-16
Ubuntu 11.10 64-bit PASS - 2012-05-16
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-05-17
Windows Vista Simona PASS - 2012-05-17
Windows 7 32-bit Mihaela PASS - 2012-05-17
Windows 7 64-bit Paul PASS - 2012-05-17
Mac OS X 10.6.8 VladG PASS - 2012-05-17
Mac OS X 10.7.4 VladG PASS - 2012-05-17
Ubuntu 12.04 32-bit Virgil PASS - 2012-05-17
Ubuntu 11.10 64-bit Ioana PASS - 2012-05-17

Results in Litmus

Regression Testing

Run the Firefox 13.0 Regression Tests and Firefox 13.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit Paul PASS - 2012-05-17
Mac OS X 10.6.8 VladG PASS - 2012-05-18
Ubuntu 11.10 64-bit Mihaela PASS - 2012-05-17
Windows XP Simona PASS - 2012-05-17
  1. [DONE] Exploratory testing Etherpad
  2. [DONE] Plugins testing Etherpad

Feature Testing

Run the Firefox 13.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
Mandatory ASLR on Windows for binary components Virgil Dicu PASS - 2012-05-17 n/a n/a
New Tab Page Virgil Dicu PASS - 2012-05-17 PASS - 2012-05-17 PASS - 2012-05-17
Home Tab Vlad Ghetiu PASS - 2012-05-17 PASS - 2012-05-17 PASS - 2012-05-17
Clean up user profile Ioana Budnar PASS - 2012-05-17 PASS - 2012-05-17 PASS - 2012-05-17
Enable Smooth Scrolling Mihaela Velimiroviciu PASS - 2012-05-17 PASS - 2012-05-17 PASS - 2012-05-17
Automatic Session Restore with Tabs on Demand Mihaela Velimiroviciu PASS - 2012-05-17 PASS - 2012-05-17 PASS - 2012-05-17

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-05-18 PASS - 2012-05-18
Windows Vista PASS - 2012-05-18 PASS - 2012-05-18
Windows 7 32-bit PASS - 2012-05-18 PASS - 2012-05-18
Windows 7 64-bit PASS - 2012-05-18 PASS - 2012-05-18
Mac OS X 10.6 PASS - 2012-05-18 PASS - 2012-05-18
Mac OS 10.7 PASS - 2012-05-18 PASS - 2012-05-18
Ubuntu 11.10 32-bit PASS - 2012-05-18 PASS - 2012-05-18
Ubuntu 11.10 64-bit PASS - 2012-05-18 PASS - 2012-05-18
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 51 34 60%
QA+ SG: for the Security team to verify security fixes 7 23 23%
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • [AT RISK] bug 749010 Perma-orange Linux64 test_memoryReporters.xul crash above mozilla::dom::workers::WorkerPrivate::BlockAndCollectRuntimeStats on Beta 13
    • Requires a Linux64-opt build -- not sure where to get that (2012-05-15)
  • [ON TRACK] bug 755974 Block AMD Radeon HD 6290/6300/6310/6320 for Direct3D 9
    • Marcia is testing the staged block (2012-05-17)

Issues Found

ID Summary Priority Status
756080 The Bookmarks toolbar is set as visible automatically when resetting Firefox with at least 5 bookmarks toolbar items -- NEW
756111 Emoticons are not displayed on the Facebook chat -- RESOLVED

Open; Resolved; Total (50% complete)


Known Issues

ID Summary Priority Status
585971 autocomplete matches unrelated hosts when searching for www.example.com -- NEW
725296 Browser is unresponsive while Silverlight content is loading -- NEW
725614 Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content -- NEW
749112 "Live Bookmarks Loading" message is displayed at the bottom of the RSS feeds -- NEW
753684 Removing an add-on installed twice doesn't actually remove it -- NEW
753770 Printing progress bar is not properly displayed -- NEW
754829 In Full Screen mode - YouTube videos player changes dimmension -- NEW
730829 Restore "Open all in tabs" and "open <site>" to the bottom of livemarks popups (was: "Open all in tabs" missing after bug 613588) -- ASSIGNED
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
726943 Fullscreen button in HTML video makes the browser go fullscreen but not the video -- VERIFIED
727702 Undoing applying a Persona reverts back to default theme instead of old Persona -- VERIFIED
753889 stage.mo -> releases.mo breaks staging beta builds for update testing -- VERIFIED
753680 Cannot enable all personas from Add-ons Manager list -- VERIFIED

Open; Resolved; Total (14.29% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 5

Builds FTP, HTTP
ID 20120523114940
Changeset 5de6e4bd8ede
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 70%)
  • [DONE] Security fix verifications (target 50%)
  • [DONE] Update tests on releasetest
  • {{ |}} Get stability sign-off (marcia)
  • {{ |}} Get final sign-off (bmoss)
  • {{ |}} Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-05-23
Windows Vista PASS - 2012-05-23
Windows 7 32-bit PASS - 2012-05-23
Windows 7 64-bit PASS - 2012-05-23
Mac OS X 10.6 PASS - 2012-05-23
Mac OS X 10.7 PASS - 2012-05-23
Ubuntu 11.10 32-bit PASS - 2012-05-23
Ubuntu 11.10 -bit PASS - 2012-05-23
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-05-24
Windows Vista Simona PASS - 2012-05-24
Windows 7 32-bit Mihaela PASS - 2012-05-24
Windows 7 64-bit Paul PASS - 2012-05-24
Mac OS X 10.6.8 VladG PASS - 2012-05-24
Mac OS X 10.7.4 VladG PASS - 2012-05-24
Ubuntu 12.04 32-bit Virgil PASS - 2012-05-24
Ubuntu 11.10 64-bit Ioana PASS - 2012-05-24

Results in Litmus

Regression Testing

Run the Firefox 13.0 Regression Tests and Firefox 13.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit Mihaela PASS - 2012-05-24
Mac OS X 10.6.8 VladG PASS - 2012-05-24
Ubuntu 11.10 32-bit Virgil PASS - 2012-05-24
Windows XP Ioana PASS - 2012-05-24
  1. [DONE] Exploratory testing Etherpad
  2. [DONE] Plugins testing Etherpad

Feature Testing

Run the Firefox 13.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
Mandatory ASLR on Windows for binary components Virgil Dicu result - YYYY-MM-DD n/a n/a
New Tab Page Virgil Dicu result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD
Home Tab Vlad Ghetiu result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD
Clean up user profile Ioana Budnar result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD
Enable Smooth Scrolling Mihaela Velimiroviciu result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD
Automatic Session Restore with Tabs on Demand Mihaela Velimiroviciu result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-05-24 PASS - 2012-05-24
Windows Vista PASS - 2012-05-24 PASS - 2012-05-24
Windows 7 32-bit PASS - 2012-05-24 PASS - 2012-05-24
Windows 7 64-bit PASS - 2012-05-24 PASS - 2012-05-24
Mac OS X 10.6 PASS - 2012-05-24 PASS - 2012-05-24
Mac OS 10.7 PASS - 2012-05-24 PASS - 2012-05-24
Ubuntu 11.10 32-bit PASS - 2012-05-24 PASS - 2012-05-24
Ubuntu 11.10 64-bit PASS - 2012-05-24 PASS - 2012-05-24
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 65 23 74%
QA+ SG: for the Security team to verify security fixes 22 8 73%
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • [AT RISK] bug 753251 Crash in nsGfxScrollFrameInner::ScrollToImpl
    • QA testing dead-ended, need more leads
  • [ON TRACK] bug 756289 Netflix video errors "you are viewing from more than one tab"
    • QA testing unsuccessful, reached out to a bug commenter
  • [ON TRACK] bug 757284 Issues with (large?) uploads stopping halfway (youtube, mediafire)
    • Bisect found, is QAWANTED still needed?
  • [ON TRACK] bug 719117 Crash @ mozalloc_abort | NS_DebugBreak_P | nsIFrame::GetOffsetToCrossDoc
    • Reproduced, what next?
  • [ON TRACK] bug 640904 Crash in JSAutoEnterCompartment::enter
    • QA testing unsuccessful with Blekko toolbar, any other leads?

Issues Found

ID Summary Priority Status
756080 The Bookmarks toolbar is set as visible automatically when resetting Firefox with at least 5 bookmarks toolbar items -- NEW
756111 Emoticons are not displayed on the Facebook chat -- RESOLVED

Open; Resolved; Total (50% complete)


Known Issues

ID Summary Priority Status
585971 autocomplete matches unrelated hosts when searching for www.example.com -- NEW
725296 Browser is unresponsive while Silverlight content is loading -- NEW
725614 Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content -- NEW
749112 "Live Bookmarks Loading" message is displayed at the bottom of the RSS feeds -- NEW
753684 Removing an add-on installed twice doesn't actually remove it -- NEW
753770 Printing progress bar is not properly displayed -- NEW
754829 In Full Screen mode - YouTube videos player changes dimmension -- NEW
730829 Restore "Open all in tabs" and "open <site>" to the bottom of livemarks popups (was: "Open all in tabs" missing after bug 613588) -- ASSIGNED
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
726943 Fullscreen button in HTML video makes the browser go fullscreen but not the video -- VERIFIED
727702 Undoing applying a Persona reverts back to default theme instead of old Persona -- VERIFIED
753889 stage.mo -> releases.mo breaks staging beta builds for update testing -- VERIFIED
753680 Cannot enable all personas from Add-ons Manager list -- VERIFIED

Open; Resolved; Total (14.29% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 6

Builds FTP, HTTP
ID 20120528154913
Changeset 5dd4dde1d4eb
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 100%)
  • [ON TRACK] Security fix verifications (target 100%)
  • [DONE] Update tests on releasetest
  • {{ |}} Get stability sign-off (marcia)
  • {{ |}} Get final sign-off (bmoss)
  • [DONE] Update tests on beta

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests

Platform Result: report Failures: known bugs
Windows XP PASS - 2012-05-29 0 - no fails
Windows Vista PASS - 2012-05-29 0 - no fails
Windows 7 32-bit PASS - 2012-05-29 0 - no fails
Windows 7 64-bit PASS - 2012-05-29 0 - no fails
Mac OS X 10.6 PASS - 2012-05-29 0 - no fails
Mac OS X 10.7 PASS - 2012-05-29 0 - no fails
Ubuntu 11.10 32-bit PASS - 2012-05-29 0 - no fails
Ubuntu 11.10 -bit PASS - 2012-05-29 0 - no fails

Endurance Tests

Platform Result: report Max Memory
Windows Vista PASS - 2012-05-29 78 MB
Windows 7 32-bit PASS - 2012-05-29 90 MB
Windows 7 64-bit PASS - 2012-05-29 76 MB
Mac OS X 10.6 PASS - 2012-05-29 151 MB
Mac OS X 10.7 PASS - 2012-05-29 140 MB
Ubuntu 11.10 32-bit PASS - 2012-05-29 59 MB
Ubuntu 11.10 -bit PASS - 2012-05-29 96 MB

Update Tests

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-05-29 PASS - 2012-05-29
Windows Vista PASS - 2012-05-29 PASS - 2012-05-29
Windows 7 32-bit PASS - 2012-05-29 PASS - 2012-05-29
Windows 7 64-bit PASS - 2012-05-29 PASS - 2012-05-29
Mac OS X 10.6 PASS - 2012-05-29 PASS - 2012-05-29
Mac OS X 10.7 PASS - 2012-05-29 PASS - 2012-05-29
Ubuntu 11.10 32-bit PASS - 2012-05-29 PASS - 2012-05-29
Ubuntu 11.10 -bit PASS - 2012-05-29 PASS - 2012-05-29

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result: report
Windows XP Ioana PASS - 2012-05-29
Windows Vista Simona PASS - 2012-05-29
Windows 7 32-bit Mihaela PASS - 2012-05-29
Windows 7 64-bit Paul PASS - 2012-05-29
Mac OS X 10.6.8 VladG PASS - 2012-05-29
Mac OS X 10.7.4 VladG PASS - 2012-05-29
Ubuntu 12.04 32-bit Virgil PASS - 2012-05-29
Ubuntu 11.10 64-bit Ioana PASS - 2012-05-29

Focus Testing

Focus Area Win32 Mac Linux
Special Case: YouTube Caching PASS - 2012-05-29 PASS - 2012-05-30 PASS - 2012-05-29
Printing FAIL - 2012-05-29 PASS - 2012-05-29 PASS - 2012-05-29
Plug-ins FAIL - 2012-05-29 PASS - 2012-05-29 PASS - 2012-05-29
Top-sites PASS - 2012-05-30 PASS - 2012-05-30 PASS - 2012-05-30
Mac OSX 10.8 N/A PASS - 2012-05-30 N/A
Windows 8 FAIL - 2012-05-30 N/A N/A

Special Case: YouTube Caching

  • Test: bug 757262
    • Install Flash 11.2 and Firefox Beta, and open 2 windows
    • First window, watch an hour long YouTube video (or longer), and let the video finish
    • Once video is done, switch to the second window and do your usual browsing for another hour
    • After an hour, switch back to the first window and load any non-video website in the tab you were watching the video
  • Anticipated Result: audio from the video plays back unexpectedly for a few seconds
  • Results:
    • Windows 7: PASS (Mihaela)
    • Ubuntu 12.04: PASS (Virgil)
    • Mac OSX 10.6: PASS (VladG)
  • Details:

Printing

  • Tests:
    • Print previews
    • Print to file
    • PDFs, Flash content, SVG content
    • Rich websites (especially Flash content re: bug 747055)
    • keep an eye out for crashes
  • Results:
    • Windows 7: FAIL (Mihaela)
      • See bugs bellow
    • Ubuntu 12.04: PASS (Paul)
    • Mac OSX 10.6: PASS (VladG)
      • There is no "print to file" option on Mac, just save as pdf
  • Details & Bugs:
    • bug 759334 - Print to file creates a file with unknown format
    • bug 134002 - Content of plugins (flash, java applets) doesn't get rendered for printing or print preview
    • bug 184537 - Print preview doesn't dynamically update page number after scrolling or adjusting settings (doesn't reflect page in view)

Plug-ins

  • Tests:
    • Flash (11.2, 11.3)
    • Java
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, or other rich interfaces
  • Results:
    • Windows 7: FAIL (Simona)
    • Ubuntu 12.04: PASS (Virgil)
    • Mac OSX 10.6: PASS (VladG)
  • Details & Bugs:
    • bug 725614 - Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content
    • bug 759689 - Flash content is not loaded using Flash 11.3 Beta
    • On Windows 7 - Flash content is not loaded using Flash 11.3 beta. Issue is reproducible on Windows 7 with no Service Pack. Flash 11.3 works fine on Windows 7 with SP 1. Flash 11.2 does not cause any problems.

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • keep an eye out for crashes
  • Results:
    • Windows 7: PASS (Mihaela)
    • Ubuntu 12.04: PASS (Paul)
    • Mac OSX 10.6: PASS (VladG)
  • Details & Bugs:

Mac OS X 10.8

  • Tests:
    • Try all the menu item functions
    • Gestures
    • Plug-in compatibility
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: PASS (VladG)
  • Details & Bugs:

Windows 8

  • Tests:
    • Try all the menu item functions
    • Plug-in compatibility
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: FAIL (Paul)
  • Details & Bugs:

Exploratory testing

  • Exploratory testing on add-ons, plugins, top sites and new features Etherpad

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 106 13 89%
QA+ SG: for the Security team to verify security fixes 22 9 71%
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 379 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • bug 719117 Crash @ mozalloc_abort NS_DebugBreak_P nsIFrame::GetOffsetToCrossDoc
    • VERIFIED FIXED
  • bug 756289 Netflix video errors "you are viewing from more than one tab"
    • Marcia, any information you can add to aid debugging?
  • bug 757262 Youtube keeps playing audio after closing
    • IU has an unconfirmed testcase which takes ~1hr to test, can it be reduced?
    • Softvision tested but was unable to reproduce the issue on Win/Linux, still need to test Mac

Issues Found

ID Summary Priority Status
759689 [adbe 3210913] Flash content is not loaded using Flash 11.3 Beta -- RESOLVED
760119 Switching back from custom install to standard doesn't reset the installation folder -- RESOLVED

Open; Resolved; Total (100% complete)


Known Issues

ID Summary Priority Status
585971 autocomplete matches unrelated hosts when searching for www.example.com -- NEW
725296 Browser is unresponsive while Silverlight content is loading -- NEW
725614 Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content -- NEW
749112 "Live Bookmarks Loading" message is displayed at the bottom of the RSS feeds -- NEW
753684 Removing an add-on installed twice doesn't actually remove it -- NEW
753770 Printing progress bar is not properly displayed -- NEW
754829 In Full Screen mode - YouTube videos player changes dimmension -- NEW
756080 The Bookmarks toolbar is set as visible automatically when resetting Firefox with at least 5 bookmarks toolbar items -- NEW
730829 Restore "Open all in tabs" and "open <site>" to the bottom of livemarks popups (was: "Open all in tabs" missing after bug 613588) -- ASSIGNED
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
756111 Emoticons are not displayed on the Facebook chat -- RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
726943 Fullscreen button in HTML video makes the browser go fullscreen but not the video -- VERIFIED
727702 Undoing applying a Persona reverts back to default theme instead of old Persona -- VERIFIED
753889 stage.mo -> releases.mo breaks staging beta builds for update testing -- VERIFIED
753680 Cannot enable all personas from Add-ons Manager list -- VERIFIED

Open; Resolved; Total (18.75% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 7

Builds FTP, HTTP
ID 20120531155942
Changeset 91bf09009594
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 100%)
  • [ON TRACK] Security fix verifications (target 100%)
  • [DONE] Update tests on releasetest
  • [DONE] Get stability sign-off (marcia)
  • [DONE] Get final sign-off (bmoss)
  • [DONE] Update tests on beta
Issues of Note

[NOT BLOCKING] bug 760415 Firefox crashes if locking and unlocking Windows XP while on http://chrome.angrybirds.com/

  • Will fix for Firefox 14, QA is working on a regression range
  • Only affects those with WebGL turned on (very small userbase)
  • Intermittent in Firefox 11, prevalent in Firefox 12 (if it were serious, we'd have heard about it already)


Automated Tests

Results of the Mozmill automation testruns.

Functional Tests

Platform Result: report Failures: known bugs
Windows XP PASS - 2012-05-31 0 - no fails
Windows Vista PASS - 2012-05-31 0 - no fails
Windows 7 32-bit PASS - 2012-05-31 0 - no fails
Windows 7 64-bit PASS - 2012-05-31 0 - no fails
Mac OS X 10.6 PASS - 2012-05-31 0 - no fails
Mac OS X 10.7 PASS - 2012-05-31 0 - no fails
Ubuntu 11.10 32-bit PASS - 2012-05-31 1 - testAboutPrivateBrowsing.js
Ubuntu 11.10 -bit PASS - 2012-05-31 0 - no fails

Update Tests

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-01 PASS - 2012-06-01
Windows Vista PASS - 2012-06-01 PASS - 2012-06-01
Windows 7 32-bit PASS - 2012-06-01 PASS - 2012-06-01
Windows 7 64-bit PASS - 2012-06-01 PASS - 2012-06-01
Mac OS X 10.6 PASS - 2012-06-01 PASS - 2012-06-01
Mac OS X 10.7 PASS - 2012-06-01 PASS - 2012-06-01
Ubuntu 11.10 32-bit PASS - 2012-06-01 PASS - 2012-06-01
Ubuntu 11.10 -bit PASS - 2012-06-01 PASS - 2012-06-01

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result: report
Windows XP Ioana FAIL - 2012-06-01
Windows 7 64-bit Paul PASS - 2012-06-01
Mac OS X 10.6.8 VladG PASS - 2012-06-01
Mac OS X 10.7.4 VladG PASS - 2012-06-01
Ubuntu 11.10 64-bit Ioana PASS - 2012-06-01
Ubuntu 12.04 32-bit Virgil PASS - 2012-06-01

Focus Testing

Focus Area Win32 Mac Linux
Plug-ins PASS - 2012-06-01 PASS - 2012-06-01 PASS - 2012-06-01
Top-sites PASS - 2012-06-01 PASS - 2012-06-01 PASS - 2012-06-01
Mac OSX 10.8 N/A PASS - 2012-06-01 N/A

[RESOLVED] Priority: bug 760190

NOTE - focus here no longer needed, please refocus on usual sign-off

bug 760190 OOM crash in `anonymous namespace'::CSSParserImpl::ParseSelector or ParseSelectorGroup
Try anything you can think of to reproduce the crash -- we have no leads and this is a blocker!

  • Install Java (6u31 seems to have the highest correlation)
  • Windows XP and 7
  • Start with Firefox 13.0b6
  • Start testing http://scratch.mit.edu/ and then branch out to other heavy Java sites
  • Once we have STR, test 13.0b7 -- we need to see if it's fixed or not

[UPDATE] Correlated to 32-bit Windows with Java 6 on a netbook with 2GB of RAM (test under similar conditions)

[UPDATE] We now have STR and hardware, no need to test this; just focus on Plugins testing as below

Plug-ins

  • Tests:
    • Flash (11.2, 11.3)
    • Java
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, maps, street view, or other rich interfaces
  • Results:
    • Windows 7: PASS (Simona)
    • Ubuntu 12.04: PASS (Virgil)
    • Mac OSX 10.6: PASS (VladG)
  • Details & Bugs:
    • bug 759689 - Flash content is not loaded using Flash 11.3 Beta
    • bug 675645 - Firefox freezes when using Pop out while in Fullscreen mode on Youtube

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • keep an eye out for crashes
  • Results:
    • Windows 7: PASS (Simona)
    • Ubuntu 12.04: PASS (Virgil)
    • Mac OSX 10.6: PASS (VladG)
  • Details & Bugs:

Mac OS X 10.8

  • Tests:
    • Before testing run codesign -vv Firefox.app on the file to ensure it is signed
      • I getting the following message: valid on disk, satisfies its Designates Requirements
      • The build is signed
    • Try all the menu item functions, including
    • Gestures
    • Plug-in compatibility (as above)
    • Top-site compatibility (as above)
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: PASS (VladG)
  • Details & Bugs:

Bug Fix Verifications

P1 Verifications (DO THESE FIRST!)
  • bug 757262 Youtube keeps playing audio after closing
    • Can't verify due to unreproducible case, will defer to Beta feedback
  • bug 759788 Java related Crash (sg:crit)
    • Currently testing...
P2 Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 107 12 87%
QA+ SG: for the Security team to verify security fixes 24 7 77%
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 379 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • bug 756289 Netflix video errors "you are viewing from more than one tab"

Issues Found

ID Summary Priority Status
760417 When selecting "Pop out" for HTML5 videos, the window is opened in full screen mode and the "Full screen" button is still available on its right bottom side -- NEW
760415 Firefox crashes if locking and unlocking Windows XP while on http://chrome.angrybirds.com/ -- RESOLVED
759689 [adbe 3210913] Flash content is not loaded using Flash 11.3 Beta -- RESOLVED
760119 Switching back from custom install to standard doesn't reset the installation folder -- RESOLVED

Open; Resolved; Total (75% complete)


Release Candidate

Builds FTP, HTTP
ID 20120601045813
Changeset 2b643ea8edf9
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 100%)
    • 2 remaining fixes pending Engineering assistance to test
  • [DONE] Security fix verifications (target 100%)
  • [DONE] Update tests on releasetest
  • [DONE] Get stability sign-off (marcia)
  • [DONE] Get final sign-off (bmoss)
  • [DONE] Update tests on release

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests

Platform Result: report Failures: known bugs
Windows XP PASS - 2012-06-01 0 - no fails
Windows Vista PASS - 2012-06-01 0 - no fails
Windows 7 32-bit PASS - 2012-06-01 0 - no fails
Windows 7 64-bit PASS - 2012-06-01 0 - no fails
Mac OS X 10.6 PASS - 2012-06-01 0 - no fails
Mac OS X 10.7 PASS - 2012-06-01 0 - no fails
Ubuntu 11.10 32-bit PASS - 2012-06-01 0 - no fails
Ubuntu 11.10 -bit PASS - 2012-06-01 0 - no fails

Update Tests

Firefox 10 through 12 -> 13
Platform Releasetest: report Release: report
Windows XP PASS - 2012-06-02 PASS - 2012-06-05
Windows Vista PASS - 2012-06-02 PASS - 2012-06-05
Windows 7 32-bit PASS - 2012-06-02 PASS - 2012-06-05
Windows 7 64-bit PASS - 2012-06-02 PASS - 2012-06-05
Mac OS X 10.6 PASS - 2012-06-02 PASS - 2012-06-05
Mac OS X 10.7 PASS - 2012-06-02 PASS - 2012-06-05
Ubuntu 11.10 32-bit PASS - 2012-06-02 PASS - 2012-06-05
Ubuntu 11.10 -bit PASS - 2012-06-02 PASS - 2012-06-05
Firefox 4 through 9 -> 12
Platform Releasetest: report Release: report
Windows 2000 PASS - 2012-06-02 PASS - 2012-06-05
Windows XP PASS - 2012-06-02 PASS - 2012-06-05
Windows Vista PASS - 2012-06-02 PASS - 2012-06-05
Windows 7 32-bit PASS - 2012-06-02 PASS - 2012-06-05
Windows 7 64-bit PASS - 2012-06-02 PASS - 2012-06-05
Mac OS X 10.6 PASS - 2012-06-02 PASS - 2012-06-05
Mac OS X 10.7 PASS - 2012-06-02 PASS - 2012-06-05
Ubuntu 11.10 32-bit PASS - 2012-06-02 PASS - 2012-06-05
Ubuntu 11.10 -bit PASS - 2012-06-02 PASS - 2012-06-05

Smoketests

Run the Firefox 13.0 Smoketests in Litmus and report results below.

Platform Assignee Result: report
Windows XP Otilia PASS - 2012-06-02
Windows Vista VladG PASS - 2012-06-02
Windows 7 32-bit Mihaela PASS - 2012-06-02
Windows 7 64-bit Otilia PASS - 2012-06-02
Mac OS X 10.6.8 VladG PASS - 2012-06-02
Mac OS X 10.7.4 VladG PASS - 2012-06-02
Ubuntu 11.10 32-bit Ioana PASS - 2012-06-02
Ubuntu 11.10 64-bit Ioana PASS - 2012-06-02

Focus Testing

Focus Area Win32 Mac Linux
Plug-ins PASS - 2012-06-02 PASS - 2012-06-02 PASS - 2012-06-02
Top-sites PASS - 2012-06-02 PASS - 2012-06-02 PASS - 2012-06-02
Mac OSX 10.8 N/A PASS - 2012-06-02 N/A

Plug-ins

  • Tests:
    • Flash (11.2, 11.3)
    • Java
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, or other rich interfaces
  • Results:
    • Windows 7: PASS (Mihaela)
    • Ubuntu 11.10: PASS (Ioana)
    • Mac OSX 10.6: PASS (VladG)
  • Details & Bugs:

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • keep an eye out for crashes
  • Results:
    • Windows 7: PASS (Mihaela)
    • Ubuntu 11.10: PASS (Ioana)
    • Mac OSX 10.6: PASS (VladG)
  • Details & Bugs:

Mac OS X 10.8

  • Tests:
    • Try all the menu item functions
    • Gestures
    • Plug-in compatibility
      • With html5 enabled, in you tube, the videos are flickering green along with some video overlay. Perhaps this is caused by the video card drivers for OS X 10.8. I'll investigate further.
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: PASS with issues (VladG)
  • Details & Bugs:

Exploratory testing

  • Tests:
    • Add-ons, SPDY, Smooth scrolling, Software update, Reset Firefox, Windows 8, Plugins, On-demand Tab Restore
  • Results: (etherpad)
    • Windows 7: PASS (Mihaela)
    • Ubuntu 11.10: PASS (Ioana)
    • Mac OSx 10.6: PASS (VladG)
  • Details & Bugs:
    • bug 760810 - Google images results pages don't scroll smoothly with PgUp/PgDn and Up/Down key arrows

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox13 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 112 2 98%
QA+ SG: for the Security team to verify security fixes 30 0 100%
status-firefox13:fixed: for QA lead to triage for potential QA+ bugs 379 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Speculative fixes to be verified via post-release feedback
  • bug 756289 Netflix video errors "you are viewing from more than one tab"
  • bug 757262 Youtube keeps playing audio after closing
Tracking for 13.0.1
  • bug 760190 OOM crash in `anonymous namespace'::CSSParserImpl::ParseSelector
    • Likely fallout from fixed #1 top-crash (bug 719117)
  • bug 709860 OOM crash in nsIDNService::Init @ mozilla::Preferences::GetBranch
    • Long-standing crash seems to have escalated in Fx13, Marcia is investigating
Add-on Specific
  • bug 760960 crash in nsNPAPIPluginInstance::GetJSObject with LightShot plugin
    • Staging a block for the LightShot add-on, developer has taken it down from AMO

Issues Found

ID Summary Priority Status
760810 Google images results pages don't scroll smoothly with PgUp/PgDn and Up/Down key arrows -- RESOLVED

Open; Resolved; Total (100% complete)

Issues Found

Known Issues

ID Summary Priority Status
585971 autocomplete matches unrelated hosts when searching for www.example.com -- NEW
725296 Browser is unresponsive while Silverlight content is loading -- NEW
725614 Application shortcut keys fail to work on pages containig Java/Shockwave/PDF apps after focus is restored to other browser content -- NEW
749112 "Live Bookmarks Loading" message is displayed at the bottom of the RSS feeds -- NEW
753684 Removing an add-on installed twice doesn't actually remove it -- NEW
753770 Printing progress bar is not properly displayed -- NEW
754829 In Full Screen mode - YouTube videos player changes dimmension -- NEW
756080 The Bookmarks toolbar is set as visible automatically when resetting Firefox with at least 5 bookmarks toolbar items -- NEW
760417 When selecting "Pop out" for HTML5 videos, the window is opened in full screen mode and the "Full screen" button is still available on its right bottom side -- NEW
587140 Tab groups can push other groups across screen boundaries, making them invisible or causing groups to overlap P3 NEW
730829 Restore "Open all in tabs" and "open <site>" to the bottom of livemarks popups (was: "Open all in tabs" missing after bug 613588) -- ASSIGNED
760415 Firefox crashes if locking and unlocking Windows XP while on http://chrome.angrybirds.com/ -- RESOLVED
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
756111 Emoticons are not displayed on the Facebook chat -- RESOLVED
759689 [adbe 3210913] Flash content is not loaded using Flash 11.3 Beta -- RESOLVED
760119 Switching back from custom install to standard doesn't reset the installation folder -- RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
726943 Fullscreen button in HTML video makes the browser go fullscreen but not the video -- VERIFIED
727702 Undoing applying a Persona reverts back to default theme instead of old Persona -- VERIFIED
753889 stage.mo -> releases.mo breaks staging beta builds for update testing -- VERIFIED
753680 Cannot enable all personas from Add-ons Manager list -- VERIFIED

Open; Resolved; Total (28.57% complete)


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

13.0.1

Builds FTP, HTTP
ID 20120614114901
Changeset f48d675ffa9f

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests

Platform Result: report Failures: known bugs
Windows XP PASS - 2012-06-14 0 - no fails
Windows Vista PASS - 2012-06-14 0 - no fails
Windows 7 32-bit PASS - 2012-06-14 0 - no fails
Windows 7 64-bit PASS - 2012-06-14 0 - no fails
Mac OS X 10.6 PASS - 2012-06-14 0 - no fails
Mac OS X 10.7 PASS - 2012-06-14 0 - no fails
Ubuntu 11.10 32-bit PASS - 2012-06-14 0 - no fails
Ubuntu 11.10 -bit PASS - 2012-06-14 0 - no fails

Update Tests

Firefox 10.0 through 13.0 -> 13.0.1
Platform Releasetest: report Release: report
Windows XP PASS - 2012-06-15 PASS - 2012-06-15
Windows Vista PASS - 2012-06-15 PASS - 2012-06-15
Windows 7 32-bit PASS - 2012-06-15 PASS - 2012-06-15
Windows 7 64-bit PASS - 2012-06-15 PASS - 2012-06-15
Mac OS X 10.6 PASS - 2012-06-15 PASS - 2012-06-15
Mac OS X 10.7 PASS - 2012-06-15 PASS - 2012-06-15
Ubuntu 11.10 32-bit PASS - 2012-06-15 PASS - 2012-06-15
Ubuntu 11.10 -bit PASS - 2012-06-15 PASS - 2012-06-15
Firefox 4.0 through 9.0 -> 12.0
  • RELEASETEST
    • Firefox 4.0 win32 en-US: PASS - 2012-06-15
    • Firefox 8.0 mac de: PASS - 2012-06-15
    • Firefox 9.0 linux pt-BR: FAIL - 2012-06-15
      • bug 765312 - FF4 to FF9 Linux 32/64-bit users 404 for complete.mars
  • RELEASE
    • Firefox 4.0 win32 en-US: PASS - 2012-06-15
    • Firefox 8.0 mac de: PASS - 2012-06-15
    • Firefox 9.0 linux pt-BR: PASS - 2012-06-15
What's New Page
  • RELEASETEST
    • Firefox 13.0 win32 en-US - PASS - 2012-06-15 (no WN page)
    • Firefox 12.0 mac ja-JP-mac - PASS - 2012-06-15
    • Firefox 11.0 linux de - PASS - 2012-06-15
  • RELEASE
    • Firefox 13.0 win32 en-US - PASS - 2012-06-15 (no WN page)
    • Firefox 12.0 mac ja-JP-mac - PASS - 2012-06-15
    • Firefox 11.0 linux de - PASS - 2012-06-15

Manual Testing

Focus Area Win32 Mac Linux
Plug-ins PASS - 2012-06-15 PASS - 2012-06-15 PASS - 2012-06-15
Top-sites PASS - 2012-06-15 PASS - 2012-06-15 PASS - 2012-06-15
Mac OSX 10.8 N/A PASS - 2012-06-15 N/A

Details

Plug-ins

  • Tests:
    • Flash 11.3
    • Java 7u4
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content
  • Results:
    • Windows 7: PASS (Paul)
    • Ubuntu 12.04: PASS (Virgil)
    • Mac OSX 10.6.8: PASS (VladG)
  • Details & Bugs:

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Twitter: tweet, follow, unfollow, media sharing, trends
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • Webmail: Hotmail, Gmail, Yahoo
    • keep an eye out for crashes
  • Results:
    • Windows 7: PASS (Paul)
    • Ubuntu 12.04: PASS (Virgil)
    • Mac OSX 10.6.8: PASS (VladG)
  • Details & Bugs:

Mac OS X 10.8 Code-signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS -VladG
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS -VladG
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS -VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS -VladG
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS -VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS -VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS -VladG

Do the following testing to make sure the build works as expected...

  • Tests:
    • Try all the menu item functions
    • Gestures
    • Plug-in compatibility
      • With html5 enabled, in you tube, the videos are flickering green along with some video overlay. Perhaps this is caused by the video card drivers for OS X 10.8. I'll investigate further.
      • I've got no flickering and the videos are playing as expected in html5. The test machine has a AMD Radeon HD 6630M 256 MB video card - VladG
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: PASS with issues (VladG)
  • Details & Bugs:

Bug Verifications

The following should be solved via Blocklisting:

  • [ON TRACK] bug 763459 - RealPlayer Browser Record plugin breaks Youtube since Fx13
  • [ON TRACK] bug 762062 - Firefox startup crash in JS_HoldPrincipals (Skype 6.0.0.10201)

Fixed bugs needing Verification:

  • [DROPPED] bug 756277 - XMLHttpRequest started from Cc["@mozilla.org/xmlextras/xmlhttprequest;1"] ends up bound to some random window
  • [DONE] bug 756850 - Some combinations of Hebrew vowels and consonants not displayed in certain fonts with harfbuzz
  • [DONE] bug 747683 [adbe 3172259] Crash Report [@ LdrShutdownProcess ] with Flash 11.3 "Protected Mode for Firefox"

Backout bugs needing Confirmation:

  • [DONE] bug 733614 - CSS 'height' property on a multi-column block is not respected
  • [DONE] bug 736731, bug 764546 - Hotmail no longer auto-updates (due to "globalStorage" removal)

Third-party bugs we are tracking but won't resolve on our end:

  • bug 681521 - Firefox Crash [@ JS_GetContextPrivate ] (Trusteer)
  • bug 761598 - Crash in RapportTanzan13 with Trusteer Rapport and LastPass
  • bug 709860 - OOM crash in nsIDNService::Init @ mozilla::Preferences::GetBranch (Avast)
  • bug 759689 - [adbe 3210913] Flash content is not loaded using Flash 11.3 Beta

13.0.2

Builds [ FTP], [ HTTP]
ID
Changeset []

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests

Platform Result: [ report] Failures: known bugs
Windows XP result - YYYY-MM-DD 0 - no fails
Windows Vista result - YYYY-MM-DD 0 - no fails
Windows 7 32-bit result - YYYY-MM-DD 0 - no fails
Windows 7 64-bit result - YYYY-MM-DD 0 - no fails
Mac OS X 10.6 result - YYYY-MM-DD 0 - no fails
Mac OS X 10.7 result - YYYY-MM-DD 0 - no fails
Ubuntu 11.10 32-bit result - YYYY-MM-DD 0 - no fails
Ubuntu 11.10 -bit result - YYYY-MM-DD 0 - no fails

Update Tests

Firefox 10.0 through 13.0 -> 13.0.1
Platform Releasetest: [ report] Release: [ report]
Windows XP result - YYYY-MM-DD result - YYYY-MM-DD
Windows Vista result - YYYY-MM-DD result - YYYY-MM-DD
Windows 7 32-bit result - YYYY-MM-DD result - YYYY-MM-DD
Windows 7 64-bit result - YYYY-MM-DD result - YYYY-MM-DD
Mac OS X 10.6 result - YYYY-MM-DD result - YYYY-MM-DD
Mac OS X 10.7 result - YYYY-MM-DD result - YYYY-MM-DD
Ubuntu 11.10 32-bit result - YYYY-MM-DD result - YYYY-MM-DD
Ubuntu 11.10 -bit result - YYYY-MM-DD result - YYYY-MM-DD
Firefox 4.0 through 9.0 -> 12.0
  • RELEASETEST
    • Firefox 4.0 win32 en-US: result - YYYY-MM-DD
    • Firefox 8.0 mac de: result - YYYY-MM-DD
    • Firefox 9.0 linux pt-BR: result - YYYY-MM-DD
  • RELEASE
    • Firefox 4.0 win32 en-US: result - YYYY-MM-DD
    • Firefox 8.0 mac de: result - YYYY-MM-DD
    • Firefox 9.0 linux pt-BR: result - YYYY-MM-DD
  • Automatic Spotcheck
What's New Page
  • RELEASETEST
    • Firefox 13.0 win32 en-US - result - YYYY-MM-DD (no WN page)
    • Firefox 12.0 mac ja-JP-mac - result - YYYY-MM-DD
    • Firefox 11.0 linux de - result - YYYY-MM-DD
  • RELEASE
    • Firefox 13.0 win32 en-US - result - YYYY-MM-DD (no WN page)
    • Firefox 12.0 mac ja-JP-mac - result - YYYY-MM-DD
    • Firefox 11.0 linux de - result - YYYY-MM-DD

Manual Testing

Focus Area Win32 Mac Linux
Plug-ins result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD
Top-sites result - YYYY-MM-DD result - YYYY-MM-DD result - YYYY-MM-DD
Mac OSX 10.8 N/A result - YYYY-MM-DD N/A

Plug-ins

  • Tests:
    • Flash 11.3
    • Java 7u4
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content
  • Results:
    • Windows 7: result - YYYY-MM-DD (tester)
    • Ubuntu 12.04: result - YYYY-MM-DD (tester)
    • Mac OSX 10.6.8: result - YYYY-MM-DD (tester)
  • Details & Bugs:

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Twitter: tweet, follow, unfollow, media sharing, trends
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • Webmail: Hotmail, Gmail, Yahoo
    • keep an eye out for crashes
  • Results:
    • Windows 7: result - YYYY-MM-DD (tester)
    • Ubuntu 12.04: result - YYYY-MM-DD (tester)
    • Mac OSX 10.6.8: result - YYYY-MM-DD (tester)
  • Details & Bugs:

Mac OS X 10.8 Code-signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • result - YYYY-MM-DD (tester)
  • Open a terminal window and cd to the folder Firefox.app is located
    • result - YYYY-MM-DD (tester)
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • result - YYYY-MM-DD (tester)
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • result - YYYY-MM-DD (tester)
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • result - YYYY-MM-DD (tester)
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • result - YYYY-MM-DD (tester)
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • result - YYYY-MM-DD (tester)

Do the following testing to make sure the build works as expected...

  • Tests:
    • Try all the menu item functions
    • Gestures
    • Plug-in compatibility
      • With html5 enabled, in you tube, the videos are flickering green along with some video overlay. Perhaps this is caused by the video card drivers for OS X 10.8. I'll investigate further.
      • I've got no flickering and the videos are playing as expected in html5. The test machine has a AMD Radeon HD 6630M 256 MB video card - VladG
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: result - YYYY-MM-DD (tester)
  • Details & Bugs:

Bug Verifications

Add your name beside whatever you are testing so people don't duplicate your results.

  • bug 686335 - Bad plugin search order prevents upgrading Flash without an OS restart
  • bug 758363 - Implement NPN_ReloadPlugins for out-of-process Plugins
NOTE that testing any of these in a VM requires the VM to be run in Protected Mode

My advice would be to take a snapshot of the VM before starting and rolling back before each test they can be run from a clean snapshot.

NOTE that not all of these plugin versions are available for Mac and Linux.

Just test with whatever old versions you can find and update to the latest. Report the version numbers in your results.

Builds

Tests

Flash 10.2 (blocklisted)

Flash 10.3 (enabled & disabled)

  • Download and install Flash 10.3
  • Load some Flash content in tabs before updating
  • Pave-over install Flash 11.3.300.262
  • Reload tabs with Flash
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.2 (enabled & disabled)

  • Download and install Flash 11.2
  • Load some Flash content in tabs before updating
  • Pave-over install Flash 11.3.300.262
  • Reload tabs with Flash
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 (enabled & disabled)

  • Download and install Flash 11.3.300.257
  • Load some Flash content in tabs before updating
  • Pave-over Flash 11.3.300.262
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)

  • Download and install Flash 11.2
  • Start Firefox and open a page with flash content
  • Download but do not install Flash 11.3
  • Open a command line to the download folder and run:
    • install_flash_player.exe -install -iv 9
  • Reload the page with Flash content once the installer finishes
  • Uninstall Flash
  • Repeat the test, loading Flash content in a new tab after update
  • Repeat the test, loading Flash content in a new window after update

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)

  • Download and install Flash 11.2
  • Start Firefox and open a page with flash content
  • Download but do not install Flash 11.3
  • Open a command line to the download folder and run:
    • install_flash_player.exe -install -iv 9
  • Reload the page with Flash content once the installer finishes
  • Uninstall Flash
  • Repeat the test, loading Flash content in a new tab after update
  • Repeat the test, loading Flash content in a new window after update

Java 6u31 (enabled & disabled)

  • Download and install Java 6u31
  • Load some Java content in tabs before updating
  • Pave-over install Java 7u5
  • Reload tabs with Java content after updating
  • Uninstall Java and retest with Java disabled before updating

Java 7u2 (enabled & disabled)

  • Download and install Java 7u2
  • Load some Java content in tabs before updating
  • Pave-over install Java 7u5
  • Reload tabs with Java content after updating
  • Uninstall Java and retest with Java disabled before updating

Results

Expected
  • Plugin updates without any problems
  • Latest version of the plugin is listed in about:plugins
  • Content loads (or reloads) after update without crashing
  • NOTE: Multiple versions may be listed in about:plugins
    • OKAY as long as plugin content is using the latest version
    • Multiple entries can be cleared by restarting the OS
Windows 7
  • Firefox 16.0a1 2012-06-22 (ashughes)
    • PASS Flash 10.2 (blocklisted)
    • PASS Flash 10.3 (enabled & disabled)
    • PASS Flash 11.2 (enabled & disabled)
    • ISSUE Flash 11.3 (enabled & disabled)
      • Multiple entries, crash when not using auto-update
    • ISSUE Java 6u31 (enabled & disabled)
      • Firefox requires restart before updated plug-in is activated
    • PASS Java 7u2 (enabled & disabled)
    • PASS Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)
    • ISSUE Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)
      • dom.ipc.plugins.enabled=false disables Flash (unreproducible)
Windows XP
  • Firefox 16.0a1 2012-06-22 (ashughes)
    • PASS Update Flash 10.2 -> 11.3.300.262 (enabled) using the installer method
    • PASS Update Flash 11.2 -> 11.3.300.262 (enabled) using the auto-update method
    • ISSUE Update Java 6u31 -> 7u5 (disabled)
      • Firefox requires restart before updated plug-in is activated
    • PASS Update Java 7u2 -> 7u5 (enabled)
Mac OSX 10.6
  • Firefox 16.0a1 2012-06-22 (ashughes)
    • PASS Update Flash 10.2 -> 11.3.300.257 (enabled) via pave-over install
    • PASS Update Flash 11.2 -> 11.3.300.257 (enabled) via pave-over install
    • Note: Max Flash version is 11.3.300.257
    • Could not find a way to do Java updates reliably -- could not find an intuitive way to install out of date plugins and upgrade
Ubuntu 12.04
  • Firefox 13.0.1 Tryserver (virgil)
    • PASS Flash 10.2 r159 -> 11.2.205.236 (enabled & disabled) via pave-over
    • PASS Flash 11.2.205.235 -> 11.2.205.236 (enabled & disabled) via pave-over
    • PASS Flash 10.2 r159 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
    • PASS Flash 11.2.205.235 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
  • Notes
    • Flash Plugin: Replace flash by overwriting usr/lib/mozilla/plugins/libflashplayer.so
    • IcedTea Java Plugin: was unable to get an older version working in the OS, Software Centre only provides the latest.
    • Oracle Java: Could not get older versions to install using this method

Issues Found

  • Disabled Java requires restart of Firefox to re-enable after update
  • Side-by-Side Flash versions in about:config when paving over Flash 11.3.300.257 64-bit with 11.3.300.262 32-bit; resolved only by OS restart; does not seem to affect functionality/stability
  • Side-by-Side Flash versions in about:config when simulating 11.2->11.3 auto-update with install_flash_player.exe -install -iv 9
  • dom.ipc.plugins.enabled set to false appears to disable Flash
  • Hang/crash when watching Youtube on update (bug 686335#c63) -- not reproducible using auto-update

12.0 to 13.0.1 Partial Updates

RE bug 768576
Platform Betatest: report Releasetest: report Release: report
Windows XP PASS - 2012-06-27 PASS - 2012-06-28 PASS - 2012-06-28
Windows Vista PASS - 2012-06-27 PASS - 2012-06-28 PASS - 2012-06-28
Windows 7 32-bit PASS - 2012-06-27 PASS - 2012-06-28 PASS - 2012-06-28
Windows 7 64-bit PASS - 2012-06-27 PASS - 2012-06-28 PASS - 2012-06-28
Mac OS X 10.6 PASS - 2012-06-27 PASS - 2012-06-28 PASS - 2012-06-28
Mac OS X 10.7 PASS - 2012-06-27 PASS - 2012-06-28 PASS - 2012-06-28
Ubuntu 11.10 32-bit n/a n/a n/a
Ubuntu 11.10 -bit n/a n/a n/a

Issues

  • [FIXED] bug 769156 - 12->13.0.1 updates on releasetest fail due to 13.0.2build1 staging
    • Resolved by bhearsum deleting 13.0.1->13.0.2 snippets on releasetest channel
    • NOTE: All releasetest failures were caused by this bug, cannot be removed from the report