Releases/Firefox 17/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 16 | Firefox 17 | Firefox 18 »

Contents

Firefox 17 Test Plan

Summary

Milestone Firefox 17
Status Released on Tuesday, November 20, 2012
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 17 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 17. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-07-16: [DONE] Nightly Merge
  • 2012-08-27: [DONE] Aurora Merge
  • 2012-10-08: [DONE] Pre-Beta feature sign-offs
  • 2012-10-11: [DONE] Beta 1
  • 2012-10-18: {{|}} Beta 2
  • 2012-10-25: {{|}} Beta 3
  • 2012-11-01: {{|}} Beta 4
  • 2012-11-08: {{|}} Beta 5
  • 2012-11-15: {{|}} Beta 6
  • 2012-11-19: {{|}} Release Candidate
  • 2012-11-20: {{|}} Final

Nightly

Merge

Functional Automation - result
Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Improve display of location bar results Virgil Dicu On track
Firefox Social Integration Anthony Hughes On track
HTML Tree Editor Mihaela Velimiroviciu
Silent Update not now prompt Ioana Budnar Release target removed.

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated

Bugzilla query error


Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated


Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (SoftVision) - Last Updated

Result Spreadsheet

Aurora Merge

NOTE - OSX 10.5 should not be able to install nor update to Firefox 17
Builds nightly
Changeset tip (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-08-28
Windows Vista PASS - 2012-08-28
Windows 7 32-bit PASS - 2012-08-28
Windows 7 64-bit PASS - 2012-08-28
Windows 8 32-bit N/A (pending IT)
Windows 8 64-bit N/A (pending IT)
Mac OS X 10.6 PASS - 2012-08-28
Mac OS X 10.7 PASS - 2012-08-28
Mac OS X 10.8 PASS - 2012-08-28
Ubuntu 12.04 32-bit PASS - 2012-08-28
Ubuntu 12.04 64-bit PASS - 2012-08-28
Test Failures (known bugs)

Manual Testing

Operating System Tester Plug-in Compatibility Web Compatibility OSX 10.5 EOL
Windows XP Simona PASS - 2012-08-29 PASS - 2012-08-29 N/A
Windows 7 Ioana PASS - 2012-08-29 PASS - 2012-08-29 N/A
Windows 8 Paul PASS - 2012-08-30 PASS - 2012-08-30 N/A
Mac OS X VladG PASS - 2012-08-30 PASS - 2012-08-30 PASS - 2012-08-30
Ubuntu 12.04 Ioana PASS - 2012-08-30 PASS - 2012-08-30 N/A

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

OSX 10.5 EOL

Verify the following:

  1. Mac OSX 10.5 cannot download Firefox 17 (I'm not sure if this is scoped but try it anyway)
  2. Mac OSX 10.5 cannot start Firefox 17
  3. Mac OSX 10.5 cannot update to Firefox 17
  • The build can be downloaded and installed
  • Aurora didn't start, showing a incompatibility message
  • The update channel seems to be close so I didn't manage to perform an update

Feature Sign-offs

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 17
  • Select Runs: Firefox 17 Aurora Merge Feature sign off
Feature Owner Win32 Mac Linux
Improve display of location bar results Virgil Dicu PASS PASS PASS
Firefox Social Integration Anthony Hughes PASS PASS PASS
Details

Report any detailed results or bugs here...

Update Testing

Spotcheck updates from a previous Aurora to Aurora 17.

Platform Auroratest Aurora
Windows 7 64-bit PASS - 2012-08-30 PASS - 2012-08-30
Windows 8 64-bit PASS - 2012-08-30 PASS - 2012-08-30
Mac OS X 10.7 PASS - 2012-08-30 PASS - 2012-08-30
Ubuntu 12.04 64-bit PASS - 2012-08-30 PASS - 2012-08-30
Issues

Bug Triage

  1. VERIFYME (5): verify the fix and do some exploratory testing around the issue
  2. FIXED (139): triage these bugs and test to ensure the fix is effective
  3. QAWANTED (6): triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (8): triage these bugs for potential new regressions
Results

Beta 1

Builds #1: FTP, HTTP #2: FTP, HTTP
ID #1: 20121009144933 #2: 20121010150351
Changeset 6b83222781e3 (pushlog)
Advisory

RelEng issues in generating the builds (see bug 796959 comment 5)

Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build #1 Results (report) Build #2 Results (report)
Windows XP PASS - 2012-10-09 PASS - 2012-10-10
Windows Vista PASS - 2012-10-09 PASS - 2012-10-10
Windows 7 32-bit PASS - 2012-10-09 PASS - 2012-10-10
Windows 7 64-bit PASS - 2012-10-09 PASS - 2012-10-10
Windows 8 32-bit PASS - 2012-10-09 PASS - 2012-10-10
Windows 8 64-bit PASS - 2012-10-09 PASS - 2012-10-10
Mac OS X 10.6 PASS - 2012-10-09 PASS - 2012-10-10
Mac OS X 10.7 PASS - 2012-10-09 PASS - 2012-10-10
Mac OS X 10.8 PASS - 2012-10-09 PASS - 2012-10-10
Ubuntu 12.04 32-bit PASS - 2012-10-09 PASS - 2012-10-10
Ubuntu 12.04 64-bit PASS - 2012-10-09 PASS - 2012-10-10
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows 7 Manuela, Ioana PASS - 2012-10-11
Windows 8 Virgil PASS - 2012-10-11
Mac OSX 10.7 Ioana, Mihaela PASS - 2012-10-11
Mac OSX 10.8 Virgil PASS - 2012-10-11
Ubuntu 12.04 Manuela PASS - 2012-10-11
Special Cases
Mac OSX 10.5 de-support Virgil PASS - 2012-10-11
Update regression tests (requires updates on releasetest) ashughes PASS - 2012-10-11

Update Testing

  • Partial: 16.0b6, 16.0b5
  • Complete: <= 16.0b4
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-10-11 PASS - 2012-10-11
Windows Vista PASS - 2012-10-11 PASS - 2012-10-11
Windows 7 32-bit PASS - 2012-10-11 PASS - 2012-10-11
Windows 7 64-bit PASS - 2012-10-11 PASS - 2012-10-11
Windows 8 32-bit PASS - 2012-10-11 PASS - 2012-10-11
Windows 8 64-bit PASS - 2012-10-11 PASS - 2012-10-11
Mac OS X 10.6 PASS - 2012-10-11 PASS - 2012-10-11
Mac OS X 10.7 PASS - 2012-10-11 PASS - 2012-10-11
Mac OS X 10.8 PASS - 2012-10-11 PASS - 2012-10-11
Ubuntu 12.04 32-bit PASS - 2012-10-11 PASS - 2012-10-11
Ubuntu 12.04 64-bit PASS - 2012-10-11 PASS - 2012-10-11
Issues

Feature Sign-off

Feature owners to test and sign-off overall quality of their features following uplift to Beta.

Feature Owner Win32 Mac Linux
Improve display of location bar results Virgil Dicu n/a release target should be updated n/a release target should be updated n/a release target should be updated
HTML tree editor Mihaela Velimiroviciu PASS - 2012-10-11 PASS - 2012-10-11 PASS - 2012-10-11
Firefox Social Integration Anthony Hughes n/a - delayed to Beta 2 n/a - delayed to Beta 2 n/a - delayed to Beta 2

Build #2 Verifications

Beta 2

Builds FTP, HTTP
ID 20121017073013
Changeset 2ba4becf6e35 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results (report)
Windows XP PASS - 2012-10-17
Windows Vista PASS - 2012-10-17
Windows 7 32-bit PASS - 2012-10-17
Windows 7 64-bit PASS - 2012-10-17
Windows 8 32-bit PASS - 2012-10-17
Windows 8 64-bit PASS - 2012-10-17
Mac OS X 10.6 PASS - 2012-10-17
Mac OS X 10.7 PASS - 2012-10-17
Mac OS X 10.8 PASS - 2012-10-17
Ubuntu 12.04 32-bit PASS - 2012-10-17
Ubuntu 12.04 64-bit PASS - 2012-10-17
Test Failures (known bugs)
  • testPrivateBrowsing/testDownloadManagerClosed.js (known)

Manual Testing

Run the Firefox 17b2 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Moztrap > Desktop Firefox > 17 > Firefox 17b2 Regression Tests > Operating System > Platform > Run Tests

Platform Tester Result (report)
Windows 7 Manuela PASS - 2012-10-18
Windows 8 Simona PASS - 2012-10-18
Mac OSX 10.7 Paul PASS - 2012-10-18
Mac OSX 10.8 Mihaela PASS - 2012-10-18
Ubuntu 12.04 Virgil PASS - 2012-10-18
Special Cases
Mac OSX 10.5 de-support Virgil PASS - 2012-10-18
Update regression tests (requires updates on releasetest) ashughes PASS - 2012-10-18

Update Testing

  • Partial: 17.0b1, 16.0b6
  • Complete: <=16.0b5
Platform Releasetest (report) Beta ([ report])
Windows XP PASS - 2012-10-18 result - YYYY-MM-DD
Windows Vista PASS - 2012-10-18 result - YYYY-MM-DD
Windows 7 32-bit PASS - 2012-10-18 result - YYYY-MM-DD
Windows 7 64-bit PASS - 2012-10-18 result - YYYY-MM-DD
Windows 8 32-bit PASS - 2012-10-18 result - YYYY-MM-DD
Windows 8 64-bit PASS - 2012-10-18 result - YYYY-MM-DD
Mac OS X 10.6 PASS - 2012-10-18 result - YYYY-MM-DD
Mac OS X 10.7 PASS - 2012-10-18 result - YYYY-MM-DD
Mac OS X 10.8 PASS - 2012-10-18 result - YYYY-MM-DD
Ubuntu 12.04 32-bit PASS - 2012-10-18 result - YYYY-MM-DD
Ubuntu 12.04 64-bit PASS - 2012-10-18 result - YYYY-MM-DD
Manual Spotchecks (due to Jenkins outage)
Update Windows 7 Windows 8 Mac OSX 10.7 Mac OSX 10.8 Ubuntu 12.04
17.0b1 -> 17.0b2 en-US PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
17.0b1 -> 17.0b2 de PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
16.0b6 -> 17.0b2 en-US PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
16.0b6 -> 17.0b2 zh-TW PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
16.0b1 -> 17.0b2 en-US PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
16.0b1 -> 17.0b2 es-ES PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
15.0b6 -> 17.0b2 en-US PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
15.0b6 -> 17.0b2 pt-BR PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22 PASS - 2012-10-22
Issues

Feature Sign-off

Feature owners to test and sign-off overall quality of their features following uplift to Beta.

Feature Owner Win32 Mac Linux
Firefox Social Integration Anthony Hughes PASS - 2012-10-18 PASS - 2012-10-18 PASS - 2012-10-18
Issues
  • Server-side issues not blocking 17b2 (should be fixed with next server-side push)
    • Like button activity not pushing to website
    • Full page loading within panel (friends and notifications button)

Beta 3

Builds FTP, HTTP
ID 20121023124120
Changeset f8a6138bcfba (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results (report)
Windows XP PASS - 2012-10-24
Windows Vista PASS - 2012-10-24
Windows 7 32-bit PASS - 2012-10-24
Windows 7 64-bit PASS - 2012-10-24
Windows 8 32-bit PASS - 2012-10-24
Windows 8 64-bit PASS - 2012-10-24
Mac OS X 10.6 PASS - 2012-10-24
Mac OS X 10.7 PASS - 2012-10-24
Mac OS X 10.8 PASS - 2012-10-24
Ubuntu 12.04 32-bit PASS - 2012-10-24
Ubuntu 12.04 64-bit PASS - 2012-10-24
Test Failures (known bugs)
  • testPrivateBrowsing/testDownloadManagerClosed.js (known)

Manual Testing

Run the Firefox 17b3 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Moztrap > Desktop Firefox > 17 > Firefox 17b3 Regression Tests > Operating System > Platform > Run Tests

Platform Tester Result (report)
Windows 7 Simona PASS - 2012-10-24
Windows 8 Manuela PASS - 2012-10-24
Mac OSX 10.7 Virgil PASS - 2012-10-24
Mac OSX 10.8 Mihaela PASS - 2012-10-24
Ubuntu 12.04 Ioana PASS - 2012-10-24
Special Cases
Mac OSX 10.5 de-support Virgil PASS - 2012-10-24
Update regression tests (requires updates on releasetest) ashughes result - YYYY-MM-DD

Bug Fix Verifications

Update Testing

  • Partial: 17.0b2, 17.0b1
  • Complete: <=16.0b6
Platform Releasetest (report) Beta ([ report])
Windows XP PASS - 2012-10-24 result - YYYY-MM-DD
Windows Vista PASS - 2012-10-24 result - YYYY-MM-DD
Windows 7 32-bit PASS - 2012-10-24 result - YYYY-MM-DD
Windows 7 64-bit PASS - 2012-10-24 result - YYYY-MM-DD
Windows 8 32-bit PASS - 2012-10-24 result - YYYY-MM-DD
Windows 8 64-bit PASS - 2012-10-24 result - YYYY-MM-DD
Mac OS X 10.6 PASS - 2012-10-24 result - YYYY-MM-DD
Mac OS X 10.7 PASS - 2012-10-24 result - YYYY-MM-DD
Mac OS X 10.8 PASS - 2012-10-24 result - YYYY-MM-DD
Ubuntu 12.04 32-bit PASS - 2012-10-24 result - YYYY-MM-DD
Ubuntu 12.04 64-bit PASS - 2012-10-24 result - YYYY-MM-DD
Issues

Beta 4

Builds FTP, HTTP
ID 20121031065642
Changeset 486335dcb4a2 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results (report)
Windows XP PASS - 2012-11-01
Windows Vista PASS - 2012-11-01
Windows 7 32-bit PASS - 2012-11-01
Windows 7 64-bit PASS - 2012-11-01
Windows 8 32-bit PASS - 2012-11-01
Windows 8 64-bit PASS - 2012-11-01
Mac OS X 10.6 PASS - 2012-11-01
Mac OS X 10.7 PASS - 2012-11-01
Mac OS X 10.8 PASS - 2012-11-01
Ubuntu 12.04 32-bit PASS - 2012-11-01
Ubuntu 12.04 64-bit PASS - 2012-11-01
Test Failures (known bugs)
  • testPrivateBrowsing/testDownloadManagerClosed.js (known)

Manual Testing

Run the Firefox 17b4 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Moztrap > Desktop Firefox > 17 > Firefox 17b4 Regression Tests > Operating System > Platform > Run Tests

If you have an AMD or NVIDIA graphics card, please install the Beta drivers during your testing
  • AMD Catalyst 12.11 Beta (download)
    • AMD Catalyst 12.11 Beta graphics driver is not available for Mac OS X.
  • NVidia GeForce 310.33 Beta (download)
    • NVidia GeForce 310.33 Beta graphics driver is not compatible with Windows 8.
    • NVidia GeForce 310.33 Beta graphics driver is not available for Mac OS X.
  • keep your eye out for rendering issues and crashes
Platform Tester Result (report)
Windows 7 Paul PASS - 2012-11-01
Windows 8 Simona PASS - 2012-11-02
Mac OSX 10.7 Virgil PASS - 2012-11-01
Mac OSX 10.8 Mihaela PASS - 2012-11-01
Ubuntu 12.04 Ioana PASS - 2012-11-02
Special Cases
Mac OSX 10.5 de-support Virgil PASS - 2012-11-01
Update regression tests (requires updates on releasetest) ashughes PASS - 2012-11-02

Bug Fix Verifications

Update Testing

  • Partial: 17.0b3, 17.0b2
  • Complete: <=17.0b1
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-11-02 PASS - 2012-11-02
Windows Vista PASS - 2012-11-02 PASS - 2012-11-02
Windows 7 32-bit PASS - 2012-11-02 PASS - 2012-11-02
Windows 7 64-bit PASS - 2012-11-02 PASS - 2012-11-02
Windows 8 32-bit PASS - 2012-11-02 PASS - 2012-11-02
Windows 8 64-bit PASS - 2012-11-02 PASS - 2012-11-02
Mac OS X 10.6 PASS - 2012-11-02 PASS - 2012-11-02
Mac OS X 10.7 PASS - 2012-11-02 PASS - 2012-11-02
Mac OS X 10.8 PASS - 2012-11-02 PASS - 2012-11-02
Ubuntu 12.04 32-bit PASS - 2012-11-02 PASS - 2012-11-02
Ubuntu 12.04 64-bit PASS - 2012-11-02 PASS - 2012-11-02
Issues

Beta 5

Builds FTP, HTTP
ID 20121106195758
Changeset 036da70b61f9 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results (report)
Windows XP PASS - 2012-11-06
Windows Vista PASS - 2012-11-06
Windows 7 32-bit PASS - 2012-11-06
Windows 7 64-bit PASS - 2012-11-06
Windows 8 32-bit PASS - 2012-11-06
Windows 8 64-bit PASS - 2012-11-06
Mac OS X 10.6 PASS - 2012-11-06
Mac OS X 10.7 PASS - 2012-11-06
Mac OS X 10.8 PASS - 2012-11-06
Ubuntu 12.04 32-bit PASS - 2012-11-06
Ubuntu 12.04 64-bit PASS - 2012-11-06
Test Failures (known bugs)

Manual Testing

Special Instructions

While testing this Beta, please install the latest AMD or NVidia beta drivers (if you have a GPU) and Flash 11.5 (if you are on Windows). Report any visual problems, crashes, or hangs in Bugzilla and add them to the New Issues section below.

Regression Tests

Run the Firefox 17b5 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Moztrap > Desktop Firefox > 17 > Firefox 17b5 Regression Tests > Operating System > Platform > Run Tests

Platform Tester Result (report) Bugs
Windows 7 Manuela PASS - 2012-11-07
Windows 8 Simona PASS - 2012-11-07
Mac OSX 10.8 Virgil PASS - 2012-11-07
Ubuntu 12.04 Mihaela PASS - 2012-11-07
Mac OSX 10.5 de-support Paul PASS - 2012-11-07
Update regression tests ashughes PASS - 2012-11-07

Social API

In addition to normal dogfooding (sidebar, activity stream, chat, toolbar), please heavily focus on testing around Private Browsing. In this Beta, the Social API should be disabled in Private Browsing mode. Test to make sure Social API cannot be enabled in Private Browsing, your Social API session is restored when exiting Private Browsing, and that Private Browsing mode works as per normal for all other use cases (Bookmarks, History, Cache, Logins, Cookies, etc)

Platform Tester Result Bugs
Windows 7 Simona PASS - 2012-11-08 807217#c43 807675#c3
Mac OSX 10.8 Virgil PASS - 2012-11-08
Ubuntu 12.04 Ioana PASS - 2012-11-08 807217#c45

Click-to-Play

Firefox 17 includes a click-to-play blocking feature which should show an overlay when plug-in content loads. Clicking the overlay should play the content. This feature has not been deployed for all plug-ins however. Flash <=11.3, Java 7 <=7u8, Java 6 <=6u36, Reader <=10.1.3, Silverlight <=5.1.10410.0 should all be CTP blocked. All the latest versions of the plug-ins should not be blocked.

Recent regressions indicate not all sites respect CTP blocking (CNN and LA Times for example). Please check various websites with plug-in content. If you can, work through the top-100 list and fill in that table with your results. However, don't be afraid to go outside that list if there are other sites you can think of.

Platform Tester Result Bugs
Windows 7 Virgil, Manuela FAIL - 2012-11-07 809792, 809858
Mac OSX 10.8 Manuela, Paul FAIL - 2012-11-07 809792
Ubuntu 12.04 Ioana, Paul FAIL - 2012-11-08 809785, 809793, 809860

Bug Fix Verifications

Update Testing

  • Partial: 17.0b4, 17.0b3
  • Complete: <=17.0b2
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-11-07 PASS - 2012-11-08
Windows Vista PASS - 2012-11-07 PASS - 2012-11-08
Windows 7 32-bit PASS - 2012-11-07 PASS - 2012-11-08
Windows 7 64-bit PASS - 2012-11-07 PASS - 2012-11-08
Windows 8 32-bit PASS - 2012-11-07 PASS - 2012-11-08
Windows 8 64-bit PASS - 2012-11-07 PASS - 2012-11-08
Mac OS X 10.6 PASS - 2012-11-07 PASS - 2012-11-08
Mac OS X 10.7 PASS - 2012-11-07 PASS - 2012-11-08
Mac OS X 10.8 PASS - 2012-11-07 PASS - 2012-11-08
Ubuntu 12.04 32-bit PASS - 2012-11-07 PASS - 2012-11-08
Ubuntu 12.04 64-bit PASS - 2012-11-07 PASS - 2012-11-08
Issues

Beta 6

Builds FTP, HTTP
ID 20121113065533
Changeset 0dbbf9b3175c (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results (report)
Windows XP 4 failures - 2012-11-12
Windows Vista 0 failures - 2012-11-12
Windows 7 32-bit 0 failures - 2012-11-12
Windows 7 64-bit 0 failures - 2012-11-12
Windows 8 32-bit 0 failures - 2012-11-12
Windows 8 64-bit 0 failures - 2012-11-12
Mac OS X 10.6 0 failures - 2012-11-12
Mac OS X 10.7 0 failures - 2012-11-12
Mac OS X 10.8 0 failures - 2012-11-12
Ubuntu 12.04 32-bit 0 failures - 2012-11-12
Ubuntu 12.04 64-bit 0 failures - 2012-11-12
Test Failures (known bugs)
  • bug 811407 Awesomebar tests failed for 17.0b6 candidate on Windows XP resulting in Disconnect error

Manual Testing

Special Instructions

While testing this Beta, please install the latest AMD or NVidia beta drivers (if you have a GPU) and Flash 11.5 (if you are on Windows). Report any visual problems, crashes, or hangs in Bugzilla and add them to the New Issues section below.

Regression Tests

Run the Firefox 17b6 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Moztrap > Desktop Firefox > 17 > Firefox 17b6 Regression Tests > Operating System > Platform > Run Tests

Platform Tester Result (report) Bugs
Windows 7 Virgil PASS - 2012-11-14
Windows 8 Simona PASS - 2012-11-14
Mac OSX 10.8 Ioana PASS - 2012-11-14
Ubuntu 12.04 Manuela PASS - 2012-11-14
Mac OSX 10.5 de-support Mihaela PASS - 2012-11-14
Update regression tests ashughes PASS - 2012-11-14

Awesomebar

Check the Awesomebar for potential regressions. In particular, bookmarking, searching, filtering, security information, and switch-to-tab.

Platform Tester Result Bugs
Windows XP Virgil PASS - 2012-11-14
Windows 8 Simona PASS - 2012-11-14
Mac OSX 10.8 Mihaela PASS - 2012-11-14
Ubuntu 12.04 Manuela PASS - 2012-11-14

Private Browsing

Test Private Browsing mode and check for any potential regressions. In particular history, session and cache leaking into the normal browsing session.

Platform Tester Result Bugs
Windows XP Paul PASS - 2012-11-14
Windows 8 Virgil PASS - 2012-11-14
Mac OSX 10.8 Mihaela PASS - 2012-11-14
Ubuntu 12.04 Manuela PASS - 2012-11-14

Bug Fix Verifications

Downloads Panel

Social API

VERIFYME (bugs) - aiming to verify all blocker/critical/major bugs

Update Testing

  • Partial: 17.0b5, 17.0b4
  • Complete: <=17.0b3
Platform Releasetest (report) Beta (report)
Windows XP PASS - 2012-11-14 PASS - 2012-11-14
Windows Vista PASS - 2012-11-14 PASS - 2012-11-14
Windows 7 32-bit PASS - 2012-11-14 PASS - 2012-11-14
Windows 7 64-bit PASS - 2012-11-14 PASS - 2012-11-14
Windows 8 32-bit PASS - 2012-11-14 PASS - 2012-11-14
Windows 8 64-bit PASS - 2012-11-14 PASS - 2012-11-14
Mac OS X 10.6 PASS - 2012-11-14 PASS - 2012-11-14
Mac OS X 10.7 PASS - 2012-11-14 PASS - 2012-11-14
Mac OS X 10.8 PASS - 2012-11-14 PASS - 2012-11-14
Ubuntu 12.04 32-bit PASS - 2012-11-14 PASS - 2012-11-14
Ubuntu 12.04 64-bit PASS - 2012-11-14 PASS - 2012-11-14
Issues

Release Candidate

Builds FTP, HTTP FTP, HTTP
ID 20121116115405 20121119183901
Changeset 0af23f8db35d (pushlog) 919435c6f654 (pushlog)
Checklist (respin for bug 813264)

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build#1 Results (report) Build#2 Results (report)
Windows XP PASS - 2012-11-16 1 failure - 2012-11-19
Windows Vista PASS - 2012-11-16 PASS - 2012-11-19
Windows 7 32-bit PASS - 2012-11-16 PASS - 2012-11-19
Windows 7 64-bit PASS - 2012-11-16 PASS - 2012-11-19
Windows 8 32-bit PASS - 2012-11-16 PASS - 2012-11-19
Windows 8 64-bit PASS - 2012-11-16 PASS - 2012-11-19
Mac OS X 10.6 1 failure - 2012-11-16 PASS - 2012-11-19
Mac OS X 10.7 1 failure - 2012-11-16 PASS - 2012-11-19
Mac OS X 10.8 2 failures - 2012-11-16 1 failure - 2012-11-19
Ubuntu 12.04 32-bit 1 failure - 2012-11-16 PASS - 2012-11-19
Ubuntu 12.04 64-bit 1 failure - 2012-11-16 PASS - 2012-11-19
Test Failures (known bugs)
  • Build 1
    • /restartTests/testAddons_uninstallExtension/test4.js - Shutdown expected but none detected before end of test (bug 783484)
    • /testPreferences/testPreferredLanguage.js -could not find element Link: Documenti (bug 812435)
  • Build 2
    • /testAwesomeBar/testSuggestHistory.js (bug 813449)
    • /restartTests/testAddons_uninstallExtension/test4.js (bug 783484)

Manual Testing

Regression Tests

Run the Firefox 17 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Build 1 Results
Platform Tester Result (report) Bugs
Windows 7 Manuela PASS - 2012-11-19
Windows 8 Simona PASS - 2012-11-19
Mac OSX 10.8 Paul PASS - 2012-11-19
Ubuntu 12.04 Mihaela PASS - 2012-11-19
Mac OSX 10.5 de-support Ioana PASS - 2012-11-19
Update regression tests PASS - 2012-11-19
Build 2 Results

Conduct focused regression testing around third-party binary add-ons and exploratory testing around web compatibility. Yandex, AVG, Bandoo, and Skype are some add-ons to test, but try to test outside that scope and find some other third-party binary add-ons. In terms of websites, try some of the Alexa Top-sites but also try to visit sites you might normally use outside of that scope (banking, video, webmail, news, games, etc).

  • [FAIL] Windows 7 (Manuela, Ioana)
    • Addons Tested:
      • Hotspot Shield - PASS
      • Bandoo for Firefox - PASS
      • Skype Click to Call - FAIL: this addon is present in Firefox, but it doesn't work at all (after being forced as compatible and enabled). No phone numbers are highlighted (although they were when opening the same profile/pages etc in Firefox 15.0.1).
      • Yandex Toolbar - PASS
    • Websites Tested: facebook.com, amazon.com, twitter.com, qq.com, live.com, wikipedia.org, google.com, yahoo.com, baidu.com, youtube.com, miniclip.com, trailers.apple.com - PASS
    • Scenarios Tested: tested the addons for basic functionality, tested the websites for functionality, rendering and performance issues (with the addons installed).
  • [PASS] Windows 8 (Simona)
  • [FAIL] Mac OSX 10.8 (Paul)
  • [FAIL] Ubuntu 12.04 (Mihaela)
    • Addons Tested:
      • 4shared.com Community Toolbar3.15.1.0 - unstable (sometimes new tabs opened from + button are not shown until a browser restart)
      • BitTorrentBar Community Toolbar3.15.1.0 - PASS
      • Facebook Toolbar1.8.2 - PASS
      • Hotspot Shield Community Toolbar3.15.1.0 - breaks session restore (restore previous session button not displayed in about:home page, nor under History menu)
      • IncrediMail MediaBar 2 Community Toolbar3.15.1.0 - unstable (sometimes new tabs opened from + button are not shown until a browser restart)
      • McAfee SiteAdvisor28.0 - PASS
      • Yahoo! Toolbar2.5.1.20121012015120 - PASS
      • Yandex Community Toolbar3.15.1.0 - unstable (sometimes new tabs opened from + button are not shown until a browser restart), breaks session restore (restore previous session button not displayed in about:home page, nor under History menu)
    • Websites Tested: google.com, facebook.com, www.youtube.com (flash and html), maps.google.com, mail.yahoo.com, yahoo.com
    • Scenarios Tested: install/enable/disable add-on, enable/disable toolbar, add-ons basic functionality, websites basic functionality, pinned tabs
  • Issues Found:

Additional Spotchecks

  • [PASS] Mac OSX 10.5 Desupport (Virgil)

Bug Fix Verifications

VERIFYME (bugs) - aiming to verify all blocker/critical/major bugs

QAWANTED

  • bug 675260 crash in DefaultFreeEntry coming from ShutdownNSS (ashughes)
  • bug 770238 Fatal jemalloc assertion in js::gc::Arena::finalize<JSString> (marcia)
  • bug 812307 crash in DataMngrHlpFF17.dll @ nsVoidArray::~nsVoidArray (juanb)
  • bug 812456 Extension block request: DataMngr (virgil)
  • bug 812683 Win7 startup crash in d3d11.dll (after installing IE10?) (ioana)

Update Testing

  • Partial: 16.0.2, 15.0.1, 14.0.1
  • Complete: 16.0
Platform betatest (report) release (report)
Windows XP PASS - 2012-11-20 PASS - 2012-11-20
Windows Vista PASS - 2012-11-20 PASS - 2012-11-20
Windows 7 32-bit PASS - 2012-11-20 PASS - 2012-11-20
Windows 7 64-bit PASS - 2012-11-20 PASS - 2012-11-20
Windows 8 32-bit FAIL - 2012-11-20 PASS - 2012-11-20
Windows 8 64-bit FAIL - 2012-11-20 PASS - 2012-11-20
Mac OS X 10.6 PASS - 2012-11-20 PASS - 2012-11-20
Mac OS X 10.7 PASS - 2012-11-20 PASS - 2012-11-20
Mac OS X 10.8 PASS - 2012-11-20 PASS - 2012-11-20
Ubuntu 12.04 32-bit PASS - 2012-11-20 PASS - 2012-11-20
Ubuntu 12.04 64-bit PASS - 2012-11-20 PASS - 2012-11-20
Issues
  • Windows 8 VMs failed with "cannot connect to jsbridge extension, port 24242"
    • Manual spotchecks succeeded

17.0.1

Builds FTP, HTTP
ID 20121128204232
Changeset c23c45132139 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Build#1 Results
Windows XP PASS - 2012-11-29 (functional, restart)
Windows Vista PASS - 2012-11-29 (functional, restart)
Windows 7 32-bit PASS - 2012-11-29 (functional, restart)
Windows 7 64-bit PASS - 2012-11-29 (functional, restart)
Windows 8 32-bit PASS - 2012-11-29 (functional, restart)
Windows 8 64-bit PASS - 2012-11-29 (functional, restart)
Mac OS X 10.6 PASS - 2012-11-29 (functional, restart)
Mac OS X 10.7 PASS - 2012-11-29 (functional, restart)
Mac OS X 10.8 1 failure - 2012-11-29 (functional, restart)
Ubuntu 12.04 32-bit PASS - 2012-11-29 (functional, restart)
Ubuntu 12.04 64-bit 1 failure - 2012-11-29 (functional, restart)
Test Failures (known bugs)
  • [known] restartTests/testAddons_uninstallExtension/test4.js (bug 783484)
  • [known] restartTests/testAddons_pluginDisabledAfterRestart/test2.js (bug 780957)

Manual Testing

Regression Tests

Run the Firefox 17 Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Build 1 Results
Platform Tester Result (report) Bugs
Windows XP Paul PASS - 2012-11-29
Windows 8 Ioana PASS - 2012-11-29
Mac OSX 10.8 Virgil PASS - 2012-11-29
Ubuntu 12.04 Simona PASS - 2012-11-29
Mac OSX 10.5 de-support Paul PASS - 2012-11-29
Update regression tests PASS - 2012-11-29

Bug Fix Verifications

[VERIFIED FIXED] bug 806192 crash in inDOMUtils::GetBindingURLs with Firebug and ShareMeNot

Windows 7 Mac OSX Ubuntu
Firefox does not crash when loading Firebug scripts PASS (Manuela) PASS (Virgil) PASS (Simona)
Test
  • Start Firefox with a new profile
  • Install Firebug from addons.mozilla.org
  • Install Sharemenot from addons.mozilla.org
  • Navigate to http://asu.edu
  • Activate Firebug
  • Click the Scripts panel
  • Click the enable link in Firebug
  • Click the reload link in Firebug
  • Note: broken behaviour can be observed in Firefox 17.0b6
Notes

[VERIFIED FIXED] bug 815042 leaving private browsing with social enabled doesn't reset all social components

Windows 7 Mac OSX Ubuntu
Social API usable in normal and private browsing modes PASS (Manuela) PASS (Virgil) PASS (Simona)
Social API does not automatically restore state when switching modes PASS (Manuela) PASS( Virgil) PASS (Simona)
Notes
  • Social API can't be enabled from the Facebook activation page. Commented and reopened bug 807915. Social API can be enabled only by setting the preferences social.active and social.enabled to true in about:config.
  • When exiting Private Browsing mode, Social API restores to the state it was in before entering PB mode.

[VERIFIED FIXED] bug 815743 Backout the UA changes in bug 588909

Windows 7 Mac OSX Ubuntu
UA string contains Gecko/20100101 in about: PASS (Manuela) PASS (Virgil) PASS (Simona)
general.useragent.complexOverride.moodle defaults to false in about:config PASS (Manuela) PASS (Virgil) PASS (Simona)
opensourcecms.com shoes rich-text textareas PASS (Manuela) PASS (Virgil) PASS (Simona)
moodle.org maintains authenticated session PASS (Manuela) PASS (Virgil) PASS (Simona)
Test
Notes
  • Didn't find the "Full page description" textarea, did find the "Front page description" textarea instead.

[VERIFIED FIXED] bug 814101 Font Issue in FF 17

Windows 7 (ATI) Windows 7 (NVidia) Windows 8 (ATI) Windows 8 (NVidia)
about:support shows Direct2D enabled = true PASS (Mihaela) PASS(Paul) PASS (Mihaela) PASS(Paul)
about:support shows GPU Accelerated Windows = 1/1 Direct3D 10 PASS(Mihaela) PASS(Paul) PASS (Mihaela) PASS(Paul)
ClearType ON displays anti-aliased text PASS (Mihaela) PASS(Paul) PASS (Mihaela) PASS(Paul)
ClearType OFF does not display anti-aliased text PASS (Mihaela) PASS(Paul) PASS (Mihaela) PASS(Paul)
Test
  • Open about:config and make sure to set layers.acceleration.disabled=false, gfx.content.azure.enabled=true
  • Open Windows Control Panel > Appearance and Personalization > Display > Adjust ClearType Text
  • Turn on ClearType and go to a website
  • Use the Magnifier tool in Windows to zoom in on the text (500% worked for me)
  • Check for anti-aliased text (might appear blurry)
  • Go back to Control Panel and turn off ClearType
  • Use the Magnifier tool in Windows to zoom in on the text (500% worked for me)
  • Check for un-anti-aliased text (no blurring)
  • Note: broken behaviour can be observed in Firefox 17.0
Notes
  • Environments:
    • Win 7 32 bit, AMD Radeon HD 6450
    • Win 7 64bit, Nvidia GeForce 210
    • Win 8 64bit, AMD Radeon HD 6450
    • Win 8 32bit, Nvidia GeForce 210

[VERIFIED FIXED] bug 814554 Firefox 17 silently stops processing permissions.sqlite

Windows 7 Mac OSX Ubuntu
Cookies permissions persist across Firefox restart PASS (Manuela) PASS (Virgil) PASS (Simona)
Test
  • Start Firefox 17.0.1 with a new profile
  • Select Options > Privacy > History > Firefox will: use custom settings for history
  • Change cookies pref to Keep until: ask me every time
  • Quit Firefox and replace the permissions.sqlite in your profile folder with this one
  • Start Firefox and verify Options > Privacy > History > Cookies Exceptions shows only "schafmail.de - Allow for Session"
  • Visit http://www.openstreetmap.org/
  • Check the box for "use my choice for all cookies from this site" and click Allow
  • Verify Options > Privacy > History > Cookies Exceptions has added "www.openstreetmap.org - Allow" to the list
  • Restart Firefox and verify that both schafmail.de and www.openstreetmap.org are listed
  • Visit http://www.openstreetmap.org/ and verify you are not prompted to accept the cookie again
  • Note: broken behaviour can be seen in Firefox 17.0'
Notes

Update Testing

  • Partial: 17.0, 16.0.2, 15.0.1
  • Complete: 16.0
Platform betatest (report) release (report)
Windows XP PASS - 2012-11-29 PASS - 2012-11-30
Windows Vista PASS - 2012-11-29 PASS - 2012-11-30
Windows 7 32-bit PASS - 2012-11-29 PASS - 2012-11-30
Windows 7 64-bit PASS - 2012-11-29 PASS - 2012-11-30
Windows 8 32-bit (manual re:bug 803489) PASS - 2012-11-29 PASS - 2012-11-30
Windows 8 64-bit (manual re:bug 803489) PASS - 2012-11-29 PASS - 2012-11-30
Mac OS X 10.6 PASS - 2012-11-29 PASS - 2012-11-30
Mac OS X 10.7 PASS - 2012-11-29 PASS - 2012-11-30
Mac OS X 10.8 PASS - 2012-11-29 PASS - 2012-11-30
Ubuntu 12.04 32-bit PASS - 2012-11-29 PASS - 2012-11-30
Ubuntu 12.04 64-bit PASS - 2012-11-29 PASS - 2012-11-30
Spotchecks
  • [DONE] Verify background updates still work (chunk size and ping interval changed for 17.0.1)
Issues
  • Very intermittent issue with 15.0.1->17.0.1 fallback updates resulting in connection error to update server

Bugs

Triage

  1. VERIFYME: verify the fix and do some exploratory testing around the issue
  2. FIXED: triage these bugs and test to ensure the fix is effective
  3. QAWANTED: triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED: triage these bugs for potential new regressions

New Issues

Anything in bold is a potential blocker.

  • bug 813531 An unknown error while printing SVG file
  • bug 813553 [Mac OS] Firefox freezes after installing add-ons which launch a pop-up after installation
  • bug 813563 Cannot restore previous session (with Yandex Community Toolbar3.15.1.0 and Hotspot Shield Community Toolbar3.15.1.0)
  • bug 813566 [Mac OS] Mouse cursor disappears when switching tabs
  • bug 813568 Skype Click to Call doesn't work on Firefox 17, although it is enabled
  • bug 813572 New tab is not added to the tabs strip (with 4shared.com Community Toolbar, IncrediMail MediaBar 2 Community Toolbar)
  • bug 813594 Gaping tabs strip after installing Facebook Toolbar

Known Issues

Anything in bold is a potential blocker.

Click-to-Play

  • bug 809785 Plugin content can't be activated on sohu.com because it is covered by an <a> element
  • bug 809793 The Click-to-Play message is cropped on taobao.com
  • bug 809858 Click to play overlay not working on paypal.com

Graphics

  • bug 686782 GFX glitches on Windows 8 with D2D
  • bug 772837 Content of drop-down lists (awesomebar history) blink on secondary monitor with HWA

HTML5 Video

  • bug 760417 Full-screen button visible in pop-out full-screen mode
  • bug 786972 Can't replay video by clicking on the progress bar after playback ends
  • bug 803057 HTML5 videos continues to play after Pop Out
  • bug 708557 Html5 youtube context menu not persistent when right clicking

Layout

  • FIXED bug 785754 Buttons on some sites are not rendered correctly
  • FIXED bug 803074 Rendering is broken after restoring skew() support

Plugins

  • bug 667399 RealPlayer G2 LiveConnect plugin crash
  • WFM bug 786655 Flash context menu does not own focus
  • bug 786970 Windowed plugin content gets displayed out of it's container when scrolling the browser window
  • FIXED? bug 788423 [adbe 3325067] Flash hang in F_1152915508 on dropbox.com (test Nov 7, 2012)
  • bug 797116 Win8 Flash hang in F_1152915508 while playing Knight Hunters
  • WFM bug 798295 Copy actions are sometimes unresponsive in Flash
  • bug 800928 Incorrect display of Vimeo context menu in a tab dragged to a new window
  • bug 800940 Incorrect positioning of IMDB video player in a tab dragged to a new window
  • bug 797830 [flash 11.5] Right clicking the Flash content pauses the video when using the 11.5 beta version

Printing

  • bug 638796 Non-functional print button in Firefox Menu when viewing a PDF file

Social API

  • bug 789707 Make sure we're handling offline mode
  • FIXED bug 799856 crash in mozilla::InternalInvalidateThebesLayersInSubtree, canceled login to Facebook
  • bug 804622 Sidebar asks me to log in even though I'm logged in
  • bug 807673 Social chat panels are not displayed right when resizing Firefox vertically
  • bug 807675 Friends in the Social sidebar are sometimes marked as offline without actually going offline
  • bug 807915 Can't activate Social API from the Facebook activation page
  • bug 807935 The chat window persists in DOM full-screen mode
  • FIXED? bug 807981 Facebook new friend request notification link is not opened in new tab (test Nov 14)
  • bug 807997 Nothing happens when attempting to open a contact in Facebook sidebar with about: page open
  • bug 808001 Clicking on notification pop-ups opens them in the focused tab, not in a new tab
  • bug 811727 Can't turn on facebook messenger from facebook notification

Switch-to-Tab

  • bug 811668 "Switch to tab" switches to other tab group even if the same page is opened in current group

WebGL

  • bug 787056 Google MapsGL blurry on machines where Chrome MapsGL isn't supported

Toolbars

  • bug 518089 A hidden state of the toolbar which user added does not persist after reboot, When a multi-byte letter is included in a name of the toolbar
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_17/Test_Plan&oldid=491289"