Releases/Firefox 15/Test Plan

From MozillaWiki
Jump to navigation Jump to search

« Firefox 14 | Firefox 15 | Firefox 16 »

Firefox 15 Test Plan

Summary

Milestone Firefox 15
Status Currently in Beta -- shipping on August 28, 2012
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)

The following is the test plan for Firefox 15 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 15. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-04-24: [DONE] Nightly Merge
  • 2012-06-05: [DONE] Aurora Merge
  • 2012-06-22: [DONE] Aurora Testday
  • 2012-07-17: [DONE] Pre-Beta feature sign-offs
  • 2012-07-19: [ON TRACK] Beta 1
  • 2012-07-26: {{|}} Beta 2
  • 2012-08-02: {{|}} Beta 3
  • 2012-08-09: {{|}} Beta 4
  • 2012-08-16: {{|}} Beta 5
  • 2012-08-23: {{|}} Beta 6
  • 2012-08-27: {{|}} Release Candidate
  • 2012-08-28: {{|}} Final

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Show PDF Inline Mihaela Velimiroviciu ON TRACK MozTrap
Background Updates Vlad Ghetiu ON TRACK MozTrap
Debugger Ioana Budnar ON TRACK MozTrap
Responsive View Virgil Dicu ON TRACK MozTrap
Detailed Element Layout View Paul Silaghi ON TRACK MozTrap
Opt-in Activation for Plugins R2 Paul Silaghi ON TRACK MozTrap
Accessibility Improvements for Mac Vlad Ghetiu ON TRACK
Windows Start-up Performance Improvements Jason Smith ON TRACK
In-content preferences Mihaela Velimiroviciu ON TRACK MozTrap
Incremental Garbage Collection Ioana Budnar DELAYED to Fx?? MozTrap
Web Apps Integration to Desktop Jason Smith DELAYED to Fx16
Speedy Session Restore Paul Silaghi DELAYED to Fx16
Panel-based Download Manager Simona Badau DELAYED to Fx16 MozTrap
Developer Toolbar Virgil Dicu DELAYED to Fx16 MozTrap
Firefox Social Integration Anthony Hughes DELAYED to Fx16

Nightly

Merge

Functional Automation - result
Update Spotchecks - result


Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
  • No issues found

Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet


Aurora

Aurora Merge Sign-off

Automated Tests

Results of the Mozmill automation testruns.

Functional Tests
Platform Result: report Failures: known bugs
Windows XP PASS - 2012-06-06 no fails
Windows Vista PASS - 2012-06-06 no fails
Windows 7 32-bit PASS - 2012-06-06 no fails
Windows 7 64-bit PASS - 2012-06-06 1 fail
Mac OS X 10.6 PASS - 2012-06-06 2 fails
Mac OS X 10.7 PASS - 2012-06-06 1 fail
Ubuntu 11.10 32-bit PASS - 2012-06-06 no fails
Ubuntu 11.10 -bit PASS - 2012-06-06 no fails
Bugs
bug 761569 - testPrivateBrowsing/testCloseWindow.js
bug 761984 - testSearch/testFocusAndSearch.js
Endurance Tests
Platform Result: report Max Memory
Windows XP PASS - 2012-06-06 151 MB (+n/a from 14.0a2)
Windows Vista PASS - 2012-06-06 134 MB (+61% from 14.0a2)
Windows 7 32-bit PASS - 2012-06-06 139 MB (+46% from 14.0a2)
Windows 7 64-bit PASS - 2012-06-06 116 MB (+47% from 14.0a2)
Mac OS X 10.6 PASS - 2012-06-06 144 MB (+41% from 14.0a2)
Mac OS X 10.7 PASS - 2012-06-06 122 MB (+21% from 14.0a2)
Ubuntu 11.10 32-bit PASS - 2012-06-06 105 MB (+46% from 14.0a2)
Ubuntu 11.10 -bit PASS - 2012-06-06 129 MB (+30% from 14.0a2)
Bugs
bug 762215 - [cpg] Very large memory regression between Firefox 14 and 15

Feature Sign-offs

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 15
  • Select Runs: Firefox 15 Aurora Merge Feature sign off
Feature Owner Win32 Mac Linux
Show PDF inline Mihaela Velimiroviciu PASS PASS PASS
Firefox Social Integration Simona Badau n/a n/a n/a
Incremental garbage collection Ioana Budnar FAIL PASS PASS
Silent Update: Background updates Vlad Ghetiu PASS PASS PASS
Windows Start-up Performance Improvements Jason Smith PASS n/a n/a
Opt-in activation for plugins (Phase 2) Paul Silaghi PASS PASS PASS
In-content preferences Mihaela Velimiroviciu PASS PASS PASS
Accessibility Improvements for Mac Vlad Ghetiu n/a n/a n/a
Developer Tool: Debugger Ioana Budnar PASS PASS PASS
Developer Tool: Responsive View Virgil Dicu PASS PASS PASS
Developer Tool: Detailed Element Layout View Paul Silaghi PASS PASS PASS
Web Apps Integration to Desktop Jason Smith PASS PASS PASS
Details

Report any detailed results or bugs here...

  • Firefox Social Integration Feature - The feature is still in definition, nothing is landed on Firefox 15.

Smoketests

Run the smoketests in MozTrap and report results below.

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 15
  • Select Runs: Firefox 15 Aurora Merge Smoketests

Please email me feedback on using MozTrap and the test cases themselves

Platform Assignee Result: report
Windows 7 64-bit Simona PASS - 2012-06-08
Mac OS X 10.7.4 VladG PASS - 2012-06-08
Ubuntu 11.10 64-bit Mihaela PASS - 2012-06-08

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG

Do some smoketesting and exploratory testing to make sure the build works as expected.

  • [DONE] - VladG - ran the smoketests from MozTrap
  • [DONE] - VladG - exploratory on Youtube, Facebook, flash, silverlight, banking sites, etc

Exploratory Testing

Suggested areas to focus testing:

  • Plug-ins
    • PASS - Ioana, Simona
  • Audio/Video
    • FAIL - Ioana, Simona
    • 762903 - In Full screen "Hide Controls" is not applied on OGG videos
  • Social Networking
    • PASS - Ioana
  • Banking
  • Session Restore
    • PASS - Simona

Update Tests

auroratest
  • Win32: 14.0a2 en-US 2012-06-01 -> 15.0a2 en-US 2012-06-08 (PASS)
  • MacOS: 14.0a2 en-US 2012-06-02 -> 15.0a2 en-US 2012-06-08 (PASS)
  • Linux: 14.0a2 en-US 2012-05-31 -> 15.0a2 en-US 2012-06-08 (PASS)


Bugs Found

Full Query
ID Summary Priority Status
762868 Large text files are not printed entirely -- RESOLVED
762903 In Full screen "Hide Controls" is not applied on OGG videos -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Unprefixing Border-Image

Overview

In Firefox 15, the border-image is being unprefixed and will experience syntax changes as a result. The main syntax change is that border-image's border-style default is now none, not solid. However, as already noted from early testing, this results in regressions in top apps or sites. We need carefully identify what top sites are affected by this syntax change, determine the severity of the problem, and evangelize fixes to the site or app developer.

Top Sites Study

John Jensen did a study in the past that analyzed use of vendor prefixes across the web. We need to first take a look here to identify the list of sites that did end up using moz-border-image and see if the most important sites on the list regress by the new syntax changes. Testing of this then needs to exploratory test the following sites in this document here, specifically analyzing the layout of the site in regards to how border-image is used. For more information about this property, see this MDN document.

Important Bugs

Full Query
ID Summary Priority Status
730530 alliances.commandandconquer.com - Buttons not visible in Tiberium Alliances -- RESOLVED
748253 Lord of Ultima -moz-border-image does not render on buttons for FF 13 Aurora & FF 14 Nightly - Works on FF 12 Beta 6 -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Bug Triage

Query 06.18 - 06.22 06.25 - 06.29 07.02 - 07.06 07.09 - 07.13 Total
UNCONFIRMED 18 triaged 9 triaged 0 triaged 0 triaged 27 triaged
QAWANTED n/a n/a 1 resolved 6 resolved 7 resolved
FIXED 3 verified 0 verified 0 verified 0 verified 3 verified

Beta 1

Builds FTP, HTTP
ID 20120717110313
Changeset dc005f41ebdd
Checklist

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Platform Result: report
Windows XP PASS - 2012-07-17
Windows Vista PASS - 2012-07-17
Windows 7 32-bit PASS - 2012-07-17
Windows 7 64-bit PASS - 2012-07-17
Mac OS X 10.6 PASS - 2012-07-17
Mac OS X 10.7 PASS - 2012-07-17
Ubuntu 11.10 32-bit PASS - 2012-07-17
Ubuntu 11.10 64-bit PASS - 2012-07-17
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 1 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-07-18
Windows 7 64-bit Paul PASS - 2012-07-18
Mac OS X 10.7 Mihaela PASS - 2012-07-18
Ubuntu 12.04 64-bit Ioana PASS - 2012-07-18

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-18
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-18
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-18
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-18
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-18
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-18
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-18

Spot check build after this test - PASS - VladG - 2012-07-08

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
    • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Mihaela, Simona PASS - 2012-07-19 PASS - 2012-07-19 PASS - 2012-07-19
Windows 7 64-bit Paul PASS - 2012-07-18 PASS - 2012-07-18 PASS - 2012-07-18
Mac OS X 10.7 VladG PASS - 2012-07-18 PASS - 2012-07-18 PASS - 2012-07-18
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-18 PASS - 2012-07-18 PASS - 2012-07-18

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-18 PASS - 2012-07-19
Windows Vista PASS - 2012-07-18 PASS - 2012-07-19
Windows 7 32-bit PASS - 2012-07-18 PASS - 2012-07-19
Windows 7 64-bit PASS - 2012-07-18 PASS - 2012-07-19
Mac OS X 10.6 PASS - 2012-07-18 PASS - 2012-07-19
Mac OS 10.7 PASS - 2012-07-18 PASS - 2012-07-19
Ubuntu 11.10 32-bit PASS - 2012-07-18 PASS - 2012-07-19
Ubuntu 11.10 64-bit PASS - 2012-07-18 PASS - 2012-07-19
Issues

New Issues

Bug Status
bug 775082 Video controls disappear after dragging the tabs Not a Fx15 regression, does not block release
bug 775114 Firefox does not hand off to Ubuntu archive utility for tar files Not a Fx15 regression, does not block release
bug 775488 Radio live streaming doesn't work with NoScript installed Not a Fx15 regression, does not block release

Beta 2

Builds FTP, HTTP
ID 20120724191344
Changeset 4d2c274ea4dc
Checklist

Automated Tests

The following are the detailed PASSs of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-07-26
Windows Vista PASS - 2012-07-26
Windows 7 32-bit PASS - 2012-07-26
Windows 7 64-bit PASS - 2012-07-26
Mac OS X 10.6 PASS - 2012-07-26
Mac OS X 10.7 PASS - 2012-07-26
Ubuntu 11.10 32-bit PASS - 2012-07-26
Ubuntu 11.10 64-bit PASS - 2012-07-26
Test Failures (known bugs)

Manual Smoketests

Run the Firefox 15 Beta 2 Smoketests in MozTrap and report the results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-07-25
Windows 7 32-bit Mihaela PASS - 2012-07-25
Mac OS X 10.7 VladG PASS - 2012-07-25
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-25

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-25
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-25
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-25
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-25
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-25
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-25
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-25

Spot check build after this test - PASS - VladG - 2012-07-25

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Regression Testing

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo
Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-07-26 PASS - 2012-07-26 PASS - 2012-07-26
Windows 7 32-bit Mihaela PASS - 2012-07-25 PASS - 2012-07-26 PASS - 2012-07-26
Mac OS X 10.7 VladG PASS - 2012-07-25 PASS - 2012-07-25 PASS - 2012-07-25
Ubuntu 12.04 64-bit Ioana PASS - 2012-07-25 PASS - 2012-07-25 PASS - 2012-07-26

Update Testing

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-26 RESULT - 2012-07-26
Windows Vista PASS - 2012-07-26 RESULT - 2012-07-26
Windows 7 32-bit PASS - 2012-07-26 RESULT - 2012-07-26
Windows 7 64-bit PASS - 2012-07-26 RESULT - 2012-07-26
Mac OS X 10.6 PASS - 2012-07-26 RESULT - 2012-07-26
Mac OS 10.7 PASS - 2012-07-26 RESULT - 2012-07-26
Ubuntu 11.10 32-bit PASS - 2012-07-26 RESULT - 2012-07-26
Ubuntu 11.10 64-bit PASS - 2012-07-26 RESULT - 2012-07-26
Issues

New Issues in Beta 2

Bug Status

Beta Feature Sign-off

Run the Firefox 15.0 pre-beta sign off tests in MozTrap and report results below.

Feature Owner Win32 Mac Linux
Show PDF Inline Mihaela Velimiroviciu disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
Silent Update: Background updates Vlad Ghetiu PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Debugger Ioana Budnar PASS - 2012-07-13 PASS - 2012-07-17 PASS - 2012-07-13
Responsive Mode Virgil Dicu PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Detailed Element Layout Paul Silaghi PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Opt-in activation for plugins Phase 2 Paul Silaghi PASS - 2012-07-13 PASS - 2012-07-13 PASS - 2012-07-13
Accessibility Improvements for Mac Vlad Ghetiu N/A - 2012-07-13 PASS - 2012-07-13 N/A - 2012-07-13
Start-up Performance Improvements Jason Smith N/A - 2012-07-13 PASS - 2012-07-17 N/A - 2012-07-13
Panel-based Download Manager Simona Badau disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
Incremental Garbage Collection Ioana Budnar disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
Firefox Developer Toolbar Virgil Dicu disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13
In-content Preferences Mihaela Velimiroviciu disabled - 2012-07-13 disabled - 2012-07-13 disabled - 2012-07-13

Notes:

Beta Bug Triage

Query 07.16 - 07.20 07.23 - 07.27 07.30 - 08.03 08.06 - 08.10 08.13 - 08.17 08.20 - 08.24 Total
UNCONFIRMED 9 triaged 7 triaged 16 / 19
QAWANTED 3 triaged 6 triaged 9 / 10
FIXED 11 verified 4 verified 15 / 112

Verification Spreadsheet

Beta Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Known Issues

Full Query
ID Summary Priority Status
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
675260 crash in DefaultFreeEntry coming from ShutdownNSS P3 RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
726206 Firefox crash @ _dwrite_draw_glyphs_to_gdi_surface_gdi while printing -- RESOLVED
757978 Uninstalling Firefox leaves behind a Nightly\webapprt folder -- RESOLVED
760417 When selecting "Pop out" for HTML5 videos, the window is opened in full screen mode and the "Full screen" button is still available on its right bottom side -- RESOLVED
762868 Large text files are not printed entirely -- RESOLVED
770207 On Windows the "Feedback" button is not highlighted when hovering the mouse over it -- RESOLVED
775082 Video controls disappear after dragging the tabs -- RESOLVED

9 Total; 0 Open (0%); 9 Resolved (100%); 0 Verified (0%);