Releases/Firefox 17/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 389: Line 389:


=== Manual Testing ===
=== Manual Testing ===
==== Plug-in Compatibility ====
Run the [https://moztrap.mozilla.org/manage/cases/?filter-suite=114 Desktop Firefox Regression Tests] in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.
Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.
* Adobe Flash Player
**[http://get.adobe.com/flashplayer/ 11.3 Release]
**[http://labs.adobe.com/downloads/flashplayer11-4.html Flash 11.4 beta] (make sure you can autoupdate)
* [http://www.microsoft.com/silverlight/ Microsoft Silverlight]
* [http://www.real.com/ RealPlayer]
** Real Player plugin crashed when opening pages with content managed by the plugin: [http://www.slp3d2.com/launchsiteglobal/test.cfm click], [http://cf.unc.edu/skillstest/take/realplayer.htm click] - {{bug|667399}}
* [http://www.apple.com/quicktime/download/ QuickTime]


==== Add-on Compatibility ====
{|class="querytable"
Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.
|class="header" | Platform
* [https://addons.mozilla.org/en-US/firefox/compatibility/15.0 Top AMO add-ons]
|class="header" | Tester
* AVG toolbar (installed via AVG software)
|class="header" | Result
** AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
|-
** AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
| Windows XP
* Norton toolbar (installed via Norton software)
|
** Not currently available. Investigating if we can get this software.
| '''result''' - YYYY-MM-DD
* [http://sg.toolbar.yahoo.com/ Yahoo toolbar]
|-
* [http://bar.yandex.ru/en/ Yandex toolbar]
| Windows Vista
* [http://babylon-toolbar.software.informer.com/download/ Babylon toolbar]
|
* [http://lavasoft.com/ Ad-Aware Antivirus]
| '''result''' - YYYY-MM-DD
** The toolbar can't be installed on Firefox.
|-
 
| Windows 7
==== Web Compatibility ====
|
Test the following websites for rendering issues, functional issues, performance issues, and crashes
| '''result''' - YYYY-MM-DD
# [http://www.alexa.com/topsites Overall Top 10]
|-
# [http://www.alexa.com/topsites/countries Top by Country] (pick a random site from three random countries)
| Windows 8
# [http://www.alexa.com/topsites/category Top by Category] (pick a random site from three random categories)
|
# Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
| '''result''' - YYYY-MM-DD
# Try Google's [http://support.google.com/maps/bin/answer.py?hl=en&answer=1630790 MapsGL demo]
|-
# Try YouTube's [http://www.youtube.com/html5 HTML5 demo]
| Mac OSX 10.5
 
|
==== Web Mail ====
| '''result''' - YYYY-MM-DD
Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.
|-
# [https://mail.google.com/ Google Mail]
| Mac OSX 10.6
# [https://mail.yahoo.com/ Yahoo Mail]
|
# [https://www.hotmail.com Hotmail]
| '''result''' - YYYY-MM-DD
# [https://www.outlook.com/ Outlook.com]
|-
 
| Mac OSX 10.7
==== Mac 10.8 Signing ====
|
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
| '''result''' - YYYY-MM-DD
* Download the .dmg file and drag the Firefox.app file to a location on your computer
|-
* Open a terminal window and cd to the folder Firefox.app is located
| Mac OSX 10.8
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
|
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
| '''result''' - YYYY-MM-DD
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
|-
* Click "open" and browse to any website, Firefox should start and you should be able to browse
| Ubuntu 12.04
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
|
 
| '''result''' - YYYY-MM-DD
Spot check build after this test
|}
* Try all the menu item functions
* Gestures
* Plug-in compatibility
* Top-site compatibility
* Add-on compatibility
* Software update
 
==== Mac OSX 32-on-64-bit ====
 
==== Stub Installer ====
 
==== Breakpad End-to-End ====
Test the end-to-end functionality in the crash reporting feature, across operating systems.
* Use the [http://ted.mielczarek.org/mozilla/crashme.html Crash Me add-on] to trigger a crash
* Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
* Repeatedly crash Firefox and submit your reports
* Verify that your session restores properly
* Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
* Try adding comments to your crash report and verify the crash-stats.com report shows the comment


=== Update Testing ===
=== Update Testing ===

Revision as of 18:18, 3 October 2012

« Firefox 16 | Firefox 17 | Firefox 18 »

Firefox 17 Test Plan

Summary

Milestone Firefox 17
Status Currently in Aurora -- merging to Beta on October 9, 2012
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 17 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 17. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-07-16: {{|}} Nightly Merge
  • 2012-08-27: {{|}} Aurora Merge
  • 2012-10-08: {{|}} Pre-Beta feature sign-offs
  • 2012-10-11: {{|}} Beta 1
  • 2012-10-18: {{|}} Beta 2
  • 2012-10-25: {{|}} Beta 3
  • 2012-11-01: {{|}} Beta 4
  • 2012-11-08: {{|}} Beta 5
  • 2012-11-15: {{|}} Beta 6
  • 2012-11-19: {{|}} Release Candidate
  • 2012-11-20: {{|}} Final

Nightly

Merge

Functional Automation - result
Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Improve display of location bar results Virgil Dicu On track
Firefox Social Integration Anthony Hughes On track
HTML Tree Editor Mihaela Velimiroviciu
Silent Update not now prompt Ioana Budnar Release target removed.

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated

Bugzilla query error

Query options must be valid JSON.1


Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated


Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet

Aurora Merge

NOTE - OSX 10.5 should not be able to install nor update to Firefox 17
Builds nightly
Changeset tip (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-08-28
Windows Vista PASS - 2012-08-28
Windows 7 32-bit PASS - 2012-08-28
Windows 7 64-bit PASS - 2012-08-28
Windows 8 32-bit N/A (pending IT)
Windows 8 64-bit N/A (pending IT)
Mac OS X 10.6 PASS - 2012-08-28
Mac OS X 10.7 PASS - 2012-08-28
Mac OS X 10.8 PASS - 2012-08-28
Ubuntu 12.04 32-bit PASS - 2012-08-28
Ubuntu 12.04 64-bit PASS - 2012-08-28
Test Failures (known bugs)

Manual Testing

Operating System Tester Plug-in Compatibility Web Compatibility OSX 10.5 EOL
Windows XP Simona PASS - 2012-08-29 PASS - 2012-08-29 N/A
Windows 7 Ioana PASS - 2012-08-29 PASS - 2012-08-29 N/A
Windows 8 Paul PASS - 2012-08-30 PASS - 2012-08-30 N/A
Mac OS X VladG PASS - 2012-08-30 PASS - 2012-08-30 PASS - 2012-08-30
Ubuntu 12.04 Ioana PASS - 2012-08-30 PASS - 2012-08-30 N/A

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

OSX 10.5 EOL

Verify the following:

  1. Mac OSX 10.5 cannot download Firefox 17 (I'm not sure if this is scoped but try it anyway)
  2. Mac OSX 10.5 cannot start Firefox 17
  3. Mac OSX 10.5 cannot update to Firefox 17
  • The build can be downloaded and installed
  • Aurora didn't start, showing a incompatibility message
  • The update channel seems to be close so I didn't manage to perform an update

Feature Sign-offs

  • Log in to moztrap.mozilla.org
  • Select Run Tests
  • Select Product: Desktop Firefox
  • Select Versions: 17
  • Select Runs: Firefox 17 Aurora Merge Feature sign off
Feature Owner Win32 Mac Linux
Improve display of location bar results Virgil Dicu PASS PASS PASS
Firefox Social Integration Anthony Hughes PASS PASS PASS
Details

Report any detailed results or bugs here...

Update Testing

Spotcheck updates from a previous Aurora to Aurora 17.

Platform Auroratest Aurora
Windows 7 64-bit PASS - 2012-08-30 PASS - 2012-08-30
Windows 8 64-bit PASS - 2012-08-30 PASS - 2012-08-30
Mac OS X 10.7 PASS - 2012-08-30 PASS - 2012-08-30
Ubuntu 12.04 64-bit PASS - 2012-08-30 PASS - 2012-08-30
Issues

Bug Triage

  1. VERIFYME (5): verify the fix and do some exploratory testing around the issue
  2. FIXED (139): triage these bugs and test to ensure the fix is effective
  3. QAWANTED (6): triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (8): triage these bugs for potential new regressions
Results

Beta 1

Builds FTP, HTTP
ID
Changeset ID (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results ([ report])
Windows XP result - YYYY-MM-DD
Windows Vista result - YYYY-MM-DD
Windows 7 32-bit result - YYYY-MM-DD
Windows 7 64-bit result - YYYY-MM-DD
Windows 8 32-bit result - YYYY-MM-DD
Windows 8 64-bit result - YYYY-MM-DD
Mac OS X 10.6 result - YYYY-MM-DD
Mac OS X 10.7 result - YYYY-MM-DD
Mac OS X 10.8 result - YYYY-MM-DD
Ubuntu 12.04 32-bit result - YYYY-MM-DD
Ubuntu 12.04 64-bit result - YYYY-MM-DD
Test Failures (known bugs)

Manual Testing

Run the Desktop Firefox Regression Tests in MozTrap and report the results below. Do some exploratory testing around the testcases to try to exercise some edge-cases.

Platform Tester Result
Windows XP result - YYYY-MM-DD
Windows Vista result - YYYY-MM-DD
Windows 7 result - YYYY-MM-DD
Windows 8 result - YYYY-MM-DD
Mac OSX 10.5 result - YYYY-MM-DD
Mac OSX 10.6 result - YYYY-MM-DD
Mac OSX 10.7 result - YYYY-MM-DD
Mac OSX 10.8 result - YYYY-MM-DD
Ubuntu 12.04 result - YYYY-MM-DD

Update Testing

  • Partial:
  • Complete:
Platform Releasetest ([ report]) Beta ([ report])
Windows XP result - YYYY-MM-DD result - YYYY-MM-DD
Windows Vista result - YYYY-MM-DD result - YYYY-MM-DD
Windows 7 32-bit result - YYYY-MM-DD result - YYYY-MM-DD
Windows 7 64-bit result - YYYY-MM-DD result - YYYY-MM-DD
Windows 8 32-bit result - YYYY-MM-DD result - YYYY-MM-DD
Windows 8 64-bit result - YYYY-MM-DD result - YYYY-MM-DD
Mac OS X 10.6 result - YYYY-MM-DD result - YYYY-MM-DD
Mac OS X 10.7 result - YYYY-MM-DD result - YYYY-MM-DD
Mac OS X 10.8 result - YYYY-MM-DD result - YYYY-MM-DD
Ubuntu 12.04 32-bit result - YYYY-MM-DD result - YYYY-MM-DD
Ubuntu 12.04 64-bit result - YYYY-MM-DD result - YYYY-MM-DD
Issues

Bug Triage

  1. VERIFYME (3): verify the fix and do some exploratory testing around the issue
  2. FIXED (221): triage these bugs and test to ensure the fix is effective
  3. QAWANTED (3): triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (3): triage these bugs for potential new regressions
Results

Feature Sign-off

Feature owners to test and sign-off overall quality of their features following uplift to Beta.

Feature Owner Win32 Mac Linux
Improve display of location bar results Virgil Dicu RESULTS - yyyy-mm-dd RESULTS - yyyy-mm-dd RESULTS - yyyy-mm-dd
HTML tree editor Mihaela Velimiroviciu RESULTS - yyyy-mm-dd RESULTS - yyyy-mm-dd RESULTS - yyyy-mm-dd
Firefox Social Integration Anthony Hughes RESULTS - yyyy-mm-dd RESULTS - yyyy-mm-dd RESULTS - yyyy-mm-dd

New Issues

Full Query
ID Summary Priority Status
786970 Windowed plugin content gets displayed out of it's container when scrolling the browser window -- RESOLVED
786972 Can't replay the same HTML5 youtube video by clicking on the progress bar after when it ends -- RESOLVED
787056 MapsGL maps.google.com renders blurry on machines Chrome doesn't allow MapsGL enabling on P3 REOPENED

3 Total; 1 Open (33.33%); 2 Resolved (66.67%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
468855 Bringing up a context-menu on a Flash video player pauses the video -- RESOLVED
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
686782 GFX glitches on windows 8 (with D2D) -- RESOLVED
760417 When selecting "Pop out" for HTML5 videos, the window is opened in full screen mode and the "Full screen" button is still available on its right bottom side -- RESOLVED
778819 Toggle Fullscreen in context menu stops workng Flash 11.3.300.268 -- RESOLVED
785754 Buttons on some sites are not rendered right and are like half cut off -- VERIFIED

6 Total; 0 Open (0%); 5 Resolved (83.33%); 1 Verified (16.67%);

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_17/Test_Plan&oldid=476183"