MozillaWiki

Releases/Firefox 17/Test Plan: Difference between revisions

Page Discussion

Releases/Firefox 17/Test Plan (view source)

Revision as of 23:43, 30 November 2012

914 bytes added ,  30 November 2012
→‎Update Testing
 
(33 intermediate revisions by 7 users not shown)
Line 9: Line 9:
  |-
  |-
  | '''Status'''
  | '''Status'''
  | Currently in '''Beta''' -- merging to Released on November 20, 2012
  | '''Released''' on Tuesday, November 20, 2012
  |-
  |-
  | '''Lead'''
  | '''Lead'''
Line 1,851: Line 1,851:
  |-
  |-
  | '''ID'''
  | '''ID'''
  | id
  | 20121128204232
  |-
  |-
  | '''Changeset'''
  | '''Changeset'''
  | [http://hg.mozilla.org/releases/mozilla-release/rev/id id] ([http://hg.mozilla.org/releases/mozilla-release/pushloghtml?fromchange=919435c6f654&tochange=id pushlog])
  | [http://hg.mozilla.org/releases/mozilla-release/rev/c23c45132139 c23c45132139] ([http://hg.mozilla.org/releases/mozilla-release/pushloghtml?fromchange=919435c6f654&tochange=c23c45132139 pushlog])
  |}
  |}


; Checklist (respin for {{bug|813264}})
; Checklist
* {{|}} [[Releases/Firefox_17/Test_Plan#Automated_Testing_9|Automated Testing]]
* {{done|}} [[Releases/Firefox_17/Test_Plan#Automated_Testing_9|Automated Testing]]
* {{|}} [[Releases/Firefox_17/Test_Plan#Manual_Testing_9|Manual Testing]]
* {{done|}} [[Releases/Firefox_17/Test_Plan#Manual_Testing_9|Manual Testing]]
* {{|}} [[Releases/Firefox_17/Test_Plan#Update_Testing_9|Update Testing]] on betatest
* {{done|}} [[Releases/Firefox_17/Test_Plan#Manual_Testing_9|Bug Fix Verifications]]
* {{|}} Functional sign-off
* {{done|}} [[Releases/Firefox_17/Test_Plan#Update_Testing_9|Update Testing]] on betatest
* {{done|}} Sign-off email sent to drivers
* {{|}} [[Releases/Firefox_17/Test_Plan#Update_Testing_9|Update Testing]] on release
* {{|}} [[Releases/Firefox_17/Test_Plan#Update_Testing_9|Update Testing]] on release


Line 1,869: Line 1,870:
  |-
  |-
  | class="header" style="width:20%" | Platform
  | class="header" style="width:20%" | Platform
  | class="header" style="width:30%" | Build#1 Results ([ report])
  | class="header" style="width:30%" | Build#1 Results
  |-
  |-
  | Windows XP
  | Windows XP
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716718499d functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716718b5ca restart])
  |-
  |-
  | Windows Vista
  | Windows Vista
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717e614 functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d37167187a06 restart])
  |-
  |-
  | Windows 7 32-bit
  | Windows 7 32-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717cb86 functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d371671815ed restart])
  |-
  |-
  | Windows 7 64-bit
  | Windows 7 64-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717cac0 functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717f43d restart])
  |-
  |-
  | Windows 8 32-bit
  | Windows 8 32-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717deed functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d37167181013 restart])
  |-
  |-
  | Windows 8 64-bit
  | Windows 8 64-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717da97 functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d371671800a9 restart])
  |-
  |-
  | Mac OS X 10.6
  | Mac OS X 10.6
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716716a9ef functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d371671725a9 restart])
  |-
  |-
  | Mac OS X 10.7
  | Mac OS X 10.7
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716716edf4 functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716717372a restart])
  |-
  |-
  | Mac OS X 10.8
  | Mac OS X 10.8
  | '''result''' - YYYY-MM-DD
  | '''1 failure''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d371671688af functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716716fa4b restart])
  |-
  |-
  | Ubuntu 12.04 32-bit
  | Ubuntu 12.04 32-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d37167168403 functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d3716716dea6 restart])
  |-
  |-
  | Ubuntu 12.04 64-bit
  | Ubuntu 12.04 64-bit
  | '''result''' - YYYY-MM-DD
  | '''1 failure''' - 2012-11-29 ([http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d371671667fd functional], [http://mozmill-ci.blargon7.com/#/functional/report/352218d7e3125c857fc1d37167169a92 restart])
  |}
  |}


; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
* '''[known]''' restartTests/testAddons_uninstallExtension/test4.js ({{bug|783484}})
* '''[known]''' restartTests/testAddons_pluginDisabledAfterRestart/test2.js ({{bug|780957}})


=== Manual Testing ===
=== Manual Testing ===
Line 1,920: Line 1,923:
  | Windows XP
  | Windows XP
  | Paul
  | Paul
  | '''In progress''' - 2012-11-29
  | '''PASS''' - 2012-11-29
  |  
  |  
  |-
  |-
  | Windows 8
  | Windows 8
  | Ioana
  | Ioana
  | '''In progress''' - 2012-11-29
  | '''PASS''' - 2012-11-29
  |  
  |  
  |-
  |-
  | Mac OSX 10.8
  | Mac OSX 10.8
  | Virgil
  | Virgil
  | '''In progress''' - 2012-11-29
  | '''PASS''' - 2012-11-29
  |  
  |  
  |-
  |-
  | Ubuntu 12.04
  | Ubuntu 12.04
  | Simona
  | Simona
  | '''In progress''' - 2012-11-29
  | '''PASS''' - 2012-11-29
  |  
  |  
  |-
  |-
  | Mac OSX 10.5 de-support
  | Mac OSX 10.5 de-support
  | Paul
  | Paul
  | '''In progress''' - 2012-11-29
  | '''PASS''' - 2012-11-29
  |  
  |  
  |-
  |-
  | Update regression tests
  | Update regression tests
  | tester
  |  
  | '''results''' - 2012-11-29
  | '''PASS''' - 2012-11-29
  |  
  |  
  |}
  |}
Additional Spotchecks
* ['''result'''] Mac OSX 10.5 Desupport (tester)


=== Bug Fix Verifications ===
=== Bug Fix Verifications ===
; NOTICE all of these verifications need to be performed against 17.0.1 and [https://wiki.mozilla.org/Releases/Firefox_17_ESR/Test_Plan#Bug_Fix_Verifications_2 17.0.1esr]
==== [VERIFIED FIXED] {{bug|806192}} crash in inDOMUtils::GetBindingURLs with Firebug and ShareMeNot ====
 
==== {{bug|806192}} crash in inDOMUtils::GetBindingURLs with Firebug and ShareMeNot ====
{| class="querytable"
{| class="querytable"
  | class="header" style="width:55% |  
  | class="header" style="width:55% |  
Line 1,982: Line 1,980:
  |}
  |}


==== {{bug|815042}} leaving private browsing with social enabled doesn't reset all social components ====
==== [VERIFIED FIXED] {{bug|815042}} leaving private browsing with social enabled doesn't reset all social components ====
{| class="querytable"
{| class="querytable"
  | class="header" style="width:55% |  
  | class="header" style="width:55% |  
Line 2,001: Line 1,999:
  | colspan="4" | '''Notes'''
  | colspan="4" | '''Notes'''
* Social API can't be enabled from the Facebook activation page. Commented and reopened {{bug|807915}}. Social API can be enabled only by setting the preferences social.active and social.enabled to true in about:config.
* Social API can't be enabled from the Facebook activation page. Commented and reopened {{bug|807915}}. Social API can be enabled only by setting the preferences social.active and social.enabled to true in about:config.
* When exiting Private Browsing mode, Social API restores to the state it was in before entering PB mode.
  |}
  |}


==== {{bug|815743}} Backout the UA changes in bug 588909 ====
==== [VERIFIED FIXED] {{bug|815743}} Backout the UA changes in bug 588909 ====
{| class="querytable"
{| class="querytable"
  | class="header" style="width:55% |  
  | class="header" style="width:55% |  
Line 2,051: Line 2,050:
  |}
  |}


==== {{bug|814101}} Font Issue in FF 17 ====
==== [VERIFIED FIXED] {{bug|814101}} Font Issue in FF 17 ====
{| class="querytable"
{| class="querytable"
  | class="header" |  
  | class="header" |  
Line 2,060: Line 2,059:
  |-
  |-
  | about:support shows Direct2D enabled = true
  | about:support shows Direct2D enabled = true
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  |-
  |-
  | about:support shows GPU Accelerated Windows = 1/1 Direct3D 10
  | about:support shows GPU Accelerated Windows = 1/1 Direct3D 10
  | result (Mihaela)
  | PASS(Mihaela)
  | result (Paul)
  | PASS(Paul)
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  |-
  |-
  | ClearType ON displays anti-aliased text
  | ClearType ON displays anti-aliased text
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  |-
  |-
  | ClearType OFF does not display anti-aliased text
  | ClearType OFF does not display anti-aliased text
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  | result (Mihaela)
  | PASS (Mihaela)
  | result (Paul)
  | PASS(Paul)
  |-
  |-
  | colspan="5" | '''Test'''
  | colspan="5" | '''Test'''
Line 2,095: Line 2,094:
  |-
  |-
  | colspan="4" | '''Notes'''
  | colspan="4" | '''Notes'''
*  
* Environments:
** Win 7 32 bit, AMD Radeon HD 6450
** Win 7 64bit, Nvidia GeForce 210
** Win 8 64bit, AMD Radeon HD 6450
** Win 8 32bit, Nvidia GeForce 210
  |}
  |}


==== {{bug|813264}} Bug 809865 or Bug 810726 broke Skype click-to-call ====
==== [VERIFIED FIXED] {{bug|814554}} Firefox 17 silently stops processing permissions.sqlite ====
{| class="querytable"
| class="header" style="width:55% |
| class="header" style="width:15% | Windows XP
| class="header" style="width:15% | Windows 7
| class="header" style="width:15% | Windows 8
|-
| Skype Click-to-Call installs and functions in Firefox 17.0.1
| result (Paul)
| FAIL (Manuela)
| result (Ioana)
|-
| colspan="4" | '''Test'''
* Install the Skype click-to-call toolbar ([http://www.skype.com/intl/en/get-skype/on-your-computer/click-to-call/windows/ download])
* Browse the web and click on phone number links
* '''''Note: installation may requiring forcing version compatibility'''''
* '''''Note: broken behaviour can be observed in Firefox 17.0'''''
|-
| colspan="4" | '''Notes'''
*
|}
 
==== {{bug|814554}} Firefox 17 silently stops processing permissions.sqlite ====
{| class="querytable"
{| class="querytable"
  | class="header" style="width:55% |  
  | class="header" style="width:55% |  
Line 2,128: Line 2,109:
  |-
  |-
  | Cookies permissions persist across Firefox restart
  | Cookies permissions persist across Firefox restart
  | result (Manuela)
  | PASS (Manuela)
  | result (Virgil)
  | PASS (Virgil)
  | result (Simona)
  | PASS (Simona)
  |-
  |-
  | colspan="4" | '''Test'''
  | colspan="4" | '''Test'''
Line 2,146: Line 2,127:
  |-
  |-
  | colspan="4" | '''Notes'''
  | colspan="4" | '''Notes'''
*  
*Please see more details about how the verification was done here: https://bugzilla.mozilla.org/show_bug.cgi?id=814554#c33
|}
 
==== {{bug|816197}} Need hotfix to mitigate TMP bustage in bug 813763 ====
{| class="querytable"
| class="header" style="width:55% |
| class="header" style="width:15% | Windows 7
| class="header" style="width:15% | Mac OSX
| class="header" style="width:15% | Ubuntu
|-
| Hotfix removed when TMP >0.4.0.3 installed
| result (Manuela)
| result (Virgil)
| result (Simona)
|-
| Hotfix removed when TMP not installed
| result (Manuela)
| result (Virgil)
| result (Simona)
|-
| Hotfix displays [https://support.mozilla.org/en-US/kb/tab-mix-plus-incompatible-firefox-17-use-dev-build this page] on first startup with TMP <=0.4.0.3 installed
| result (Manuela)
| result (Virgil)
| result (Simona)
|-
| Hotfix displays [https://support.mozilla.org/en-US/kb/tab-mix-plus-incompatible-firefox-17-use-dev-build this page] on second startup with TMP <=0.4.0.3 installed
| result (Manuela)
| result (Virgil)
| result (Simona)
|-
| Hotfix removed on third startup with TMP <=0.4.0.3 installed
| result (Manuela)
| result (Virgil)
| result (Simona)
|-
| Hotfix removed if TMP <=0.4.0.3 installed but disabled
| result (Manuela)
| result (Virgil)
| result (Simona)
|-
| colspan="4" | '''Test'''
* Be sure to check the hotfix with Firefox 17.0 and 17.0.1
|-
| colspan="4" | '''Notes'''
*
  |}
  |}


=== Update Testing ===
=== Update Testing ===
* Partial:  
* Partial: 17.0, 16.0.2, 15.0.1
* Complete:  
* Complete: 16.0


{|class="querytable" style="width:60%"
{|class="querytable" style="width:60%"
  |-
  |-
  | class="header" | Platform
  | class="header" | Platform
  | class="header" | betatest ([ report])
  | class="header" | betatest ([http://mozmill-ondemand.blargon7.com/#/update/detail?branch=17.0&channel=betatest&from=2012-11-29&to=2012-11-29&target=17.0.1 report])
  | class="header" | release ([ report])
  | class="header" | release ([http://mozmill-ondemand.blargon7.com/#/update/detail?branch=17.0&channel=release&from=2012-11-30&to=2012-11-30&target=17.0.1 report])
  |-
  |-
  | Windows XP
  | Windows XP
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Windows Vista
  | Windows Vista
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Windows 7 32-bit
  | Windows 7 32-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Windows 7 64-bit
  | Windows 7 64-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Windows 8 32-bit
  | Windows 8 32-bit (manual re:{{bug|803489}})
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Windows 8 64-bit
  | Windows 8 64-bit (manual re:{{bug|803489}})
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Mac OS X 10.6
  | Mac OS X 10.6
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Mac OS X 10.7
  | Mac OS X 10.7
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Mac OS X 10.8
  | Mac OS X 10.8
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Ubuntu 12.04 32-bit
  | Ubuntu 12.04 32-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |-
  |-
  | Ubuntu 12.04 64-bit
  | Ubuntu 12.04 64-bit
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-29
  | '''result''' - YYYY-MM-DD
  | '''PASS''' - 2012-11-30
  |}
  |}
; Spotchecks
* {{done|}} Verify background updates still work (chunk size and ping interval changed for 17.0.1)


; Issues
; Issues
* Very intermittent issue with 15.0.1->17.0.1 fallback updates resulting in connection error to update server


== Bugs ==
== Bugs ==
Ashughes
Confirmed users
14,525

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/490749...491289"