Releases/Firefox 14/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(198 intermediate revisions by 9 users not shown)
Line 9: Line 9:
  |-
  |-
  | '''Status'''
  | '''Status'''
  | Currently in '''Aurora ''' -- merging to Beta on June 4, 2012
  | Currently in '''Beta''' -- Shipping July 17, 2012. Tentative.
  |-
  |-
  | '''Lead'''
  | '''Lead'''
Line 29: Line 29:
* 2012-03-13: {{done|}} Nightly Merge
* 2012-03-13: {{done|}} Nightly Merge
* 2012-04-24: {{done|}} Aurora Merge
* 2012-04-24: {{done|}} Aurora Merge
* 2012-06-05: {{|}} Pre-Beta feature sign-offs
* 2012-06-05: {{done|}} Pre-Beta feature sign-offs
* 2012-06-07: {{|}} Beta 1
* 2012-06-07: {{done|}} Beta 6 (Starting at version 6 to sync with Firefox Mobile)
* 2012-06-14: {{|}} Beta 2
* 2012-06-14: {{done|}} Beta 7
* 2012-06-21: {{|}} Beta 3
* 2012-06-21: {{done|}} Beta 8
* 2012-06-28: {{|}} Beta 4
* 2012-06-28: {{done|}} Beta 9
* 2012-07-05: {{|}} Beta 5
* 2012-07-05: {{done|}} Beta 10
* 2012-07-11: {{|}} Beta 6
* 2012-07-11: {{done|}} Beta 11
* 2012-07-13: {{|}} Release Candidate
* 2012-07-13: {{ok|}} Release Candidate
* 2012-07-17: {{|}} Final
* 2012-07-17: {{|}} Final


Line 52: Line 52:
* [[QA/Firefox migrators| Firefox Migrators]] (Paul Silaghi)
* [[QA/Firefox migrators| Firefox Migrators]] (Paul Silaghi)
* [[DevTools/Features/PseudoClassLock| Pseudo-Class Lock]] (Ioana Budnar)
* [[DevTools/Features/PseudoClassLock| Pseudo-Class Lock]] (Ioana Budnar)
* [[Platform/Features/MouseCapture| Support for Mouse Capture / Mouse Lock]] (Virgil Dicu)
* [[Platform/Features/MouseCapture| Support for Mouse Capture / Mouse Lock]] (Virgil Dicu) - Signed off
* [[Opt-in activation for plugins | Opt-in activation for plugins]] (Paul Silaghi); Initial minimal implementation landed for 14 (opt-in), further work in 15 and 16.
* [[Opt-in activation for plugins | Opt-in activation for plugins]] (Paul Silaghi); Initial minimal implementation landed for 14 (opt-in), further work in 15 and 16.


Line 83: Line 83:
# Record the event in this wiki below
# Record the event in this wiki below


; Current Status (Mihaela Velimiroviciu) - Last Updated  
; Current Status (Mihaela Velimiroviciu) - Last Updated: 2012-06-25
* {{bug|752887}} - Increased memory usage for endurance tests on Linux Aurora branch
* {{bug|752887}} - Increased memory usage for endurance tests on Linux Aurora branch
* {{bug|768026}} - Higher memory consumption on Win 6.0 on Beta branch since the merge


== Functional Automation ==
== Functional Automation ==
Line 412: Line 413:
  |-
  |-
  | '''ID'''
  | '''ID'''
  | TBD
  | 20120605113340
  |-
  |-
  | '''Changeset'''
  | '''Changeset'''
Line 419: Line 420:


; Checklist
; Checklist
* {{ok|}} Functional automated tests, synced to Litmus
* {{done|}} Functional automated tests, synced to Litmus
* {{ok|}} Manual tests
* {{done|}} Manual tests
* {{ok|}} Bug fix verifications (target 50%)
* {{ok|}} Bug fix verifications (target 50%)
* {{ok|}} Security fix verifications (target 25%)
* {{ok|}} Security fix verifications (target 25%)
* {{ok|}} Update tests on releasetest
* {{done|}} Update tests on releasetest
* {{ok|}} Get final sign-off from Bob Moss
* {{done|}} Get final sign-off from Bob Moss
* {{ok|}} Update tests on beta
* {{done|}} Update tests on beta


=== Automated Tests ===
=== Automated Tests ===
Line 432: Line 433:
  |-
  |-
  | class="header" | Platform
  | class="header" | Platform
  | class="header" | Result: [http://mozmill-ondemand.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-06-05&to=2012-06-07 report]
  | class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-06-05&to=2012-06-06 report]
  |-
  |-
  | Windows XP
  | Windows XP
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Windows Vista
  | Windows Vista
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Windows 7 32-bit
  | Windows 7 32-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Windows 7 64-bit
  | Windows 7 64-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Mac OS X 10.6
  | Mac OS X 10.6
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Mac OS X 10.7
  | Mac OS X 10.7
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Ubuntu 11.10 32-bit
  | Ubuntu 11.10 32-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Ubuntu 11.10 64-bit
  | Ubuntu 11.10 64-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |}
  |}


Line 476: Line 477:
  | Windows XP
  | Windows XP
  | Simona
  | Simona
  | '''In progress''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  | Windows Vista
  | Windows Vista
Line 506: Line 507:
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Windows 7 32-bit
  | Windows 7 32-bit (es-AR)
  | Gabriela
  | Gabriela
  | '''In progress''' - 2012-06-06  
  | '''PASS''' - 2012-06-06  
  |}
  |}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=472 Litmus]''
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=472 Litmus]''
Line 536: Line 537:
  | tester
  | tester
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
|-
| Windows 7 32-bit (es-AR)
| Gabriela
| '''PASS''' - 2012-06-07
  |}
  |}


==== Exploratory Testing ====
==== Exploratory Testing ====


# {{ok|}}Exploratory testing on plugins, addons, bookmarks etc [https://mozqa.etherpad.mozilla.org/Firefox14-exploratory Etherpad]
# {{done|}}Exploratory testing on plugins, addons, bookmarks etc [https://mozqa.etherpad.mozilla.org/Firefox14-exploratory Etherpad]
# {{ok|}}Exploratory testing on the new features [https://mozqa.etherpad.mozilla.org/Firefox14-features Etherpad]
# {{done|}}Exploratory testing on the new features [https://mozqa.etherpad.mozilla.org/Firefox14-features Etherpad]


==== Feature Testing ====
==== Feature Testing ====
Line 555: Line 560:
  |[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
  |[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
  | Vlad Ghetiu
  | Vlad Ghetiu
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  |[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
  |[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
  | Paul Silaghi
  | Paul Silaghi
| '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
| '''PASS''' - 2012-06-06
  |-
  |-
  |[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
  |[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
  | Paul Silaghi
  | Paul Silaghi
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''In Progress''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''N/A'''
  | '''N/A'''
  |-
  |-
  |[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
  |[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
  | Ioana Budnar
  | Ioana Budnar
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  |[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
  |[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
  | Virgil Dicu
  | Virgil Dicu
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULTS''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  |-
  |-
  |}
  |}
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
* Open a terminal window and cd to the folder Firefox.app is located
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
* Click "open" and browse to any website, Firefox should start and you should be able to browse
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
Spot check build after this test.


=== Update Tests ===
=== Update Tests ===
Line 591: Line 608:
  |-
  |-
  | class="header" | Platform
  | class="header" | Platform
  | class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=releasetest&from=2012-06-06&to=2012-06-06&target=14.0 report]
  | class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-06-05&to=2012-06-06 report]
  | class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-06-06&to=2012-06-06&target=14.0 report]
  | class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-06-06&to=2012-06-06&target=14.0 report]
  |-
  |-
  | Windows XP
  | Windows XP
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Windows Vista
  | Windows Vista
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Windows 7 32-bit
  | Windows 7 32-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Windows 7 64-bit
  | Windows 7 64-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Mac OS X 10.6
  | Mac OS X 10.6
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Mac OS 10.7
  | Mac OS 10.7
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Ubuntu 11.10 32-bit
  | Ubuntu 11.10 32-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |-
  |-
  | Ubuntu 11.10 64-bit
  | Ubuntu 11.10 64-bit
  | '''RESULT''' - 2012-06-06
  | '''PASS''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  | '''RESULT''' - 2012-06-06
  |}
  |}
Line 631: Line 648:
=== Bug Fix Verifications ===
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox13 flag set to ''verified''
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]


Line 641: Line 658:
  |-
  |-
  | '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
  | '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
  | 51
  | 14
  | 34
  | 25
  | 60%
  | 36%
  |-
  |-
  | '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
  | '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
Line 669: Line 686:
<bugzilla>
<bugzilla>
{
{
  "id": "761990"
  "id": "761990, 762011, 762456, 762483"
}
}
</bugzilla>
</bugzilla>
Line 676: Line 693:
<bugzilla>
<bugzilla>
{
{
  "id": "711742, 747858, 754829"
  "id": "711742, 747858, 754829, 760171, 760175, 675645"
}
</bugzilla>
 
=== Beta 6 Feedback ===
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
 
Builds can be found here: [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b6-candidates/build2/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b6-candidates/build2/ HTTP]
 
You can post your feedback below or in [https://etherpad.mozilla.org/fx14-dogfooding this etherpad].
 
Be sure to install the [https://addons.mozilla.org/en-US/firefox/addon/add-on-compatibility-reporter/ Add-on Compatibility Reporter] if you want to report incompatibilities.
 
== Beta 7 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b7-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b7-candidates/build1/ HTTP]
|-
| '''ID'''
| TBD
|-
| '''Changeset'''
| [https://hg.mozilla.org/releases/mozilla-beta/rev/89ec8943347a 89ec8943347a]
|}
 
; Checklist
* {{done|}} Functional automated tests, synced to Litmus
* {{done|}} Manual tests
* {{done|}} Bug fix verifications (target 50%)
* {{ok|}} Security fix verifications (target 25%)
* {{done|}} Update tests on releasetest
* {{ok|}} Get final sign-off from Bob Moss
* {{ok|}} Update tests on beta
 
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-06-12&to=2012-06-13 report]
|-
| Windows XP
| '''PASS''' - 2012-06-12
|-
| Windows Vista
| '''PASS''' - 2012-06-12
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-12
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-12
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-12
|-
| Mac OS X 10.7
| '''PASS''' - 2012-06-12
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-12
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-12
|}
 
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
 
=== Manual Tests ===
The following are the detailed results of the manual testing.
 
==== Smoketests ====
Run the '''Firefox 14.0 Smoketests''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=474 Litmus] and report results below.
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result
|-
| Windows XP
| tester
| '''RESULT''' - 2012-06-13
|-
| Windows Vista
| tester
| '''RESULT''' - 2012-06-13
|-
| Windows 7 32-bit
| Simona
| '''In progress''' - 2012-06-13
|-
| Windows 7 64-bit
| tester
| '''RESULT''' - 2012-06-13
|-
| Mac OS X 10.6.8
| VladG
| '''PASS''' - 2012-06-13
|-
| Mac OS X 10.7.4
| tester
| '''RESULT''' - 2012-06-13
|-
| Ubuntu 12.04 32-bit
| tester
| '''RESULT''' - 2012-06-13
|-
| Ubuntu 11.10 64-bit
| tester
| '''RESULT''' - 2012-06-13
|-
| Windows 7 32-bit (es-AR)
| Gabriela
| '''RESULT''' - 2012-06-13
|}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=474 Litmus]''
 
==== Regression Testing ====
Run the '''Firefox 14.0 Regression Tests''' and  '''Firefox 14.0 Web App Compatibility''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=474 Litmus] and report the results below.
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result: [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=474 report]
|-
| Windows 7 32-bit
| tester
| '''RESULT''' - 2012-06-12
|-
| Mac OS X 10.6.8
| tester
| '''RESULT''' - 2012-06-12
|-
| Ubuntu 11.10 64-bit
| tester
| '''RESULT''' - 2012-06-12
|-
| Windows XP
| tester
| '''RESULT''' - 2012-06-12
|-
| Windows 7 32-bit (es-AR)
| Gabriela
| '''PASS''' - 2012-06-07
|}
 
==== Exploratory Testing ====
 
# {{done|}}Exploratory testing on plugins, addons, bookmarks etc [https://mozqa.etherpad.mozilla.org/Fx14b7-exploratory Etherpad]
# {{done|}}Exploratory testing on the new features [https://mozqa.etherpad.mozilla.org/Fx14b7-features Etherpad]
 
==== Feature Testing ====
Run the  '''Firefox 14.0 Features Sign-off''' tests in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=474 Litmus] and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
 
{| class="querytable"
| class="header" | Feature
| class="header" | Owner
| class="header" | Win32
| class="header" | Mac
| class="header" | Linux
|-
|[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
| Vlad Ghetiu
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
|[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
| Paul Silaghi
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
|[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
| Paul Silaghi
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
| '''N/A'''
|-
|[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
| Ioana Budnar
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
|[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
| Virgil Dicu
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
|[[Releases/Firefox_14/Test_Plan#Mac_10.8_Signing | Mac OS 10.8 Signing]]
| Required
| n/a
| '''RESULT''' - 2012-06-12
| n/a
|-
|}
 
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
**'''PASS''' - VladG
* Open a terminal window and cd to the folder Firefox.app is located
**'''PASS''' - VladG
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
**'''PASS''' - VladG
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
**'''PASS''' - VladG - the output is containing "com.apple.FinderInfo" and "com.apple.quarantine"
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
**'''PASS''' - VladG
* Click "open" and browse to any website, Firefox should start and you should be able to browse
**'''PASS''' - VladG
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
**'''PASS''' - VladG
Spot check build after this test.
 
=== Update Tests ===
The following are details of update verifications
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-06-13&to=2012-06-13 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-06-14&to=2012-06-14&target=14.0 report]
|-
| Windows XP
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Windows Vista
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Mac OS 10.7
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-12
| '''RESULT''' - 2012-06-12
|}
 
; Issues
 
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
 
{| class="querytable"
| class="header" | Query
| class="header" | Triaged
| class="header" | Remaining
| class="header" | % Complete
|-
| '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
| 20
| 19
| 51%
|-
| '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
| 7
| 23
| 23%
|-
| '''[http://bit.ly/NCG2UT status-firefox14:fixed]''': for QA lead to triage for potential QA+ bugs
| 297
| 0
| 100%
|}
 
''[https://docs.google.com/spreadsheet/ccc?key=0AjuYQoXoeC5KdENxZDNoRlYwZENWOXFDN01DUVpfN0E#gid=0 Softvision verifications etherpad]''
 
=== QAWANTED ===
[http://bit.ly/L7NnJw QAWANTED Bugs]
* Need monitoring of crash stats for {{bug|760190}}
* {{bug|746036#c13}} needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
* {{bug|761990}} Feedback button regression - investigated and updated (Simona).
* {{bug|748047}} needs verification (reset profile) - verified as fixed (Ioana).
* {{bug|748569}} needs verification (migration code)
* Need steps to reproduce {{bug|709860}} - investigated and commented, no STR yet (Virgil)
* {{bug|755728}} low priority investigation - investigated and updated (Paul).
 
=== Issues Found ===
<bugzilla>
{
"id": "764366, 764385, 764388"
}
</bugzilla>
 
=== Known Issues ===
<bugzilla>
{
"id": "711742, 747858, 754829, 760171, 760175, 675645, 761990, 762011, 762456, 762483, 587874, 764062"
}
</bugzilla>
 
=== Beta 7 Feedback ===
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
 
Builds can be found here: [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b7-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b7-candidates/build1/ HTTP]
 
You can post your feedback below or in [https://etherpad.mozilla.org/fx14-dogfooding this etherpad].
 
Be sure to install the [https://addons.mozilla.org/en-US/firefox/addon/add-on-compatibility-reporter/ Add-on Compatibility Reporter] if you want to report incompatibilities.
 
== Beta 8 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b8-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b8-candidates/build1/ HTTP]
|-
| '''ID'''
| 20120619191901
|-
| '''Changeset'''
| [http://hg.mozilla.org/releases/mozilla-beta/rev/f8d3886db65a Changeset]
|}
 
; Checklist
* {{done|}} Functional automated tests, synced to Litmus
* {{done|}} Manual tests
* {{ok|}} Bug fix verifications (target 50%)
* {{ok|}} Security fix verifications (target 25%)
* {{done|}} Update tests on releasetest
* {{ok|}} Get final sign-off from Bob Moss
* {{ok|}} Update tests on beta
 
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-06-19&to=2012-06-20 report]
|-
| Windows XP
| '''PASS''' - 2012-06-20
|-
| Windows Vista
| '''PASS''' - 2012-06-20
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-20
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-20
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-20
|-
| Mac OS X 10.7
| '''PASS''' - 2012-06-20
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-20
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-20
|}
 
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
 
=== Manual Tests ===
The following are the detailed results of the manual testing.
 
==== Smoketests ====
Run the '''Firefox 14.0 Smoketests''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=475 Litmus] and report results below.
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result
|-
| Windows XP
| Simona
| '''PASS''' - 2012-06-19
|-
| Windows Vista
| tester
| '''RESULT''' - 2012-06-19
|-
| Windows 7 32-bit
| tester
| '''RESULT''' - 2012-06-19
|-
| Windows 7 64-bit
| tester
| '''RESULT''' - 2012-06-19
|-
| Mac OS X 10.6.8
| tester
| '''RESULT''' - 2012-06-19
|-
| Mac OS X 10.7.4
| tester
| '''RESULT''' - 2012-06-19
|-
| Ubuntu 12.04 32-bit
| tester
| '''RESULT''' - 2012-06-19
|-
| Ubuntu 11.10 64-bit
| tester
| '''RESULT''' - 2012-06-19
|-
| Windows 8 32-bit build (en-US)
| juanb
| '''PASS''' - 2012-06-20
|}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=475 Litmus]''
 
==== Regression Testing ====
Run the '''Firefox 14.0 Regression Tests''' and  '''Firefox 14.0 Web App Compatibility''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=475 Litmus] and report the results below.
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result: [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=475 report]
|-
| Windows 7 32-bit
| tester
| '''RESULT''' - 2012-06-19
|-
| Mac OS X 10.6.8
| tester
| '''RESULT''' - 2012-06-19
|-
| Ubuntu 11.10 64-bit
| tester
| '''RESULT''' - 2012-06-19
|-
| Windows XP
| tester
| '''RESULT''' - 2012-06-19
|}
 
==== Exploratory Testing ====
 
# {{done|}}Exploratory testing on plugins, addons, bookmarks, new features etc [https://mozqa.etherpad.mozilla.org/Fx14b8-exploratory Etherpad]
 
==== Feature Testing ====
Run the  '''Firefox 14.0 Features Sign-off''' tests in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=475 Litmus] and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
 
{| class="querytable"
| class="header" | Feature
| class="header" | Owner
| class="header" | Win32
| class="header" | Mac
| class="header" | Linux
|-
|[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
| Vlad Ghetiu
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
|[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
| Paul Silaghi
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
|[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
| Paul Silaghi
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
| '''N/A'''
|-
|[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
| Ioana Budnar
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
|[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
| Virgil Dicu
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
|[[Releases/Firefox_14/Test_Plan#Mac_10.8_Signing | Mac OS 10.8 Signing]]
| Required
| n/a
| '''RESULT''' - 2012-06-19
| n/a
|-
|}
 
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
**'''PASS''' (Ioana)
* Open a terminal window and cd to the folder Firefox.app is located
**'''PASS''' (Ioana)
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
**'''PASS''' (Ioana)
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
**'''PASS''' (Ioana)
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
**'''PASS''' (Ioana)
* Click "open" and browse to any website, Firefox should start and you should be able to browse
**'''PASS''' (Ioana)
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
**'''PASS''' (Ioana)
 
Spot check build after this test. [https://mozqa.etherpad.mozilla.org/Fx14b8-exploratory Results]
 
=== Update Tests ===
The following are details of update verifications
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-06-19&to=2012-06-19 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-06-19&to=2012-06-19&target=14.0 report]
|-
| Windows XP
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Windows Vista
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Windows 7 32-bit
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Windows 7 64-bit
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Mac OS X 10.6
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Mac OS 10.7
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Ubuntu 11.10 32-bit
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|-
| Ubuntu 11.10 64-bit
| '''RESULT''' - 2012-06-19
| '''RESULT''' - 2012-06-19
|}
 
; Issues
 
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
 
{| class="querytable"
| class="header" | Query
| class="header" | Triaged
| class="header" | Remaining
| class="header" | % Complete
|-
| '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
| 7
| 32
| 89%
|-
| '''[http://bit.ly/MaJ96c SG with ''testcase'']''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
| 7
| 23
| 23%
|-
| '''[http://bit.ly/NCG2UT status-firefox14:fixed]''': for QA lead to triage for potential QA+ bugs
| 297
| 0
| 100%
|}
 
''[https://docs.google.com/spreadsheet/ccc?key=0AjuYQoXoeC5KdENxZDNoRlYwZENWOXFDN01DUVpfN0E#gid=0 Softvision verifications etherpad]''
 
=== QAWANTED ===
[http://bit.ly/L7NnJw QAWANTED Bugs]
* Need monitoring of crash stats for {{bug|760190}}
* {{bug|746036#c13}} needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
* {{bug|761990}} Feedback button regression - investigated and updated (Simona).
* {{bug|748047}} needs verification (reset profile) - verified as fixed (Ioana).
* {{bug|748569}} needs verification (migration code)
* Need steps to reproduce {{bug|709860}} - investigated and commented, no STR yet (Virgil)
* {{bug|755728}} low priority investigation - investigated and updated (Paul).
* {{bug|764157}} regression range wanted - found regression range/updated (Virgil)
* {{bug|754000}} performance regression - investigated and updated (Paul).
 
=== Issues Found ===
<bugzilla>
{
"id": "766548"
}
</bugzilla>
 
=== Known Issues ===
<bugzilla>
{
"id": "711742, 747858, 754829, 760171, 760175, 675645, 761990, 762011, 762456, 762483"
}
}
</bugzilla>
</bugzilla>
Line 683: Line 1,328:
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.


Builds can be found here: [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b6-candidates/build2/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b6-candidates/build2/ HTTP]
Builds can be found here: [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b8-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b8-candidates/build1/ HTTP]


You can post your feedback below or in [https://etherpad.mozilla.org/fx14-dogfooding this etherpad].
You can post your feedback below or in [https://etherpad.mozilla.org/fx14-dogfooding this etherpad].


Be sure to install the [https://addons.mozilla.org/en-US/firefox/addon/add-on-compatibility-reporter/ Add-on Compatibility Reporter] if you want to report incompatibilities.
Be sure to install the [https://addons.mozilla.org/en-US/firefox/addon/add-on-compatibility-reporter/ Add-on Compatibility Reporter] if you want to report incompatibilities.
== Beta 9 ==
: NOTE this is a chemspill Beta to mitigate requirement to restart OS on Flash update
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b9-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b9-candidates/build1/ HTTP]
|-
| '''ID'''
| 20120624012213
|-
| '''Changeset'''
| [http://hg.mozilla.org/releases/mozilla-beta/rev/d050090e578c d050090e578c]
|}
; Checklist
* {{done|}} Functional automated tests
* {{ok|}} Manual tests
* {{ok|}} Bug fix verifications
* {{done|}} Update tests on releasetest
* {{ |}} Peer sign-off
* {{ |}} Update tests on beta
* '''Currently blocked on update testing due to {{bug|767854}} (server issue)'''
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-06-23&to=2012-06-24 report]
|-
| Windows XP
| '''PASS''' - 2012-06-24
|-
| Windows Vista
| '''PASS''' - 2012-06-24
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-24
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-24
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-24
|-
| Mac OS X 10.7
| '''PASS''' - 2012-06-24
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-24
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-24
|}
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
* /restartTests/testDefaultBookmarks/test1.js: {{bug|767819}}
* /testPrivateBrowsing/testAboutPrivateBrowsing.js: {{bug|767820}}
* /testCookies/testRemoveAllCookies.js: {{bug|767821}}
* /testSecurity/testGreyLarry.js: {{bug|767822}}
=== Manual Testing ===
Run manual smoke tests in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=477 Litmus].
* Win 7 x86 - '''PASS''' - 2012-06-25 (Mihaela)
{| class="querytable"
| class="header" style="width:30%" | Focus Area
| class="header" style="width:20%" | Win7-32
| class="header" style="width:20%" | WinXP
| class="header" style="width:20%" | Mac
| class="header" style="width:20%" | Linux
|-
| Plug-ins
| '''PASS''' - 2012-06-25
| '''PASS''' - 2012-06-25
| '''PASS''' - 2012-06-25
| '''PASS''' - 2012-06-25
|-
| Top-sites
| '''PASS''' - 2012-06-25
| '''PASS''' - 2012-06-25
| '''PASS''' - 2012-06-25
| '''PASS''' - 2012-06-25
|-
| Mac OSX 10.8
| N/A
| N/A
| '''PASS''' - 2012-06-25
| N/A
|}
==== Exploratory testing etherpad: https://mozqa.etherpad.mozilla.org/14b9Exploratory ====
==== Plug-ins ====
* Tests:
** Flash 11.3
** Java 6, 7
** Silverlight
** QuickTime
** Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content, updating from older plugin versions (particularly with content already loaded and running), try reloading/new tab/new window with plugin content after update
* Results:
** Windows XP: '''PASS''' - 2012-06-25 (Simona)
** Windows 7: '''PASS''' - 2012-06-25 (Paul)
** Ubuntu 12.04: '''PASS''' - 2012-06-25 (Virgil)
** Mac OSX 10.6.8: '''PASS''' - 2012-06-25  (Vladg)
* Details & Bugs:
==== Top Site Compatibility ====
* Tests:
** YouTube: video playback, 3D, uploading large files (>100MB), commenting
** Facebook: chat, status, pictures, video, games, timeline, groups
** Twitter: tweet, follow, unfollow, media sharing, trends
** Banking website authentication and transactions
** E-Store authentication, shopping cart, and checkout
** Webmail: Hotmail, Gmail, Yahoo
** ''keep an eye out for crashes''
* Results:
** Windows XP: '''PASS''' - 2012-06-25 (Simona)
** Windows 7: '''PASS''' - 2012-06-25 (Paul)
** Ubuntu 12.04: '''PASS''' - 2012-06-25 (Virgil)
** Mac OSX 10.6.8: '''PASS''' - 2012-06-25  (Vladg)
* Details & Bugs:
==== Mac OS X 10.8 Code-signing ====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
** '''PASS''' - 2012-06-25 - VladG
* Open a terminal window and cd to the folder Firefox.app is located
** '''PASS''' - 2012-06-25 - VladG
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
** '''PASS''' - 2012-06-25 - VladG
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
** '''PASS''' - 2012-06-25 - VladG
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
** '''PASS''' - 2012-06-25 - VladG
* Click "open" and browse to any website, Firefox should start and you should be able to browse
** '''PASS''' - 2012-06-25 - VladG
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, Firefox should start up and browse without any issues
** '''PASS''' - 2012-06-25 - VladG
Do the following testing to make sure the build works as expected...
* Tests:
** Try all the menu item functions
** Gestures
** Plug-in compatibility
***When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
** Top-site compatibility
** Add-on compatibility
** Software update, especially with rich profiles and fallback updates (use ''betatest'' channel)
* Results: '''PASS''' - 2012-06-25 - VladG
* Details & Bugs:
=== Update Tests ===
The following are details of update verifications
{|class="querytable" style="width:80%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=releasetest&from=2012-06-25&to=2012-06-25&target=14.0 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-06-25&to=2012-06-26&target=14.0 report]
|-
| Windows XP
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Windows Vista
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Mac OS 10.7
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-24
| '''PASS''' - 2012-06-25
|}
; Issues
=== Bug Verifications ===
Add your name beside whatever you are testing so people don't duplicate your results.
* {{bug|686335}} - Bad plugin search order prevents upgrading Flash without an OS restart
* {{bug|758363}} - Implement NPN_ReloadPlugins for out-of-process Plugins
; NOTE that testing any of these in a VM requires the VM to be run in Protected Mode
My advice would be to take a snapshot of the VM before starting and rolling back before each test they can be run from a clean snapshot.
; NOTE that not all of these plugin versions are available for Mac and Linux.
Just test with whatever old versions you can find and update to the latest. Report the version numbers in your results.
==== Tests ====
Flash 10.2 (blocklisted)
* Download and install [http://fpdownload.macromedia.com/get/flashplayer/installers/archive/fp_10.2.159.1_archive.zip Flash 10.2]
* Blocklist Flash by [https://wiki.mozilla.org/Blocklisting/Testing#Forcing_a_Blocklist_Ping forcing a ping]
* Load some Flash content in tabs before updating
* Pave-over install [http://get.adobe.com/flashplayer Flash 11.3.300.262]
* Reload tabs with Flash
Flash 10.3 (enabled & disabled)
* Download and install [http://fpdownload.macromedia.com/get/flashplayer/installers/archive/fp_10.3.183.20_archive.zip Flash 10.3]
* Load some Flash content in tabs before updating
* Pave-over install [http://get.adobe.com/flashplayer Flash 11.3.300.262]
* Reload tabs with Flash
* Uninstall Flash and retest with Flash disabled before updating
Flash 11.2 (enabled & disabled)
* Download and install [http://fpdownload.macromedia.com/get/flashplayer/installers/archive/fp_11.2.202.235_archive.zip Flash 11.2]
* Load some Flash content in tabs before updating
* Pave-over install [http://get.adobe.com/flashplayer Flash 11.3.300.262]
* Reload tabs with Flash
* Uninstall Flash and retest with Flash disabled before updating
Flash 11.3 (enabled & disabled)
* Download and install [http://www.filehippo.com/download_flashplayer_firefox/ Flash 11.3.300.257]
* Load some Flash content in tabs before updating
* Pave-over [http://get.adobe.com/flashplayer Flash 11.3.300.262]
* Uninstall Flash and retest with Flash disabled before updating
Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)
* Download and install [http://fpdownload.macromedia.com/get/flashplayer/installers/archive/fp_11.2.202.235_archive.zip Flash 11.2]
* Start Firefox and open a page with flash content
* Download but do not install [http://download.macromedia.com/pub/flashplayer/current/support/install_flash_player.exe Flash 11.3 Auto-update Installer]
* Open a command line to the download folder and run:
** install_flash_player.exe -install -iv 9
* Reload the page with Flash content once the installer finishes
* Uninstall Flash
* Repeat the test, loading Flash content in a new tab after update
* Repeat the test, loading Flash content in a new window after update
Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)
* Download and install [http://fpdownload.macromedia.com/get/flashplayer/installers/archive/fp_11.2.202.235_archive.zip Flash 11.2]
* Start Firefox and open a page with flash content
* Download but do not install [http://download.macromedia.com/pub/flashplayer/current/support/install_flash_player.exe Flash 11.3 Auto-update Installer]
* Open a command line to the download folder and run:
** install_flash_player.exe -install -iv 9
* Reload the page with Flash content once the installer finishes
* Uninstall Flash
* Repeat the test, loading Flash content in a new tab after update
* Repeat the test, loading Flash content in a new window after update
Java 6u31 (enabled & disabled)
* Download and install [http://www.oldapps.com/java.php?old_java=7272?download Java 6u31]
* Load some Java content in tabs before updating
* Pave-over install [http://www.java.com/en/download/ Java 7u5]
* Reload tabs with Java content after updating
* Uninstall Java and retest with Java disabled before updating
Java 7u2 (enabled & disabled)
* Download and install [http://www.oldapps.com/java.php?old_java=7024?download Java 7u2]
* Load some Java content in tabs before updating
* Pave-over install [http://www.java.com/en/download/ Java 7u5]
* Reload tabs with Java content after updating
* Uninstall Java and retest with Java disabled before updating
==== Results ====
Expected
* Plugin updates without any problems
* Latest version of the plugin is listed in about:plugins
* Content loads (or reloads) after update without crashing
* NOTE: Multiple versions may be listed in about:plugins
** OKAY as long as plugin content is using the latest version
** Multiple entries can be cleared by restarting the OS
Windows 7 (ashughes)
* '''PASS''' Flash 10.2 -> 11.3.300.262 (blocklisted) via pave-over
* '''PASS''' Flash 10.3 -> 11.3.300.262 (enabled) via pave-over
* '''PASS''' Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
* '''PASS''' Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
* '''PASS''' Flash 11.3 -> 11.3.300.262 (disabled) via auto-update
* '''PASS''' Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
* '''PASS''' Java 6u31 (disabled) -- ''requires Firefox restart''
* '''PASS''' Java 7u2 (enabled)
Windows XP (ashughes)
* '''PASS''' Update Flash 10.2 -> 11.3.300.262 (disabled) via pave-over
* '''PASS''' Update Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
* '''PASS''' Update Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
* '''PASS''' Update Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
* '''PASS''' Update Java 6u31 -> 7u5 (disabled)
* '''PASS''' Update Java 7u2 -> 7u5 (enabled)
Mac OSX 10.6 (ashughes)
* '''PASS''' Update Flash 10.2 -> 11.3.300.257 (disabled) via pave-over
* '''PASS''' Update Flash 11.2 -> 11.3.300.257 (enabled) via pave-over
* '''PASS''' Update Flash 11.2 -> 11.3.300.257 (dom.ipc.plugins.enabled=false) via pave-over
* '''result''' Update Java 6u* -> 7u5 (disabled) via Software Update
* '''result''' Update Java 7u* -> 7u5 (enabled) via Software Update
* Flash:
** Flash installer forces Firefox to quit before updating
** Disabling before update does not automatically enable after update
Ubuntu 12.04 (ashughes)
* '''PASS''' Flash 11.2.205.233 -> 11.2.205.236 (enabled) via pave-over
* '''PASS''' Flash 11.2.205.233 -> 11.2.205.236 (disabled) via pave-over
* '''PASS''' Flash 11.2.205.233 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
* '''result''' Update Oracle Java 6u* -> 7u5 (disabled) via pave-over
* '''result''' Update IcedTea Java Plugin (enabled) via Software Update
* Flash Plugin:
** Overwriting when Firefox is running will result in an expected recoverable plugin crash
** Disabling a plugin before update does not automatically re-enable after pave-over update
* Java: As of yet unable to find a reliable way to install/update an old version (Ubuntu only provides the latest)
==== Known Issues ====
* Windows: Disabled Java requires restart of Firefox to re-enable after update
* Side-by-Side Flash versions in about:config when paving over Flash 11.3.300.257 64-bit with 11.3.300.262 32-bit; resolved only by OS restart; does not seem to affect functionality/stability
** NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
* Side-by-Side Flash versions in about:config when simulating 11.2->11.3 auto-update with ''install_flash_player.exe -install -iv 9''
** NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
* Hang/crash when watching Youtube after pave-over update ([https://bugzilla.mozilla.org/show_bug.cgi?id=686335#c63 bug 686335#c63]) -- not reproducible using auto-update
=== Issues Found ===
<bugzilla>
{
"id": "768002"
}
</bugzilla>
=== Known Issues ===
<bugzilla>
{
"id": "711742, 747858, 754829, 760171, 760175, 675645, 761990, 762011, 762456, 762483"
}
</bugzilla>
== Beta 10 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b10-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b10-candidates/build1/ HTTP]
|-
| '''ID'''
| 20120628060610
|-
| '''Changeset'''
| [http://hg.mozilla.org/releases/mozilla-beta/rev/e4445f905091 e4445f905091]
|}
; Checklist
* {{done|}} Functional automated tests, synced to Litmus
* {{done|}} Manual tests
* {{ok|}} Bug fix verifications (target 50%)
* {{ok|}} Security fix verifications (target 25%)
* {{done|}} Update tests on releasetest
* {{ok|}} Get final sign-off from Bob Moss
* {{ok|}} Update tests on beta
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-06-28&to=2012-06-28 report]
|-
| Windows XP
| '''PASS''' - 2012-06-28
|-
| Windows Vista
| '''PASS''' - 2012-06-28
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-28
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-28
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-28
|-
| Mac OS X 10.7
| '''PASS''' - 2012-06-28
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-28
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-28
|}
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
=== Manual Tests ===
The following are the detailed results of the manual testing.
==== Smoketests ====
Run the '''Firefox 14.0 Smoketests''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=479 Litmus] and report results below.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result
|-
| Windows XP
| Simona
| '''PASS''' - 2012-06-28
|-
|}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=479 Litmus]''
==== Regression Testing ====
Run the '''Firefox 14.0 Regression Tests''' and  '''Firefox 14.0 Web App Compatibility''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=478 Litmus] and report the results below.
Results of manual testing (details in exploratory etherpad):
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result: [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=478 report]
|-
| Windows 7 32-bit
| Simona
| '''PASS''' - 2012-06-29
|-
| Mac OS X 10.6.8
| Vlad
| '''PASS''' - 2012-06-29
|-
| Ubuntu 11.10 64-bit
| Virgil
| '''PASS''' - 2012-06-29
|-
| Windows 7 64-bit
| Paul
| '''PASS''' - 2012-06-29
|}
==== Exploratory Testing ====
{{done|}}Exploratory testing on plugins, addons, bookmarks etc [https://mozqa.etherpad.mozilla.org/Firefox14beta10 Etherpad]
==== Feature Testing ====
Run the  '''Firefox 14.0 Features Sign-off''' tests in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=478 Litmus] and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
{| class="querytable"
| class="header" | Feature
| class="header" | Owner
| class="header" | Win32
| class="header" | Mac
| class="header" | Linux
|-
|[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
| Vlad Ghetiu
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
|[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
| Paul Silaghi
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
|[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
| Paul Silaghi
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
| '''N/A'''
|-
|[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
| Ioana Budnar
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
|[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
| Virgil Dicu
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
|[[Releases/Firefox_14/Test_Plan#Mac_10.8_Signing | Mac OS 10.8 Signing]]
| VladG
| n/a
| '''PASS''' - 2012-06-29
| n/a
|-
|}
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
**'''PASS''' - VladG - 2012-06-29
* Open a terminal window and cd to the folder Firefox.app is located
**'''PASS''' - VladG - 2012-06-29
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
**'''PASS''' - VladG - 2012-06-29
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
**'''PASS''' - VladG - 2012-06-29
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
**'''PASS''' - VladG - 2012-06-29
* Click "open" and browse to any website, Firefox should start and you should be able to browse
**'''PASS''' - VladG - 2012-06-29
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
**'''PASS''' - VladG - 2012-06-29
Spot check build after this test.
*Try all the menu item functions
*Gestures
*Plug-in compatibility
**'''When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"'''
*Top-site compatibility
*Add-on compatibility
*Software update, especially with rich profiles and fallback updates (use betatest channel)
Results: '''PASS''' - 2012-06-29 - VladG
=== Update Tests ===
The following are details of update verifications
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-06-28&to=2012-06-28 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-06-28&to=2012-06-28&target=14.0 report]
|-
| Windows XP
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Windows Vista
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Windows 7 32-bit
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Windows 7 64-bit
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Mac OS X 10.6
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Mac OS 10.7
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-06-28
| '''RESULT''' - 2012-06-28
|}
; Issues
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
{| class="querytable"
| class="header" | Query
| class="header" | Triaged
| class="header" | Remaining
| class="header" | % Complete
|-
| '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
| 31
| 8
| 79%
|-
| '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
|
|
|
|-
| '''[http://bit.ly/NCG2UT status-firefox14:fixed]''': for QA lead to triage for potential QA+ bugs
|
|
|
|}
''[https://docs.google.com/spreadsheet/ccc?key=0AjuYQoXoeC5KdENxZDNoRlYwZENWOXFDN01DUVpfN0E#gid=0 Softvision verifications etherpad]''
=== QAWANTED ===
[http://bit.ly/L7NnJw QAWANTED Bugs]
* Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
=== Issues Found ===
<bugzilla>
{
"id": ""
}
</bugzilla>
== Beta 11 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b11-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b11-candidates/build1/ HTTP]
|-
| '''ID'''
| 20120704090211
|-
| '''Changeset'''
| [http://hg.mozilla.org/releases/mozilla-beta/rev/48842c93fcb2 48842c93fcb2]
|}
; Checklist
* {{done|}} Functional automated tests, synced to Litmus
* {{done|}} Manual tests
* {{done|}} Bug fix verifications (target 50%)
* {{ok|}} Security fix verifications (target 25%)
* {{ok|}} Update tests on releasetest
* {{ok|}} Get final sign-off from Bob Moss
* {{ok|}} Update tests on beta
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-07-03&to=2012-07-04 report]
|-
| Windows XP
| '''PASS''' - 2012-07-04
|-
| Windows Vista
| '''PASS''' - 2012-07-04
|-
| Windows 7 32-bit
| '''PASS''' - 2012-07-04
|-
| Windows 7 64-bit
| '''PASS''' - 2012-07-04
|-
| Mac OS X 10.6
| '''PASS''' - 2012-07-04
|-
| Mac OS X 10.7
| '''PASS''' - 2012-07-04
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-07-04
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-07-04
|}
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
=== Manual Tests ===
The following are the detailed results of the manual testing.
==== Smoketests ====
Run the '''Firefox 14.0 Smoketests''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=479 Litmus] and report results below.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result
|-
| Windows XP
| Ioana
| '''PASS''' - 2012-07-05
|}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=479 Litmus]''
==== Exploratory Testing ====
# {{done|}}Exploratory testing on plugins, new features etc [https://mozqa.etherpad.mozilla.org/Fx14b11-exploratory Etherpad]
==== Feature Testing ====
Run the  '''Firefox 14.0 Features Sign-off''' tests in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=479 Litmus] and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
{| class="querytable"
| class="header" | Feature
| class="header" | Owner
| class="header" | Win32
| class="header" | Mac
| class="header" | Linux
|-
|[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
| Vlad Ghetiu
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
|[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
| Paul Silaghi
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
|[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
| Paul Silaghi
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
| '''N/A'''
|-
|[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
| Ioana Budnar
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
|[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
| Virgil Dicu
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
|[[Releases/Firefox_14/Test_Plan#Mac_10.8_Signing | Mac OS 10.8 Signing]]
| Required
| n/a
| '''RESULT''' - 2012-07-04
| n/a
|-
|}
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
**'''PASS''' - VladG - 2012-07-05
* Open a terminal window and cd to the folder Firefox.app is located
**'''PASS''' - VladG - 2012-07-05
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
**'''PASS''' - VladG - 2012-07-05
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
**'''PASS''' - VladG - 2012-07-05
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
**'''PASS''' - VladG - 2012-07-05
* Click "open" and browse to any website, Firefox should start and you should be able to browse
**'''PASS''' - VladG - 2012-07-05
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
**'''PASS''' - VladG - 2012-04-05
Spot check build after this test: '''PASS''' - VladG - 2012-04-05
*Try all the menu item functions
*Gestures
*Plug-in compatibility
*Top-site compatibility
*Add-on compatibility
=== Update Tests ===
The following are details of update verifications
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-07-03&to=2012-07-04 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-07-03&to=2012-07-04&target=14.0 report]
|-
| Windows XP
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Windows Vista
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Windows 7 32-bit
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Windows 7 64-bit
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Mac OS X 10.6
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Mac OS 10.7
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Ubuntu 11.10 32-bit
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|-
| Ubuntu 11.10 64-bit
| '''RESULT''' - 2012-07-04
| '''RESULT''' - 2012-07-04
|}
; Issues
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
{| class="querytable"
| class="header" | Query
| class="header" | Triaged
| class="header" | Remaining
| class="header" | % Complete
|-
| '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
| 31
| 8
| 79%
|-
| '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
|
|
|
|-
| '''[http://bit.ly/NCG2UT status-firefox14:fixed]''': for QA lead to triage for potential QA+ bugs
|
|
|
|}
''[https://docs.google.com/spreadsheet/ccc?key=0AjuYQoXoeC5KdENxZDNoRlYwZENWOXFDN01DUVpfN0E#gid=0 Softvision verifications etherpad]''
=== QAWANTED ===
[http://bit.ly/L7NnJw QAWANTED Bugs]
* Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
=== Issues Found ===
<bugzilla>
{
"id": ""
}
</bugzilla>
=== Known Issues ===
<bugzilla>
{
"id": ""
}
</bugzilla>
== Beta 12 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0b12-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0b12-candidates/build1/ HTTP]
|-
| '''ID'''
| 20120710123126
|-
| '''Changeset'''
| [http://icanhascheezburger.com TBD]
|}
; Checklist
* {{done|}} Functional automated tests, synced to Litmus
* {{done|}} Manual tests
* {{done|}} Bug fix verifications (target 50%)
* {{ok|}} Security fix verifications (target 25%)
* {{done|}} Update tests on releasetest
* {{ok|}} Get final sign-off from Bob Moss
* {{ok|}} Update tests on beta
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-07-10&to=2012-07-10 report]
|-
| Windows XP
| '''PASS''' - 2012-07-10
|-
| Windows Vista
| '''PASS''' - 2012-07-10
|-
| Windows 7 32-bit
| '''PASS''' - 2012-07-10
|-
| Windows 7 64-bit
| '''PASS''' - 2012-07-10
|-
| Mac OS X 10.6
| '''PASS''' - 2012-07-10
|-
| Mac OS X 10.7
| '''PASS''' - 2012-07-10
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-07-10
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-07-10
|}
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
=== Manual Tests ===
The following are the detailed results of the manual testing.
==== Smoketests ====
Run the '''Firefox 14.0 Smoketests''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=480 Litmus] and report results below.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result
|-
| Windows 7 32-bit
| Simona
| '''PASS''' - 2012-07-11
|}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=480 Litmus]''
==== Exploratory Testing ====
# {{done|}}Exploratory testing on plugins, new features etc [https://mozqa.etherpad.mozilla.org/Fx14b12-exploratory Etherpad]
==== Feature Testing ====
Run the  '''Firefox 14.0 Features Sign-off''' tests in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=480 Litmus] and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
{| class="querytable"
| class="header" | Feature
| class="header" | Owner
| class="header" | Win32
| class="header" | Mac
| class="header" | Linux
|-
|[[Firefox/Features/URL Autocomplete/Test Plan|URL Autocomplete]]
| Vlad Ghetiu
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
|-
|[[Opt-in activation for plugins/Test Plan|Opt-in activation for plugins]]
| Paul Silaghi
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
|-
|[[QA/Firefox migrators |Firefox migrators - Safari Migrator]]
| Paul Silaghi
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
| '''N/A'''
|-
|[[DevTools/Features/PseudoClassLock/Test Plan |Pseudo-Class Lock ]]
| Ioana Budnar
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
|-
|[[Platform/Features/MouseCapture/Test Plan |Support for Mouse Capture / Mouse Lock ]]
| Virgil Dicu
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
| '''RESULT''' - 2012-07-10
|-
|[[Releases/Firefox_14/Test_Plan#Mac_10.8_Signing | Mac OS 10.8 Signing]]
| Required
| n/a
| '''PASS''' - 2012-07-11
| n/a
|-
|}
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
**'''PASS''' - VladG - 2012-07-11
* Open a terminal window and cd to the folder Firefox.app is located
**'''PASS''' - VladG - 2012-07-11
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
**'''PASS''' - VladG - 2012-07-11
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
**'''PASS''' - VladG - 2012-07-11
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
**'''PASS''' - VladG - 2012-07-11
* Click "open" and browse to any website, Firefox should start and you should be able to browse
**'''PASS''' - VladG - 2012-07-11
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
**'''PASS''' - VladG - 2012-07-11
Spot check build after this test - '''PASS''' - VladG - 2012-07-11
*Try all the menu item functions
*Gestures
*Plug-in compatibility
**When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
*Top-site compatibility
*Add-on compatibility
*Software update
=== Update Tests ===
The following are details of update verifications
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-07-10&to=2012-07-11 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-07-10&to=2012-07-11&target=14.0 report]
|-
| Windows XP
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Windows Vista
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Windows 7 32-bit
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Windows 7 64-bit
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Mac OS X 10.6
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Mac OS 10.7
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-07-11
| '''PASS''' - 2012-07-11
|}
; Issues
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
{| class="querytable"
| class="header" | Query
| class="header" | Triaged
| class="header" | Remaining
| class="header" | % Complete
|-
| '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
| 36
| 3
| 92%
|-
| '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
|
|
|
|-
| '''[http://bit.ly/NCG2UT status-firefox14:fixed]''': for QA lead to triage for potential QA+ bugs
|
|
|
|}
''[https://docs.google.com/spreadsheet/ccc?key=0AjuYQoXoeC5KdENxZDNoRlYwZENWOXFDN01DUVpfN0E#gid=0 Softvision verifications etherpad]''
=== QAWANTED ===
[http://bit.ly/L7NnJw QAWANTED Bugs]
* Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
=== Issues Found ===
<bugzilla>
{
"id": "772786, 772842, 773236"
}
</bugzilla>
=== Known Issues ===
<bugzilla>
{
"id": "770207, 711545, 675645, 763237, 763237"
}
</bugzilla>
== Firefox 14.0.1 Release Candidates  ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/firefox/nightly/14.0.1-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/firefox/nightly/14.0.1-candidates/build1/ HTTP]
|-
| '''ID'''
| 20120713134347
|-
| '''Changeset'''
| [http://hg.mozilla.org/releases/mozilla-release/rev/e5728a4e106c e5728a4e106c]
|}
; Checklist
* {{done|}} Functional automated tests
* {{done|}} Manual tests
* {{done|}} Bug fix verifications
* {{ok|}} Security fix verifications
* {{done|}} Update tests on releasetest
* {{ok|}} Get final sign-off
* {{ok|}} Update tests on release
=== Automated Tests ===
The following are the detailed results of the automated BFT test runs.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Result: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=14.0&platform=All&from=2012-07-13&to=2012-07-14 report]
|-
| Windows XP
| '''PASS''' - 2012-07-15
|-
| Windows Vista
| '''PASS''' - 2012-07-15
|-
| Windows 7 32-bit
| '''PASS''' - 2012-07-15
|-
| Windows 7 64-bit
| '''PASS''' - 2012-07-15
|-
| Mac OS X 10.6
| '''PASS''' - 2012-07-15
|-
| Mac OS X 10.7
| '''PASS''' - 2012-07-15
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-07-15
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-07-15
|}
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
=== Manual Tests ===
The following are the detailed results of the manual testing.
==== Smoketests ====
Run the '''Firefox 14.0 Smoketests''' in [https://litmus.mozilla.org/run_tests.cgi?test_run_id=482 Litmus] and report results below.
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Assignee
| class="header" | Result
|-
| Windows 7 32-bit
| Mihaela
| '''PASS''' - 2012-07-16
|-
| Mac OS X 10.7.4
| Ioana
| '''PASS''' - 2012-07-16
|-
| Ubuntu 12.04 32-bit
| Virgil
| '''PASS''' - 2012-07-16
|-
|}
''Results in [https://litmus.mozilla.org/test_run_report.cgi?test_run_id=482 Litmus]''
==== Exploratory Testing ====
# {{done|}}Exploratory testing on plugins, new features, web compatibility etc [https://mozqa.etherpad.mozilla.org/Fx14-0-1-exploratory Etherpad]
==== Feature Testing ====
By now we should have all features signed off.
===== Mac 10.8 Signing =====
NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test ([https://people.mozilla.com/~stmichaud/Gatekeeper/MLGatekeeper.html more info]).
* Download the .dmg file and drag the Firefox.app file to a location on your computer
**'''PASS''' - VladG - 2012-07-16
* Open a terminal window and cd to the folder Firefox.app is located
**'''PASS''' - VladG - 2012-07-16
* Run ''codesign -vv Firefox.app'' in terminal, output should contain ''valid on disk'' and ''satisfies its Designated Requirement''
**'''PASS''' - VladG - 2012-07-16
* Run ''xattr Firefox.app'' in terminal, output should contain ''com.apple.quarantine''
**'''PASS''' - VladG - 2012-07-16
* Double click the Firefox.app icon, you should see a warning about the program being provided by an ''unknown developer'' or having been ''downloaded from the Internet''
**'''PASS''' - VladG - 2012-07-16
* Click "open" and browse to any website, Firefox should start and you should be able to browse
**'''PASS''' - VladG - 2012-07-16
* Quit Firefox and double click the Firefox.app icon again, you should ''not'' be reprompted for the warning in step 5, firefox should start up and browse without any issues
**'''PASS''' - VladG - 2012-07-16
Spot check build after this test.
Try all the menu item functions - '''PASS''' - VladG - 2012-07-16
*Gestures
*Plug-in compatibility
*Top-site compatibility
*Add-on compatibility
*Software update
=== Update Tests ===
The following are details of update verifications
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/overview?branch=14.0&channel=releasetest&from=2012-07-15&to=2012-07-11 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=14.0&channel=beta&from=2012-07-15&to=2012-07-11&target=14.0 report]
|-
| Windows XP
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Windows Vista
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Windows 7 32-bit
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Windows 7 64-bit
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Mac OS X 10.6
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Mac OS 10.7
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Ubuntu 11.10 32-bit
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|-
| Ubuntu 11.10 64-bit
| '''PASS''' - 2012-07-15
| '''RESULT''' - 2012-07-15
|}
; Issues
=== Bug Fix Verifications ===
The following are the detailed results of verifications of bug fixes landed in Beta.
* Any fix you very should have the status-firefox14 flag set to ''verified''
* If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
{| class="querytable"
| class="header" | Query
| class="header" | Triaged
| class="header" | Remaining
| class="header" | % Complete
|-
| '''[http://bit.ly/KbYn7h QA+]''': for testers to verify functional fixes
| 36
| 3
| 92%
|-
| '''[http://bit.ly/LjZyWi QA+ SG]''': for the [[Desktop_Firefox#Security_Triage_Team | Security team]] to verify security fixes
|
|
|
|-
| '''[http://bit.ly/NCG2UT status-firefox14:fixed]''': for QA lead to triage for potential QA+ bugs
|
|
|
|}
''[https://docs.google.com/spreadsheet/ccc?key=0AjuYQoXoeC5KdENxZDNoRlYwZENWOXFDN01DUVpfN0E#gid=0 Softvision verifications etherpad]''
=== QAWANTED ===
[http://bit.ly/L7NnJw QAWANTED Bugs]
* Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.
=== Issues Found ===
<bugzilla>
{
"id": "774212, 774244"
}
</bugzilla>
=== Known Issues ===
<bugzilla>
{
"id": "692446, 667399"
}
</bugzilla>

Latest revision as of 21:45, 16 July 2012

« Firefox 13 | Firefox 14 | Firefox 15 »

Firefox 14 Test Plan

Summary

Milestone Firefox 14
Status Currently in Beta -- Shipping July 17, 2012. Tentative.
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 14 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 14. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-03-13: [DONE] Nightly Merge
  • 2012-04-24: [DONE] Aurora Merge
  • 2012-06-05: [DONE] Pre-Beta feature sign-offs
  • 2012-06-07: [DONE] Beta 6 (Starting at version 6 to sync with Firefox Mobile)
  • 2012-06-14: [DONE] Beta 7
  • 2012-06-21: [DONE] Beta 8
  • 2012-06-28: [DONE] Beta 9
  • 2012-07-05: [DONE] Beta 10
  • 2012-07-11: [DONE] Beta 11
  • 2012-07-13: [ON TRACK] Release Candidate
  • 2012-07-17: {{|}} Final

Features

See the Feature Release Tracking wiki for a general overview

[DONE]
[ON TRACK]
[AT RISK]


[MISSED]

Nightly Merge

Functional Automation - result
Update Spotchecks - result

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Juan Becerra of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated
2012-06-25
  • bug 752887 - Increased memory usage for endurance tests on Linux Aurora branch
  • bug 768026 - Higher memory consumption on Win 6.0 on Beta branch since the merge

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Juan Becerra of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
  • No issues found

Unconfirmed Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 404 [1] => Not Found ) ) 1


P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible
  1. Check this <insert link> for UNCONFIRMED bugs found for this Firefox Release's Beta Timeframe
  2. Read each bug and determine what product and component they belong under in bugzilla
  3. Move the bug to the right product and component
  4. CC any applicable team members that may be interested in the bug
P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle
  1. Check the <insert link> bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Juan Becerra of the NEW issue
P3 Task - Confirm UNCONFIRMED Bugs from Other Releases
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Juan Becerra of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet

Aurora Merge

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-04-26
Windows Vista PASS - 2012-04-26
Windows 7 32-bit PASS - 2012-04-26
Windows 7 64-bit PASS - 2012-04-26
Mac OS X 10.6 PASS - 2012-04-26
Mac OS X 10.7 PASS - 2012-04-26
Ubuntu 11.10 32-bit PASS - 2012-04-26
Ubuntu 11.10 64-bit PASS - 2012-04-26
Test Failures
  • /testPasswordManager/testPasswordNotificationBar.js (bug 749156)
  • /testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
  • /testSecurity/testBlueLarry.js (bug 748686)
  • /testSecurity/testGreenLarry.js (bug 748686)
  • /testSecurity/testGreyLarry.js (bug 748686)

Updates

The following are details of update verifications for Firefox 12 merge to Aurora

Auroratest
  • [ON TRACK] [ashughes] Windows XP:
  • [ON TRACK] [ashughes] Windows 7:
  • [ON TRACK] [ashughes] Mac OS X 10.6:
  • [ON TRACK] [ashughes] Ubuntu 11.04:
Aurora
  • [ON TRACK] [ashughes] Windows XP:
  • [ON TRACK] [ashughes] Windows 7:
  • [ON TRACK] [ashughes] Mac OS X 10.6:
  • [ON TRACK] [ashughes] Ubuntu 11.04:
Issues Found

Manual Testing

Smoketests

Run the Aurora Merge Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-04-27
Windows Vista Simona PASS - 2012-04-27
Windows 7 32-bit Paul PASS - 2012-04-27
Windows 7 64-bit Simona PASS - 2012-04-27
Mac OS X 10.6.8 Paul PASS - 2012-04-27
Mac OS X 10.7.3 Paul PASS - 2012-04-27
Ubuntu 11.10 32-bit Virgil PASS - 2012-04-27
Ubuntu 11.10 64-bit Mihaela PASS - 2012-04-27

Results in Litmus

Feature Sign-offs

Run the Aurora 14.0 sign-offs tests in Litmus and report results below.

Feature Owner Win32 Mac Linux
Panel-based Download Manager Simona Badau PASS - 2012-04-27 PASS - 2012-04-27 PASS - 2012-04-27
Windows 8 Metro Firefox Vlad Ghetiu N/A N/A N/A
Cycle Collector Perf improvements Virgil Dicu PASS - 2012-04-27 PASS - 2012-04-30 PASS - 2012-04-30
Incremental Garbage Collection Ioana Budnar PASS - 2012-04-27 PASS - 2012-04-27 PASS - 2012-04-27
Hang Detector and Reporter Simona Badau N/A N/A N/A
URL Autocomplete Vlad Ghetiu PASS - 2012-04-27 PASS - 2012-04-27 PASS - 2012-04-27
Opt-in activation for plugins Paul Silaghi PASS - 2012-04-27 PASS - 2012-04-30 PASS - 2012-04-30
Show PDF Inline Mihaela Velimiroviciu N/A N/A N/A
Firefox migrators Paul Silaghi PASS - 2012-04-30 PASS - 2012-04-30 N/A
Silent Update: Background updates Vlad Ghetiu N/A N/A N/A
Web Apps integration Jason Smith FAIL - 2012-05-01 FAIL - 2012-05-01 N/A - 2012-05-01
Pseudo-Class Lock Ioana Budnar PASS - 2012-04-30 PASS - 2012-04-30 PASS - 2012-04-30
Support for Mouse Capture / Mouse Lock Virgil Dicu FAIL - 2012-04-30 FAIL - 2012-04-30 FAIL - 2012-04-30

Results in Litmus


Notes:

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Pre-beta features sign offs

Run the Firefox 14.0 pre-beta sign off tests in Litmus and report results below.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu PASS - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31
Opt-in activation for plugins Paul Silaghi PASS - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31
Firefox migrators - Safari Migrator Paul Silaghi PASS - 2012-05-31 PASS - 2012-05-31 N/A - 2012-05-31
Pseudo-Class Lock Ioana Budnar FAIL - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31
Support for Mouse Capture / Mouse Lock Virgil Dicu PASS - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31

Results in Litmus

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 6

Builds FTP, HTTP
ID 20120605113340
Changeset 89ec8943347a
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [DONE] Get final sign-off from Bob Moss
  • [DONE] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-06
Windows Vista PASS - 2012-06-06
Windows 7 32-bit PASS - 2012-06-06
Windows 7 64-bit PASS - 2012-06-06
Mac OS X 10.6 PASS - 2012-06-06
Mac OS X 10.7 PASS - 2012-06-06
Ubuntu 11.10 32-bit PASS - 2012-06-06
Ubuntu 11.10 64-bit PASS - 2012-06-06
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

  • Working on cloning the branch as of 06-04 @11:50pm Pacific
Platform Assignee Result
Windows XP Simona PASS - 2012-06-06
Windows Vista tester RESULT - 2012-06-06
Windows 7 32-bit tester RESULT - 2012-06-06
Windows 7 64-bit tester RESULT - 2012-06-06
Mac OS X 10.6.8 tester RESULT - 2012-06-06
Mac OS X 10.7.4 tester RESULT - 2012-06-06
Ubuntu 12.04 32-bit tester RESULT - 2012-06-06
Ubuntu 11.10 64-bit tester RESULT - 2012-06-06
Windows 7 32-bit (es-AR) Gabriela PASS - 2012-06-06

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit tester RESULT - 2012-06-06
Mac OS X 10.6.8 tester RESULT - 2012-06-06
Ubuntu 11.10 64-bit tester RESULT - 2012-06-06
Windows XP tester RESULT - 2012-06-06
Windows 7 32-bit (es-AR) Gabriela PASS - 2012-06-07

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
  2. [DONE] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Opt-in activation for plugins Paul Silaghi PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Firefox migrators - Safari Migrator Paul Silaghi PASS - 2012-06-06 PASS - 2012-06-06 N/A
Pseudo-Class Lock Ioana Budnar PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Support for Mouse Capture / Mouse Lock Virgil Dicu PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test.

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-06 RESULT - 2012-06-06
Windows Vista PASS - 2012-06-06 RESULT - 2012-06-06
Windows 7 32-bit PASS - 2012-06-06 RESULT - 2012-06-06
Windows 7 64-bit PASS - 2012-06-06 RESULT - 2012-06-06
Mac OS X 10.6 PASS - 2012-06-06 RESULT - 2012-06-06
Mac OS 10.7 PASS - 2012-06-06 RESULT - 2012-06-06
Ubuntu 11.10 32-bit PASS - 2012-06-06 RESULT - 2012-06-06
Ubuntu 11.10 64-bit PASS - 2012-06-06 RESULT - 2012-06-06
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 14 25 36%
QA+ SG: for the Security team to verify security fixes 7 23 23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 6 Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 7

Builds FTP, HTTP
ID TBD
Changeset 89ec8943347a
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-12
Windows Vista PASS - 2012-06-12
Windows 7 32-bit PASS - 2012-06-12
Windows 7 64-bit PASS - 2012-06-12
Mac OS X 10.6 PASS - 2012-06-12
Mac OS X 10.7 PASS - 2012-06-12
Ubuntu 11.10 32-bit PASS - 2012-06-12
Ubuntu 11.10 64-bit PASS - 2012-06-12
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP tester RESULT - 2012-06-13
Windows Vista tester RESULT - 2012-06-13
Windows 7 32-bit Simona In progress - 2012-06-13
Windows 7 64-bit tester RESULT - 2012-06-13
Mac OS X 10.6.8 VladG PASS - 2012-06-13
Mac OS X 10.7.4 tester RESULT - 2012-06-13
Ubuntu 12.04 32-bit tester RESULT - 2012-06-13
Ubuntu 11.10 64-bit tester RESULT - 2012-06-13
Windows 7 32-bit (es-AR) Gabriela RESULT - 2012-06-13

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit tester RESULT - 2012-06-12
Mac OS X 10.6.8 tester RESULT - 2012-06-12
Ubuntu 11.10 64-bit tester RESULT - 2012-06-12
Windows XP tester RESULT - 2012-06-12
Windows 7 32-bit (es-AR) Gabriela PASS - 2012-06-07

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
  2. [DONE] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Opt-in activation for plugins Paul Silaghi RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-06-12 RESULT - 2012-06-12 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Mac OS 10.8 Signing Required n/a RESULT - 2012-06-12 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - the output is containing "com.apple.FinderInfo" and "com.apple.quarantine"
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG

Spot check build after this test.

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-12 RESULT - 2012-06-12
Windows Vista PASS - 2012-06-12 RESULT - 2012-06-12
Windows 7 32-bit PASS - 2012-06-12 RESULT - 2012-06-12
Windows 7 64-bit PASS - 2012-06-12 RESULT - 2012-06-12
Mac OS X 10.6 PASS - 2012-06-12 RESULT - 2012-06-12
Mac OS 10.7 PASS - 2012-06-12 RESULT - 2012-06-12
Ubuntu 11.10 32-bit PASS - 2012-06-12 RESULT - 2012-06-12
Ubuntu 11.10 64-bit PASS - 2012-06-12 RESULT - 2012-06-12
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 20 19 51%
QA+ SG: for the Security team to verify security fixes 7 23 23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Need monitoring of crash stats for bug 760190
  • bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
  • bug 761990 Feedback button regression - investigated and updated (Simona).
  • bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
  • bug 748569 needs verification (migration code)
  • Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
  • bug 755728 low priority investigation - investigated and updated (Paul).

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 7 Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 8

Builds FTP, HTTP
ID 20120619191901
Changeset Changeset
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-20
Windows Vista PASS - 2012-06-20
Windows 7 32-bit PASS - 2012-06-20
Windows 7 64-bit PASS - 2012-06-20
Mac OS X 10.6 PASS - 2012-06-20
Mac OS X 10.7 PASS - 2012-06-20
Ubuntu 11.10 32-bit PASS - 2012-06-20
Ubuntu 11.10 64-bit PASS - 2012-06-20
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Simona PASS - 2012-06-19
Windows Vista tester RESULT - 2012-06-19
Windows 7 32-bit tester RESULT - 2012-06-19
Windows 7 64-bit tester RESULT - 2012-06-19
Mac OS X 10.6.8 tester RESULT - 2012-06-19
Mac OS X 10.7.4 tester RESULT - 2012-06-19
Ubuntu 12.04 32-bit tester RESULT - 2012-06-19
Ubuntu 11.10 64-bit tester RESULT - 2012-06-19
Windows 8 32-bit build (en-US) juanb PASS - 2012-06-20

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit tester RESULT - 2012-06-19
Mac OS X 10.6.8 tester RESULT - 2012-06-19
Ubuntu 11.10 64-bit tester RESULT - 2012-06-19
Windows XP tester RESULT - 2012-06-19

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, addons, bookmarks, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Opt-in activation for plugins Paul Silaghi RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-06-19 RESULT - 2012-06-19 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Mac OS 10.8 Signing Required n/a RESULT - 2012-06-19 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS (Ioana)
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS (Ioana)
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS (Ioana)
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS (Ioana)
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS (Ioana)
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS (Ioana)
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS (Ioana)

Spot check build after this test. Results

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP RESULT - 2012-06-19 RESULT - 2012-06-19
Windows Vista RESULT - 2012-06-19 RESULT - 2012-06-19
Windows 7 32-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Windows 7 64-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Mac OS X 10.6 RESULT - 2012-06-19 RESULT - 2012-06-19
Mac OS 10.7 RESULT - 2012-06-19 RESULT - 2012-06-19
Ubuntu 11.10 32-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Ubuntu 11.10 64-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 7 32 89%
SG with testcase: for the Security team to verify security fixes 7 23 23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Need monitoring of crash stats for bug 760190
  • bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
  • bug 761990 Feedback button regression - investigated and updated (Simona).
  • bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
  • bug 748569 needs verification (migration code)
  • Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
  • bug 755728 low priority investigation - investigated and updated (Paul).
  • bug 764157 regression range wanted - found regression range/updated (Virgil)
  • bug 754000 performance regression - investigated and updated (Paul).

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 9

NOTE this is a chemspill Beta to mitigate requirement to restart OS on Flash update
Builds FTP, HTTP
ID 20120624012213
Changeset d050090e578c
Checklist
  • [DONE] Functional automated tests
  • [ON TRACK] Manual tests
  • [ON TRACK] Bug fix verifications
  • [DONE] Update tests on releasetest
  • {{ |}} Peer sign-off
  • {{ |}} Update tests on beta
  • Currently blocked on update testing due to bug 767854 (server issue)

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-24
Windows Vista PASS - 2012-06-24
Windows 7 32-bit PASS - 2012-06-24
Windows 7 64-bit PASS - 2012-06-24
Mac OS X 10.6 PASS - 2012-06-24
Mac OS X 10.7 PASS - 2012-06-24
Ubuntu 11.10 32-bit PASS - 2012-06-24
Ubuntu 11.10 64-bit PASS - 2012-06-24
Test Failures (known bugs)
  • /restartTests/testDefaultBookmarks/test1.js: bug 767819
  • /testPrivateBrowsing/testAboutPrivateBrowsing.js: bug 767820
  • /testCookies/testRemoveAllCookies.js: bug 767821
  • /testSecurity/testGreyLarry.js: bug 767822

Manual Testing

Run manual smoke tests in Litmus.

  • Win 7 x86 - PASS - 2012-06-25 (Mihaela)
Focus Area Win7-32 WinXP Mac Linux
Plug-ins PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25
Top-sites PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25
Mac OSX 10.8 N/A N/A PASS - 2012-06-25 N/A

Exploratory testing etherpad: https://mozqa.etherpad.mozilla.org/14b9Exploratory

Plug-ins

  • Tests:
    • Flash 11.3
    • Java 6, 7
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content, updating from older plugin versions (particularly with content already loaded and running), try reloading/new tab/new window with plugin content after update
  • Results:
    • Windows XP: PASS - 2012-06-25 (Simona)
    • Windows 7: PASS - 2012-06-25 (Paul)
    • Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
    • Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
  • Details & Bugs:

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Twitter: tweet, follow, unfollow, media sharing, trends
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • Webmail: Hotmail, Gmail, Yahoo
    • keep an eye out for crashes
  • Results:
    • Windows XP: PASS - 2012-06-25 (Simona)
    • Windows 7: PASS - 2012-06-25 (Paul)
    • Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
    • Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
  • Details & Bugs:

Mac OS X 10.8 Code-signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - 2012-06-25 - VladG
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - 2012-06-25 - VladG
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - 2012-06-25 - VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - 2012-06-25 - VladG
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - 2012-06-25 - VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - 2012-06-25 - VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, Firefox should start up and browse without any issues
    • PASS - 2012-06-25 - VladG

Do the following testing to make sure the build works as expected...

  • Tests:
    • Try all the menu item functions
    • Gestures
    • Plug-in compatibility
      • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: PASS - 2012-06-25 - VladG
  • Details & Bugs:

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-24 PASS - 2012-06-25
Windows Vista PASS - 2012-06-24 PASS - 2012-06-25
Windows 7 32-bit PASS - 2012-06-24 PASS - 2012-06-25
Windows 7 64-bit PASS - 2012-06-24 PASS - 2012-06-25
Mac OS X 10.6 PASS - 2012-06-24 PASS - 2012-06-25
Mac OS 10.7 PASS - 2012-06-24 PASS - 2012-06-25
Ubuntu 11.10 32-bit PASS - 2012-06-24 PASS - 2012-06-25
Ubuntu 11.10 64-bit PASS - 2012-06-24 PASS - 2012-06-25
Issues

Bug Verifications

Add your name beside whatever you are testing so people don't duplicate your results.

  • bug 686335 - Bad plugin search order prevents upgrading Flash without an OS restart
  • bug 758363 - Implement NPN_ReloadPlugins for out-of-process Plugins
NOTE that testing any of these in a VM requires the VM to be run in Protected Mode

My advice would be to take a snapshot of the VM before starting and rolling back before each test they can be run from a clean snapshot.

NOTE that not all of these plugin versions are available for Mac and Linux.

Just test with whatever old versions you can find and update to the latest. Report the version numbers in your results.

Tests

Flash 10.2 (blocklisted)

Flash 10.3 (enabled & disabled)

  • Download and install Flash 10.3
  • Load some Flash content in tabs before updating
  • Pave-over install Flash 11.3.300.262
  • Reload tabs with Flash
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.2 (enabled & disabled)

  • Download and install Flash 11.2
  • Load some Flash content in tabs before updating
  • Pave-over install Flash 11.3.300.262
  • Reload tabs with Flash
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 (enabled & disabled)

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)

  • Download and install Flash 11.2
  • Start Firefox and open a page with flash content
  • Download but do not install Flash 11.3 Auto-update Installer
  • Open a command line to the download folder and run:
    • install_flash_player.exe -install -iv 9
  • Reload the page with Flash content once the installer finishes
  • Uninstall Flash
  • Repeat the test, loading Flash content in a new tab after update
  • Repeat the test, loading Flash content in a new window after update

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)

  • Download and install Flash 11.2
  • Start Firefox and open a page with flash content
  • Download but do not install Flash 11.3 Auto-update Installer
  • Open a command line to the download folder and run:
    • install_flash_player.exe -install -iv 9
  • Reload the page with Flash content once the installer finishes
  • Uninstall Flash
  • Repeat the test, loading Flash content in a new tab after update
  • Repeat the test, loading Flash content in a new window after update

Java 6u31 (enabled & disabled)

  • Download and install Java 6u31
  • Load some Java content in tabs before updating
  • Pave-over install Java 7u5
  • Reload tabs with Java content after updating
  • Uninstall Java and retest with Java disabled before updating

Java 7u2 (enabled & disabled)

  • Download and install Java 7u2
  • Load some Java content in tabs before updating
  • Pave-over install Java 7u5
  • Reload tabs with Java content after updating
  • Uninstall Java and retest with Java disabled before updating

Results

Expected

  • Plugin updates without any problems
  • Latest version of the plugin is listed in about:plugins
  • Content loads (or reloads) after update without crashing
  • NOTE: Multiple versions may be listed in about:plugins
    • OKAY as long as plugin content is using the latest version
    • Multiple entries can be cleared by restarting the OS

Windows 7 (ashughes)

  • PASS Flash 10.2 -> 11.3.300.262 (blocklisted) via pave-over
  • PASS Flash 10.3 -> 11.3.300.262 (enabled) via pave-over
  • PASS Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
  • PASS Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
  • PASS Flash 11.3 -> 11.3.300.262 (disabled) via auto-update
  • PASS Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
  • PASS Java 6u31 (disabled) -- requires Firefox restart
  • PASS Java 7u2 (enabled)

Windows XP (ashughes)

  • PASS Update Flash 10.2 -> 11.3.300.262 (disabled) via pave-over
  • PASS Update Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
  • PASS Update Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
  • PASS Update Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
  • PASS Update Java 6u31 -> 7u5 (disabled)
  • PASS Update Java 7u2 -> 7u5 (enabled)

Mac OSX 10.6 (ashughes)

  • PASS Update Flash 10.2 -> 11.3.300.257 (disabled) via pave-over
  • PASS Update Flash 11.2 -> 11.3.300.257 (enabled) via pave-over
  • PASS Update Flash 11.2 -> 11.3.300.257 (dom.ipc.plugins.enabled=false) via pave-over
  • result Update Java 6u* -> 7u5 (disabled) via Software Update
  • result Update Java 7u* -> 7u5 (enabled) via Software Update
  • Flash:
    • Flash installer forces Firefox to quit before updating
    • Disabling before update does not automatically enable after update

Ubuntu 12.04 (ashughes)

  • PASS Flash 11.2.205.233 -> 11.2.205.236 (enabled) via pave-over
  • PASS Flash 11.2.205.233 -> 11.2.205.236 (disabled) via pave-over
  • PASS Flash 11.2.205.233 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
  • result Update Oracle Java 6u* -> 7u5 (disabled) via pave-over
  • result Update IcedTea Java Plugin (enabled) via Software Update
  • Flash Plugin:
    • Overwriting when Firefox is running will result in an expected recoverable plugin crash
    • Disabling a plugin before update does not automatically re-enable after pave-over update
  • Java: As of yet unable to find a reliable way to install/update an old version (Ubuntu only provides the latest)

Known Issues

  • Windows: Disabled Java requires restart of Firefox to re-enable after update
  • Side-by-Side Flash versions in about:config when paving over Flash 11.3.300.257 64-bit with 11.3.300.262 32-bit; resolved only by OS restart; does not seem to affect functionality/stability
    • NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
  • Side-by-Side Flash versions in about:config when simulating 11.2->11.3 auto-update with install_flash_player.exe -install -iv 9
    • NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
  • Hang/crash when watching Youtube after pave-over update (bug 686335#c63) -- not reproducible using auto-update

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 10

Builds FTP, HTTP
ID 20120628060610
Changeset e4445f905091
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-28
Windows Vista PASS - 2012-06-28
Windows 7 32-bit PASS - 2012-06-28
Windows 7 64-bit PASS - 2012-06-28
Mac OS X 10.6 PASS - 2012-06-28
Mac OS X 10.7 PASS - 2012-06-28
Ubuntu 11.10 32-bit PASS - 2012-06-28
Ubuntu 11.10 64-bit PASS - 2012-06-28
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Simona PASS - 2012-06-28

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below. Results of manual testing (details in exploratory etherpad):

Platform Assignee Result: report
Windows 7 32-bit Simona PASS - 2012-06-29
Mac OS X 10.6.8 Vlad PASS - 2012-06-29
Ubuntu 11.10 64-bit Virgil PASS - 2012-06-29
Windows 7 64-bit Paul PASS - 2012-06-29

Exploratory Testing

[DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Opt-in activation for plugins Paul Silaghi RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-06-28 RESULT - 2012-06-28 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Mac OS 10.8 Signing VladG n/a PASS - 2012-06-29 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-06-29
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-06-29
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-06-29
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-06-29
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-06-29
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-06-29
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-06-29

Spot check build after this test.

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
    • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
  • Top-site compatibility
  • Add-on compatibility
  • Software update, especially with rich profiles and fallback updates (use betatest channel)

Results: PASS - 2012-06-29 - VladG

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-28 RESULT - 2012-06-28
Windows Vista PASS - 2012-06-28 RESULT - 2012-06-28
Windows 7 32-bit PASS - 2012-06-28 RESULT - 2012-06-28
Windows 7 64-bit PASS - 2012-06-28 RESULT - 2012-06-28
Mac OS X 10.6 PASS - 2012-06-28 RESULT - 2012-06-28
Mac OS 10.7 PASS - 2012-06-28 RESULT - 2012-06-28
Ubuntu 11.10 32-bit PASS - 2012-06-28 RESULT - 2012-06-28
Ubuntu 11.10 64-bit PASS - 2012-06-28 RESULT - 2012-06-28
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 31 8 79%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 11

Builds FTP, HTTP
ID 20120704090211
Changeset 48842c93fcb2
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [ON TRACK] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-07-04
Windows Vista PASS - 2012-07-04
Windows 7 32-bit PASS - 2012-07-04
Windows 7 64-bit PASS - 2012-07-04
Mac OS X 10.6 PASS - 2012-07-04
Mac OS X 10.7 PASS - 2012-07-04
Ubuntu 11.10 32-bit PASS - 2012-07-04
Ubuntu 11.10 64-bit PASS - 2012-07-04
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-07-05

Results in Litmus

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Opt-in activation for plugins Paul Silaghi RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-07-04 RESULT - 2012-07-04 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Mac OS 10.8 Signing Required n/a RESULT - 2012-07-04 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-05
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-05
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-05
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-05
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-05
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-05
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-04-05

Spot check build after this test: PASS - VladG - 2012-04-05

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP RESULT - 2012-07-04 RESULT - 2012-07-04
Windows Vista RESULT - 2012-07-04 RESULT - 2012-07-04
Windows 7 32-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Windows 7 64-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Mac OS X 10.6 RESULT - 2012-07-04 RESULT - 2012-07-04
Mac OS 10.7 RESULT - 2012-07-04 RESULT - 2012-07-04
Ubuntu 11.10 32-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Ubuntu 11.10 64-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 31 8 79%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Beta 12

Builds FTP, HTTP
ID 20120710123126
Changeset TBD
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-07-10
Windows Vista PASS - 2012-07-10
Windows 7 32-bit PASS - 2012-07-10
Windows 7 64-bit PASS - 2012-07-10
Mac OS X 10.6 PASS - 2012-07-10
Mac OS X 10.7 PASS - 2012-07-10
Ubuntu 11.10 32-bit PASS - 2012-07-10
Ubuntu 11.10 64-bit PASS - 2012-07-10
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows 7 32-bit Simona PASS - 2012-07-11

Results in Litmus

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Opt-in activation for plugins Paul Silaghi RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-07-10 RESULT - 2012-07-10 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Mac OS 10.8 Signing Required n/a PASS - 2012-07-11 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-11
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-11
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-11
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-11
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-11
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-11
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-11

Spot check build after this test - PASS - VladG - 2012-07-11

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
    • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-11 PASS - 2012-07-11
Windows Vista PASS - 2012-07-11 PASS - 2012-07-11
Windows 7 32-bit PASS - 2012-07-11 PASS - 2012-07-11
Windows 7 64-bit PASS - 2012-07-11 PASS - 2012-07-11
Mac OS X 10.6 PASS - 2012-07-11 PASS - 2012-07-11
Mac OS 10.7 PASS - 2012-07-11 PASS - 2012-07-11
Ubuntu 11.10 32-bit PASS - 2012-07-11 PASS - 2012-07-11
Ubuntu 11.10 64-bit PASS - 2012-07-11 PASS - 2012-07-11
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 36 3 92%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Firefox 14.0.1 Release Candidates

Builds FTP, HTTP
ID 20120713134347
Changeset e5728a4e106c
Checklist
  • [DONE] Functional automated tests
  • [DONE] Manual tests
  • [DONE] Bug fix verifications
  • [ON TRACK] Security fix verifications
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off
  • [ON TRACK] Update tests on release

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-07-15
Windows Vista PASS - 2012-07-15
Windows 7 32-bit PASS - 2012-07-15
Windows 7 64-bit PASS - 2012-07-15
Mac OS X 10.6 PASS - 2012-07-15
Mac OS X 10.7 PASS - 2012-07-15
Ubuntu 11.10 32-bit PASS - 2012-07-15
Ubuntu 11.10 64-bit PASS - 2012-07-15
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows 7 32-bit Mihaela PASS - 2012-07-16
Mac OS X 10.7.4 Ioana PASS - 2012-07-16
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-16

Results in Litmus

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, new features, web compatibility etc Etherpad

Feature Testing

By now we should have all features signed off.

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-16
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-16
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-16
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-16
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-16
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-16
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-16

Spot check build after this test.

Try all the menu item functions - PASS - VladG - 2012-07-16

  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-15 RESULT - 2012-07-15
Windows Vista PASS - 2012-07-15 RESULT - 2012-07-15
Windows 7 32-bit PASS - 2012-07-15 RESULT - 2012-07-15
Windows 7 64-bit PASS - 2012-07-15 RESULT - 2012-07-15
Mac OS X 10.6 PASS - 2012-07-15 RESULT - 2012-07-15
Mac OS 10.7 PASS - 2012-07-15 RESULT - 2012-07-15
Ubuntu 11.10 32-bit PASS - 2012-07-15 RESULT - 2012-07-15
Ubuntu 11.10 64-bit PASS - 2012-07-15 RESULT - 2012-07-15
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining % Complete
QA+: for testers to verify functional fixes 36 3 92%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1


Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_14/Test_Plan&oldid=451273"