View source for QA/MergeCompatibilityTesting

= Merge Compatibility Testing =

== Introduction ==

The fast release cycle has proven to be challenging to validate platform and web compatibility from release to release. Regressions are more likely to occur undetected than in the prior longer release model prior to shipping Firefox 5. To get a better assessment of Firefox compatibility, QA in conjunction with our testing community and with outsourced testing partners have been building up experience and expertise in conducting both web and platform compatibility test runs.

== Objective ==

The objecting of the Merge Compatibility Project is to perform a comprehensive set of regression tests for both web and platform compatibility across the current Firefox release channels. Web compatibility refers to crowd-sourced manual tests of the most visited global web sites and perform user based scenario interactions. The selected web sites are grouped by functionality. For instance if it is a financial site, the objective would be for a tester to log into to the site a perform an actual transaction and verify the results of the transaction. If it is a music site, the tester should select and play an audio file on the site and so on. The objective is  detect deeper issues that may arise for a realistic user interaction with the sites features, pages and capabilities.

The objective of platform compatibility is to ensure Firefox behaves correct across the spectrum of supported hardware and platform configurations. Historically we have relied on our test community to provide the coverage of testing across the matrix of hardware platforms and supported OSes. This reliance has proven to lack completeness and therefore requires a different approach. The proposed approach is to use an outsource vendor ([http://www.ibeta.com Ibeta]) to supply the machines and configurations which to run graphics and targeted OS compatibility tests.

== Implementation ==
The merge compatibility testing will be conducted on each release channel (nightly, aurora and beta) 3-4 weeks following a channel software merge event. The compatibility testing is broken down into two distinct focus areas. The first area is web compatibility testing using a crowd-sourced approach.  

== Web Compatibility Testing ==
;Desktop Firefox Web Compatibility 2011-08-16
* [https://docs.google.com/spreadsheet/ccc?key=0AvdU4bzekK2qdDRyRTBIWGtJNTFuNm1sd2FjYi01RXc&hl=en_US#gid=30 results spreadsheet]
* [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=utest-fx&columnlist=bug_severity%2Cpriority%2Cop_sys%2Cassigned_to%2Cbug_status%2Cresolution%2Cshort_desc%2Cstatus_whiteboard%2Copendate&list_id=1611009 reported bugs]

;Desktop Firefox Web Compatibility 2011-10-25
* [https://docs.google.com/spreadsheet/ccc?key=0AvdU4bzekK2qdFJCQ2VOY0V4WGhnZ2gyZ1MwLW9lTXc&authkey=CICAkvcH&pli=1#gid=29 results spreadsheet]

;Desktop Firefox Web Compatibility 2011-12-05
* [https://docs.google.com/spreadsheet/ccc?key=0AvdU4bzekK2qdG1LaHZFbEFmQUdITTh3MzU0aUw3VHc#gid=17 results spreadsheet]

== Platform Compatibility Testing ==

=== 12-02-2011 Graphics and Platform Compatibility and Blocklist Testing ===

;Summary

For the Firefox 9 timeframe we will be conducting graphics and platform compatibility testing as well as testing the graphics Firefox blocklisting feature due to recent changes in the feature implementation. 

;Objectives
# Verify recently updated blocklist feature is working correctly
# Verify blocklist feature is working with a selected set of real hardware platforms and graphics configurations.
# Perform graphics compatibility tests across all current channels (Firefox 9,10,11) and performed on a selectewd set of real hardware platforms and graphics configurations.
# Experiment with 3rd party software, plugin and binary addons configurations for testing firefox operability.

<b>The following tests will be conducted in priority order:</b>

{| class="fullwidth-table"
 | style="background:#EFEFEF; width:25%" | '''Test Description'''
 | style="background:#EFEFEF; width:25%" | '''Test Plan'''
 | style="background:#EFEFEF; width:25%" | '''Test Results'''
 | style="background:#EFEFEF; width:25%" | '''Notes'''
 |-
 |<b>Graphics & 3rd party Software Compatibility Tests for Firefox 9</b>
 | [https://wiki.mozilla.org/QA/MergeCompatibilityTesting/GraphicsCompatibilityTest-12-2011 compatibility test plan]
 | Add test results
 |
 |- 
 |<b>Blocklist testing for Nightly Channel(Firefox 11)</b>
 | [https://wiki.mozilla.org/QA/MergeCompatibilityTesting/BlocklistTest-12-2011 blocklist test plan]
 | Add test results
 |Doug Sherk may suppy a try-build version for IBeta to use for Nighly (mozilla-central). We will use this version for nightly graphics compatibility testing
 |-
 |<b>Graphics & 3rd party Software Compatibility Tests for Firefox 10</b>
 | [https://wiki.mozilla.org/QA/MergeCompatibilityTesting/GraphicsCompatibilityTest-12-2011 compatibility test plan]
 | Add test results
 |
 |-
 |<b>Graphics & 3rd party Software Compatibility Tests for Firefox 11</b>
 | [https://wiki.mozilla.org/QA/MergeCompatibilityTesting/GraphicsCompatibilityTest-12-2011 compatibility test plan]
 | Add test results
 |
 |}

== Results ==

== Links and Resources ==

* [https://mozqa.sync.in/ibeta-firefox-compatibility-testing on going meeting etherpad]
* [https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers]
* [https://wiki.mozilla.org/Blocklisting/Testing https://wiki.mozilla.org/Blocklisting/Testing]
* [https://docs.google.com/spreadsheet/ccc?key=0Aj-5ZZLSQ6mrdHZOSTl4ZE1tNkFfeThHUlNPckRrRHc&hl=en_US#gid=1 Machine Configuration Matrix]
* [https://docs.google.com/spreadsheet/ccc?key=0Aj-5ZZLSQ6mrdEdBWUlvdkNGUU1kQTRNUUM3TWZWUEE&hl=en_US#gid=0 Merge Compatibility Test Definitions]
* [http://people.mozilla.org/~jgriffin/grafxbot-9.0.00.xpi http://people.mozilla.org/~jgriffin/grafxbot-9.0.00.xpi]
* [https://docs.google.com/spreadsheet/ccc?key=0Al4xZaJkos_adElPQ3ZIbWcycS1IVHFKWFd0UkxpMGc&hl=en_US#gid=0 Third Party Crash Landscape spreadsheet]
* [https://tbpl.mozilla.org/?tree=Try&rev=6b1a6ab681a6 Doug's try build 1]