Confirmed users
486
edits
QA/Firefox3/Strategy: Difference between revisions
→Test Tools
| (10 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
<small>[[QA/Firefox3|« QA/Firefox3]]</small> | |||
= QA Test Strategy – Firefox = | = QA Test Strategy – Firefox = | ||
| Line 5: | Line 5: | ||
== Overview == | == Overview == | ||
This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3. It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy. Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, Green test pass on test cases in both execution and automation, and a thorough bug verification window of all | This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3. It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy. Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, 'Green'-colored test pass on test cases in both execution and automation, and a thorough bug verification window of all Blocking bugs and Critical bugs. | ||
NOTE: Updates to this document continue to change as Mozilla continues to refine development process. | NOTE: Updates to this document continue to change as Mozilla continues to refine development process. | ||
== Areas Covered == | == Areas Covered == | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Feature_Test_Deliverables Featured Areas Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Feature_Test_Deliverables Featured Areas Tests] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Web_Compatibility_Test_Deliverables Web Compatibility Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Web_Compatibility_Test_Deliverables Web Compatibility Tests ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#User_Performance_Test_Deliverables User Performance Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#User_Performance_Test_Deliverables User Performance Tests ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Configurations_Test_Deliverables Configurations Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Configurations_Test_Deliverables Configurations Tests ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Security_Test_Deliverables Security Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Security_Test_Deliverables Security Tests ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Accessibility_Tests_Deliverables Accessibility Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Accessibility_Tests_Deliverables Accessibility Tests ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Bug_Verifications_Deliverables Bug Verifications | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Bug_Verifications_Deliverables Bug Verifications ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Distribution_Test_Deliverables Distribution Tests | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Distribution_Test_Deliverables Distribution Tests ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#L10N_Test_Deliverables L10N Test Deliverables] | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#L10N_Test_Deliverables L10N Test Deliverables] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Updates_Test_Deliverables Updates Test | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Updates_Test_Deliverables Updates Test ] | ||
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Regression_Tests Regression Tests] | * [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Regression_Tests Regression Tests] | ||
| Line 40: | Line 40: | ||
== Test Tools == | == Test Tools == | ||
This is | This is ongoing set of Test Tools used by the QA team for automation and execution of the Firefox Product. Some of these tools are developed in-house, and are only accessible within the internal Mozilla network. The following list of tools shown here contain a short description of the tool, as well as further explanation on how the relevant tools are used for specific Test Areas below. | ||
* [http://litmus.mozilla.org/ Litmus Test Case Manger] - | * [http://litmus.mozilla.org/ Litmus Test Case Manger] - | ||
| Line 68: | Line 68: | ||
* BuildBot - | * BuildBot - | ||
** test system for building. owned by Robcee. client / server interaction. Client requests execution activities. Server sets tests up, run results. (eg. parallel make system for builds. able to build pieces independently and can piece it together later) Like a next-gen tinderbox. | ** test system for building. owned by Robcee. client / server interaction. Client requests execution activities. Server sets tests up, run results. (eg. parallel make system for builds. able to build pieces independently and can piece it together later) Like a next-gen tinderbox. | ||
* [http://developer.mozilla.org/en/docs/Mozilla_automated_testing#Reftest Reftests] - | |||
** Unit test framework, easy to run in the browser, easy to write. The testcases don't rely on external files. Mainly useful for layout tests, only. | |||
* [http://quality.mozilla.org/projects/automation/minotaur Minotaur] - | * [http://quality.mozilla.org/projects/automation/minotaur Minotaur] - | ||
** Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox. | ** Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox. | ||
== | == Categories == | ||
=== Feature | === Feature Tests === | ||
;Summary | ;Summary | ||
| Line 107: | Line 110: | ||
Test Results for executed test cases will be tracked within the litmus tests. For automation results, unit tests that are created in mochikit will be ran on tinderboxen. Other test results | Test Results for executed test cases will be tracked within the litmus tests. For automation results, unit tests that are created in mochikit will be ran on tinderboxen. Other test results | ||
=== Web Compatibility | === Web Compatibility Tests === | ||
;Summary | ;Summary | ||
The Web Compatibility | The Web Compatibility Tests focuses on regression and compatibility with popular websites and web applications. | ||
;Planning & Scheduling | ;Planning & Scheduling | ||
| Line 125: | Line 128: | ||
Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus. | Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus. | ||
=== User Performance | === User Performance Tests === | ||
;Summary | ;Summary | ||
The User Performance | The User Performance Tests are not to be confused with Browser performance tests that are run with automation. Instead, we will concentrate on day to day usage of Firefox and any performance issues that we see. These would includes areas like downloading, idling, and other Firefox daily usage cases. It will also incorporate some failover scenarios that would add a few stress cases and error handling cases. | ||
;Planning & Scheduling | ;Planning & Scheduling | ||
| Line 143: | Line 146: | ||
Test cases will exist in litmus and the public wiki. Any issues found will be tracked in Bugzilla. | Test cases will exist in litmus and the public wiki. Any issues found will be tracked in Bugzilla. | ||
=== Configurations | === Configurations Tests === | ||
;Summary | ;Summary | ||
The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other. This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine. | The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other. This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine. | ||
| Line 159: | Line 162: | ||
Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus. Virtual Machines are tracked on the file server and available for download with VMWare licensing. | Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus. Virtual Machines are tracked on the file server and available for download with VMWare licensing. | ||
=== Security | === Security Tests === | ||
;Summary | ;Summary | ||
Security tests are critical for catching external attacks through Firefox to the user. Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox. The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again. | Security tests are critical for catching external attacks through Firefox to the user. Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox. The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again. | ||
| Line 176: | Line 179: | ||
Test results will be tracked via automation through eggplant and jesse's fuzzer tools. There are a few security tests also tracked in litmus to be ran manually. Also a handful of security bugs in Bugzilla have test cases to be run independently. (Example: [https://bugzilla.mozilla.org/show_bug.cgi?id=389106 Bug 389106]) | Test results will be tracked via automation through eggplant and jesse's fuzzer tools. There are a few security tests also tracked in litmus to be ran manually. Also a handful of security bugs in Bugzilla have test cases to be run independently. (Example: [https://bugzilla.mozilla.org/show_bug.cgi?id=389106 Bug 389106]) | ||
=== Accessibility Tests | === Accessibility Tests === | ||
;Summary | ;Summary | ||
| Line 191: | Line 194: | ||
Test cases will exist in Litmus and the public wiki. Any issues found will be tracked in Bugzilla. | Test cases will exist in Litmus and the public wiki. Any issues found will be tracked in Bugzilla. | ||
=== Bug Verifications | === Bug Verifications === | ||
;Summary | ;Summary | ||
New code is landing every night into the tree, and current functionality is constantly being update and overridden. Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups. | New code is landing every night into the tree, and current functionality is constantly being update and overridden. Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups. | ||
| Line 207: | Line 210: | ||
Bugs will be marked verified or reactivated for regression. All results tracked in Bugzilla. | Bugs will be marked verified or reactivated for regression. All results tracked in Bugzilla. | ||
=== Distribution | === Distribution Tests === | ||
;Summary | ;Summary | ||
Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds. Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release. This is to preserve the validation and regression of the product. | Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds. Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release. This is to preserve the validation and regression of the product. | ||
| Line 221: | Line 224: | ||
Results are tracked on the intranet wiki page and reported to the support group for Firefox partners. | Results are tracked on the intranet wiki page and reported to the support group for Firefox partners. | ||
=== L10N | === L10N Tests === | ||
;Summary | ;Summary | ||
The L10N | The L10N Tests will prepare a set of standard Firefox testing requirements. Its purpose is to ensure that third party localizers are properly checking the validity of the build before checking in to the tree. It also will provide a quick automated check of configurations and values, and report results. | ||
;Planning & Scheduling | ;Planning & Scheduling | ||
| Line 232: | Line 235: | ||
Test Automation will be running through a tool that will effectly validate customizations and configurations. | Test Automation will be running through a tool that will effectly validate customizations and configurations. | ||
A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales. The localizer is also responsible for unit tests prior to checkin. | A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales. The localizer is also responsible for unit tests prior to checkin. Click to see the existing [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:L10N_Test_Plan L10N Test Plan]. | ||
;Test Framework | ;Test Framework | ||
| Line 246: | Line 249: | ||
Test results are tracked in different forms. The localizer will track their unit cases and litmus for the custom l10n test suite. Test execution team will report results in Litmus. Test automation will track results at ''<insert location here>''. Smartware will also track results at ''<insert location here>''. | Test results are tracked in different forms. The localizer will track their unit cases and litmus for the custom l10n test suite. Test execution team will report results in Litmus. Test automation will track results at ''<insert location here>''. Smartware will also track results at ''<insert location here>''. | ||
=== Updates | === Updates Tests === | ||
;Summary | ;Summary | ||
Update Testing is the process of verifying minor security patches and major version releases from a prior version. The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links. | Update Testing is the process of verifying minor security patches and major version releases from a prior version. The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links. | ||
| Line 261: | Line 264: | ||
Results are reported via wiki and email. There are also live updates tests against the production server after it has been released. | Results are reported via wiki and email. There are also live updates tests against the production server after it has been released. | ||
== | == Collecting Feedback == | ||
There are multiple ways we | There are multiple ways we collect comments and feedback for Firefox releases. The goal of feedback helps QA track issues found in development, regression, and post-release analysis. | ||
;Reporting tools: | ;Reporting tools: | ||