MozillaWiki

QA/Firefox3/Strategy: Difference between revisions

Page Discussion

QA/Firefox3/Strategy (view source)

Revision as of 18:49, 10 January 2008

5,375 bytes added ,  10 January 2008
→‎Test Tools
 
(33 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{draft}}
<small>[[QA/Firefox3|&laquo; QA/Firefox3]]</small>


= QA Test Strategy – Firefox =
= QA Test Strategy – Firefox =
Line 5: Line 5:
== Overview ==
== Overview ==


This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3.  It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy.  Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, Green test pass on test cases in both execution and automation, and a thorough bug verification window of all Pri 1 and top Pri 2 tier bugs.
This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3.  It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy.  Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, 'Green'-colored test pass on test cases in both execution and automation, and a thorough bug verification window of all Blocking bugs and Critical bugs.
 
NOTE: Updates to this document continue to change as Mozilla continues to refine development process.


== Areas Covered ==
== Areas Covered ==
* Regression Tests
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Feature_Test_Deliverables Featured Areas Tests]
* Featured Areas Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Web_Compatibility_Test_Deliverables Web Compatibility Tests ]
* Web Compatibility Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#User_Performance_Test_Deliverables User Performance Tests ]
* User Performance Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Configurations_Test_Deliverables Configurations Tests ]
* Configurations Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Security_Test_Deliverables Security Tests ]
* Security Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Accessibility_Tests_Deliverables Accessibility Tests ]
* Stress Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Bug_Verifications_Deliverables Bug Verifications ]
* Accessibility Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Distribution_Test_Deliverables Distribution Tests ]
* Bug Verifications Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#L10N_Test_Deliverables L10N Test Deliverables]
* Distribution Tests Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Updates_Test_Deliverables Updates Test ]
* L10N Test Deliverables
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Regression_Tests Regression Tests]
* Updates Test Deliverables


== Areas not Covered ==
== Areas not Covered ==


* Feature Unit Tests
* Feature Unit Tests  
* String Localization Tests
** These are normally covered by feature developers, not QA.  However, we do have a Javascript test suite that will collectively find bugs with in-testsuite javascript test cases that Bob Clary will run on occasion.  Need more definition here on what QA/Dev should own.
* In depth functionality of Plugins, Extensions, and Themes
 
* String Localization Tests  
** Many of these tests are owned by localizers and SmartWare, our third party testing company.  See more in the L10n Test Deliverables section.
 
* In depth functionality of Plugins, Extensions, and Themes  
** Many extensions are hosted on AMO, but are not developed and tested by mozilla.  We rely on our volunteers to manage these third party extensions.  Mozilla QA will focus on high level Addon management, error handling, and a few top extensions supported.  But extension functionality will not be covered by Mozilla QA.


== Schedule ==
== Schedule ==
Line 34: Line 40:


== Test Tools ==  
== Test Tools ==  
This is a comprehensive set of Test Tools used by the QA team for automation and execution of the Firefox Product. Some of these tools are developed in-house, and are only accessible within the internal Mozilla network.
This is ongoing set of Test Tools used by the QA team for automation and execution of the Firefox Product. Some of these tools are developed in-house, and are only accessible within the internal Mozilla network.  The following list of tools shown here contain a short description of the tool, as well as further explanation on how the relevant tools are used for specific Test Areas below.


* [http://litmus.mozilla.org/ Litmus Test Case Manger] -  
* [http://litmus.mozilla.org/ Litmus Test Case Manger] -  
Line 48: Line 54:
** Bob Clary again.  Takes a huge list of locales and OS installs.  highly error prone.  Script will go to webpage, download each build, installs, runs a test, and closes it. fully automated.  This is executed only after we go live.    (all.html - "other system and languages")
** Bob Clary again.  Takes a huge list of locales and OS installs.  highly error prone.  Script will go to webpage, download each build, installs, runs a test, and closes it. fully automated.  This is executed only after we go live.    (all.html - "other system and languages")


* L10n search verifier -  
* <strike> L10n search verifier - </strike> ''This has been replaced by Minotaur tool. (see below)'' <strike>
** Don't want to break localized products that may have mistakes.  Search engines, bookmarks, RSS feeds that are pre-configured.  Idea is to catch changes before it's checked in.  Diffs the region.properties settings for each locale against rules defined in the l10n requirements.  (http://wiki.mozilla.org/Firefox2/L10n_Requirements)
** Don't want to break localized products that may have mistakes.  Search engines, bookmarks, RSS feeds that are pre-configured.  Idea is to catch changes before it's checked in.  Diffs the region.properties settings for each locale against rules defined in the l10n requirements.  (http://wiki.mozilla.org/Firefox2/L10n_Requirements)</strike>


* Performance testing -
* [http://quality.mozilla.org/projects/automation/talos Talos] - Performance Testing Project
** TP (page load time), TS (page start time). Ran against performance machines and records metrics and performance trends.
** Talos is a performance testing project. With a framework written in Python it runs Ts (startup test) and Tp (page load test) while monitoring memory and cpu usage. Ts is a simple Javascript web page that times the loading and then exits. Tp consists of a Javascript web page that cycles through a set of locally served web pages. The web page set is a collection of the top 500 web pages as listed by Alexa; they have been 'cleaned' so that all of their content is locally served.


* [https://intranet.mozilla.org/QA#Eggplant Eggplant] -  
* [https://intranet.mozilla.org/QA#Eggplant Eggplant] -  
Line 63: Line 69:
** test system for building.  owned by Robcee.  client / server interaction.  Client requests execution activities.  Server sets tests up, run results.  (eg. parallel make system for builds.  able to build pieces independently and can piece it together later)  Like a next-gen tinderbox.
** test system for building.  owned by Robcee.  client / server interaction.  Client requests execution activities.  Server sets tests up, run results.  (eg. parallel make system for builds.  able to build pieces independently and can piece it together later)  Like a next-gen tinderbox.


* [http://wiki.mozilla.org/User:Ctalbert:Minotaur Minotaur] -  
* [http://developer.mozilla.org/en/docs/Mozilla_automated_testing#Reftest Reftests] -
** Unit test framework, easy to run in the browser, easy to write. The testcases don't rely on external files. Mainly useful for layout tests, only.
 
* [http://quality.mozilla.org/projects/automation/minotaur Minotaur] -  
** Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox.
** Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox.


== Deliverables ==
== Categories ==


=== Feature Test Deliverables ===
=== Feature Tests ===


;Summary
;Summary
Line 79: Line 88:


We will track the design, landing, test plan development, test case development, and running of the test cases for the initial feature testing via this [http://spreadsheets2.google.com/ccc?key=p-9pXWWpA7Bih6yFR9FYQWw&hl=en_US tracking spreadsheet].
We will track the design, landing, test plan development, test case development, and running of the test cases for the initial feature testing via this [http://spreadsheets2.google.com/ccc?key=p-9pXWWpA7Bih6yFR9FYQWw&hl=en_US tracking spreadsheet].
Current drafts of the '''Firefox 3 Feature Test Plans''' are listed [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan#Feature_Focused_Areas here] under "Test Plan Location" column.


The expectation is that the major features are well tested during the Alpha phase or maybe B1 at the latest.  Then testing during later betas is focused on verifying bug fixes, testing  additional edge cases that are identified (polish testing), and regression testing using smoke tests, BFTs, FFTs and automated runs.
The expectation is that the major features are well tested during the Alpha phase or maybe B1 at the latest.  Then testing during later betas is focused on verifying bug fixes, testing  additional edge cases that are identified (polish testing), and regression testing using smoke tests, BFTs, FFTs and automated runs.
Line 97: Line 108:


;Results
;Results
Test Results for executed test cases will be tracked within the litmus tests.  For automation results, unit tests that are created in mochikit will be ran on tinderboxes
Test Results for executed test cases will be tracked within the litmus tests.  For automation results, unit tests that are created in mochikit will be ran on tinderboxen.  Other test results


=== Web Compatibility Test Deliverables ===
=== Web Compatibility Tests ===


;Summary
;Summary
The Web Compatibility Test Deliverables focuses on regression and compatibility with popular websites and web applications.   
The Web Compatibility Tests focuses on regression and compatibility with popular websites and web applications.   


;Planning & Scheduling
;Planning & Scheduling
Line 108: Line 119:


;Test Framework
;Test Framework
* [http://www.alexa.com top sites]
* [http://www.alexa.com Top Sites] - coverage of a few popular sites that drives most user traffic
* Top extensions (see shaver list)
* [http://wiki.mozilla.org/Firefox:2.0_QA_Activities:Extension_Results Top extensions] - coverage of Top firefox extensions used by the public
* Top plugins
* Top themes - coverage of Top firefox themes used by the public
* Top themes
* Top plugins - coverage of Top web plugins used by the public
* Protocol handling
* Protocol handling - regression of a popular list of protocols supported on the web.  (eg. mailto://, https://, ftp://) Ties into [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=56&subgroup_id=820 litmus content handling] test cases. 


;Results
;Results
Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus.
Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus.


=== User Performance Test Deliverables ===
=== User Performance Tests ===


;Summary
;Summary
The User Performance Test Deliverables is not to be confused with Browser performance tests that are run with automation.  Instead, we will concentrate on day to day usage of Firefox and any performance issues that we see.  These would includes areas like downloading, idling, and other Firefox daily usage cases.  It will also incorporate some failover scenarios that would add a few stress cases and error handling cases.  
The User Performance Tests are not to be confused with Browser performance tests that are run with automation.  Instead, we will concentrate on day to day usage of Firefox and any performance issues that we see.  These would includes areas like downloading, idling, and other Firefox daily usage cases.  It will also incorporate some failover scenarios that would add a few stress cases and error handling cases.  


;Planning & Scheduling
;Planning & Scheduling
Line 135: Line 146:
Test cases will exist in litmus and the public wiki.  Any issues found will be tracked in Bugzilla.
Test cases will exist in litmus and the public wiki.  Any issues found will be tracked in Bugzilla.


=== Configurations Test Deliverables ===
=== Configurations Tests ===
;Summary
;Summary
The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other.  This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine.   
The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other.  This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine.   
Line 151: Line 162:
Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus.  Virtual Machines are tracked on the file server and available for download with VMWare licensing.
Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus.  Virtual Machines are tracked on the file server and available for download with VMWare licensing.


=== Security Test Deliverables ===
=== Security Tests ===
;Summary
;Summary
Security tests are critical for catching external attacks through Firefox to the user.  Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox.  The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again.
Security tests are critical for catching external attacks through Firefox to the user.  Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox.  The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again.
Line 168: Line 179:
Test results will be tracked via automation through eggplant and jesse's fuzzer tools.  There are a few security tests also tracked in litmus to be ran manually.  Also a handful of security bugs in Bugzilla have test cases to be run independently.  (Example: [https://bugzilla.mozilla.org/show_bug.cgi?id=389106 Bug 389106])
Test results will be tracked via automation through eggplant and jesse's fuzzer tools.  There are a few security tests also tracked in litmus to be ran manually.  Also a handful of security bugs in Bugzilla have test cases to be run independently.  (Example: [https://bugzilla.mozilla.org/show_bug.cgi?id=389106 Bug 389106])


=== Stress Tests Deliverables ===
=== Accessibility Tests ===
* Fail-over Litmus scenarios (e.g. Pulling out the plug on download)
 
=== Accessibility Tests Deliverables ===
;Summary
;Summary


Line 186: Line 194:
Test cases will exist in Litmus and the public wiki.  Any issues found will be tracked in Bugzilla.
Test cases will exist in Litmus and the public wiki.  Any issues found will be tracked in Bugzilla.


=== Bug Verifications Deliverables ===
=== Bug Verifications ===
;Summary
;Summary
New code is landing every night into the tree, and current functionality is constantly being update and overridden.  Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups.   
New code is landing every night into the tree, and current functionality is constantly being update and overridden.  Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups.   
Line 202: Line 210:
Bugs will be marked verified or reactivated for regression.  All results tracked in Bugzilla.
Bugs will be marked verified or reactivated for regression.  All results tracked in Bugzilla.


=== Distribution Test Deliverables ===
=== Distribution Tests ===
;Summary
;Summary
Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds.  Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release.  This is to preserve the validation and regression of the product.  
Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds.  Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release.  This is to preserve the validation and regression of the product.  
Line 216: Line 224:
Results are tracked on the intranet wiki page and reported to the support group for Firefox partners.
Results are tracked on the intranet wiki page and reported to the support group for Firefox partners.


=== L10N Test Deliverables ===
=== L10N Tests ===
;Summary
;Summary
The L10N Test Suite will prepare a set of standard Firefox testing requirements.  Its purpose is to ensure that third party localizers are properly checking the validity of the build before checking in to the tree.  It also will provide a quick automated check of configurations and values, and report results.   
The L10N Tests will prepare a set of standard Firefox testing requirements.  Its purpose is to ensure that third party localizers are properly checking the validity of the build before checking in to the tree.  It also will provide a quick automated check of configurations and values, and report results.   


;Planning & Scheduling
;Planning & Scheduling
String freeze for Firefox 3 will be targeting a time frame between B1 and B2.  Test Execution will be developing a few test cases that they determine would be important to be checked during localization testing in their main Firefox 3 features.  A customized l10n test suite will be created for the localizer to run on private builds prior to checkin.   
String freeze for Firefox 3 will be targeting a time frame between B1 and B2.  Test Execution will be developing a few test cases that they determine would be important to be checked during localization testing in their main Firefox 3 features.  A customized l10n test suite will be created for the localizer to run on private builds prior to checkin.   
The prioritized list of locales for Firefox 3 is listed [http://wiki.mozilla.org/Firefox3/Firefox_Requirements#Locale_Support here].


Test Automation will be running through a tool that will effectly validate customizations and configurations.   
Test Automation will be running through a tool that will effectly validate customizations and configurations.   


A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales.  The localizer is also responsible for unit tests prior to checkin.  
A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales.  The localizer is also responsible for unit tests prior to checkin. Click to see the existing [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:L10N_Test_Plan L10N Test Plan].


;Test Framework
;Test Framework
Line 239: Line 249:
Test results are tracked in different forms.  The localizer will track their unit cases and litmus for the custom l10n test suite.  Test execution team will report results in Litmus.  Test automation will track results at ''<insert location here>''.  Smartware will also track results at ''<insert location here>''.
Test results are tracked in different forms.  The localizer will track their unit cases and litmus for the custom l10n test suite.  Test execution team will report results in Litmus.  Test automation will track results at ''<insert location here>''.  Smartware will also track results at ''<insert location here>''.


=== Updates Test Deliverables ===
=== Updates Tests ===
;Summary
;Summary
Update Testing is the process of verifying minor security patches and major version releases from a prior version.  The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links.
Update Testing is the process of verifying minor security patches and major version releases from a prior version.  The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links.
Line 254: Line 264:
Results are reported via wiki and email.  There are also live updates tests against the production server after it has been released.
Results are reported via wiki and email.  There are also live updates tests against the production server after it has been released.


== Reporting ==
== Collecting Feedback ==
There are multiple ways we report results and feedback for Firefox releases.  The goal of reporting results helps QA track issues found in development, regression, and post-release feedback.
There are multiple ways we collect comments and feedback for Firefox releases.  The goal of feedback helps QA track issues found in development, regression, and post-release analysis.


;Reporting tools:
;Reporting tools:
Line 274: Line 284:


;Manual framework:
;Manual framework:
* Smoketests - A subset of litmus tests ran to ensure the quality of a nightly build.  Executed daily.
* [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=54&subgroup_id= Smoketests] - A subset of litmus tests ran to ensure the quality of a nightly build.  Executed daily.
* BFTs - Basic Functional Tests. A more comprehensive subset of litmus tests that cover more regression than smoketests, but not a full suite.  Executed once a week.
* [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=55&subgroup_id= Basic Functional Tests] - A more comprehensive subset of litmus tests that cover more regression than smoketests, but not a full suite.  Executed once a week.
* FFTs - Full Functional Tests. A thorough set of litmus tests that cover each and every test case that is created to cover full functional regression and feature functionality.  Executed at least once per release.
* [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=56&subgroup_id= Full Functional Tests] - A thorough set of litmus tests that cover each and every test case that is created to cover full functional regression and feature functionality.  Executed at least once per release.
* Eggplant Cases - An automated independent tool that takes the smoketests across all platforms and executes them on demand.   
* Eggplant Cases - An automated independent tool that takes the smoketests across all platforms and executes them on demand.   
* Focused feature areas across supported Firefox platforms  
* Focused feature areas across supported Firefox platforms  
Line 296: Line 306:


== Community Involvement ==
== Community Involvement ==
The goal is to actively involve more community members in testing Minefield, Gran Paradiso, and ultimately Firefox 3 as it moves into beta and to final release.
There are many active participants in the community that are enthusiastic about Firefox and want to contribute to our quality assurance efforts.  However, getting them involved and maximizing the value of their time and effort requires a lot of coordination, planning, and enforcing of the correct processes and procedures.  Since such activities create constant challenges for our QA team,  we are working on a new strategy to promote more community involvement. 
Current avenues of community involvement include:
* Friday Test Days
** increased the frequency of these events to allow people to show up each week to try out new features and help us run through test cases.  Activities include triaging unconfirmed bugs, running tests in Litmus with trunk builds (and alphas/betas), Q&A/Help sessions.  We will be focusing on a new Firefox 3 feature/component every week.
* Tuesday Bug Days
** a full day each week dedicated to working with various bug lists in Bugzilla.  Activities will rotate between unconfirmed bug triage, bug verifications, test case writing, and reproducing crash bugs.


This goal is to actively involve more community members to assist with testing the firefox project in different aspects of the area.   There are many active participants in the community that are enthusiastic with Firefox and want to give back their time and effort to help test this project.  However, this involves heavy coordination, planning of work areas, and enforcing the correct process and procedures constantly produces challenges. There is a seperate community team in QA that is currently working on incorporating a new strategy to promote more community involvement.  
* QMO - [http://quality.mozilla.org quality.mozilla.org]
** Using the website to promote news and articles on what the Mozilla QA team is working on
** Provide events and forums for the community to get involved.
** Create closer relationships with regular contributors and help them become leaders for our events and projects.
** Promote the [http://wiki.mozilla.org/Litmus:Extension QA Extension] to get more people connected to our tools and processes.  We will be setting up a "product page" on QMO and start the process for getting it polished enough for AMO.
** Update/migrate outdated QA pages on mozilla.org and showcase them on QMO
* Campus Evangelism - [https://www.mozilla.com/en-US/university/ Mozilla University]
** Leverage the Campus Reps program to reach out to students and professors at schools.
** Reach out to folks interested in open source and help them discover Mozilla project and get involved in Firefox development and testing.


Current avenues of community involvement includes:
* Better "Beta" product pages and programs
* friday testdays - a full day dedicated to testing firefox and thunderbird.  Activities include triage unconfirmed bugs, running tests in litmus, QnA, testing the trunk builds, and many other firefox-related topics
** Update [http://www.mozilla.org/projects/minefield/ Minefield] and [http://www.mozilla.org/projects/granparadiso/ Gran Paradiso] pages to direct folks to QMO and other useful channels for getting involved and providing feedback.
* tuesday bugdays - a full day dedicated to triaging bugs unconfirmed and confirmed.
** Coordinate community testing with marketing/pr plans for Firefox 3 Beta release
* QMO - [quality.mozilla.org quality.mozilla.org]: website to promote news and articles on mozilla QA.
* Campus evangelism - [https://www.mozilla.com/en-US/university/ Targeting the university students] with an exposure to testing open source and particularly firefox
* Others?
Mw22
Confirmed users
486

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/68078...79576"