MozillaWiki

QA/Firefox3/Strategy: Difference between revisions

Page Discussion

QA/Firefox3/Strategy (view source)

Revision as of 18:49, 10 January 2008

16,098 bytes added ,  10 January 2008
→‎Test Tools
 
(82 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{draft}}
<small>[[QA/Firefox3|&laquo; QA/Firefox3]]</small>


= QA Test Strategy – Firefox =
= QA Test Strategy – Firefox =
Line 5: Line 5:
== Overview ==
== Overview ==


This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3.  It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy.  Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, Green test pass on test cases in both execution and automation, and a thorough bug verification window of all Pri 1 and top Pri 2 tier bugs.
This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3.  It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy.  Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, 'Green'-colored test pass on test cases in both execution and automation, and a thorough bug verification window of all Blocking bugs and Critical bugs.
 
NOTE: Updates to this document continue to change as Mozilla continues to refine development process.


== Areas Covered ==
== Areas Covered ==
* Featured Areas Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Feature_Test_Deliverables Featured Areas Tests]
* Web Compatibility Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Web_Compatibility_Test_Deliverables Web Compatibility Tests ]
* User Performance Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#User_Performance_Test_Deliverables User Performance Tests ]
* Assorted Profiles Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Configurations_Test_Deliverables Configurations Tests ]
* Security Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Security_Test_Deliverables Security Tests ]
* Stress Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Accessibility_Tests_Deliverables Accessibility Tests ]
* Accessibility Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Bug_Verifications_Deliverables Bug Verifications ]
* Distribution Tests Suite
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Distribution_Test_Deliverables Distribution Tests ]
* L10N Test Suites
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#L10N_Test_Deliverables L10N Test Deliverables]
* Updates Test Suites
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Updates_Test_Deliverables Updates Test ]
* Bug Verifications
* [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_QA_Strategy#Regression_Tests Regression Tests]


== Areas not Covered ==
== Areas not Covered ==


* Feature Unit Tests
* Feature Unit Tests  
* String Localization Tests
** These are normally covered by feature developers, not QA.  However, we do have a Javascript test suite that will collectively find bugs with in-testsuite javascript test cases that Bob Clary will run on occasion.  Need more definition here on what QA/Dev should own.
* In depth functionality of Plugins, Extensions, and Themes
 
* String Localization Tests  
** Many of these tests are owned by localizers and SmartWare, our third party testing company.  See more in the L10n Test Deliverables section.
 
* In depth functionality of Plugins, Extensions, and Themes  
** Many extensions are hosted on AMO, but are not developed and tested by mozilla.  We rely on our volunteers to manage these third party extensions.  Mozilla QA will focus on high level Addon management, error handling, and a few top extensions supported.  But extension functionality will not be covered by Mozilla QA.


== Schedule ==
== Schedule ==


[http://tinyurl.com/2vwjx2 Firefox3 Schedule]
The release schedule is broken up by milestones, determined by the Firefox team.  Each milestone will contain feature enhancements, bug fixes, and updates.  Most testing of focused areas and regression areas will come in the latter half of the schedule, during the latter parts of alpha up until the final release candidate.  The following [http://spreadsheets5.google.com/fm?id=o01795881176923626235.6561840264411999137.06157383049480795802.4746102257499821515&hl=en&fmcmd=23&gid=2 test schedule] depicts how QA will be breaking down the test cycles.


The release schedule is broken up by milestones, which consists of feature checkins, bug fixes, and
''Questions:'' Are there plans to branch?  If so, what would the schedule look like?


''Questions:'' Is there plans to branch? If so, what would the schedule look like?
== Test Tools ==
This is ongoing set of Test Tools used by the QA team for automation and execution of the Firefox Product. Some of these tools are developed in-house, and are only accessible within the internal Mozilla network. The following list of tools shown here contain a short description of the tool, as well as further explanation on how the relevant tools are used for specific Test Areas below.


== Components ==
* [http://litmus.mozilla.org/ Litmus Test Case Manger] -
** Manages a suite of test cases used for smoketests, BFTs, FFTs, regression, and functional areas.  Public usage.


=== Feature Test Suite ===
* JS Test Suite -
** JavaScript project engine.  Code name: Spidermonkey.  Owned by Bob Clary and written in JavaScript, used to test different javascript API's and functions.
 
* Security Test library -
** Bob Clary finds ways to test security issues: automates ways to throw noise at "fuzz testers"; triggers randomness to hit edge cases.  The way we use it, we target security issues.  (eg. take HTML structure of webpage, look at tags in HTML, and find cases to execute within the tags.)  Ways to exploit memory, leaks, execution.  Tries to automate crashing tests by generating HTML and js code to crash the browser. 
 
* Download Checker -
** Bob Clary again.  Takes a huge list of locales and OS installs.  highly error prone.  Script will go to webpage, download each build, installs, runs a test, and closes it. fully automated.  This is executed only after we go live.    (all.html - "other system and languages")
 
* <strike> L10n search verifier - </strike> ''This has been replaced by Minotaur tool. (see below)'' <strike>
** Don't want to break localized products that may have mistakes.  Search engines, bookmarks, RSS feeds that are pre-configured.  Idea is to catch changes before it's checked in.  Diffs the region.properties settings for each locale against rules defined in the l10n requirements.  (http://wiki.mozilla.org/Firefox2/L10n_Requirements)</strike>
 
* [http://quality.mozilla.org/projects/automation/talos Talos] - Performance Testing Project
** Talos is a performance testing project. With a framework written in Python it runs Ts (startup test) and Tp (page load test) while monitoring memory and cpu usage. Ts is a simple Javascript web page that times the loading and then exits. Tp consists of a Javascript web page that cycles through a set of locally served web pages. The web page set is a collection of the top 500 web pages as listed by Alexa; they have been 'cleaned' so that all of their content is locally served.
 
* [https://intranet.mozilla.org/QA#Eggplant Eggplant] -
** Graphical replay; Tracy owns it.  Litmus smoke tests on it.  Identify a spot, string a list of actions, and play back.
 
* [http://developer.mozilla.org/en/docs/Mochitest MochiTests] -
** Built on the Mochikit platform. Main developer - robert sayer. unit test framework, easy way to run the browser. Scripting actions to quickly create test cases and tie to executables using JavaScript. Also supports accessibility layers.
 
* BuildBot -
** test system for building.  owned by Robcee.  client / server interaction.  Client requests execution activities.  Server sets tests up, run results.  (eg. parallel make system for builds.  able to build pieces independently and can piece it together later)  Like a next-gen tinderbox.
 
* [http://developer.mozilla.org/en/docs/Mozilla_automated_testing#Reftest Reftests] -
** Unit test framework, easy to run in the browser, easy to write. The testcases don't rely on external files. Mainly useful for layout tests, only.
 
* [http://quality.mozilla.org/projects/automation/minotaur Minotaur] -
** Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox.
 
== Categories ==
 
=== Feature Tests ===


;Summary
;Summary
Line 47: Line 88:


We will track the design, landing, test plan development, test case development, and running of the test cases for the initial feature testing via this [http://spreadsheets2.google.com/ccc?key=p-9pXWWpA7Bih6yFR9FYQWw&hl=en_US tracking spreadsheet].
We will track the design, landing, test plan development, test case development, and running of the test cases for the initial feature testing via this [http://spreadsheets2.google.com/ccc?key=p-9pXWWpA7Bih6yFR9FYQWw&hl=en_US tracking spreadsheet].
Current drafts of the '''Firefox 3 Feature Test Plans''' are listed [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan#Feature_Focused_Areas here] under "Test Plan Location" column.


The expectation is that the major features are well tested during the Alpha phase or maybe B1 at the latest.  Then testing during later betas is focused on verifying bug fixes, testing  additional edge cases that are identified (polish testing), and regression testing using smoke tests, BFTs, FFTs and automated runs.
The expectation is that the major features are well tested during the Alpha phase or maybe B1 at the latest.  Then testing during later betas is focused on verifying bug fixes, testing  additional edge cases that are identified (polish testing), and regression testing using smoke tests, BFTs, FFTs and automated runs.


;Test Suite
;Test Framework
Automation:
Automation:
* Bugzilla bugs
* Test cases within Bugzilla Bugs - some testcases are Unit tests, API-level, and can only be executed within a development environment.
* Mochikit
* Mochikit
* Eggplant
* Reftests
* Reftests
* Browser Chrome
* Browser Chrome
Line 61: Line 103:


Execution:
Execution:
* Litmus (BFTs, FFTs, smoketests, focused suite)
* Litmus (BFTs, FFTs, smoketests, Focused Tests)
* Test cases within Bugzilla Bugs
* Test cases within Bugzilla Bugs - some testcases are written and executed within feature bugs that are done manually.
* Eggplant - a set of functional tests ran and results reported to litmus


;Results
;Results
Test Results for executed test cases will be tracked within the litmus test suite.  For automation results, unit tests that are created in mochikit will be ran on tinderboxes
Test Results for executed test cases will be tracked within the litmus tests.  For automation results, unit tests that are created in mochikit will be ran on tinderboxen.  Other test results


=== Web Compatibility Test Suite ===
=== Web Compatibility Tests ===


;Summary
;Summary
The Web Compatibility Test Suite focuses on regression and compatibility with popular websites and web applications.   
The Web Compatibility Tests focuses on regression and compatibility with popular websites and web applications.   


;Planning & Scheduling
;Planning & Scheduling
We will be gathering the most relevant and widely used web applications today.  Web compatibility tests will be broken down by new feature tests for Firefox 3 specifically, as well as regression across the top extensions, plugins, and themes posted on AMO.  Tests will be expected to run during the later stages of alpha all the way up to RC2.
We will be gathering the most relevant and widely used web applications today.  Web compatibility tests will be broken down by new feature tests for Firefox 3 specifically, as well as regression across the top extensions, plugins, and themes posted on AMO.  Tests will be expected to run during the later stages of alpha all the way up to RC2.


;Test Suite
;Test Framework
* [http://www.alexa.com top sites]
* [http://www.alexa.com Top Sites] - coverage of a few popular sites that drives most user traffic
* top extensions (see shaver list)
* [http://wiki.mozilla.org/Firefox:2.0_QA_Activities:Extension_Results Top extensions] - coverage of Top firefox extensions used by the public
* top plugins
* Top themes - coverage of Top firefox themes used by the public
* Top themes
* Top plugins - coverage of Top web plugins used by the public
* Protocol handling
* Protocol handling - regression of a popular list of protocols supported on the web.  (eg. mailto://, https://, ftp://) Ties into [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=56&subgroup_id=820 litmus content handling] test cases. 


;Results
;Results
Test results are tracked in execution of FFTs, BFTs, and smoketests in litmus.
Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus.


=== User Performance Test Suite ===
=== User Performance Tests ===


;Summary
;Summary
The User Performance Test Suite is not to be confused with Browser performance tests that are run with automation.  Instead, we will concentrate on day to day usage of firefox and any performance issues that we see.  These would includes areas like downloading, idling, and other firefox daily usage cases.  
The User Performance Tests are not to be confused with Browser performance tests that are run with automation.  Instead, we will concentrate on day to day usage of Firefox and any performance issues that we see.  These would includes areas like downloading, idling, and other Firefox daily usage cases.  It will also incorporate some failover scenarios that would add a few stress cases and error handling cases.  


;Planning & Scheduling
;Planning & Scheduling
Different scenarios will be conducted and posted on litmus or a public wiki.  Since this test suite involves a different mix of computing environments, we will ask for community assistance to help.  The timing can exist around the Beta schedule, for about 1 or 2 cycles.   
Different scenarios will be conducted and posted on litmus or a public wiki.  Since these frameworks involves a different mix of computing environments, we will ask for community assistance to help.  The timing can exist around the Beta schedule, for about 1 or 2 cycles.   


;Test Suite
;Test Framework
* performance against other browsers?  Do we have benchmark comparisons?
* Performance against other browsers?  Do we have benchmark comparisons?
* Hard drives, throttling CPU’s
* Hard drives, throttling CPU’s
* Internet connection (wireless, dial up, LAN)
* Internet connection (wireless, dial up, LAN)
* Tab switching tests (martijn’s reftests)
* [https://bugzilla.mozilla.org/attachment.cgi?id=269215 Tab switching test]
* Fail-over error-handling testing scenarios in Litmus


;Results
;Results
Test cases will exist in litmus and the public wiki.  Any issues found will be tracked in bugzilla.
Test cases will exist in litmus and the public wiki.  Any issues found will be tracked in Bugzilla.


=== Profile Suite Test Suite ===
=== Configurations Tests ===
;Summary
;Summary
The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other.  This test suite will focus on running common litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine.   
The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other.  This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine.   


;Planning & Scheduling
;Planning & Scheduling
QA is developing virtualization test suites that contains images of various profiles across XP, vista, and Linux virtual machines.  This will help to assure a clean environment for testing, as well as cross functional platform operation.   
QA is developing virtualization test suites that contains images of various profiles across XP, vista, and Linux virtual machines.  This will help to assure a clean environment for testing, as well as cross functional platform operation.   
The [http://wiki.mozilla.org/MozillaQualityAssurance:Environments Test Environments TestPlan] will be divided up among the team and ran during the Beta stages of the overall schedule.
The [http://wiki.mozilla.org/MozillaQualityAssurance:Environments Environments TestPlan] will be divided up among the team and run during the Beta stages of the overall schedule.


;Test Suite
;Test Framework
* [https://intranet.mozilla.org/QA:Virtualization VM environment]
* [https://intranet.mozilla.org/QA:Virtualization Virtual Machines environment]
* Assorted Profile settings (see marcia’s link to all the different profile combos)  
* Mixed Profile settings (see marcia’s link to all the different profile combos)  
* Platforms specific (mac, vista, linux, xp)
* Platforms specific (mac, vista, linux, xp)


;Results
;Results
Issues found will be tracked in bugzilla, and test results will be recorded in Litmus.  Virtual Machines are tracked on the file server and available for download with VMWare licensing.
Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus.  Virtual Machines are tracked on the file server and available for download with VMWare licensing.


=== Security Test Suite ===
=== Security Tests ===
;Summary
;Summary
Security Tests are critical for catching external attacks through Firefox to the user.  Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox.  The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again.
Security tests are critical for catching external attacks through Firefox to the user.  Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox.  The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again.


;Planning & Scheduling
;Planning & Scheduling
QA will be creating new tests for new security features added to Firefox 3.  Tests will be run in the beta to RC stage, and a test plan will identify the strategy.  There are also security cases automated within Eggplant that cover simple security checks like cookie manager, website passwords, etc..  Development has a series of unit cases and fuzzer tools that they will run on individual environments.   
QA will be creating new tests for new security features added to Firefox 3.  Tests will be run in the beta to RC stage, and a test plan will identify the strategy.  There are also security cases automated within Eggplant that cover simple security checks like cookie manager, website passwords, etc..  Development has a series of unit cases and fuzzer tools that they will run on individual environments.   


 
;Test Framework
;Test Suite
* Browser Security tests at [http://bcheck.scanit.be/bcheck/ Scanit]
* security cases in litmus
* [http://test.bclary.com/tests/mozilla.org/security/ Bob Clary's security tests] - Javascript cases created and automated on Eggplant.
* Tracy’s security eggplant tests
* [http://test.bclary.com/tests/mozilla.org/security/ Bob Clary's security tests]
* Platforms with anti virus software, firewalls, global passwords, vista parental controls
* Platforms with anti virus software, firewalls, global passwords, vista parental controls
* Jesse’s Fuzzer Tools  - [https://bugzilla.mozilla.org/attachment.cgi?id=240710&action=edit jsparsefuzz.js]
* Jesse’s Fuzzer Tools  - [https://bugzilla.mozilla.org/attachment.cgi?id=240710&action=edit jsparsefuzz.js]
* Security bugs in bugzilla
* Security bugs in Bugzilla


;Results
;Results
Test results will be tracked via automation through eggplant and jesse's fuzzer tools.  There are a few security tests also tracked in litmus to be ran manually.  Also a handful of security bugs in bugzilla have test cases to be ran independently.   (example: [https://bugzilla.mozilla.org/show_bug.cgi?id=389106 Bug 389106])
Test results will be tracked via automation through eggplant and jesse's fuzzer tools.  There are a few security tests also tracked in litmus to be ran manually.  Also a handful of security bugs in Bugzilla have test cases to be run independently. (Example: [https://bugzilla.mozilla.org/show_bug.cgi?id=389106 Bug 389106])


=== Stress Testing ===
=== Accessibility Tests ===
* failover litmus scenarios (eg. Pulling out the plug on download)
 
=== Accessibility Test Suite ===
;Summary
;Summary


Line 146: Line 185:


;Planning & Scheduling
;Planning & Scheduling
Testing of Accessibility features will be within the Alpha to Beta stage of the schedule.   
Testing of Accessibility features will be within the Alpha to Beta stage of the schedule.  TimK will be heading up the feature testing ([http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan:UIAccessibility current Draft]), but also rely heavily on community participation.  There are also plans to put in more automation around a11y software to run through test scenarios.


;Test Suite
;Test Framework
* Accessiblity areas against new features
* Accessibility areas against new features
* Regression of areas
* Regression of areas, manual/ automation


;Results
;Results
Test cases will exist in litmus and the public wiki.  Any issues found will be tracked in bugzilla.
Test cases will exist in Litmus and the public wiki.  Any issues found will be tracked in Bugzilla.


=== Bug Verifications ===
=== Bug Verifications ===
;Summary
;Summary
New code is landing every night into the tree, and current functionality is constantly being update and overridden.  Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups. 


;Planning & Scheduling
;Planning & Scheduling
To prioritize and narrow the list of bugs, QA will focus primarily on Firefox bugs that represent highest severity, and carry the "blocking" flag.  This is more so a "goal" to verify every blocking and priority 1 bug, and will be the benchmark to "ship" or "stop" the release.


;Test Suite
;Test Framework
* Process to watch bugs on nightly channel, landing per feature
* Process to watch bugs on nightly channel, landing per feature
* Cross watching other team member’s bugs
* Cross watching other team member’s bugs
Line 167: Line 208:


;Results
;Results
Bugs will be marked verified or reactivated for regression.  All results tracked in Bugzilla.


=== Distribution Test Suite ===
=== Distribution Tests ===
;Summary
;Summary
Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds.  Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release.  This is to preserve the validation and regression of the product.


;Planning & Scheduling
;Planning & Scheduling
New test cases will be developed for new distribution requirements. Current regression tests documented in the [https://intranet.mozilla.org/Firefox:Distribution:TestPlan Partner Test Plan], will be ran during the later beta stages of the release. 


;Test Suite
;Test Framework
* DEX Requirements (new configurations)
* DEX Requirements (new configurations)
* Partner Major/Minor Updates  
* Partner Major/Minor Updates  


;Results
;Results
Results are tracked on the intranet wiki page and reported to the support group for Firefox partners.


=== L10N Test Suite ===
=== L10N Tests ===
;Summary
;Summary
The L10N Tests will prepare a set of standard Firefox testing requirements.  Its purpose is to ensure that third party localizers are properly checking the validity of the build before checking in to the tree.  It also will provide a quick automated check of configurations and values, and report results. 


;Planning & Scheduling
;Planning & Scheduling
String freeze for Firefox 3 will be targeting a time frame between B1 and B2.  Test Execution will be developing a few test cases that they determine would be important to be checked during localization testing in their main Firefox 3 features.  A customized l10n test suite will be created for the localizer to run on private builds prior to checkin. 


;Test Suite
The prioritized list of locales for Firefox 3 is listed [http://wiki.mozilla.org/Firefox3/Firefox_Requirements#Locale_Support here].
 
Test Automation will be running through a tool that will effectly validate customizations and configurations. 
 
A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales.  The localizer is also responsible for unit tests prior to checkin.  Click to see the existing [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:L10N_Test_Plan L10N Test Plan].
 
;Test Framework
Automation
* Minotaur
 
Execution
* Standard litmus test suite for localizer to run
* Standard litmus test suite for localizer to run
* Work with 3rd party Smartware to identify test cases
* Work with 3rd party Smartware to identify test cases
Line 190: Line 247:


;Results
;Results
Test results are tracked in different forms.  The localizer will track their unit cases and litmus for the custom l10n test suite.  Test execution team will report results in Litmus.  Test automation will track results at ''<insert location here>''.  Smartware will also track results at ''<insert location here>''.


=== Updates Test Suite ===
=== Updates Tests ===
;Summary
;Summary
Update Testing is the process of verifying minor security patches and major version releases from a prior version.  The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links.


;Planning & Scheduling
;Planning & Scheduling
Update testing comes after a minor or major build has landed and ready for verifications.  It does not include nightly and hourly updates on the trunk.  Responsibilities are divided up by locales and OS's, and ran through a thorough list of configuration checks as well as smoketests and spot checks.  It is also checked against the beta server (staging) and production (bouncer).  Update testing occurs no earlier than beta 2 stage and is tested on ever release thereafter.


;Test Suite
;Test Framework
* Minor updates (partial, failed)
* Minor updates (partial, failed)
* Major updates (complete)
* Major updates (complete)
* Spot Checks
* Spot Checks, Configurations


;Results
;Results
Results are reported via wiki and email.  There are also live updates tests against the production server after it has been released.
== Collecting Feedback ==
There are multiple ways we collect comments and feedback for Firefox releases.  The goal of feedback helps QA track issues found in development, regression, and post-release analysis.
;Reporting tools:
* [http://talkback-public.mozilla.org/reports/firefox/ Talkback / Crash Reporter] - For Fx2.0 and below
* [http://crash-stats.mozilla.com/ Breakpad / Crash Reporter] - Firefox 3
* Hendrix Feedback
** [http://groups.google.com/group/mozilla.feedback/topics Mozilla.feedback]
** [http://groups.google.com/group/mozilla.feedback.firefox.prerelease/topics?lnk=gschg Mozilla.feedback.firefox.prerelease]
** [http://groups.google.com/group/mozilla.feedback.firefox/topics?lnk=gschg Mozilla.feedback.firefox]
* [http://reporter.mozilla.org/ reporter.mozilla.org] - Broken Websites reported by users via Firefox Help > Report Broken Web Site menu
== Regression Tests ==
Regression Tests are constantly executed over the course of a release.  The purposes of these tests are to plant cross-platform overlapping tests, to ensure stability and reliability of the builds post nightly checkins.  The QA Tiger Team was formed for this reason, to ensure nothing is easily missed.  See the detailed description and schedule [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan:Fx3_QA_Daily_Smoketest#Tiger_Team_Testing_Squad Here].
=== Manual Tests ===
Manual Regression tests are ran weekly and at every milestone marker.  The majority of the test cases are recorded in litmus and available for anyone to execute.  The QA Tiger team has been executing weekly test runs on Smoketests, BFTs, and focused areas.  Any of these tests can be publicly executed and reported against Litmus.
;Manual framework:
* [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=54&subgroup_id= Smoketests] - A subset of litmus tests ran to ensure the quality of a nightly build.  Executed daily.
* [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=55&subgroup_id= Basic Functional Tests] - A more comprehensive subset of litmus tests that cover more regression than smoketests, but not a full suite.  Executed once a week.
* [http://litmus.mozilla.org/show_test.cgi?searchType=by_category&product_id=1&branch_id=15&testgroup_id=56&subgroup_id= Full Functional Tests] - A thorough set of litmus tests that cover each and every test case that is created to cover full functional regression and feature functionality.  Executed at least once per release.
* Eggplant Cases - An automated independent tool that takes the smoketests across all platforms and executes them on demand. 
* Focused feature areas across supported Firefox platforms
* Feature review of nightly checkins found on tinderboxen. see [http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=day&mindate=&maxdate=&cvsroot=%2Fcvsroot Bonsai Query].
=== Automation Tests ===
Both Daily and triggered automation cases exist in different frameworks that run a suite of cases like security, performance, and API level test cases. 
;Automation Framework:
* Buildbot
* Mochitests
* Reftests
* Browser Chrome
* XPCShell
* Performance tests
* Automated Memory Leaks (rob sayer's unit cases)
Results are recorded on tinderboxen.  Many of these tests are run automatically on nightly checkins, and will flash orange if tests have failed.


== Community Involvment ==
== Community Involvement ==
The goal is to actively involve more community members in testing Minefield, Gran Paradiso, and ultimately Firefox 3 as it moves into beta and to final release.


This goal is to actively involve more community members to assist with testing the firefox project in different aspects of the area.  There are many active participants in the community that are enthusiastic with Firefox and want to give back their time and effort to help test this project.  However, this involves heavy coordination, planning of work areas, and enforcing the correct process and procedures constantly produces challenges.  There is a seperate community team in QA that is currently working on incorporating a new strategy to promote more community involvment.   
There are many active participants in the community that are enthusiastic about Firefox and want to contribute to our quality assurance efforts.  However, getting them involved and maximizing the value of their time and effort requires a lot of coordination, planning, and enforcing of the correct processes and procedures.  Since such activities create constant challenges for our QA team,  we are working on a new strategy to promote more community involvement.   


Current avenues of community involvement includes:
Current avenues of community involvement include:
* friday testdays - a full day dedicated to testing firefox and thunderbird.  Activities include triage unconfirmed bugs, running tests in litmus, QnA, testing the trunk builds, and many other firefox-related topics
* tuesday bugdays - a full day dedicated to triaging bugs unconfirmed and confirmed.
* QMO - quality.mozilla.org: website to promote news and articles on mozilla QA.
* Campus evangelism - Targeting the university students with an exposure to testing open source and particularly firefox
* others?


* Friday Test Days
** increased the frequency of these events to allow people to show up each week to try out new features and help us run through test cases.  Activities include triaging unconfirmed bugs, running tests in Litmus with trunk builds (and alphas/betas), Q&A/Help sessions.  We will be focusing on a new Firefox 3 feature/component every week.


== Weekly Smoketests ==
* Tuesday Bug Days
** a full day each week dedicated to working with various bug lists in Bugzilla.  Activities will rotate between unconfirmed bug triage, bug verifications, test case writing, and reproducing crash bugs.


Each week, the Tiger team will be running a series of regression tests against the nightly builds on trunk.   The purpose of these tests are to plant cross-platform overlapping tests, to ensure stability and reliability of the builds post nightly checkins. See the detailed description and schedule [http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan:Fx3_QA_Daily_Smoketest#Tiger_Team_Testing_Squad Here].
* QMO - [http://quality.mozilla.org quality.mozilla.org]
** Using the website to promote news and articles on what the Mozilla QA team is working on
** Provide events and forums for the community to get involved.
** Create closer relationships with regular contributors and help them become leaders for our events and projects.
** Promote the [http://wiki.mozilla.org/Litmus:Extension QA Extension] to get more people connected to our tools and processes.  We will be setting up a "product page" on QMO and start the process for getting it polished enough for AMO.
** Update/migrate outdated QA pages on mozilla.org and showcase them on QMO
* Campus Evangelism - [https://www.mozilla.com/en-US/university/ Mozilla University]
** Leverage the Campus Reps program to reach out to students and professors at schools.
** Reach out to folks interested in open source and help them discover Mozilla project and get involved in Firefox development and testing.


Test Suite:
* Better "Beta" product pages and programs
* Smoketests across all supported Firefox platforms
** Update [http://www.mozilla.org/projects/minefield/ Minefield] and [http://www.mozilla.org/projects/granparadiso/ Gran Paradiso] pages to direct folks to QMO and other useful channels for getting involved and providing feedback.
* Focused feature areas across supported Firefox platforms 
** Coordinate community testing with marketing/pr plans for Firefox 3 Beta release
* Feature review of nightly checkins found on tinderbox
Mw22
Confirmed users
486

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/66322...79576"