QA/Firefox3/Strategy: Difference between revisions
| Line 60: | Line 60: | ||
;Summary | ;Summary | ||
The Web Compatibility Test Suite focuses on regression and compatibility with popular websites and web applications. | The Web Compatibility Test Suite focuses on regression and compatibility with popular websites and web applications. | ||
;Planning | |||
We will be gathering the most relevant and widely used web applications today. Test breakdown of areas will | |||
;Test Suite | |||
* [http://www.alexa.com top sites] | * [http://www.alexa.com top sites] | ||
* top extensions (see shaver list) | * top extensions (see shaver list) | ||
| Line 68: | Line 72: | ||
* Protocol handling | * Protocol handling | ||
;Results | |||
Test cases will be tracked in FFTs, BFTs, and smoketests in litmus. | Test cases will be tracked in FFTs, BFTs, and smoketests in litmus. | ||
Revision as of 01:22, 22 August 2007
|
THIS PAGE IS A WORKING DRAFT | 
|
| The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page. |
QA Test Strategy – Firefox
Overview
This document is a tracking strategy that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox.
Areas Covered
- Featured Areas Tests Suite
- Web Compatibility Tests Suite
- User Performance Tests Suite
- Assorted Profiles Tests Suite
- Security Tests Suite
- Stress Tests Suite
- Accessibility Tests Suite
- Distribution Tests Suite
- L10N Test Suites
- Updates Test Suites
- Bug Verifications
Areas not Covered
Schedule
The release schedule is broken up by milestones, which consists of feature checkins, bug fixes, and
Questions: Is there plans to branch? If so, what would the schedule look like?
Components
Feature Test Suite
- Summary
The Feature Test Suite covers a thorough set of test cases that are developed by QA and dev that covers all the new features described in the Firefox 3 Product Requirements Document. QA will utilize the Test Case Template as a formatting tool to create test cases on these features. Both automation and manual test cases will be created and stored in the following locations:
- Planning
Each milestone marker in the Firefox scheduling, will contain a list of features that are noted and checked into the tree for the release. The QA Execution team will be dividing up the Firefox 3 features and developing a test plan for each module. Test plan will be under initially reviewed by peers, and posted to newsgroups for further review. Completion of testing the feature is targeting the final Beta in the cycle.
- Test Suite
Automation:
- Bugzilla bugs
- Mochikit
- Eggplant
- Reftests
- Browser Chrome
- XPCShell
- JProf
Execution:
- Litmus (BFTs, FFTs, smoketests, focused suite)
- Test cases within Bugzilla Bugs
- Results
Test Results for executed test cases will be tracked within the litmus test suite. For automation results, unit tests that are created in mochikit will be ran on tinderboxes
Web Compatibility Test Suite
- Summary
The Web Compatibility Test Suite focuses on regression and compatibility with popular websites and web applications.
- Planning
We will be gathering the most relevant and widely used web applications today. Test breakdown of areas will
- Test Suite
- top sites
- top extensions (see shaver list)
- top plugins
- Top themes
- Protocol handling
- Results
Test cases will be tracked in FFTs, BFTs, and smoketests in litmus.
User Performance Test Suite
The User Performance Test Suite is not to be confused with Browser performance tests that are run with automation.
- performance against other browsers? Do we have benchmark comparisons?
- Hard drives, throttling CPU’s
- Internet connection (wireless, dial up, LAN)
- Tab switching tests (martijn’s reftests)
Profile Suite Test Suite
- VM environment (see al’s link to different VM scenarios)
- VM server environment? (see marcia’s link to all the different profile combos)
- Platforms specific (mac, vista, linux, xp)
Security Test Suite
- security cases in litmus
- Tracy’s security eggplant tests
- Bob Clary's security tests
- Platforms with anti virus software, firewalls, global passwords, vista parental controls
- Jesse’s Fuzzer Tools - jsparsefuzz.js
Stress Testing
- failover litmus scenarios (eg. Pulling out the plug on download)
Accessibility Test Suite
- Accessiblity areas against new features
- Regression of areas
Bug Verifications
- Process to watch bugs on nightly channel, landing per feature
- Cross watching other team member’s bugs
- Dedicated time frame to work only on bug verifications during a milestone
- Updating Litmus and reftests based on bugs
Distribution Test Suite
- DEX Requirements (new configurations)
- Partner Major/Minor Updates
L10N Test Suite
- Standard litmus test suite for localizer to run
- Work with 3rd party Smartware to identify test cases
- Spot checks on Tier 1 locales
Updates Test Suite
- Minor updates (partial, failed)
- Major updates (complete)
- Spot Checks
Community Involvment
This goal is to actively involve more community members to assist with testing the firefox project in different aspects of the area. There are many active participants in the community that
Weekly Smoketests
- Tiger team (smoketesting on different platform) –Tracy to watch the results
- Focused testing (across platforms)