QA/Firefox3/Strategy

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

QA Test Strategy – Firefox

Overview

This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3. It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy. Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, Green test pass on test cases in both execution and automation, and a thorough bug verification window of all Pri 1 and top Pri 2 tier bugs.

Areas Covered

• Regression Tests
• Featured Areas Tests Deliverables
• Web Compatibility Tests Deliverables
• User Performance Tests Deliverables
• Configurations Tests Deliverables
• Security Tests Deliverables
• Stress Tests Deliverables
• Accessibility Tests Deliverables
• Bug Verifications Deliverables
• Distribution Tests Deliverables
• L10N Test Deliverables
• Updates Test Deliverables

Areas not Covered

• Feature Unit Tests
• String Localization Tests
• In depth functionality of Plugins, Extensions, and Themes

Schedule

The release schedule is broken up by milestones, determined by the Firefox team. Each milestone will contain feature enhancements, bug fixes, and updates. Most testing of focused areas and regression areas will come in the latter half of the schedule, during the latter parts of alpha up until the final release candidate. The following test schedule depicts how QA will be breaking down the test cycles.

Questions: Are there plans to branch? If so, what would the schedule look like?

Test Tools

This is a comprehensive set of Test Tools used by the QA team for automation and execution of the Firefox Product. Some of these tools are developed in-house, and are only accessible within the internal Mozilla network.

• Litmus Test Case Manger -
  • Manages a suite of test cases used for smoketests, BFTs, FFTs, regression, and functional areas. Public usage.

• JS Test Suite -
  • JavaScript project engine. Code name: Spidermonkey. Owned by Bob Clary and written in JavaScript, used to test different javascript API's and functions.

• Security Test library -
  • Bob Clary finds ways to test security issues: automates ways to throw noise at "fuzz testers"; triggers randomness to hit edge cases. The way we use it, we target security issues. (eg. take HTML structure of webpage, look at tags in HTML, and find cases to execute within the tags.) Ways to exploit memory, leaks, execution. Tries to automate crashing tests by generating HTML and js code to crash the browser.

• Download Checker -
  • Bob Clary again. Takes a huge list of locales and OS installs. highly error prone. Script will go to webpage, download each build, installs, runs a test, and closes it. fully automated. This is executed only after we go live. (all.html - "other system and languages")

• L10n search verifier -
  • Don't want to break localized products that may have mistakes. Search engines, bookmarks, RSS feeds that are pre-configured. Idea is to catch changes before it's checked in. Diffs the region.properties settings for each locale against rules defined in the l10n requirements. (http://wiki.mozilla.org/Firefox2/L10n_Requirements)

• Performance testing -
  • TP (page load time), TS (page start time). Ran against performance machines and records metrics and performance trends.

• Eggplant -
  • Graphical replay; Tracy owns it. Litmus smoke tests on it. Identify a spot, string a list of actions, and play back.

• MochiTests -
  • Built on the Mochikit platform. Main developer - robert sayer. unit test framework, easy way to run the browser. Scripting actions to quickly create test cases and tie to executables using JavaScript. Also supports accessibility layers.

• BuildBot -
  • test system for building. owned by Robcee. client / server interaction. Client requests execution activities. Server sets tests up, run results. (eg. parallel make system for builds. able to build pieces independently and can piece it together later) Like a next-gen tinderbox.

• Minotaur -
  • Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox.

Deliverables

Feature Test Deliverables

Summary

Our feature testing approach is to match major features to test engineers and have the test engineering work with the development engineers during development and as the feature lands to ensure the features are well tested.

Planning & Scheduling

Each milestone marker in the Firefox scheduling, will contain a list of features that are noted and checked into the tree for the release. The QA Execution team will be dividing up the Firefox 3 features and developing a test plan for each module. Test plan will be under initially reviewed by peers, and posted to newsgroups for further review. Completion of testing the feature is targeting the final Beta in the cycle.

Developers focus on unit tests. QA will focus on areas where the unit test burden is very onerous and focus on user scenarios, API testing, integration testing exploratory testing, and system level testing. QA will utilize this Test Case Template as a formatting tool to create test cases on these features. Both automation and manual test cases will be created and stored within litmus, bugzilla, and CVS.

We will track the design, landing, test plan development, test case development, and running of the test cases for the initial feature testing via this tracking spreadsheet.

The expectation is that the major features are well tested during the Alpha phase or maybe B1 at the latest. Then testing during later betas is focused on verifying bug fixes, testing additional edge cases that are identified (polish testing), and regression testing using smoke tests, BFTs, FFTs and automated runs.

Test Framework

Automation:

• Test cases within Bugzilla Bugs - some testcases are Unit tests, API-level, and can only be executed within a development environment.
• Mochikit
• Reftests
• Browser Chrome
• XPCShell
• JProf

Execution:

• Litmus (BFTs, FFTs, smoketests, Focused Tests)
• Test cases within Bugzilla Bugs - some testcases are written and executed within feature bugs that are done manually.
• Eggplant - a set of functional tests ran and results reported to litmus

Results

Test Results for executed test cases will be tracked within the litmus tests. For automation results, unit tests that are created in mochikit will be ran on tinderboxes

Web Compatibility Test Deliverables

Summary

The Web Compatibility Test Deliverables focuses on regression and compatibility with popular websites and web applications.

Planning & Scheduling

We will be gathering the most relevant and widely used web applications today. Web compatibility tests will be broken down by new feature tests for Firefox 3 specifically, as well as regression across the top extensions, plugins, and themes posted on AMO. Tests will be expected to run during the later stages of alpha all the way up to RC2.

Test Framework

• top sites
• Top extensions (see shaver list)
• Top plugins
• Top themes
• Protocol handling

Results

Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus.

User Performance Test Deliverables

Summary

The User Performance Test Deliverables is not to be confused with Browser performance tests that are run with automation. Instead, we will concentrate on day to day usage of Firefox and any performance issues that we see. These would includes areas like downloading, idling, and other Firefox daily usage cases. It will also incorporate some failover scenarios that would add a few stress cases and error handling cases.

Planning & Scheduling

Different scenarios will be conducted and posted on litmus or a public wiki. Since these frameworks involves a different mix of computing environments, we will ask for community assistance to help. The timing can exist around the Beta schedule, for about 1 or 2 cycles.

Test Framework

• Performance against other browsers? Do we have benchmark comparisons?
• Hard drives, throttling CPU’s
• Internet connection (wireless, dial up, LAN)
• Tab switching test
• Fail-over error-handling testing scenarios in Litmus

Results

Test cases will exist in litmus and the public wiki. Any issues found will be tracked in Bugzilla.

Configurations Test Deliverables

Summary

The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other. This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine.

Planning & Scheduling

QA is developing virtualization test suites that contains images of various profiles across XP, vista, and Linux virtual machines. This will help to assure a clean environment for testing, as well as cross functional platform operation. The Environments TestPlan will be divided up among the team and run during the Beta stages of the overall schedule.

Test Framework

• Virtual Machines environment
• Mixed Profile settings (see marcia’s link to all the different profile combos)
• Platforms specific (mac, vista, linux, xp)

Results

Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus. Virtual Machines are tracked on the file server and available for download with VMWare licensing.

Security Test Deliverables

Summary

Security tests are critical for catching external attacks through Firefox to the user. Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox. The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again.

Planning & Scheduling

QA will be creating new tests for new security features added to Firefox 3. Tests will be run in the beta to RC stage, and a test plan will identify the strategy. There are also security cases automated within Eggplant that cover simple security checks like cookie manager, website passwords, etc.. Development has a series of unit cases and fuzzer tools that they will run on individual environments.

Test Framework

• Browser Security tests at Scanit
• Bob Clary's security tests - Javascript cases created and automated on Eggplant.
• Platforms with anti virus software, firewalls, global passwords, vista parental controls
• Jesse’s Fuzzer Tools - jsparsefuzz.js
• Security bugs in Bugzilla

Results

Test results will be tracked via automation through eggplant and jesse's fuzzer tools. There are a few security tests also tracked in litmus to be ran manually. Also a handful of security bugs in Bugzilla have test cases to be run independently. (Example: Bug 389106)

Stress Tests Deliverables

• Fail-over Litmus scenarios (e.g. Pulling out the plug on download)

Accessibility Tests Deliverables

Summary

Accessibility Tests will be developed and ran against the new features coming within Firefox 3. These cases will ensure we are following proper A11y guidelines.

Planning & Scheduling

Testing of Accessibility features will be within the Alpha to Beta stage of the schedule. TimK will be heading up the feature testing (current Draft), but also rely heavily on community participation. There are also plans to put in more automation around a11y software to run through test scenarios.

Test Framework

• Accessibility areas against new features
• Regression of areas, manual/ automation

Results

Test cases will exist in Litmus and the public wiki. Any issues found will be tracked in Bugzilla.

Bug Verifications Deliverables

Summary

New code is landing every night into the tree, and current functionality is constantly being update and overridden. Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups.

Planning & Scheduling

To prioritize and narrow the list of bugs, QA will focus primarily on Firefox bugs that represent highest severity, and carry the "blocking" flag. This is more so a "goal" to verify every blocking and priority 1 bug, and will be the benchmark to "ship" or "stop" the release.

Test Framework

• Process to watch bugs on nightly channel, landing per feature
• Cross watching other team member’s bugs
• Dedicated time frame to work only on bug verifications during a milestone
• Updating Litmus and reftests based on bugs

Results

Bugs will be marked verified or reactivated for regression. All results tracked in Bugzilla.

Distribution Test Deliverables

Summary

Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds. Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release. This is to preserve the validation and regression of the product.

Planning & Scheduling

New test cases will be developed for new distribution requirements. Current regression tests documented in the Partner Test Plan, will be ran during the later beta stages of the release.

Test Framework

• DEX Requirements (new configurations)
• Partner Major/Minor Updates

Results

Results are tracked on the intranet wiki page and reported to the support group for Firefox partners.

L10N Test Deliverables

Summary

The L10N Test Suite will prepare a set of standard Firefox testing requirements. Its purpose is to ensure that third party localizers are properly checking the validity of the build before checking in to the tree. It also will provide a quick automated check of configurations and values, and report results.

Planning & Scheduling

String freeze for Firefox 3 will be targeting a time frame between B1 and B2. Test Execution will be developing a few test cases that they determine would be important to be checked during localization testing in their main Firefox 3 features. A customized l10n test suite will be created for the localizer to run on private builds prior to checkin.

Test Automation will be running through a tool that will effectly validate customizations and configurations.

A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales. The localizer is also responsible for unit tests prior to checkin.

Test Framework

Automation

• Minotaur

Execution

• Standard litmus test suite for localizer to run
• Work with 3rd party Smartware to identify test cases
• Spot checks on Tier 1 locales

Results

Test results are tracked in different forms. The localizer will track their unit cases and litmus for the custom l10n test suite. Test execution team will report results in Litmus. Test automation will track results at <insert location here>. Smartware will also track results at <insert location here>.

Updates Test Deliverables

Summary

Update Testing is the process of verifying minor security patches and major version releases from a prior version. The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links.

Planning & Scheduling

Update testing comes after a minor or major build has landed and ready for verifications. It does not include nightly and hourly updates on the trunk. Responsibilities are divided up by locales and OS's, and ran through a thorough list of configuration checks as well as smoketests and spot checks. It is also checked against the beta server (staging) and production (bouncer). Update testing occurs no earlier than beta 2 stage and is tested on ever release thereafter.

Test Framework

• Minor updates (partial, failed)
• Major updates (complete)
• Spot Checks, Configurations

Results

Results are reported via wiki and email. There are also live updates tests against the production server after it has been released.

Community Involvement

This goal is to actively involve more community members to assist with testing the firefox project in different aspects of the area. There are many active participants in the community that are enthusiastic with Firefox and want to give back their time and effort to help test this project. However, this involves heavy coordination, planning of work areas, and enforcing the correct process and procedures constantly produces challenges. There is a seperate community team in QA that is currently working on incorporating a new strategy to promote more community involvement.

Current avenues of community involvement includes:

• friday testdays - a full day dedicated to testing firefox and thunderbird. Activities include triage unconfirmed bugs, running tests in litmus, QnA, testing the trunk builds, and many other firefox-related topics
• tuesday bugdays - a full day dedicated to triaging bugs unconfirmed and confirmed.
• QMO - [quality.mozilla.org quality.mozilla.org]: website to promote news and articles on mozilla QA.
• Campus evangelism - Targeting the university students with an exposure to testing open source and particularly firefox
• Others?

Regression Tests

Regression Tests are constantly executed over the course of a release. The purposes of these tests are to plant cross-platform overlapping tests, to ensure stability and reliability of the builds post nightly checkins. The QA Tiger Team was formed for this reason, to ensure nothing is easily missed. See the detailed description and schedule Here.

Manual Tests

Manual Regression tests are ran weekly and at every milestone marker. The majority of the test cases are recorded in litmus and available for anyone to execute. The QA Tiger team has been executing weekly test runs on Smoketests, BFTs, and focused areas. Any of these tests can be publicly executed and reported against Litmus.

Manual framework

• Smoketests
• BFTs
• FFTs
• Eggplant Cases
• Focused feature areas across supported Firefox platforms
• Feature review of nightly checkins found on tinderboxen. see Bonsai Query.

Automation Tests

Both Daily and triggered automation cases exist in different frameworks that run a suite of cases like security, performance, and API level test cases.

Automation Framework

• Buildbot
• Mochitests
• Reftests
• Browser Chrome
• XPCShell
• Performance tests
• Automated Memory Leaks (rob sayer's unit cases)

Results are recorded on tinderboxen. Many of these tests are run automatically on nightly checkins, and will flash orange if tests have failed.