QA/Fennec1.0/TestPlan

« Mobile

Overview

Fennec 1.0 Test Strategy

The goal of this Overall Test Plan is to lay out our testing strategy, communicate it to others, and track the status.

Since Fennec is not installed or used in the same way Firefox is, the test approach needs to be modified. We will automate as much as possible. The automation we will use will be a subset (as all features are not in Fennec) of the Firefox automation with new tests developed to test specific issues of firefox. Key pieces of automation will be functionality, performance, and stability.

There will be a lot of unique tests which are device specific and Fennec specific that will be too expensive to automate. With these we will add them to litmus and run them by hand.

Items to Update

Here is a list of items that will be updated once we get more information:

Maemo 1.0 Release (RC1):

* L10n - L10n Fennec Litmus test cases
* L10n - Spot check wiki and list of locales
* Install/Update (definition of repositories) - Update with betachannel and windows mobile
* Web compatibility Litmus test cases - update with web compatibility tool when available.
* QA/Support Team Issues
* How to scale testing to multiple devices
* How to test with wifi vs wireless carrier

Post Maemo 1.0:

* Accessibility Litmus test cases
* Security Tools and test cases
* Stress Testing
* Performance regression plan (maybe just user performance, but probably more)

Schedule

• Maemo Alpha 1: October 17
• Maemo Alpha 2: December 23
• Maemo Beta 1: March 17
• WinMo Alpha1: April 17
• Maemo Beta 2 - June 26
• WinMo Alpha2: June 26
• Maemo Beta 3 - August 20
• WinMo Alpha 3 - August 20
• Maemo Beta 4 - Sept 30 - Release Test Results, QA Blockers
• Maemo Beta 5 - Nov 9 - Release Test Results, QA Blockers
• Maemo RC1 - December 30 - Release Test Results
• Maemo RC2 - January 6 - Release Test Results
• Maemo RC2 - January 7 - Release Test Results
• Maemo RC3 - January 27 - Release Test Results
• Maemo 1.0 - January 28
• WinMo Alpha 4 - February 8 - Release Test Results
• Maemo 1.0.1 - March 31 - Release Test Results

Test Areas

General Test Areas

Mobile Specific Features

Platform Specific Features

Environments

The main environment that we will be focusing on for the 1.0 release will be the Nokia maemo platform (N8xx devices). In addition to the Maemo platform, we are working on Windows Mobile and Symbian support. These will be under concurrent development but not at the same feature complete and quality of the Maemo platform.

Platforms

Source Code and Nightly Build Locations

Fennec 1.0 is currently built from trunk (mozilla-central) as well as the front end mobile-browser repositories. To minimize branching and maintenance, we will target the 1.9.2 branch (splitting from mozilla-central July 2009). All other platforms will build from the same branch.

Nightly builds for Maemo (you need both fennec*.deb and xulrunner*.deb where the timestamps match) and Windows Mobile (only the .cab file is needed) are located here: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/

Review Feedback

In this section, we will roll up all common feedback pulled from external sources that appears common and should be tracked by QA. Feedback sources can include:

• Hendrix Feedback system (mozilla.feedback newsgroups)
• Bugzilla
• Mozillazine Forums
• HTC Touch Pro forum
• Omnia forum
• xda forums - more low level devs

Automation

QA is undertaking the task of porting the unit tests from Firefox to Fennec for each platform. This is a lengthy task, but our goal is to have 100% unit tests passing by the 1.0 ship.

We have created a tracking system for the unit tests which we update on a regular basis.

Release Test Guidelines

• QA Requirements:
  • E-mail to Drivers pre/post-Release Cycle and pre/post-Respin
    • Includes:
      • Build location link
      • Blocker Bugs added to the newest release build
      • Release cycle schedule
      • Dev Contact available on-hand

• List of Disabled Features for the Release
  • Session Store
  • Private Browsing
  • Safebrowsing
  • RSS
  • Certificates Manager
  • Find In Page

• Template for results tracking page
• Updates Testing
  • HowTo:Maemo
  • HowTo:WinMo

Additional References

This section is left to add any additional references relevant to the Mobile project

Useful Information:

• Original list of [requirements]
• IRC Channels: #mobile, #wince, #symbian

Building:

• Maemo:
  • Building Fennec
  • Getting builds for N800/N810
  • note: for anything not requiring a device, a desktop build of fennec is sufficient (use Ubuntu if you need mochitests)
• Windows Mobile:
  • Building Fennec

Testing:

• Unittests on Fennec
  • Mochitests on Fennec
  • Reftests on Fennec