Firefox ESR Test Plan Template

Use the following template when creating a Firefox ESR Test Plan.

How To Guide

Copy and paste the template below to a new section of the current major ESR version test plan. For example, if you were creating a test plan for Firefox 10.0.10esr, you would copy the template to a new section called Firefox 10.0.10 ESR in the Firefox 10 ESR Test Plan found at https://wiki.mozilla.org/Releases/Firefox_10_ESR/Test_Plan. All release test plans follow a similar nomenclature.

Anywhere you see an italicized version, replace it with the version number you are testing.

Main Header: Firefox version ESR

This section details a summary of information about the build.

Subsection: Automated Tests

This section details the results of the functional automation. These tests are run automatically once builds appear on the FTP server. Status emails are sent to the mozmill-ci mailing list. Results are reported to the mozmill-ci dashboard.

Subsection: Manual Smoketests

This section details the results of manual smoketesting performed by Softvision (our outsource partner in Romania). We typically test four supported platforms to save time (2 Windows, 1 Mac, and 1 Linux).

The link to the smoketests and report needs to be updated to point to MozTrap. Every new release we clone the previous release's smoketests and adjust if necessary. These tests are designed to cover the most basic functionality and check the ESR branding. After the testing is complete you will want to review the MozTrap report for any failures and escalate to bugs if needed.

Softvision will fill out the assignee and results columns in the table.

Subsection: Update Testing

This section details the results of automated update testing. These tests need to be triggered manually by uploading a plain-text configuration file to the mozmill-ondemand server (details available upon request from ahughes@mozilla.com).

Update tests are run on all supported platforms, 3 previous versions (usually the two most recent versions and the oldest version receiving the update), and 2 locales (usually en-US and one random locale).

Before we give the go-ahead to release the build we run tests on releasetest channel. After we've given the go-ahead to release, RelEng pushes the snippets to the live channel and we run the tests again on the esr channel.

In terms of evaluating the results, a PASS means the builds successfully applied the correct update. Try to spotcheck one or two results per platform (no need to check every single result).

• any versions listed in Partial should receive a partial or partial+fallback update
• any versions listed in Complete should receive a complete or complate+fallback update
• the target build ID should match that of the release you are signing off

Subsection: Bug Fix Verifications

This section details the bug fixes which must be verified before we sign off a release. List any bugs using the bugzilla template, changing status to verified when the bug has been tested and signed-off.

Subsection: Issues

This section details any bugs found during testing this release or known to affect this release. For the most part, Softvision will fill out both of these tables. We need to evaluate the list of New issues for potential blocker regressions. Once reviewed and deemed non-critical, move the bug to the Known issues list.

Firefox version ESR

Checklist

• {{ |}} Functional automation
• {{ |}} Manual tests
• {{ |}} Bug fix verifications
• {{ |}} Update automation
• {{ |}} Peer sign-off
• {{ |}} Updates pushed/tested on ESR channel

Automated Tests

The following are the detailed results of the automated Mozmill test runs.

Test Failures (known bugs)

Manual Smoketests

Run the Firefox ESR version Smoketests in MozTrap and report the results below.

Update Testing

• Partial: versions receiving partial updates
• Complete: versions receiving complete updates

Issues

Bug Fix Verifications

The following bugs require testing and verification of fixes before signing off the release.

• bug number - testing status (QA contact)

Issues

New

• bug number - summary (reporter)

Known

• [status] bug number - summary (reporter)