QA/Feature Testing v2

Feature Testing v2.0

The information available in this page is also available as a Google Doc here.

Purpose of this document

What this document is

This document outlines the major milestones associated with the quality management of Firefox features. These milestones are centered around manual testing, evaluation and reporting.

The process described below applies ONLY for the Firefox features shipped based on the train model.

What this document is not

The information available in this document does not apply to:

• QA teams conducting automated testing
• QA teams working on off-the-train features

The process described below does NOT directly result in a final Go/No-Go decision for a Firefox feature.

Process

Overview

Phase I - PI request submission and QA resource allocation

• QA testing is requested by the expected deadline in Jira Service Desk
  • Deadline: Nightly, wk2
  • Responsible: Eng/Product
  • Description: A testing requests need to be filed directly in Jira Service Desk. By default the submitted PI requests are to be considered for QA testing in the Beta cycle.
    • Project team should highlight if they need Nightly testing or not (optional but recommended).
    • The Exception path for late submissions can be found here.

• A quality engineer is assigned to the request, based on available resources and priorities
  • Responsible: QA
  • Description: Once a quality engineer has been assigned, the associated Jira ticket will be updated and the PI requestor gets notified automatically.

Phase II - Feature documentation and pre-execution approvals

• Feature technical documentation is provided by the expected deadline
  • Deadline: Nightly, wk4
    • Past due features will be called out and deferred by default to the next release.
    • Request for exception must have approvals from head of engineering AND product.
  • Responsible: Eng/Product
  • Description: The technical documentation should be provided based on this template and should be complete and relevant to the work requested.
    • Once documentation is received, QA will set the status of the Jira ticket status to TEST PLANNING.
    • Any delay in submitting technical documentation will significantly impact QA’s ability to plan and test during the Beta cycle.
  • Reference: Guideline for Technical Documentation

• A feature kick-off meeting is scheduled once technical documentation is received
  • Deadline: Nightly, wk5
  • Responsible: QA, Eng/Product
  • Description: The kick-off meeting is the perfect place for QA to get a better understanding of the scope of the feature and clarify any questions related to it.
    • Following the kick-off meeting, QA will set the status of the Jira ticket to TEST CREATION.

• Test Plan and Test Cases are drafted and formal approvals requested
  • Deadline: Nightly, wk5
  • Responsible: QA
  • Description: Documents are created for the Test Plan and High-level Test Cases. QA will provide the links and ask for formal approval in the Jira ticket.
    • Once the links are provided, QA will set the status of the Jira ticket to WAITING FOR DEVELOPERS.
  • References: Test Plan template, Test Cases template

• Test Plan and Test Cases are reviewed and formal approvals are made
  • Deadline: Nightly, wk7
  • Responsible: Eng/Product
  • Description: TO DO

Phase III - Go/No-Go decision based on feature completeness

• Feature achieves completeness by the expected deadline
  • Deadline: Nightly, wk6
  • Responsible: Eng/Product
  • Description: here
    • The Exception path for late submissions can be found here.

• QA provides a test report for the features for which Nightly testing was requested
  • Deadline: Nightly, wk7
  • Responsible: QA
  • Description: In the case of features for which Nightly testing was requested, a test report will be sent out highlighting concerns.

• Go/No-Go decision is made based on feature completeness, quality and risk
  • Deadline: Nightly, wk7
  • Responsible: Eng/Product, Relman
  • Description: Go/no-go will be based on (input from) feature completeness (Eng/Product), quality (QA), and risk (Relman).

Phase IV - Preliminary feature quality assessment

• QA testing begins in Beta
  • Responsible: QA
  • Description: Test execution begins on the first builds available in Beta, wk1.
    • QA will also set the status of the Jira ticket to TEST EXECUTION, and provide regular updates regarding testing progress and new issues found as comments.

• QA provides a pre-Release preliminary status report
  • Deadline: Beta, wk5
  • Responsible: QA
  • Description: Two weeks before the formal Pre-Release sign off, QA will provide a preliminary report to highlight concerns early on. The report will be sent out as an email, but it will also be posted as a comment in the Jira ticket.
    • If there are major concerns reported, a meeting with Engineering will likely be required to determine the next steps.
  • Reference: Preliminary status report optional template

==== Phase V - Pre-Release test execution, sign off and follow-ups

• QA provides a formal, Pre-Release feature sign off
  • Deadline: Beta, wk7
  • Responsible: QA
  • Description: The Pre-Release sign off represents the final in-depth testing conducted by QA before the merge to Release. The sign off will highlight concerns (if any) and provide a quality status that will be factored into the Go/No-Go decision that will be made by Release Management.
    • Once the Pre-Release test execution is done, QA will set the status of the Jira ticket to SIGN-OFF EMAIL SENT, and a summary of the report will be posted in it as well.
  • Reference: Feature sign off mandatory template

• QA will continue to assist following the sign off, if follow-ups are required

• QA will set the status of the Jira ticket to RESOLVED, when the work is done

Schedule and milestones

6-week release cycle

CYCLE	MILESTONE	WEEK NO
		1	2	3	4	5	6
NIGHTLY	PI request deadline
	Feature technical documentation
	Feature kick-offs
	Test Plan & Test Case formal approvals
BETA	Pre-Release preliminary status report
	Pre-Release feature sign off

7-week release cycle

CYCLE	MILESTONE	WEEK NO
		1	2	3	4	5	6	7
NIGHTLY	PI request deadline
	Feature technical documentation
	Feature kick-offs
	Test Plan & Test Case formal approvals
BETA	Pre-Release preliminary status report
	Pre-Release feature sign off

8-week release cycle

CYCLE	MILESTONE	WEEK NO
		1	2	3	4	5	6	7	8
NIGHTLY	PI request deadline		⬤
	Feature technical documentation				⬤
	Feature kick-offs					⬤
	Test Plan & Test Case formal approvals					⬤
BETA	Pre-Release preliminary status report					⬤
	Pre-Release feature sign off							⬤

Best practices

• Getting help
  • Each feature’s QA owner should have a peer (helper) assigned to help.
    • Larger, more complex features can justify more than one QA peer.

• Maintaining documentation
  • Internal Test Plan reviews and updates should occur periodically.
    • Feature Test Plans should be updated at least once a week to keep them relevant.
  • Feature status updates should be provided periodically in the QA status documents associated to each Firefox version.

• Bug tracking
  • Weekly checks should be made for the bugs reported in the wild.
    • Since there are so many environment variations (due to various software and hardware pairings), some bugs might only be uncovered by users that have very specific environment setups.
  • In the case of highly complex features, a meta bug should be created to track all the issue reported by QA.
    • Having a separate meta bug for the issues reported by QA ensures a more efficient tracking, referencing and reporting.
  • Highly severe bugs (critical, blockers) affecting a feature should be flagged using the qablocker keyword.
    • Using this keyword in addition to setting needinfo? flags for the right people is the most efficient way of raising major concerns.

• Bug verification
  • A continuous monitoring process should be in place for new bug fixes.
    • This can be easily done by setting up Bugzilla queries, or something similar.

Sign off overview

The QA sign off is based on the quality status of the feature in question. The green, yellow or red quality status is a schedule-based conversation, NOT a general release-readiness status.