QA/Feature Testing v2

From MozillaWiki
< QA
Jump to: navigation, search

Feature Testing v2.0

The information available in this page is also available as a Google Doc here.

Purpose of this document

What this document is

This document outlines the major milestones associated with the quality management of Firefox features. These milestones are centered around manual testing, evaluation and reporting.

The process described below applies ONLY for the Firefox features shipped based on the train model.

What this document is not

The information available in this document does not apply to:

  • QA teams conducting automated testing
  • QA teams working on off-the-train features

The process described below does NOT directly result in a final Go/No-Go decision for a Firefox feature.

Process

Overview

Phase I - PI request submission and QA resource allocation

  • QA testing is requested by the expected deadline in Jira Service Desk
    • Deadline: Nightly, wk2
    • Responsible: Eng/Product
    • Description: A testing requests need to be filed directly in Jira Service Desk. By default the submitted PI requests are to be considered for QA testing in the Beta cycle.
      • Project team should highlight if they need Nightly testing or not (optional but recommended).
      • The Exception path for late submissions can be found here.
  • A quality engineer is assigned to the request, based on available resources and priorities
    • Responsible: QA
    • Description: Once a quality engineer has been assigned, the associated Jira ticket will be updated and the PI requestor gets notified automatically.

Phase II - Feature documentation and pre-execution approvals

  • Feature technical documentation is provided by the expected deadline
    • Deadline: Nightly, wk4
      • Past due features will be called out and deferred by default to the next release.
      • Request for exception must have approvals from head of engineering AND product.
    • Responsible: Eng/Product
    • Description: The technical documentation should be provided based on this template and should be complete and relevant to the work requested.
      • Once documentation is received, QA will set the status of the Jira ticket status to TEST PLANNING.
      • Any delay in submitting technical documentation will significantly impact QA’s ability to plan and test during the Beta cycle.
    • Reference: Guideline for Technical Documentation
  • A feature kick-off meeting is scheduled once technical documentation is received
    • Soft deadline: Nightly, wk5
    • Responsible: QA, Eng/Product
    • Description: The kick-off meeting is the perfect place for QA to get a better understanding of the scope of the feature and clarify any questions related to it.
      • Following the kick-off meeting, QA will set the status of the Jira ticket to TEST CREATION.
  • Test Plan and Test Cases are drafted and formal approvals requested
    • Deadline: Nightly, wk5
    • Responsible: QA
    • Description: Documents are created for the Test Plan and High-level Test Cases. QA will provide the links and ask for formal approval via email.
      • Once the links are provided, QA will set the status of the Jira ticket to WAITING FOR DEVELOPERS.
    • References: Test Plan template, Test Cases template
  • Test Plan and Test Cases are reviewed and formal approvals are made
    • Deadline: Nightly, wk7
    • Responsible: Eng/Product
    • Description: Feedback and formal approval should be given on the email thread started by QA on this matter.

Phase III - Go/No-Go decision based on feature completeness

  • Feature achieves completeness by the expected deadline
    • Deadline: Nightly, wk6
    • Responsible: Eng/Product
    • Description: here
      • The Exception path for late submissions can be found here.
  • QA provides a test report for the features for which Nightly testing was requested
  • Go/No-Go decision is made based on feature completeness, quality and risk
    • Deadline: Nightly, wk7
    • Responsible: Eng/Product, Relman
    • Description: Go/no-go will be based on (input from) feature completeness (Eng/Product), quality (QA), and risk (Relman).

Phase IV - Preliminary feature quality assessment

  • QA testing begins in Beta
    • Responsible: QA
    • Description: Test execution begins on the first builds available in Beta, wk1.
      • QA will also set the status of the Jira ticket to TEST EXECUTION, and provide regular updates regarding testing progress and new issues found as comments.
  • QA provides a pre-Release preliminary status report
    • Deadline: Beta, wk5
    • Responsible: QA
    • Description: Two weeks before the formal Pre-Release sign off, QA will provide a preliminary report to highlight concerns early on. The report will be sent out as an email, but it will also be posted as a comment in the Jira ticket.
      • If there are major concerns reported, a meeting with Engineering will likely be required to determine the next steps.
    • Reference: Preliminary status report optional template

Phase V - Pre-Release test execution, sign off and follow-ups

  • QA provides a formal, Pre-Release feature sign off
    • Deadline: Beta, wk7
    • Responsible: QA
    • Description: The Pre-Release sign off represents the final in-depth testing conducted by QA before the merge to Release. The sign off will highlight concerns (if any) and provide a quality status that will be factored into the Go/No-Go decision that will be made by Release Management.
      • Once the Pre-Release test execution is done, QA will set the status of the Jira ticket to SIGN-OFF EMAIL SENT, and a summary of the report will be posted in it as well.
    • Reference: Feature sign off mandatory template
  • QA will continue to assist following the sign off, if follow-ups are required
  • QA will set the status of the Jira ticket to RESOLVED, when the work is done

Schedule and milestones

6-week release cycle

CYCLE MILESTONE WEEK NO
1 2 3 4 5 6
NIGHTLY PI request deadline
Feature technical documentation
Feature kick-offs
Test Plan & Test Case formal approvals
BETA Pre-Release preliminary status report
Pre-Release feature sign off

7-week release cycle

CYCLE MILESTONE WEEK NO
1 2 3 4 5 6 7
NIGHTLY PI request deadline
Feature technical documentation
Feature kick-offs
Test Plan & Test Case formal approvals
BETA Pre-Release preliminary status report
Pre-Release feature sign off

8-week release cycle

CYCLE MILESTONE WEEK NO
1 2 3 4 5 6 7 8
NIGHTLY PI request deadline ⬤
Feature technical documentation ⬤
Feature kick-offs ⬤
Test Plan & Test Case formal approvals ⬤
BETA Pre-Release preliminary status report ⬤
Pre-Release feature sign off ⬤

Best practices

  • Getting help
    • Each feature’s QA owner should have a peer (helper) assigned to help.
      • Larger, more complex features can justify more than one QA peer.
  • Maintaining documentation
    • Internal Test Plan reviews and updates should occur periodically.
      • Feature Test Plans should be updated at least once a week to keep them relevant.
    • Feature status updates should be provided periodically in the QA status documents associated to each Firefox version.
  • Bug tracking
    • Weekly checks should be made for the bugs reported in the wild.
      • Since there are so many environment variations (due to various software and hardware pairings), some bugs might only be uncovered by users that have very specific environment setups.
    • In the case of highly complex features, a meta bug should be created to track all the issue reported by QA.
      • Having a separate meta bug for the issues reported by QA ensures a more efficient tracking, referencing and reporting.
    • Highly severe bugs (critical, blockers) affecting a feature should be flagged using the qablocker keyword.
      • Using this keyword in addition to setting needinfo? flags for the right people is the most efficient way of raising major concerns.
  • Bug verification
    • A continuous monitoring process should be in place for new bug fixes.
      • This can be easily done by setting up Bugzilla queries, or something similar.

Sign off overview

The QA sign off is based on the quality status of the feature in question. The green, yellow or red quality status is a schedule-based conversation, NOT a general release-readiness status.

Quality status evaluation criteria

The following aspects are taken into account when evaluating the quality status of a feature:

  • percentage of completed implementation, based on Engineering’s commitment
  • unresolved bugs count and severity
  • failed tests count and failure reason
  • blocked tests percentage and blocking reason

Quality statuses

Green

Status description
A GREEN quality status indicates a feature that meets all of QA's expectations.

Status reasons

  • Everything is going according to our plan.
  • Manual test results have met the acceptance criteria that was agreed upon with Engineering.
  • All or major and critical test objectives agreed upon with Engineering have been covered.
  • There are no issues severe enough to jeopardize the release of the feature.

Example

  • [65] [desktop] [feature] Widevine 4.10.1196.0 - pre-Release QA sign off (GREEN) - link

Yellow

Status description
A YELLOW quality status indicates a feature of questionable quality, meaning that there are issues or potential issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision.

Status reasons

  • Manual test results have mostly met the acceptance criteria that was agreed upon with Engineering, with a few minor to medium exceptions that need to be addressed.
  • Only major and critical test objectives that were agreed upon with Engineering have been covered, minor ones have been left out.
  • There are moderate issues that might jeopardize the release of the feature, but an input is required from Product, User Experience and/or Engineering to determine if they should block or not.
  • Testing progress is behind the planned timeline and we are not able to cover everything on time.

Example

  • [65] [desktop] [feature] Language Switcher - Pre-Release QA sign off (YELLOW) - link

Red

Status description
A RED quality status indicates a feature of low quality, meaning that there are severe issues that need to be addressed by the Product, User Experience and Engineering teams as part of that feature’s Go/NoGo decision.

Status reasons

  • There’s a big mismatch between manual test results and the acceptance criteria that was agreed upon with Engineering.
  • Part of or all major and critical test objectives that were agreed upon with Engineering have not been covered.
  • There are severe issues jeopardizing the release of the feature or there’s a big mismatch between what was planned for implementation and what was actually implemented.
  • We are unable to cover important parts of the feature, due to resource constraints or hardware/environment requirements that we cannot meet, which means that those areas may have important issues that we don’t know about.

Example

  • [60] [desktop] [feature] Normandy Telemetry - pre-Release QA sign off (RED) - link

Pref'd OFF

Description
Features riding a specific Firefox version in a disabled state (via preference flip) will not be receiving a sign off report from QA. This is because there’s no real impact for the end user, and so an in-depth assessment of that feature’s quality is irrelevant for the Firefox version in question.

Instead of a sign off, QA will send out a regular test report describing what was considered in scope/covered.

Examples

  • [63] [desktop] [feature] Block Autoplay v2 - Test Results - link

Sign off template

  • The mandatory sign off template for pref’d on features is available here.
  • The mandatory sign off template for pref’d off features is available here.
  • The guideline for preliminary test reports is available here.

References

  • All QA documentation available for features, structured per Firefox version: here.
  • Feature QA status per Firefox version:

History of changes

  • Aug 6, 2019 - (btot) Updated the High-Level Test Cases document template
  • April 23, 2019 - (avaida) Added an optional test report template for features requiring Nightly testing.
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Feature_Testing_v2&oldid=1216000"