B2G/QA/Triage

Overview

This document provides a summary of how we manage requests for QA support through bugzilla, more specifically for the B2G QA team.

Summary of Keywords and Flags

Whiteboard

Project identifiers such as [red tai] may be used to identify a group of bugs associated with a specific project.
More detail click here

Keywords

Note: Only one keyword should be used optimally to represent what is immediately needed by QA, except for the case of qaurgent. qaurgent is used in conjunction with a QA keyword.

steps-wanted

When reproducible steps to reproduce are needed for a certain bug.

testcase-wanted

When a reduced test case is needed for a certain bug.

regressionwindow-wanted

When a regression range is necessary to identify a first good build and first bad build for a particular bug. This should be flagged only when the following criteria is met:

• Branch checks for a bug are complete (note: this is done to ensure that we truly know that the bug is a regression)
• The bug has been determined to be a nomination or a blocker (note: this is done since windows are expensive to conduct)
• The reproduction rate is greater than or equal to 50%

verifyme

When a verification of a bug is needed to ensure that the patch on the bug actually fixed the problem.

qawanted

When QA support is needed on the bug, but none of the above keywords apply. When this keyword is used, the person flagging qawanted needs to indicate what QA support is needed. Example requests include:

• Branch checks
• Retesting to see if the issue reproduces on the latest build
• Getting clarification on the reproduction rate of the bug

qaurgent

When QA support is urgently needed on a bug for a particular request. Use this only when you think the request has a high severity (e.g. smoketest issue).

More keywords

click here

Flags

in-moztrap

When a bug or feature could benefit from having formal test case coverage. See below for a more detailed discussion on the workflow of using this flag.

QA Contact

The use of QA contact field should be set optimally when the person is actively investigating something in relation to the bug, usually in connection to one of the keywords above. The QA contact field should aim to only be set by the person who is planning on working on the bug for some piece of QA.

However, there is one exception to this rule. If a high need is raised in triage to find an owner for a bug, then triagers should set the QA contact to the QA representative in triage to find a QA owner to address the QA needs for the bug. If no QA representative is present, then Pallavi Yaramada or Al Tsai should be set as the QA contact as a fallback.

QA Requests

For addressing bugs that need QA involvement with the keywords stated above except verifyme, the table below summarizes the urgency of response we follow for QA requests.

Note: For high urgency bugs, we plan to provide an initial response on the bug within two business days.

Test Case Coverage Wanted

Summary

For requesting test case coverage, we shall make use of the in-moztrap flag. This flag aims to capture when someone thinks test case coverage would be a good for a feature or bug filed in Bugzilla. From a feature perspective for a release, this aims to help capture a secondary checklist to make sure that our QA team has captured test cases for all features for a particular release. However, for feature release test case development, usage of this flag is not supposed to be used for being the sole tracking area - a separate spreadsheet will be maintained for tracking the details of the work being done. For miscellaneous cases needing test case coverage, this process also helps find missing test case coverage and sometimes, can identify untracked features that product did not know about.

To learn more information about the workflow for test case coverage, see this diagram that describes the workflow with the in-moztrap flag.

Request Importance

The following table below summarizes the priority for test case coverage requests that come from the in-moztrap flag.

Test Case Prioritization

When creating the test cases, please tag them with the following prioritization levels within MozTrap:

• P1 - Feature smoketests
• P2 - Feature basic functional tests
• P3 - Boundary, edge, stress test cases
• P4 - Anything else that doesn't fall in the above buckets

Verifications

For addressing bugs that need verification, our QA team will follow a manual process of analyzing resolved fixed bugs, determining which bugs need verification, and verifying those bugs on the appropriate branches. If someone internal or external to the team knows that a bug is worthwhile verifying, then they will flag the bug as verifyme. The following table below summarizes what bugs should be verified and in what order.