QA/Triage Guidelines

DRAFT - THIS DOCUMENT IS A WORK IN PROGRESS

Summary

The following document is a guideline for how QA conducts triage. The purpose of this document is to standardize and document our best practices for bug triage across teams; thereby making bug triage and testing more efficient. Usage of the devices outlined herein are not a replacement for providing good bug comments. The clearer the request, the less follow-up required before someone can test, the more timely the request can be fulfilled.

Important Keywords

• qawanted: identifies a bug needing QA's help as soon as possible.
  • QA will triage these bugs daily and will respond within 24 hours.
  • A clearly defined request will be assigned using the QA Contact field
  • An unclear request will be flagged using the need-info flag
  • QA will drop this keyword will be dropped if
    • we feel the request has been fulfilled
    • we have unsuccessfully exhausted all leads
    • or the scope of the request is outside the bounds of our capabilities

• regressionwindow-wanted: identifies a bug needing a regression window before Engineering can investigate a fix.
  • Urgent requests should be accompanied by the qawanted keyword
  • QA will try to narrow the bug down to a daily window along with pushlog
  • QA may investigate a narrower window if requested, pending availability of hourly builds
  • QA will drop regressionwindow-wanted only when a regression window has been found
  • QA will drop qawanted when we find (or fail to find) a regression window
  • QA will add regressionwindow-wanted if testing is blocked on needing a regression window

• steps-wanted: identifies a bug needing steps to reproduce.
  • Urgent requests should be accompanied by the qawanted keyword
  • QA will try to find steps to reproduce including environmental factors where appropriate (add-ons, hardware, 3rd-party software, etc)
  • QA will drop steps-wanted only if steps to reproduce have been found
  • QA will drop qawanted when we find (or fail to find) the steps to reproduce
  • QA will add steps-wanted if testing is blocked on needing steps to reproduce

• testcase-wanted: identifies a bug needing a minimized testcase
  • Urgent requests should be accompanied by the qawanted keyword
  • QA will try to attach a minimized testcase as soon as possible, assuming we have reproducible steps including URLs
  • QA will drop testcase-wanted only when the a minimized testcase is attached to the bug
  • QA will drop qawanted when we find (or fail to find) a minimized testcase
  • QA will add testcase-wanted if testing is blocked on needing a testcase

• verifyme: identifies a bug with a fix needing verification
  • Urgent requests should be accompanied by the qawanted keyword
  • QA will try to verify the fix against all products and branches where the patch landed
  • QA will drop verifyme only when the fix has been verified against all fixed products and branches
  • QA will drop qawanted when we verify (or fail to verify) the fix
  • We will track product/branch verifications using status flags

• needURLs: identifies a bug needing URLs which reproduce the bug
  • QA will use this keyword when testing is blocked on needing URLs (most commonly topcrash bugs)

Important Flags

• in-moztrap: identifies a bug needing a manual test in Moztrap
  • ?: Anyone may set this flag to ask for a test to be written (be sure to add a requestee on the QA team)
  • +: Requestee will set this flag and provide a link when the Moztrap test has been written
  • -: Requestee will set this flag when a Moztrap is not needed

• in-qa-testsuite: identifies a bug needing an automated test in one of QA's frameworks (ex. Mozmill)
  • ?: Anyone may set this flag and a requestee on the Automation Development team to nominate the bug as needing a test
  • +: Automation Development will set this flag to indicate when a test has landed
  • -: Automation Development will set this flag to indicate when a test is not needed or cannot be developed
  • Automation Development team may ask you to file a Mozmill Tests bug upon triaging your request

• in-testsuite: identifies a bug needing an automated test in one of Engineering's frameworks (ex. mochitest)
  • ?: QA may set this flag to ask if the bug has a test or needs one landed
  • +: QA may set this flag if we see a patch landed with a test
  • -: Engineering may set this flag to indicate when a test is not needed or cannot be developed
  • QA may assist in developing an in-testsuite test on a case-by-case basis depending on personal time and ability

• need-info: identifies a bug needing information from a specific person
  • ?: Anyone may set this flag and requestee to ask for follow up information on a bug
  • Requestee will drop this flag when they have addressed the request

• status-<product>-<version>: identifies the status of a bug for a particular product and version
  • Use ? to request testing if a product version is affected, QA will triage these bugs for confirmation
  • Use fixed to indicate a product version has been fixed, QA will triage theses bugs for verification
  • Use disabled to indicate a product version has had a feature disabled
  • QA will change ? to affected when we confirm a product version is affected by a bug
  • QA will change fixed to verified when we confirm a product version is fixed
  • QA will change fixed to affected when we find a product version is not fixed
  • QA will change disabled to verified when we confirm a feature disabled in a product version

• tracking-<product>-<version>: identifies a bug that is actively tracked by Release Drivers for a particular product version
  • ?: QA will set this flag to nominate a bug for tracking, Release Management will usually be the decider
  • +: QA will track and investigate these bugs throughout the product cycle
  • -: QA will not track these bugs

• relnote-firefox: identifies a bug needing a release note for the current release
  • ?: QA will set this flag to request a release note mention for a bug
  • Release Management will usually decide if the bug needs noting and approve (+) or deny (-)

Important Whiteboard Tags

• [qa-:<product>]: identifies a bug which cannot or will not be tested by QA against a particular product
  • [qa-:desktop]: Firefox on desktop platforms
  • [qa-:mobile]: Firefox on mobile platforms
  • [qa-:b2g]: FirefoxOS
  • Tags should be replaced with qawanted keyword if new information comes to light which either makes this testable or important to test

Other Important Fields

• QA Contact: identifies the person in QA who is responsible for testing the bug, from confirmation through to verification.

Knowing Who's Who

List QA people and their area of responsibility...

Process Differentiation

Here we can outline process devices which are unique to certain teams (ie. blocking flags for B2G)...