User:Ashughes/QA Bug Triage

From MozillaWiki
Jump to: navigation, search

Remaining Work

The following items I still need to add to this document

  • QA status:fixed triage (VERIFYME on important bugs)
  • QA pushlog fixed triage (VERIFYME on important bugs)
  • Role for devs (QAWANTED when marking a bug fixed)
  • Role for mgmt (QAWANTED for tracking noms, uplift noms, relnote noms)
  • How community can get involved

Purpose

The purpose of this document is increase productivity of getting bugs fixed and verified by:

  • increased understanding of QA bug fix verification process
  • increased contribution by volunteers
  • increased collaboration with QA partners (ex. developers, release-drivers)

Summary

QA will use the QAWANTED keyword to track bugs needing immediate QA attention. People outside of QA should get used to using the QAWANTED keyword, in conjunction with a keyword categorizing the request, to alert QA of a bug needing attention.

  • NEEDURLS: when URLs are needed
  • STEPSWANTED: when steps to reproduce are required
  • REGRESSIONWINDOW-WANTED: when a regression range (fixed or broken) is needed
  • VERIFYME: when verification of a fix is needed

QA will use the [qa-] whiteboard tag to track fixed bugs QA has already triaged and has deemed unnecessary to verify or is not able to verify (usually due to lack of builds). People outside of QA should get used to using this whiteboard tag if there are bugs you know we should ignore.

QA will use the [qa:good-first-bug] whiteboard tag to track fixed bugs QA has already triaged and feels would be good contribution opportunities for new contributors.

QA will use the NEEDINFO flag whenever we are blocked on testing and need more information from someone.

QA will involve the bug reporter in cases we feel are necessary; usually when information is required or help is needed in verification.

QA Requestors

This section is for those of you requesting help from QA (developers, release managers, product managers, project managers, etc). Above all, please remember that taking the time to be detailed and specific in your request is more likely to yield successful and timely results.

What to do when you need QA to look at a bug?

  1. Add the QAWANTED keyword
  2. Set the QA Contact field (use ashughes if you're not sure)
  3. Add a comment to the bug detailing what you need from QA
  4. If appropriate, add one of the other keywords to categorize your request
What to expect
  • QA will do everything within our ability to see the request to a successful conclusion
  • QA will, if we need more information, set the NEEDINFO flag before or during testing
  • QA will, if we have exhausted our capabilities, drop a request by removing the QAWANTED keyword
  • QA will, in appropriate cases, involve the reporter in testing their bug

What to do when you need QA to verify a fix?

  1. Add the QAWANTED and VERIFYME keywords
  2. Set the QA Contact field (use ashughes if you're not sure who to use)
  3. Add a comment to the bug detailing any high risk areas QA should pay attention to
  4. If appropriate, use one of the other keywords
What to expect
  • QA will do everything within our ability to verify the fix across all patched branches
  • QA will, upon successful verification, drop the QAWANTED and VERIFYME keywords, set status flags to verified, add a comment detailing the testing
  • QA will, if we need more information, set the NEEDINFO flag, flag the person we need information from, add a comment detailing the information request, and halt testing until that information can be provided
  • QA will involve the reporter if and when necessary
  • QA will, if we have exhausted our capabilities, drop the QAWANTED and VERIFYME keywords, add the [qa-] whiteboard tag, and add a comment detailing why we cannot or will not proceed with verification

Common Reasons for Dropping a Request

  • We've successfully completed the request
  • We've run out of leads to follow
  • We are unable to get access to builds for testing
  • We are unable to get access to a web resource for testing
  • We are unable to get access to software for testing
  • We are unable to get access to hardware for testing
  • The request is outside the scope of our expertise

Bugs We Will Not Test Unless Asked

  • bugs with the in-testsuite+ flag
  • intermittent test failures
  • changes to APIs
  • bugs outside of in-product changes

QA Contributors

This section is for those of us in QA or those of you wishing to help QA with our requests.

How to Get Involved?

What to do with STEPSWANTED Bugs?

These are bugs you are trying to find steps to reproduce. In other words, what are the basic steps one must follow to witness the bug for themselves.

  • Read the bug report in full to make sure you understand the issue
  • Try to reproduce the bug with the information provided
  • Failing that, use your imagination and try to discover ways to reproduce the bug
  • If successful, add your steps to reproduce and any other useful information to the bug report in a comment
  • If unsuccessful, ask for more information in the bug by adding a comment
  • If necessary, set the NEEDINFO flag and set the bug reporter as the requestee
Commonly Useful Information
  • Firefox version(s)
  • Operating system version(s)
  • Installed add-ons with version numbers
  • Websites visited
  • Third-party software installed
  • Links to crash reports from about:crashes

What to do with REGRESSIONWINDOW-WANTED Bugs?

These bugs you are trying to find a regression range. In other words, the versions of Firefox between which the bug was introduced. At the very least, the goal is to try to get the range down to a 24-hour window.

  • Read the bug report in full to make sure you understand the issue
  • Try to reproduce the bug with the information provided
  • If you can, try to narrow down the range to a few months between Nightlies
  • Install the mozregression tool and give it the two dates you identified. This tool will allow you to narrow the range to within 24 hours and will even give you a URL to see the changes made between these two builds.
  • Add the information it provides to the bug as a comment. Something similar to this:
Last Good Build: 2012-10-23
First Bad Build: 2012-10-24
Pushlog: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=895f4b058c60&tochange=222e6877be4b

You can do the same thing manually but it is a far more tedious process. You just need to download the Nightlies individually until you find one the exact build where the bug first appeared.

In rare cases, you may actually be asked to find a fixed regression range. In this case you are dealing with a bug that existed at a time, no longer does, but we don't know when or how it was fixed. Similar to above, you need to find the last build that was broken and the first build that was fixed.

What to do with VERIFYME Bugs?

These bugs you are trying to verify that the bug has been fixed.

  • Read the bug report in full to make sure you understand the issue
  • Try to reproduce the bug with the information provided on a build that is known to be broken (usually the Nightly from the day the bug was filed)
  • If you are able to reproduce the bug on a known broken build, download a version that is thought to be fixed (usually indicated by the status-firefoxN flags)
  • If you are still able to reproduce the bug, provide the details of your testing in the bug report and ask if the bug should be reopened.
  • If the bug does NOT reproduce then you can mark the bug fixed
    • set the status-firefoxN flag to verified for the version you tested
    • add a comment indicating the details of your testing
    • set the STATUS field to VERIFIED FIXED if you verified it against the latest Nightly
Commonly Useful Information

The following information is commonly useful when leaving details about your verification testing.

  • steps followed
  • URLs tested
  • Firefox version tested
  • Operating system tested

Be advised that you should test against the platform indicated in the bug report. If it is set to ALL test against one Windows, one Mac OSX, and one Linux version if possible.

Commonly Used Keywords, Flags, Tags

Keywords
  • NEEDURLS: used to request URL(s) which reproduce the bug
  • REGRESSIONWINDOW-WANTED: used in conjunction with QAWANTED to request finding a regression range (between which Firefox versions was the bug introduced)
  • STEPSWANTED: used in conjunction with QAWANTED to request steps to reproduce the bug
  • VERIFYME: used to request verification of a fixed bug
Whiteboard Tags
  • [qa-]: used to indicate QA is unable to take further action on the bug
  • [qa:good-first-bug]: used to identify a bug QA thinks would be a good starting point for a new contributor
Flags
  • NEEDINFO: used to request more information from the reporter, a developer, or someone else involved with a bug
  • tracking-firefoxN: the bug is either tracked (+), not tracked (-), or nominated for tracking (?) in Firefox N
  • status-firefoxN: indicates the status of a bug for a particular Firefox version
    • affected: the bug is not yet fixed
    • disabled: the bug represents a feature which has been disabled
    • fixed: the bug is fixed but not verified
    • verified: the bug is fixed and verified
    • wontfix: the bug will not be fixed

Feedback

If you have something to say, please add a bullet item with your feedback, questions, or concerns; particularly if you think something is incorrect or missing. Thank you.

  • So far it looks good. Can't wait to see the proposal after the rest of the remaining items are added. There is one thing that I would like to mention though: I think that it would be really difficult for a contributor to work on the verifyme bugs that are logged on all the platforms. Maybe the tracking flags for a verifyme bug should be changed just after verifying it on a single platform or to ask contributors to change the tracking flags only after all platforms have been verified. (Simona)
  • Looks good to me. Notes:
    • We can add some of the info here in QMO as well once everything is clear.
    • It might be useful to have a guideline as to what kind of bugs will be marked as good for contributors.
    • Some of the verifyme bugs may need some extra attention - exploratory testing around the fix. Not sure if this is clear enough.(Virgil)