Confirmed users, Bureaucrats and Sysops emeriti
722
edits
QA/Triage: Difference between revisions
Jump to navigation
Jump to search
m
no edit summary
GavinSharp (talk | contribs) |
GavinSharp (talk | contribs) mNo edit summary |
||
| Line 11: | Line 11: | ||
across many organizations. The great benefit is that the various | across many organizations. The great benefit is that the various | ||
products are extremely thoroughly tested on a day to day basis and | products are extremely thoroughly tested on a day to day basis and | ||
everyone | everyone benefits from the resulting quality. New regressions are | ||
often noticed right away before other new fixes are built on top of | often noticed right away before other new fixes are built on top of | ||
the problematic code. A diversity of opinions often allows the | the problematic code. A diversity of opinions often allows the | ||
| Line 27: | Line 27: | ||
valid problems, putting these issues on track to be addressed. | valid problems, putting these issues on track to be addressed. | ||
In addition, new Mozilla | In addition, new Mozilla QAs are coming into a very complex | ||
project which may not operate in a way similar to projects they've | project which may not operate in a way similar to projects they've | ||
worked in before. | worked in before. | ||
| Line 61: | Line 61: | ||
Do the following steps in batches, if possible. Each time you submit changes to | Do the following steps in batches, if possible. Each time you submit changes to | ||
the bug it creates extra Bugmail, so it's better to do the changes in as few rounds | the bug it creates extra Bugmail, so it's better to do the changes in as few rounds | ||
as possible. These techniques really apply to bugs | as possible. These techniques really apply to bugs QAs are filing as well. | ||
# <u>Understand the bug:</u> Read very carefully to try and understand what they're talking about. | # <u>Understand the bug:</u> Read very carefully to try and understand what they're talking about. | ||
# <u>Look for duplicates:</u> A lot of bug reporters mistakenly assume their bug won't have been filed yet. Search for keywords in the bug, and try alternate spellings and synonyms. For example, if the bug is about the control key, try spelling it ctrl as well. Look through fixed bugs when you search for duplicates. | # <u>Look for duplicates:</u> A lot of bug reporters mistakenly assume their bug won't have been filed yet. Search for keywords in the bug, and try alternate spellings and synonyms. For example, if the bug is about the control key, try spelling it ctrl as well. Look through fixed bugs when you search for duplicates. | ||
| Line 68: | Line 68: | ||
# <u>Simplify the bug testcase</u>. You're the lead investigator -- so try to get enough information to turn intermittent or hard to reproduce bugs into something that is easy to reproduce. See more on this in "How to Really, Really Help Developers on Bugs" below. | # <u>Simplify the bug testcase</u>. You're the lead investigator -- so try to get enough information to turn intermittent or hard to reproduce bugs into something that is easy to reproduce. See more on this in "How to Really, Really Help Developers on Bugs" below. | ||
# <u>Ensure appropriate hardware/platform tests</u>: most bug filers will only have tested on one platform. If bug was filed for a particular platform, see if it's a bug on another platform. If it's been reproduced on at least 2 platforms that aren't both Unix/Linux, then go ahead and mark it All/All ((Hardware/OS). Most bugs are actually All/All. It's useful to get a feeling of when to do extra tests and when not to waste your time, but this comes with experience. For example, OS X uses many native Carbon form controls and menus whereas Windows and Linux do not, so form control bugs on Mac are probably just Mac-only. However, a bug in how HTML links are dealt with is almost certainly a bug on all platforms. Also, it's useful to know the QA community so that you can ask for help when you don't have the right hardware/OS to test a particular bug with. | # <u>Ensure appropriate hardware/platform tests</u>: most bug filers will only have tested on one platform. If bug was filed for a particular platform, see if it's a bug on another platform. If it's been reproduced on at least 2 platforms that aren't both Unix/Linux, then go ahead and mark it All/All ((Hardware/OS). Most bugs are actually All/All. It's useful to get a feeling of when to do extra tests and when not to waste your time, but this comes with experience. For example, OS X uses many native Carbon form controls and menus whereas Windows and Linux do not, so form control bugs on Mac are probably just Mac-only. However, a bug in how HTML links are dealt with is almost certainly a bug on all platforms. Also, it's useful to know the QA community so that you can ask for help when you don't have the right hardware/OS to test a particular bug with. | ||
# <u>Ensure a | # <u>Ensure a concise and precise summary</u>: It is reasonable to edit the summary a bit if it will help people understand the bug more quickly or will clear up something that's misspelled or incorrect. Check to see that the summary is clear, concise and describes the problem as fully and correctly as is reasonable in a short space. Make sure it contains the words people would probably search for before filing a duplicate bug. To risk stating the obvious, the spelling in the summary matters because searches will for "Escape" will fail if the reporter wrote "Escpae". | ||
# <u>Security sensitive bugs</u>: [TBD] | # <u>Security sensitive bugs</u>: [TBD] | ||
# <u>Ensure Proper Severity</u> (but don't change Priority field, which is for the developer). Here's a quick guide, but you can't get more help from Bugzilla itself | # <u>Ensure Proper Severity</u> (but don't change Priority field, which is for the developer). Here's a quick guide, but you can't get more help from Bugzilla itself | ||
| Line 74: | Line 74: | ||
#* Critical: crash and security bugs (should also be marked with keyword crash). | #* Critical: crash and security bugs (should also be marked with keyword crash). | ||
#* Major: Bugs that affect the main screen, cause dataloss, loss of ability to operate the software or are just plain awful | #* Major: Bugs that affect the main screen, cause dataloss, loss of ability to operate the software or are just plain awful | ||
#* Normal/Minor/ | #* Normal/Minor/Trivial: Most other bugs | ||
# <u>Change Product, Component and owner</u>: Many bugs get marked "Firefox" when they are really bugs in the core engine. If it's a basic XUL, HTML or layout bug that affects multiple products send it to Core. After you submit a Product change to the bug, Bugzilla will allow you to change the Component as well. When changing Product and/or Component you usually want to select the radio button "Reassign bug to default assignee and QA contact of selected component ". The exception to this is when the bug is a regression and already assigned to a particular developer who caused the regression. You can spot those situations via a comment like "Joe, this appears to be from your checkin from bug 32412". | # <u>Change Product, Component and owner</u>: Many bugs get marked "Firefox" when they are really bugs in the core engine. If it's a basic XUL, HTML or layout bug that affects multiple products send it to Core. After you submit a Product change to the bug, Bugzilla will allow you to change the Component as well. When changing Product and/or Component you usually want to select the radio button "Reassign bug to default assignee and QA contact of selected component ". The exception to this is when the bug is a regression and already assigned to a particular developer who caused the regression. You can spot those situations via a comment like "Joe, this appears to be from your checkin from bug 32412". | ||
# <u>Add Keywords</u> -- here are just a few of the important ones [https://bugzilla.mozilla.org/describekeywords.cgi the full list of keywords is here]). | # <u>Add Keywords</u> -- here are just a few of the important ones [https://bugzilla.mozilla.org/describekeywords.cgi the full list of keywords is here]). | ||
| Line 82: | Line 82: | ||
#* "dataloss": this bug will cause users to lose data | #* "dataloss": this bug will cause users to lose data | ||
#* "hang": freezes or locks up the application | #* "hang": freezes or locks up the application | ||
#* "testcase": indicates that a | #* "testcase": indicates that a simplified testcase is included (see below) | ||
# <u>List related meta bugs</u> in "this bug blocks". If an important meta bug has a difficult to remember number, give it a memorable alias. For example, here are some important accessibility-related meta bugs: | # <u>List related meta bugs</u> in "this bug blocks". If an important meta bug has a difficult to remember number, give it a memorable alias. For example, here are some important accessibility-related meta bugs: | ||
#* focusnav: related to focus and tab navigation | #* focusnav: related to focus and tab navigation | ||
| Line 92: | Line 92: | ||
One really important thing that QAs can do for developers is to narrow down to the actual markup | One really important thing that QAs can do for developers is to narrow down to the actual markup | ||
causing the bug -- this applies for QA-filed bugs as well. Many testcases | causing the bug -- this applies for QA-filed bugs as well. Many testcases | ||
filed by inexperienced bug reporters come from a website, which | filed by inexperienced bug reporters come from a website, which unfortunately may change and thus the testcase is lost. Or, the HTML from the given testcase might be huge, which makes debugging extra difficult. | ||
change and thus the testcase is lost. Or, the | |||
HTML from the given testcase might be huge, which makes debugging extra difficult. | |||
Here are some steps for using the process of elimination to create a minimal testcase from a webpage that displays a consistent bug. | Here are some steps for using the process of elimination to create a minimal testcase from a webpage that displays a consistent bug. | ||
| Line 130: | Line 128: | ||
# In the end, make sure that the bug is marked with the keyword "regression", and add a very clear comment something like: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken".<li>An experienced QA can pinpoint the actual checkin with the problem, and assign the bug to the developer who was responsible. | # In the end, make sure that the bug is marked with the keyword "regression", and add a very clear comment something like: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken".<li>An experienced QA can pinpoint the actual checkin with the problem, and assign the bug to the developer who was responsible. | ||
If you are very technical, it is possible to go even further. Some | If you are very technical, it is possible to go even further. Some QAs who know | ||
how to apply patches and build their own Mozilla have even been known to find which | how to apply patches and build their own Mozilla have even been known to find which | ||
part of a patch caused the bug, by building with or without each piece in a patch. | part of a patch caused the bug, by building with or without each piece in a patch. | ||
| Line 143: | Line 141: | ||
# Bugs that have a patch which was almost correct, and has been commented on (perhaps minused or not), but no one finished the work. Perhaps the original developer just forgot about it. | # Bugs that have a patch which was almost correct, and has been commented on (perhaps minused or not), but no one finished the work. Perhaps the original developer just forgot about it. | ||
# Bugs that have been awaiting review for over 1 month | # Bugs that have been awaiting review for over 1 month | ||
# Bugs that have had review requested, but not from a specific person. This often happens by mistake of the reviewer name was entered incorrectly, in which case Bugzilla will flag the patch for review but not assign that to someone. These review requests are almost always ignored. If you think this is a mistake, try to assign to the review request to someone appropriate (this takes experience). If you're not sure, ping the patch author and ask them if it was | # Bugs that have had review requested, but not from a specific person. This often happens by mistake of the reviewer name was entered incorrectly, in which case Bugzilla will flag the patch for review but not assign that to someone. These review requests are almost always ignored. If you think this is a mistake, try to assign to the review request to someone appropriate (this takes experience). If you're not sure, ping the patch author and ask them if it was intentional. | ||
# Bugs where the patch author asked the reviewer a question, and received no answer | # Bugs where the patch author asked the reviewer a question, and received no answer | ||
# Patches which are reviewed but no one checked them in. Believe it or not, this happens all the time. Either the developer forgot, or did not have the correct CVS permissions to make a checkin. In this case they're supposed to request help from a developer with permission to check in. If they don't know that, a perfectly good fix may just sit there rotting away. | # Patches which are reviewed but no one checked them in. Believe it or not, this happens all the time. Either the developer forgot, or did not have the correct CVS permissions to make a checkin. In this case they're supposed to request help from a developer with permission to check in. If they don't know that, a perfectly good fix may just sit there rotting away. | ||