Changes

Jump to: navigation, search

QA/Triage

16,966 bytes added, 18:47, 5 July 2006
Original text for triage guide
= Mozilla Triage Guide -- Harnessing the Flood of Community Bugs =

== Intro ==

This article is for anyone who is serious about QA in Mozilla. By
following these techniques you will help the important bugs get
fixed, as well as optimize precious developer time.

Mozilla is an amazing project. Bugs come in from just about
anywhere -- end users, developers and QA teams scattered
across many organizations. The great benefit is that the various
products are extremely thoroughly tested on a day to day basis and
everyone benefitis from the resulting quality. New regressions are
often noticed right away before other new fixes are built on top of
the problematic code. A diversity of opinions often allows the
right solution to bubble to the top, and old bugs that are still
important often come back to the surface instead of being
forgotten.

Unfortunately, even with all of this useful information, there is a
lot of noise. The community often files invalid, duplicate, already
fixed, improperly filed and relatively incomplete bugs. That
said, community bugs are an invaluable source of information, as
the testing coverage would not be nearly as wide without it. In
Mozilla, a QA must not only test and then file their own bugs, but
must also help harness the incoming flood of community bugs for
valid problems, putting these issues on track to be addressed.

In addition, new Mozilla QA's are coming into a very complex
project which may not operate in a way similar to projects they've
worked in before.

How does the QA community harness the flood of filed bugs into
better quality for Mozilla while maximizing productivity for
developers? Let's take a look ... there's <i>a lot</i> you can do without coding.

== How to Track Relevant Incoming Bugs ==
First we need to know how to look at the subset of bugs you wish to
specialize in.One way to track a subset of bugs is through
continual queries, perhaps one overy day or so. This is good if you
only plan to go through the bugs every so often, or if you need to
track something based on keywords or fields other than
product/component, or when you need to look through older bugs.

If you want something more immediate, you can choose to track a
component, such as Disability Access. This is possible via your
Bugzilla preferences. Make sure to track the relevant component in
both Core and in the products you care about, such as Firefox or
Thunderbird.

== How to Clean Incoming Bugs ==
Ensuring the bugs are properly marked up helps make sure they don't get ignored.
For example, if a bug that really affects all platforms is marked "Solaris" because
the original bug author was on a Solaris machine, most developers will ignore it
they need to focus on the bugs that affect the most people. If a bug is in the wrong
component it can also get stuck without any attention. Another common issue is if
truly major bugs are marked normal, or have a confusing summary. So, ensuring that
the bug is properly marked up is very helpful for getting it attention as well as
keeping it from being ignored by appropriate queries that developers and other QAs
run.
Do the following steps in batches, if possible. Each time you submit changes to
the bug it creates extra Bugmail, so it's better to do the changes in as few rounds
as possible. These techniques really apply to bugs QA's are filing as well.
# <u>Understand the bug:</u> Read very carefully to try and understand what they're talking about.
# <u>Look for duplicates:</u> A lot of bug reporters mistakenly assume their bug won't have been filed yet. Search for keywords in the bug, and try alternate spellings and synonyms. For example, if the bug is about the control key, try spelling it ctrl as well. Look through fixed bugs when you search for duplicates.
# <u>Ask for missing information or clarification</u>: You can't test every bug yourself, especially those with incomplete information. Try not to waste your own time. Keep asking the reporter for more information until you have enough to try it yourself. If they did not provide build information get that right away, because many bug reports are for older versions that have long been fixed.
# <u>Reproduce the bug</u>: Change the status from Unconfirmed to Confirmed if appropriate. Many times bug reports are confusing and difficult to reproduce. Often times, reading the bug description over very carefully will reveal clues, and bring up questions. If you are convinced the bug was filed in error, and requests for clarification don't net results, then mark the bug WONTFIX or INVALID. When marking WONTFIX or INVALID, be sensitive to the fact the the bug reporter may get offended. While changing the status it's often good to write something like "This appears to work as designed, please reopen and provide more information if you believe this is in error."
# <u>Simplify the bug testcase</u>. You're the lead investigator -- so try to get enough information to turn intermittent or hard to reproduce bugs into something that is easy to reproduce. See more on this in "How to Really, Really Help Developers on Bugs" below.
# <u>Ensure appropriate hardware/platform tests</u>: most bug filers will only have tested on one platform. If bug was filed for a particular platform, see if it's a bug on another platform. If it's been reproduced on at least 2 platforms that aren't both Unix/Linux, then go ahead and mark it All/All ((Hardware/OS). Most bugs are actually All/All. It's useful to get a feeling of when to do extra tests and when not to waste your time, but this comes with experience. For example, OS X uses many native Carbon form controls and menus whereas Windows and Linux do not, so form control bugs on Mac are probably just Mac-only. However, a bug in how HTML links are dealt with is almost certainly a bug on all platforms. Also, it's useful to know the QA community so that you can ask for help when you don't have the right hardware/OS to test a particular bug with.
# <u>Ensure a consise and precise summary</u>: It is reasonable to edit the summary a bit if it will help people understand the bug more quickly or will clear up something that's misspelled or incorrect. Check to see that the summary is clear, concise and describes the problem as fully and correctly as is reasonable in a short space. Make sure it contains the words people would probably search for before filing a duplicate bug. To risk stating the obvious, the spelling in the summary matters because searches will for "Escape" will fail if the reporter wrote "Escpae".
# <u>Security sensitive bugs</u>: [TBD]
# <u>Ensure Proper Severity</u> (but don't change Priority field, which is for the developer). Here's a quick guide, but you can't get more help from Bugzilla itself
#* Enhancement: ideas for improving the software.
#* Critical: crash and security bugs (should also be marked with keyword crash).
#* Major: Bugs that affect the main screen, cause dataloss, loss of ability to operate the software or are just plain awful
#* Normal/Minor/Trival: Most other bugs
# <u>Change Product, Component and owner</u>: Many bugs get marked "Firefox" when they are really bugs in the core engine. If it's a basic XUL, HTML or layout bug that affects multiple products send it to Core. After you submit a Product change to the bug, Bugzilla will allow you to change the Component as well. When changing Product and/or Component you usually want to select the radio button "Reassign bug to default assignee and QA contact of selected component ". The exception to this is when the bug is a regression and already assigned to a particular developer who caused the regression. You can spot those situations via a comment like "Joe, this appears to be from your checkin from bug 32412".
# <u>Add Keywords</u> -- here are just a few of the important ones [https://bugzilla.mozilla.org/describekeywords.cgi the full list of keywords is here]).
#* "access": mark accessibility bugs with the keyword "access". This allows accessibility bugs to not be tied to the "Disability Access" or "Keyboard navigation" components, but to the actual component with the problem. For example, the developers for Places should really be responsible for accessibility bugs in Places, but adding the keyword "access" makes sure
#* "regression": mark regressions with "regression". Was this ever not a bug? For example, did it work in Firefox 1.5? See below for more important information on how to deal with regressions.
#* "crash": pretty obvious :)
#* "dataloss": this bug will cause users to lose data
#* "hang": freezes or locks up the application
#* "testcase": indicates that a simplifed testcase is included (see below)
# <u>List related meta bugs</u> in "this bug blocks". If an important meta bug has a difficult to remember number, give it a memorable alias. For example, here are some important accessibility-related meta bugs:
#* focusnav: related to focus and tab navigation
#* caretnav: related to caret navigation
#* themea11y: affects use with high contrast accessibility themes

== How to Really, Really Help Developers on Bugs -- Minimal Testcases ==

One really important thing that QAs can do for developers is to narrow down to the actual markup
causing the bug -- this applies for QA-filed bugs as well. Many testcases
filed by inexperienced bug reporters come from a website, which unforunately may
change and thus the testcase is lost. Or, the
HTML from the given testcase might be huge, which makes debugging extra difficult.

Here are some steps for using the process of elimination to create a minimal testcase from a webpage that displays a consistent bug.
# Save the webpage on your local hard drive. Make a second copy.
# From the first copy, cut out big pieces of CSS, JS or HTML -- start with the pieces that are most likely to not have the bug For example, at first just try to strip out plain paragraphs as these are relatively simple.
# If the new version of the file still has the bug, save it as your second copy and repeat step 2.
# If the new version does not have the bug, then your second copy still contains the bug. In this case, repeat step 2 but avoid cutting out the same piece that you did last time.
# When you can no longer cut out any HTML without destroying the ability of the file to show the bug, you have a minimal testcase.

Hopefully you'll be left with a file that has only 1-5 HTML elements
in it, but still reproduces the bug. At this point you can actually try to start stripping
out irrelevant attributes as well You may end up with something like &lt;label for="myfile"&gt;Enter
file name:&lt;/label&gt;&lt;input type="file" disabled="true" id="myfile"&gt; instead
of 50k worth of HTML. From all the developers you do this for, "Thank You!".

When finished, attach the minimal testcase, mark the bug with keyword testcase, and change the bug summary if necessary.
For example, "No borders in tables at http://www.sillystatsandstuff.com"
can become something much more precise such as "tables with rowspans do not get borders".

Keep in mind that you need to use your brain throughout this process. For example,
if the bug is layout flow related, then the width of your window and font size may
impact the results.

== How to Help with Regressions -- Finding Regression Windows ==
Another great thing that QAs sometimes help developers with is finding a regression window for regressions. A regression window is the day that a bug broke. With this information we can take a look at the checkin list and through the process of elimination
see what change caused the problem. It's always important to find the regression
window, because without it a developer may apply a bandaid fix, instead of finding
the true source of a problem.

Here are some steps for finding a regressions window:
# Take a guess when the bug might have broken -- any guess will do for a start, so just use your instincts.
# Download and install a build from that date from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ or from http://archive.mozilla.org/pub/ if you need to go back even further. I recommend installing the build in a directory named something like Firefox-03-11-06 so that you can go back to it during this test.
# Run the test build. Either make sure you have no other Firefox windows open, or that you have MOZ_NO_REMOTE=1. If you don't, you will actually be running the version of Firefox already in memory. This mistake is common. To ensure you are running the build you think that you are, go to Help -&gt; About.
# Test the build and write down the results. It's common to get confused since you are testing for the absence of a bug, so good notes are important. If the bug occurs, then you know you will have to search back further in time. Otherwise you will need to search forward in time. The optimal algorithm is to test the date exactly between the last known date when the bug didn't occur and the last known date when the bug did occur. Keep dividing by two and you'll find the day.
# In the end, make sure that the bug is marked with the keyword "regression", and add a very clear comment something like: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken".<li>An experienced QA can pinpoint the actual checkin with the problem, and assign the bug to the developer who was responsible.

If you are very technical, it is possible to go even further. Some QA's who know
how to apply patches and build their own Mozilla have even been known to find which
part of a patch caused the bug, by building with or without each piece in a patch.
However, this is really not necessary, so don't worry if you're not comfortable
doing that.

Again, from all Mozilla developers to the QAs who find regression windows, "Thank You!"

== How to Deal with Ignored Patches [Needs work] == It's probably useful for someone to search for these once in a while. Often there are nearly complete fixes that "rot" because no one finishes the job, and the Mozilla code that needs to blend with it changes, thus making the fix more and more difficult to apply over time.

There are several types of bugs here:
# Bugs that have a patch which was almost correct, and has been commented on (perhaps minused or not), but no one finished the work. Perhaps the original developer just forgot about it.
# Bugs that have been awaiting review for over 1 month
# Bugs that have had review requested, but not from a specific person. This often happens by mistake of the reviewer name was entered incorrectly, in which case Bugzilla will flag the patch for review but not assign that to someone. These review requests are almost always ignored. If you think this is a mistake, try to assign to the review request to someone appropriate (this takes experience). If you're not sure, ping the patch author and ask them if it was intentiional.
# Bugs where the patch author asked the reviewer a question, and received no answer
# Patches which are reviewed but no one checked them in. Believe it or not, this happens all the time. Either the developer forgot, or did not have the correct CVS permissions to make a checkin. In this case they're supposed to request help from a developer with permission to check in. If they don't know that, a perfectly good fix may just sit there rotting away.

== How to Deal with Ignored Bugs [Needs work] ==

Options:
# Evangelize: go to newsgroups and try to stir up interest there
# Email the developer it's assigned to and ask him what his plan is for it, and if there is any way you can help, or whether it should be assigned to someone else
# If major, CC relevant one or more developers that you know of, but don't overdo it, especially for those that already seem overwhelmed
# Add new information that will make it easy to debug
# Ask a developer in your organization about the bug, see if you can get someone to take the bug if it's assigned to someone who doesn't seem to care about it (months of no activity for a normal bug, weeks for a critical bug)
# Mark it "Blocking?"
# Raise the severity
# Add or change a target milestone
# If it's really bad, ask on IRC
# Known your developers
# <u>Use the status whiteboard</u>, especially on reopened or difficult-to-understand bugs. Many bugs collect so many comments that developers naturally delay working on them -- it's a huge amount of work just to sift through all that information. Here's where the status whiteboard is helpful -- it's where you can put any information about the current wisdom that's been collected about the bug, and what it's waiting for. This does two things -- the developers can get a quick summary of what needs to get done, and it reminds them about the bug via a Bugmail if they get bugmails for status whiteboard changes. A valid status whiteboard entry might be "Fixed for testcase 1, still a problem for testcase 2".
# Mark with keyword "helpwanted"

TBD: add info about branches vs. trunk and fill out sections on ignored patches and bugs
346
edits

Navigation menu