124
edits
QA/Triage: Difference between revisions
Jump to navigation
Jump to search
→How to Help with Regressions -- Finding Regression Windows: add some more build id and bonsai info
(→How to Clean Incoming Bugs: fixing linebreaks to unhork the list numbering; also link to QA's Talkback home) |
(→How to Help with Regressions -- Finding Regression Windows: add some more build id and bonsai info) |
||
| Line 122: | Line 122: | ||
== How to Help with Regressions -- Finding Regression Windows == | == How to Help with Regressions -- Finding Regression Windows == | ||
Another great thing that QAs sometimes help developers with is finding a regression window for regressions. A regression window is the day that a bug broke. With this information we can take a look at the checkin list and through the process of elimination | Another great thing that QAs sometimes help developers with is finding a regression window for regressions. A regression window is the day that a bug broke. With this information we can take a look at the checkin list and through the process of elimination see what change caused the problem. It's always important to find the regression window, because without it a developer may apply a bandaid fix, instead of finding the true source of a problem. | ||
see what change caused the problem. It's always important to find the regression | |||
window, because without it a developer may apply a bandaid fix, instead of finding | |||
the true source of a problem. | |||
Here are some steps for finding a regressions window: | Here are some steps for finding a regressions window: | ||
| Line 132: | Line 129: | ||
# Use a separate dedicated profile, without any extensions installed and start with -profilemanager. | # Use a separate dedicated profile, without any extensions installed and start with -profilemanager. | ||
# Test the build and write down the results. It's common to get confused since you are testing for the absence of a bug, so good notes are important. If the bug occurs, then you know you will have to search back further in time. Otherwise you will need to search forward in time. The optimal algorithm is to test the date exactly between the last known date when the bug didn't occur and the last known date when the bug did occur. Keep dividing by two and you'll find the day. | # Test the build and write down the results. It's common to get confused since you are testing for the absence of a bug, so good notes are important. If the bug occurs, then you know you will have to search back further in time. Otherwise you will need to search forward in time. The optimal algorithm is to test the date exactly between the last known date when the bug didn't occur and the last known date when the bug did occur. Keep dividing by two and you'll find the day. | ||
# In the end, make sure that the bug is marked with the keyword "regression", and add a very clear comment something like: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken". When you have found a regression window within a day, then the actual hour the product was built becomes important. You can give this information with the [ | # In the end, make sure that the bug is marked with the keyword "regression", and add a very clear comment something like: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken". When you have found a regression window within a day, then the actual hour the product was built becomes important. You can give this information with the [[MozillaQualityAssurance:Build_Ids|build ID]]. Another option is to look at the time when the "firefox.exe" or "thunderbird.exe" was last modified. Then take a larger margin (by the hour) of that time. A third option, if you're paying attention when downloading builds in step 2 above, is to note the sets of numbers in the directory name; stripping the dashes from that string gives you the build ID in most cases (e.g., a build from <tt>/pub/firefox/nightly/2005-08-27-19-trunk/</tt> is a trunk build with a build ID of 2005082719). | ||
# An experienced QA can pinpoint the actual checkin with the problem, and CC the developer who was responsible to the bug. Finding out which checkin was responsible for the regression is possible with the [http://bonsai.mozilla.org/cvsqueryform.cgi Bonsai tool]. At the bottom of that page you can add dates ("between ... and ..."). When submitting the form, you get the list of checkins between those dates. | # An experienced QA can pinpoint the actual checkin with the problem, and CC the developer who was responsible to the bug. Finding out which checkin was responsible for the regression is possible with the [http://bonsai.mozilla.org/cvsqueryform.cgi Bonsai tool]. At the bottom of that page you can add dates ("between ... and ..."). When submitting the form, you get the list of checkins between those dates. When you know the build IDs of the last working and first broken builds, expand them back into dates and times to get a more precise set of results (e.g., a build ID of 2005082719 becomes 2005-08-27 19:00). | ||
# If you are very technical, it is possible to go even further. Some QAs who know how to apply patches and build their own Mozilla have even been known to find which part of a patch caused the bug, by building with or without each piece in a patch. However, this is really not necessary, so don't worry if you're not comfortable doing that. | |||
If you are very technical, it is possible to go even further. Some QAs who know | |||
how to apply patches and build their own Mozilla have even been known to find which | |||
part of a patch caused the bug, by building with or without each piece in a patch. | |||
However, this is really not necessary, so don't worry if you're not comfortable | |||
doing that. | |||
Again, from all Mozilla developers to the QAs who find regression windows, "Thank You!" | Again, from all Mozilla developers to the QAs who find regression windows, "Thank You!" | ||