QA/Regression Windows

Why Regression Windows are so Important

Another great thing that QAs sometimes help developers with is finding a regression window for previously fixed bugs that seen back on the browser. To be short, it is the day that a bug initially broke. With this information we can take a look at the check-in list and through the process of elimination see what change, or patch, caused the problem. It's always important to find the regression window, because without it a developer may apply a band-aid fix, instead of finding the true source of the problem.

The Process of Finding a Regression Window

The work done to search for a regression window can be very quick if the person has been viewing the bug list for more than a couple of months. But, if you're reading this page, then that probably means you haven't been. So, I'll just tell you right now that don't get too frustrated if it's taking a long time for you or you just aren't able to find it. If you do become too flustered to really find it yourself, go over to the #qa channel on irc.mozilla.org and ask for some help! Anyways, without further ado, here's is a general step-by-step approach to finding these windows:

1. Take a guess when the bug might have broken -- any guess will do for a start, so just use your instincts.
2. Download and install a build from that date from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/. I recommend installing the build in a directory named something like Firefox-03-11-06 so that you can go back to it during this test.
3. Use a separate dedicated profile, without any extensions installed. If you don't know how to do this, go to this handy Support@Mozilla page.
4. Test the build and write down the results. It's common to get confused since you are testing for the absence of a bug, so good notes are important. If the bug occurs, then you know you will have to search back further in time. Otherwise you will need to search forward in time. The best approach is to test the date exactly between the last known date when the bug didn't occur and the last known date when the bug did occur. Keep dividing by two and you'll find the day.
5. After you've found the regression window, make sure the bug report is marked with the keyword "regression", and add a very clear comment that looks something like the following: "March 19, 2006 trunk: works, March 20, 2006 trunk: broken". When you have found a regression window within a day, then the actual hour the product was built becomes important. You can give this information with the build ID. Another option is to look at the time when the "firefox.exe" or "thunderbird.exe" was last modified. Then take a larger margin (by the hour) of that time. A third option, if you're paying attention when downloading builds in step 2 above, is to note the sets of numbers in the directory name; stripping the dashes from that string gives you the build ID in most cases (e.g., a build from pub/mozilla.org/firefox/nightly/2005/08/2005-08-27-19-trunk/ is a trunk build with a build ID of 2005082719).
6. An experienced QA can pinpoint the actual check-in with the problem, and CC the developer who was responsible to the bug. Finding out which check-in was responsible for the regression is possible with the Bonsai tool. At the bottom of that page you can add dates ("between ... and ..."). When submitting the form, you get the list of check-ins between those dates. When you know the build IDs of the last working and first broken builds, expand them back into dates and times to get a more precise set of results (e.g., a build ID of 2005082719 becomes 2005-08-27 19:00).
7. If you are very technical, it is possible to go even further. Some QAs who know how to apply patches and build their own Mozilla have even been known to find which part of a patch caused the bug, by building with or without each piece in a patch. However, this is really not necessary, so don't worry if you're not comfortable doing that.