166
edits
QA/Bug Triage Guidelines: Difference between revisions
Jump to navigation
Jump to search
Review and updates
No edit summary |
Paul.oiegas (talk | contribs) (Review and updates) |
||
| Line 53: | Line 53: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! | ! State/Case !! Action | ||
|- | |- | ||
| Component || Where a Component is set to something other than Untriaged, leave the bug alone unless asked to do MozRegression or QAWanted | | Component || Where a Component is set to something other than Untriaged, leave the bug alone unless asked to do MozRegression or QAWanted | ||
|- | |- | ||
| Untriaged vs. Unconfirmed || The | | Untriaged vs. Unconfirmed || The goal is to reduce the Untriaged issues number, Ryan is not concerned with # of Unconfirmed. The end goal is to have all items placed in an appropriate component. If further confirmation is needed you will be asked | ||
|- | |- | ||
| New || Where there is no Component set one | | New || Where there is no Component, set one. Where it cannot be reproduced, ask the Reporter for additional information or close as Incomplete / WFM | ||
NOTE: Sometimes | NOTE: Sometimes, reporters mark issues as New without being confirmed by an Mozillian or Community Member. Read carefully to see if it needs confirmation to remain New or it needs to be changed back to Unconfirmed until it can be reproduced | ||
|- | |- | ||
| Crash || If there is no activity by Mozilla, ping or email Emma so that she can get someone to look into the issue. place the crash tag also. | | Crash || If there is no activity by Mozilla, ping or email Emma (:emceeaich) so that she can get someone to look into the issue. place the crash tag also. | ||
|- | |- | ||
| Mozilla Communications || | | Mozilla Communications || When there is a lot of Mozilla Communications within the bug, ping a developer on IRC and ask them to set the appropriate component | ||
|- | |- | ||
| Close Me || Apply at your comfort, there is not standard. If you apply, please give the reporter a chance to respond | | Close Me <date> || Apply at your comfort, there is not standard. If you apply, please give the reporter a chance to respond (at least 30 days) | ||
|- | |- | ||
| Enhancement & Feature Requests || Feature Request | | Enhancement & Feature Requests || Enhancements and Feature Request should have an appropriate Component assigned. The developers and PM in that component will decide if it will be implemented or marked as Resolved-Won't Fix | ||
NOTE: If you are sure that a Feature Request or an Enhancement is a valid one, you could also set his status to NEW, but usually this is the component's owner job. | |||
|- | |- | ||
| Context || Bugs that contain sensitive information or that may be harmful to the testers should be marked Context. These items can be reviewed with Ryan or Emma | | Context || Bugs that contain sensitive information or that may be harmful to the testers should be marked Context. These items can be reviewed with Ryan or Emma | ||
| Line 85: | Line 86: | ||
|- | |- | ||
| Reproducible || User Agent | | Reproducible || <User Agent> | ||
I have tested your issue on latest FF release (version), latest Nightly build (buid ID) and managed to reproduce it. I have <what and how it has been tested> | I have tested your issue on latest FF release (version), latest Nightly build (buid ID) and managed to reproduce it. I have <what and how it has been tested> | ||
| Line 95: | Line 96: | ||
For the cases when reporter did not added requested info and we cannot reproduce (bug has explicit STR): | For the cases when reporter did not added requested info and we cannot reproduce (bug has explicit STR): | ||
2.Considering the fact that I cannot reproduce this and the fact that the reporter did not answered to my/<name> request until now, I will mark this as Resolved-Worksforme. | 2.Considering the fact that I cannot reproduce this and the fact that the reporter did not answered to my/<name> request until now, I will mark this as Resolved-Worksforme. | ||
| Line 105: | Line 108: | ||
However, I am assigning a component to this issue in order to involve the development team and get an opinion on this. | However, I am assigning a component to this issue in order to involve the development team and get an opinion on this. | ||
|- | |- | ||
| Crash not encountered anymore || User Agent + "I have tried to reproduce this crash on latest FF release (versions) and Nightly build (build ID) but without any luck." <- optional sentence if there are STR or testcase | | Crash not encountered anymore || <User Agent> + "I have tried to reproduce this crash on latest FF release (versions) and Nightly build (build ID) but without any luck." <- optional sentence if there are STR or testcase | ||
There are no crash reports in the last 7/14/28 days in the crash signatures provided, therefore closing this bug as Resolved: Works for me. Please reopen the bug if the crash still reproduces and provide more information. | There are no crash reports in the last 7/14/28 days in the crash signatures provided, therefore closing this bug as Resolved: Works for me. Please reopen the bug if the crash still reproduces and provide more information. | ||
| Line 112: | Line 115: | ||
Information on the tool is available at http://mozilla.github.io/mozregression/. Please don't hesitate to contact us if you encounter any problems. | Information on the tool is available at http://mozilla.github.io/mozregression/. Please don't hesitate to contact us if you encounter any problems. | ||
|- | |- | ||
| MozRegression: When we find it is fixed || It | | MozRegression: When we find it is fixed || It was confirmed broken in version xx but it is fixed in Nightly <version> | ||
Use MozRegression to find where the fix occurred - | Use MozRegression to find where the fix occurred - If time permits, probably lower priority | ||
|- | |- | ||
| MozRegression: When we find it broke || If it was working in | | MozRegression: When we find it broke || If it was working in version xx and it is now broken the MozRegression should be applied to find where that break occurred | ||
|} | |} | ||
| Line 135: | Line 138: | ||
|- | |- | ||
| Using Mozzregression Tool on Mac OS || | | Using Mozzregression Tool on Mac OS || If you need to find when an issue was introduced (regression), the easiest and best way to do this is by using the Mozregression tool. | ||
In order to do this, you'll need to install a couple of things before starting the tool. Here are the steps needed depending on OS needed for testing: | In order to do this, you'll need to install a couple of things before starting the tool. Here are the steps needed depending on OS needed for testing: | ||
| Line 145: | Line 148: | ||
Here are the complete steps to install and use Mozregression trough Virtual environment: | Here are the complete steps to install and use Mozregression trough Virtual environment: | ||
1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br | 1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br> | ||
2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo easy_install pip".<br | 2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo easy_install pip".<br> | ||
3. Next you will need to install the virtual environment using the next command "sudo pip install virtualenv".<br | 3. Next you will need to install the virtual environment using the next command "sudo pip install virtualenv".<br> | ||
4. After the virtual environment was installed, you will need to provide a directory name for the virtual <br | 4. After the virtual environment was installed, you will need to provide a directory name for the virtual environment using <br>"virtualenv <directory_name>" command (eg: "virtualenv mozregression" will create the environment folder under your current profile).<br> | ||
5. Next you will need to activate the environment from a file that was created in the environment folder using "source /Users/profile_name/folder_name/bin/activate" command (eg: "source /Users/pauloiegas/mozregression/bin/activate").<br | 5. Next you will need to activate the environment from a file that was created in the environment folder using "source /Users/profile_name/folder_name/bin/activate" command (eg: "source /Users/pauloiegas/mozregression/bin/activate").<br> | ||
6. You will notice that your command line is now changed into "(mozregression) <code>". Now you will need to install the mozregression tool into the virtual environment using "pip install mozregression" command.<br | 6. You will notice that your command line is now changed into "(mozregression) < code >". Now you will need to install the mozregression tool into the virtual environment using "pip install mozregression" command.<br> | ||
7. After the installation is completed, you can check if everything is ok using a simple command like "mozregression -h" to see if the tool help is displayed (if not you did something wrong along the way).<br | 7. After the installation is completed, you can check if everything is ok using a simple command like "mozregression -h" to see if the tool help is displayed (if not, you did something wrong along the way).<br> | ||
8. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br | 8. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br> | ||
9. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional | 9. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional information that we consider to be helpful.<br> | ||
10. You will need to deactivate the virtual environment after you complete the regression using | 10. You will need to deactivate the virtual environment after you complete the regression using "deactivate" command that will return you to the normal command line.<br> | ||
11. REMEMBER ! The environment must be activated each time when you want to use the Mozregression tool.<br | 11. '''REMEMBER''' ! The environment must be activated each time when you want to use the Mozregression tool.<br> | ||
• Mac OS X (10.10 and under) | • Mac OS X (10.10 and under) | ||
1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br | 1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br> | ||
2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo easy_install pip".<br | 2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo easy_install pip".<br> | ||
3. After the Python installation is finalized, you'll need to install the latest version of mozregression by entering the next command "sudo pip2 install -U mozregression"<br | 3. After the Python installation is finalized, you'll need to install the latest version of mozregression by entering the next command "sudo pip2 install -U mozregression"<br> | ||
4. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br | 4. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br> | ||
5. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional informations that we consider to be helpful.<br | 5. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional informations that we consider to be helpful.<br> | ||
If something has been missed you can find all the mentioned information also at this link -> http://mozilla.github.io/mozregression/quickstart.html. | If something has been missed you can find all the mentioned information also at this link -> http://mozilla.github.io/mozregression/quickstart.html. | ||
| Line 173: | Line 176: | ||
| Instaling virtual Environment for Mozregression to work on Mac OS || It seams that now days, not only the Mac OS 10.11 has some conflicts between python and mozregression when trying to do a regression window. In order to resolve this conflicts, you will need to install and use mozregression trough a virtual environment. Here are the steps you need to follow in order to do that: | | Instaling virtual Environment for Mozregression to work on Mac OS || It seams that now days, not only the Mac OS 10.11 has some conflicts between python and mozregression when trying to do a regression window. In order to resolve this conflicts, you will need to install and use mozregression trough a virtual environment. Here are the steps you need to follow in order to do that: | ||
1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br | 1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Mac OS user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br> | ||
2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo easy_install pip".<br | 2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo easy_install pip".<br> | ||
3. Next you will need to install the virtual environment using the next command "sudo pip install virtualenv".<br | 3. Next you will need to install the virtual environment using the next command "sudo pip install virtualenv".<br> | ||
4. After the virtual environment was installed, you will need to provide a directory name for the virtual environment using "virtualenv <directory_name>" command (eg: "virtualenv mozregression" will create the environment folder under your current profile).<br | 4. After the virtual environment was installed, you will need to provide a directory name for the virtual environment using "virtualenv <directory_name>" command (eg: "virtualenv mozregression" will create the environment folder under your current profile).<br> | ||
5. Next you will need to activate the environment from a file that was created in the environment folder using "source /Users/profile_name/folder_name/bin/activate" command (eg: "source /Users/pauloiegas/mozregression/bin/activate").<br | 5. Next you will need to activate the environment from a file that was created in the environment folder using "source /Users/profile_name/folder_name/bin/activate" command (eg: "source /Users/pauloiegas/mozregression/bin/activate").<br> | ||
6. You will notice that your command line is now changed into "(mozregression) <code>". Now you will need to install the mozregression tool into the virtual environment using "pip install mozregression" command.<br | 6. You will notice that your command line is now changed into "(mozregression) < code >". Now you will need to install the mozregression tool into the virtual environment using "pip install mozregression" command.<br> | ||
7. After the installation is completed, you can check if everything is ok using a simple command like "mozregression -h" to see if the tool help is displayed (if not you did something wrong along the way).<br | 7. After the installation is completed, you can check if everything is ok using a simple command like "mozregression -h" to see if the tool help is displayed (if not you did something wrong along the way).<br> | ||
8. You will need to deactivate the virtual environment after you complete the regression using ""deactivate" command that will return you to the normal command line.<br | 8. You will need to deactivate the virtual environment after you complete the regression using ""deactivate" command that will return you to the normal command line.<br> | ||
9. REMEMBER ! The environment must be activated each time when you want to use the Mozregression tool.<br | 9. '''REMEMBER''' ! The environment must be activated each time when you want to use the Mozregression tool.<br> | ||
|- | |- | ||
| Using Mozzregression Tool on Windows || On Windows platform you can either use the Mozregression gui from github releases page, or you can use the tool trough command line as on Mac OS. In the next steps I will present the command line way on Windows. | | Using Mozzregression Tool on Windows || On Windows platform you can either use the Mozregression gui from github releases page, or you can use the tool trough command line as on Mac OS. In the next steps I will present the command line way on Windows. | ||
1. First, you will need to download and install Python 2.7 from ActiveState website (http://www.activestate.com/activepython/downloads).<br | 1. First, you will need to download and install Python 2.7 from ActiveState website (http://www.activestate.com/activepython/downloads).<br> | ||
2. After installing Python you will need to open a command prompt window and enter the next command "pip install -U mozregression".<br | 2. After installing Python you will need to open a command prompt window and enter the next command "pip install -U mozregression".<br> | ||
3. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br | 3. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br> | ||
4. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional informations that we consider to be helpful.<br | 4. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional informations that we consider to be helpful.<br> | ||
If I missed something you can find all the mentioned informations also at this link -> http://mozilla.github.io/mozregression/quickstart.html. | If I missed something you can find all the mentioned informations also at this link -> http://mozilla.github.io/mozregression/quickstart.html. | ||
| Line 196: | Line 199: | ||
| Using Mozzregression Tool on Linux || On Linux platform you can either use the Mozregression gui from github releases page, or you can use the tool trough command line as on Mac OS. In the next steps I will present the command line way on Linux. | | Using Mozzregression Tool on Linux || On Linux platform you can either use the Mozregression gui from github releases page, or you can use the tool trough command line as on Mac OS. In the next steps I will present the command line way on Linux. | ||
1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Linux user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br | 1. First, you will need to open a Terminal window and enter the following command "sudo su" (without quotes). You will be prompted to insert the PC user password (the password used for logging in on the Linux user). If you don't see the characters you are entering, don't worry, they are entered but due to security are hidden. After entering them press the "Enter" key.<br> | ||
2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo apt-get install python-pip". You will be asked if you are sure you want to install it and you must confirm by entering "y" key and confirm the choice after.<br | 2. After credentials are confirmed it's time to install the Python 2.7 component by entering the next command "sudo apt-get install python-pip". You will be asked if you are sure you want to install it and you must confirm by entering "y" key and confirm the choice after.<br> | ||
3. After the Python installation is finalized, you'll need to install the latest version of mozregression by entering the next command "sudo pip2 install -U mozregression"<br | 3. After the Python installation is finalized, you'll need to install the latest version of mozregression by entering the next command "sudo pip2 install -U mozregression"<br> | ||
4. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br | 4. After doing this you are ready to start digging into builds. You can do that by using the next two commands "mozregression --good <build_date>" (eg: mozregression --good 2014-12-31) when you only know the build when it was ok, or "mozregression --good <build_date> --bad <build_date>" (eg: mozregression --good 2014-12-31 --bad 2015-12-31) when you know both a good build and a bad build.<br> | ||
5. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional informations that we consider to be helpful.<br | 5. Mozregression will start downloading builds and open them to test your issue. After testing the issue you must validate the build with "good" or "bad". The tool will continue to download and open builds until the search is narrowed down to two builds and will throw a "Pushlog link" that you will need to provide in the bug comment together with any additional informations that we consider to be helpful.<br> | ||
If I missed something you can find all the mentioned informations also at this link -> http://mozilla.github.io/mozregression/quickstart.html. | If I missed something you can find all the mentioned informations also at this link -> http://mozilla.github.io/mozregression/quickstart.html. | ||
| Line 268: | Line 271: | ||
|- | |- | ||
| Manual regression and manual creation of Pushlog link || When Mozregression tool fails to work (errors when trying to open the builds for test), or when you cannot use it because you need to perform some steps outside the browser before you can test an issue, all you can do is manual regression testing. But in order to help locate the issue you will still need the "Pushlog link" that points to the exact pushes that were made in the specific day when issue appeared. In order to create the pushlog link manually, you will need the next things: | | Manual regression and manual creation of Pushlog link || When Mozregression tool fails to work (errors when trying to open the builds for test), or when you cannot use it because you need to perform some steps outside the browser before you can test an issue, all you can do is manual regression testing. But in order to help locate the issue you will still need the "Pushlog link" that points to the exact pushes that were made in the specific day when issue appeared. In order to create the pushlog link manually, you will need the next things: | ||
1. Latest good build and the first bad build where issue appeared | |||
2. You will need to either install both the builds (one at the time) or extract them. The most convenient is to extract both the zipped builds. | |||
3. Search in installation folder/build folder for the "application.ini" file (Mac OS path is /Applications/Firefox.app/Contents/MacOS/application.ini) | |||
4. Open the ".ini" files with a text editor and search for the "SourceStamp" values | |||
5. After doing this you will have to use the " http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=good_build_stamp&tochange=bad_build_stamp " link and replace the <good_build_stamp> and <bad_build_stamp> strings with the one obtained from the ini files. | |||
You should verify the link obtained, and if it works and it is related with the period when the issue appeared, you should add it to the comment for the bug where regression was performed. If you have enough experience you could also point in the comment the exact push from the pushlog page that affected the build, and CC the developer that did it (if it's still available) in order to obtain th information faster. For the moment Rayan was doing this for the pushlog links I provided but we can experience this too in the future. | |||
|- | |- | ||
| Performing quick Regression test by searching in build release notes || In cases when reporter mentions that in a previous version of Firefox his issue was not present and after updating it appeared, you can also do a quick manual regression by searching the problematic build release notes and observe the changes that have been made on it. To better explain this process I will refer to an Issue that I triaged, where the reported problem was an audio malfunction in an html5 game that he an his team develops, that appeared after updating from Firefox 39.0 to 40.0. | | Performing quick Regression test by searching in build release notes || In cases when reporter mentions that in a previous version of Firefox his issue was not present and after updating it appeared, you can also do a quick manual regression by searching the problematic build release notes and observe the changes that have been made on it. To better explain this process I will refer to an Issue that I triaged, where the reported problem was an audio malfunction in an html5 game that he an his team develops, that appeared after updating from Firefox 39.0 to 40.0. | ||