Confirmed users
14,525
edits
QA/Mozmill Test Automation/Fixing Broken Tests: Difference between revisions
QA/Mozmill Test Automation/Fixing Broken Tests (view source)
Revision as of 14:50, 25 August 2011
, 25 August 2011→Filing a Bug
| (18 intermediate revisions by 4 users not shown) | |||
| Line 8: | Line 8: | ||
|- valign="top" | |- valign="top" | ||
| '''Co-workers:''' | | '''Co-workers:''' | ||
| Anthony Hughes | | [mailto:ahughes@mozilla.com Anthony Hughes] (3.x), [mailto:gmealer@mozilla.com Geo Mealer] (4.0) | ||
|- valign="top" | |- valign="top" | ||
| '''Dates:''' | | '''Dates:''' | ||
| Line 20: | Line 20: | ||
|- valign="top" | |- valign="top" | ||
| '''Tracking Bug / Bug List:''' | | '''Tracking Bug / Bug List:''' | ||
| [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=mozmill-test-failure Open Broken Tests], [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-test-failure All Broken Tests] | | [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=mozmill-test-failure Open Broken Tests], [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-test-failure All Broken Tests] [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-branch-fail&component=Mozmill%20Tests&product=Testing Branch-only Failures] | ||
|} | |} | ||
| Line 34: | Line 34: | ||
1. You can run all existing tests on your own machine to identify failing tests. Therefore you should run the normal tests and the restart tests. See our documentation on [https://developer.mozilla.org/en/Mozmill_Tests#Running_Mozmill_Tests developer.mozilla.org] how to do that. | 1. You can run all existing tests on your own machine to identify failing tests. Therefore you should run the normal tests and the restart tests. See our documentation on [https://developer.mozilla.org/en/Mozmill_Tests#Running_Mozmill_Tests developer.mozilla.org] how to do that. | ||
2. Mozilla QA runs daily tests in the Mozilla Lab on all platforms. Test results for all those tests are send to our Brasstacks server. You can search through the results on the [http://brasstacks.mozilla.com/ | 2. Mozilla QA runs daily tests in the Mozilla Lab on all platforms. Test results for all those tests are send to our Brasstacks server. You can search through the results on the [http://mozmill-release.brasstacks.mozilla.com/#/general/reports reports page] or the [http://mozmill-release.brasstacks.mozilla.com/#/general/top top failures page]. | ||
3. <strike>Until a web dashboard will be available which will display all the correlations we track the results in the [https://spreadsheets.google.com/ccc?key=0AmkRt0ylPb8zdGpCMFB1bkU1SHp6eGh4Nkd3RzVKWEE&hl=en#gid=1 Mozmill Failures] spreadsheet</strike> See links in #2 above. | |||
If a test fails constantly make sure to run the same test again with the '--show-errors' option of the Mozmill CLI. The reported error message should be used in the bug report for a clear failure description. | If a test fails constantly make sure to run the same test again with the '--show-errors' option of the Mozmill CLI. The reported error message should be used in the bug report for a clear failure description. | ||
==Filing a Bug== | ==Filing a Bug== | ||
If a failure has been discovered, a search on Bugzilla has to be performed before filing a new bug. It's necessary to make sure that no-one else already created a bug for that failing test. There is a query which lists all [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-test-failure broken tests]. | If a failure has been discovered, a search on Bugzilla has to be performed before filing a new bug. It's necessary to make sure that no-one else already created a bug for that failing test. There is a query which lists all [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=mozmill-test-failure&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&known_name=mozmill-tests-broken&query_based_on=mozmill-tests-broken broken tests]. | ||
If none of the listed bugs covers the test or test module you are experiencing the failure in, please file a [https://bugzilla.mozilla.org/enter_bug.cgi?alias=&assigned_to=nobody%40mozilla.org&blocked=&bug_file_loc=http%3A%2F%2F&bug_severity=normal&bug_status=NEW&comment=-----%20INSTRUCTIONS%20FOR%20FILING%20A%20MOZMILL%20FAILURE%20BUG%20-----%0D%0ANOTE%3A%20REMOVE%20THIS%20SECTION%20BEFORE%20FILING%20THE%20BUG%0D%0A%0D%0A1.%20Include%20a%20URL%20above%20for%20the%20dashboard%20report%20where%20the%20failure%20occurs%0D%0A%0D%0AEXAMPLE%3A%0D%0Ahttp%3A%2F%2Fmozmill-release.brasstacks.mozilla.com%2F%23%2Ffunctional%2Ffailure%3Ftest%3D%252FtestSecurity%252FtestGreenLarry.js%26func%3DtestLarryGreen%0D%0A%0D%0A2.%20Include%20the%20test%20and%20failure%20type%20in%20the%20SUMMARY%20field%0D%0A%0D%0AEXAMPLE%3A%0D%0A%22Failure%20in%20%2FtestSecurity%2FtestGreenLarry.js%20%7C%20Timeout%20waiting%20for%20page%20loaded.%22%0D%0A%0D%0A3.%20Add%20%22%5Bmozmill-test-failure%5D%22%20to%20the%20whiteboard%20field%20below%0D%0A%0D%0A4.%20Include%20the%20test%20function%2C%20error%20message%2C%20date%20when%20the%20error%20occurred%2C%20date%20with%20the%20error%20first%20occurred%2C%20number%20of%20times%20the%20error%20has%20been%20reported%2C%20and%20the%20branches%20it%20occurs%20in%20the%20DESCRIPTION%20field%0D%0A%0D%0AEXAMPLE%3A%0D%0ATEST%3A%20testSecurity%2FtestGreenLarry.js%3AtestLarryGreen%28%29%0D%0AERROR%3A%20controller.waitForPageLoad%28%29%3A%20Timeout%20waiting%20for%20page%20loaded.%20%0D%0AWHEN%3A%202011-07-01%0D%0AFIRST%3A%202011-06-25%0D%0AFREQ%3A%205%0D%0ABRANCHES%3A%20default%2C%20aurora%2C%20beta%0D%0A%0D%0A5.%20Remove%20these%20instructions%0D%0A%0D%0A6.%20Click%20SUBMIT%20BUG%0D%0A-----%20END%20INSTRUCTIONS%20-----%0D%0A%0D%0ATEST%3A%20%0D%0AERROR%3A%20%0D%0AWHEN%3A%20%0D%0AFIRST%3A%20%0D%0AFREQ%3A%20%0D%0A&component=Mozmill%20Tests&contenttypeentry=&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&data=&dependson=&description=&flag_type-37=X&flag_type-4=X&flag_type-607=X&form_name=enter_bug&keywords=&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Linux&priority=--&product=Mozilla%20QA&qa_contact=mozmill-tests%40mozilla-qa.bugs&rep_platform=x86_64&requestee_type-325=&requestee_type-4=&requestee_type-607=&short_desc=&status_whiteboard=&target_milestone=---&version=unspecified new bug]. Make sure that the summary contains a copy of the error message and the test module it happens in. | |||
After filing a bug, you should add it to the [https://www.pivotaltracker.com/projects/289215# Pivotal] tracker (email [mailto:ahughes@mozilla.com Anthony] if you need access). | |||
* Click '''MORE''' | '''Bugzilla''' | |||
* Find your bug and drag it to '''ICEBOX''' | |||
* Change the story to a "Bug" and add the "failure" tag | |||
* Drag it to '''CURRENT''' to begin working on it | |||
==Disabling Test in Litmus== | |||
Unless you're going to be able to fix the error immediately (same day as discovered), please disable the test in Litmus. If you do not have the necessary access in Litmus, please contact one of the leads above to disable to test. | |||
# Remove the Mozmill group from the Litmus test corresponding to the broken test. | |||
# Note the Litmus test in the bug so that it can be re-enabled later. | |||
==Fixing a Broken Test== | ==Fixing a Broken Test== | ||
| Line 54: | Line 68: | ||
==Verifying a fix== | ==Verifying a fix== | ||
Once the patch has been checked-in, the bug will be marked as fixed. We will wait for the next test-run on our machine in the QA lab. If the problem has really gone we are fine and you did a great job. Otherwise we have to back-out the patch and go back to the investigation phase. But finally we should be able to fix any of the problems. | Once the patch has been checked-in, the bug will be marked as fixed. We will wait for the next test-run on our machine in the QA lab. If the problem has really gone we are fine and you did a great job. Otherwise we have to back-out the patch and go back to the investigation phase. But finally we should be able to fix any of the problems. | ||
==Re-enabling Test in Litmus== | |||
If the bug notes that a Litmus test has been disabled due to the failure, please re-enable the Litmus test after the fix is successfully verified. This is done by re-adding the Mozmill group to the Litmus test. | |||
If you do not have the necessary access to Litmus, please contact one of the leads above to re-enable the test. | |||