TestEngineering/Web/Automation/Gaia UI Testing: Difference between revisions

TestEngineering/Web/Automation/Gaia UI Testing (view source)

Revision as of 03:37, 19 November 2013

475 bytes removed , 19 November 2013

→‎Monitoring Jenkins Jobs For Failures and XPasses

Bobsilverberg

Confirmed users

425

edits

@@ Line 40: / Line 40: @@
 ### You can also use the links above to view the latest results of the job, but this summary will not show you which jobs are xpassing. To do that you need to view the HTML report which is accessed via a link in the left nav. On the report, look for the summary near the top and look for the text that says 'x unexpected passes'. If 'x' is greater than 0, this build contained xpasses. You will need to look through the HTML report to find the tests that passed unexpectedly.
 ### For each xpass that you find:
-#### Look at the manifest.ini file that contains the test in question. It should contain a reference to a bug number for which the test was xfailed.
-#### Look at the bug in question and examine it:
-##### Has the
 #### Run the test locally to see if it also passes for you.
-##### If it does then it is a candidate to be un-xfailed.
+##### If the test passes locally for you and has been passing on CI for more than one consecutive run, then the test should be un-xfailed:
-##### If it does not then further
+###### Look at the manifest.ini file that contains the test in question. It should contain a reference to a bug number for which the test was xfailed.
-#### Look at the manifest.ini file that contains the test in question. It should contain a reference to a bug number for which the test was xfailed.
+###### Look at the bug in question and examine it:
-#### Look at the bug in question and examine it: for a bug in Bugzilla that has been opened for the failure. Every failure that needs to be addressed must have a bug opened for it. If you find an open bug for a failure, examine it:
+####### If the bug has been closed, that likely explains why the test is now passing. :)
-##### If it has a pull request attached to it which is waiting to be reviewed, review it. A pull request that is waiting to be reviewed is one that has no negative reviews and less than two positive reviews.
+####### If the bug has not been closed, add a comment to the bug seeking clarification, explaining that the test is now passing.
-##### If it is unassigned, and you would like to take the time to address it, assign it to yourself and update its status to ''Assigned''. Do not do this unless you have the time to address it.
+###### Check for an existing bug to un-xfail the test. If one exists it should be listed as being blocked by the current bug.
-#### If you do not find an open bug for the failure, open one and include as much detail about the failure as possible.
+###### If no bug exists to un-xfail the test, open one.
-##### If you have time to address the failure, assign it to yourself and update its status to ''Assigned''.
+###### Create a pull request to un-xfail the test and attach it to the un-xfail bug mentioned above.
 = Keeping Branches in Sync with Master =