TestEngineering/Web/Automation/Gaia UI Testing: Difference between revisions

TestEngineering/Web/Automation/Gaia UI Testing (view source)

Revision as of 22:02, 25 November 2013

4,212 bytes added , 25 November 2013

no edit summary

Bobsilverberg

Confirmed users

425

edits

@@ Line 15: / Line 15: @@
 '''Note:''' This is very much a work in progress. Do not take anything below as gospel until it has been finalized.
+The first priority, when supporting Gaia UI Testing, is to ensure that our existing tests are accurate and reliable. To that end, we need to monitor the jobs that run in our CI and address any that are failing or are xpassing (passing but marked as expected to fail).
+To review the jobs for failures and xpasses, complete the following steps:
+# Choose the job to review. This involves choosing the branch and device.
+## Which branch to review?
+### We currently run jobs against the [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Master/ master] and [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20v1.2/ v1.2] branches of Gaia. When reviewing the job results for failures and xpasses, always review master first, and then v1.2.
+## Which device to review?
+### We currently run jobs on two different devices, [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/ Hamachi] and [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Inari/ Inari]. When reviewing the job results for failures and xpasses, always review Hamachi first, and then Inari.
+## Taking the above into consideration, the order for review should be:
+### [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Master/job/b2g.hamachi.mozilla-central.ui/lastCompletedBuild/testReport/ Hamachi master]
+### [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Hamachi/job/b2g.hamachi.mozilla-b2g26_v1_2.ui/lastCompletedBuild/testReport/ Hamachi v1.2]
+### [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Inari/job/b2g.inari.mozilla-central.ui/lastCompletedBuild/testReport/ Inari master]
+### [http://qa-selenium.mv.mozilla.com:8080/view/B2G%20Inari/job/b2g.inari.mozilla-b2g26_v1_2.ui/lastCompletedBuild/testReport/ Inari v1.2]
+# Review the results of the latest run of the job in question.
+## Review failures.
+### You can use the links above to view the latest results of the job. This will give you a quick summary of which tests are currently failing. Looking at the ''Age'' of the failure will give you an idea of whether this is a new failure, or one that has been happening for awhile. Also, failures with an ''Age'' of 1 often indicate intermittent failures.
+### For each failure that you find:
+#### If the failure details say: '''Error Message''': test failure, '''Stacktrace''': N/A, then the failure is likely actually an xpass. Keep this in mind when going through the next steps.
+#### Search for a bug in Bugzilla that has been opened for the failure. Every failure that needs to be addressed must have a bug opened for it. If you find an open bug for a failure, examine it:
+##### If it has a pull request attached to it which is waiting to be reviewed, review it. A pull request that is waiting to be reviewed is one that has no negative reviews and less than two positive reviews.
+##### If it is unassigned, and you would like to take the time to address it, assign it to yourself and update its status to ''Assigned''. Do not do this unless you have the time to address it.
+#### If you do not find an open bug for the failure, open one and include as much detail about the failure as possible.
+##### If you have time to address the failure, assign it to yourself and update its status to ''Assigned''.
+## Address xpasses.
+### For each xpass that you found during the Review failures step:
+#### Run the test locally to see if it also passes for you.
+##### If the test passes locally for you and has been passing on CI for more than one consecutive run, then the test should be un-xfailed:
+###### Look at the manifest.ini file that contains the test in question. It should contain a reference to a bug number for which the test was xfailed.
+###### Look at the bug in question and examine it:
+####### If the bug has been closed, that likely explains why the test is now passing. :)
+####### If the bug has not been closed, add a comment to the bug seeking clarification, explaining that the test is now passing.
+###### Check for an existing bug to un-xfail the test. If one exists it should be listed as being blocked by the current bug.
+###### If no bug exists to un-xfail the test, open one.
+###### Create a pull request to un-xfail the test and attach it to the un-xfail bug mentioned above.
+= Completing Requested Reviews =
 The first priority, when supporting Gaia UI Testing, is to ensure that our existing tests are accurate and reliable. To that end, we need to monitor the jobs that run in our CI and address any that are failing or are xpassing (passing but marked as expected to fail).