MozillaWiki

QA/Desktop Firefox/Walkthroughs/Automation

< QA‎ | Desktop Firefox‎ | Walkthroughs
Revision as of 20:31, 15 April 2014 by Ashughes (talk | contribs) (Created page with "= Automation = Every day we have automated tests which run against new builds, these tests are reported to couple of different dashboards. Firefox Nightly and Aurora are repor...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Automation

Every day we have automated tests which run against new builds, these tests are reported to couple of different dashboards. Firefox Nightly and Aurora are reported here, Firefox Beta and Release Candidate tests are reported here here. The results are triaged by the Automation Development team and failures are reported to Bugzilla as Mozmill Tests bugs. If a failure is found to be due to a Firefox regression QA will assist with the investigation.

Instructions for QA Leads

QA Leads should keep an eye on the affected bugs to ensure progress is being made and to keep an eye out for areas of potential regression.

1. Add a section to your test plan to track the known bugs using the following syntax (live example): 

<bugzilla>
{
 "include_fields":"id,summary,cf_status_firefox31",
 "component":"Mozmill Tests",
 "f1":"cf_status_firefox31",
 "o1":"nowordssubstr",
 "v1":"---"
}
</bugzilla>

2. Periodically review the list for affected bugs to make sure investigation stays on track, comment in the bug if you require a status update

3. Periodically review the list for fixed bugs and check the dashboards to verify the failure has been resolved

  • Mark the bug REOPENED if the test continues to fail
  • Mark the bug VERIFIED FIXED if the test has not failed recently

4. Ask for the test to be disabled if the test consistently fails on Beta/Release

5. If the Automation Development determines a failure is related to a Firefox regression:

  • Add the qawanted keyword to the regression bug
  • Assign a QA Contact to take ownership of the testing (Florin Mezei if you are unsure)
  • Make sure the regression bug is properly tracked with the status and tracking flags
  • Check back on the bug within a day or two to ensure progress is being made

Instructions for Testers

Testers have no responsibility with automation test failures unless they turn out to be Firefox regressions

1. If you are assigned to a bug for further investigation you will be expected to follow-up within 24 hours

2. Upon request of your assistance, complete the testing requested and update the bug

3. Remove the qawanted keyword once your testing is completed

4. If further assistance is required you will be notified

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Desktop_Firefox/Walkthroughs/Automation&oldid=965118"