B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions
< B2G | QA | Automation | UI
Jump to navigation
Jump to search
(Initial revision) |
|||
| (14 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
'''''This document is currently in work in progress''''' | '''''This document is currently in work in progress''''' | ||
= Current wording = | |||
* urgent job === smoke | |||
* non-urgent job === others | |||
= | = Processing an automation report = | ||
* As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job. | |||
* | * Follow the | ||
* [[B2G/QA/Automation/UI/Filing_Automation_Report|Send out the email report]]. | |||
* | |||
= | = Finding a failure in an automation report in an urgent job = | ||
== General steps == | |||
* | * Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority. | ||
* | * File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]]. | ||
* Add the failure bug to the next standup highlights, at [https://etherpad.mozilla.org/b2g-automation-daily-standup the standup notes] | |||
== | == In details == | ||
[[File:The_test_report_shows_a_test_failure.svg|Detailed flow chart|]] | |||
= | = Finding a failure in an automation report in a non-urgent job = | ||
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority. | |||
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]]. | |||
= Starting work on a bug = | |||
* Assign yourself and fill the QA whiteboard to make it appear on the [[B2G/QA/Automation/UI/Scrum|sprint page]] | |||
* Investigate if it's either an automation issue or an infrastructure one. | |||
* If the issue can be fixed, when your patch is ready, check Treeherder report before going any further. | |||
* If not, nominate it for the flaky suite | |||
* Ask 2 reviewers from either the [[B2G/QA/Automation/UI#Core_Team|core team]] or [[B2G/QA/Automation/UI#Contributors|contributors]]. Only one is needed for enabling/disabling tests. | |||
== In details == | |||
[[File:automation_failure.png|800px|Detailed flow chart 2]] | |||
= Finishing work on a bug = | |||
* Either the second reviewer or you could ask [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch#Easy_patch_submission_with_Autolander Autolander to merge your patch]. | |||
= Priorities = | |||
* Any urgent test failing due to automation issues should be fixed ASAP | |||
* Any non-urgent, non-intermittent automation bug should be fixed within a week | |||
* Any test that proves to be newly intermittent should be reviewed at next triage | |||
* Any test failing more than 50% should be disabled and put on backlog | |||
= Transition plan = | |||
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention: | |||
# Reduction of the length of the automation report (see the [[B2G/QA/Automation/UI/Filing_Automation_Report|new template]]) | |||
# Configure Jenkins to send out email on every single failures (use [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin this plugin] to configure the content of the email. | |||
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report. | |||
# Split the jobs between flaky/non-flakies. | |||
= TODO once Jenkins is able to send out emails = | |||
== Receiving an urgent failure email == | |||
Latest revision as of 20:35, 19 March 2015
This document is currently in work in progress
Current wording
- urgent job === smoke
- non-urgent job === others
Processing an automation report
- As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
- Follow the
- Send out the email report.
Finding a failure in an automation report in an urgent job
General steps
- Immediately disable the test per the disable/xfail doc. This is the top priority.
- File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
- Add the failure bug to the next standup highlights, at the standup notes
In details
Finding a failure in an automation report in a non-urgent job
- Immediately disable the test per the disable/xfail doc. This is the top priority.
- File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
Starting work on a bug
- Assign yourself and fill the QA whiteboard to make it appear on the sprint page
- Investigate if it's either an automation issue or an infrastructure one.
- If the issue can be fixed, when your patch is ready, check Treeherder report before going any further.
- If not, nominate it for the flaky suite
- Ask 2 reviewers from either the core team or contributors. Only one is needed for enabling/disabling tests.
In details
Finishing work on a bug
- Either the second reviewer or you could ask Autolander to merge your patch.
Priorities
- Any urgent test failing due to automation issues should be fixed ASAP
- Any non-urgent, non-intermittent automation bug should be fixed within a week
- Any test that proves to be newly intermittent should be reviewed at next triage
- Any test failing more than 50% should be disabled and put on backlog
Transition plan
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:
- Reduction of the length of the automation report (see the new template)
- Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
- Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
- Split the jobs between flaky/non-flakies.