B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions
< B2G | QA | Automation | UI
Jump to navigation
Jump to search
(Modified per feedback given) |
|||
| (6 intermediate revisions by 2 users not shown) | |||
| Line 10: | Line 10: | ||
= Finding a failure in an automation report in an urgent job = | = Finding a failure in an automation report in an urgent job = | ||
== General steps == | |||
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority. | * Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority. | ||
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]]. | * File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]]. | ||
* Add the failure bug to the next standup highlights, at [https://etherpad.mozilla.org/b2g-automation-daily-standup the standup notes] | * Add the failure bug to the next standup highlights, at [https://etherpad.mozilla.org/b2g-automation-daily-standup the standup notes] | ||
== In details == | |||
[[File:The_test_report_shows_a_test_failure.svg|Detailed flow chart|]] | |||
= Finding a failure in an automation report in a non-urgent job = | = Finding a failure in an automation report in a non-urgent job = | ||
| Line 20: | Line 24: | ||
= Starting work on a bug = | = Starting work on a bug = | ||
* Assign yourself and fill the QA whiteboard to make it appear on the [[B2G/QA/Automation/UI/Scrum|sprint page]] | * Assign yourself and fill the QA whiteboard to make it appear on the [[B2G/QA/Automation/UI/Scrum|sprint page]] | ||
* | * Investigate if it's either an automation issue or an infrastructure one. | ||
* If the issue can be fixed, when your patch is ready, check Treeherder report before going any further. | |||
* If not, nominate it for the flaky suite | |||
* Ask 2 reviewers from either the [[B2G/QA/Automation/UI#Core_Team|core team]] or [[B2G/QA/Automation/UI#Contributors|contributors]]. Only one is needed for enabling/disabling tests. | * Ask 2 reviewers from either the [[B2G/QA/Automation/UI#Core_Team|core team]] or [[B2G/QA/Automation/UI#Contributors|contributors]]. Only one is needed for enabling/disabling tests. | ||
== In details == | |||
[[File:automation_failure.png|800px|Detailed flow chart 2]] | |||
= Finishing work on a bug = | = Finishing work on a bug = | ||
Latest revision as of 20:35, 19 March 2015
This document is currently in work in progress
Current wording
- urgent job === smoke
- non-urgent job === others
Processing an automation report
- As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
- Follow the
- Send out the email report.
Finding a failure in an automation report in an urgent job
General steps
- Immediately disable the test per the disable/xfail doc. This is the top priority.
- File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
- Add the failure bug to the next standup highlights, at the standup notes
In details
Finding a failure in an automation report in a non-urgent job
- Immediately disable the test per the disable/xfail doc. This is the top priority.
- File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
Starting work on a bug
- Assign yourself and fill the QA whiteboard to make it appear on the sprint page
- Investigate if it's either an automation issue or an infrastructure one.
- If the issue can be fixed, when your patch is ready, check Treeherder report before going any further.
- If not, nominate it for the flaky suite
- Ask 2 reviewers from either the core team or contributors. Only one is needed for enabling/disabling tests.
In details
Finishing work on a bug
- Either the second reviewer or you could ask Autolander to merge your patch.
Priorities
- Any urgent test failing due to automation issues should be fixed ASAP
- Any non-urgent, non-intermittent automation bug should be fixed within a week
- Any test that proves to be newly intermittent should be reviewed at next triage
- Any test failing more than 50% should be disabled and put on backlog
Transition plan
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:
- Reduction of the length of the automation report (see the new template)
- Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
- Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
- Split the jobs between flaky/non-flakies.