B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions
< B2G | QA | Automation | UI
Jump to navigation
Jump to search
(Moved the triage part to B2G/QA/Automation/UI/Triage) |
(Moved the new template part to B2G/QA/Automation/UI/Filing_Automation_Report) |
||
| Line 24: | Line 24: | ||
= Transition plan = | = Transition plan = | ||
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention: | In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention: | ||
# Reduction of the length of the automation report (see the new template) | # Reduction of the length of the automation report (see the [[B2G/QA/Automation/UI/Filing_Automation_Report|new template]]) | ||
# Configure Jenkins to send out email on every single failures (use [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin this plugin] to configure the content of the email. | # Configure Jenkins to send out email on every single failures (use [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin this plugin] to configure the content of the email. | ||
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report. | # Stop to send out the manually written automation report. Product bugs will be covered in the general daily report. | ||
# Split the jobs between flaky/non-flakies. | # Split the jobs between flaky/non-flakies. | ||
Revision as of 14:07, 18 March 2015
This document is currently in work in progress
Solid suites
Objective: Keep the build green, even if there's a current bug in the product.
- Solid suites = Super sanity, sanity, smoketests
- No more report investigation every day, the team (or more) is alerted by email sent by Jenkins.
- Summarized workflow:
- Deactivate the test
- Check if this is either a manual, a general automation or a lab-only issue
- If possible, fix it. If not, call this issue out or nominate the test to the flaky suite.
- More detailed information: http://mzl.la/1vFYdTD *TODO: replace this by an actual SVG when finalized*
Flaky suite
Objectives: Make sure the features are working even when they depend from external services..
- A report is generated everyday and somebody is in charge of it.
- Summarized workflow:
- Check if this is either a manual, a general automation or a lab-only issue
- If possible, fix it. If not, call this issue out or nominate the test for removal.
- More information: http://mzl.la/1zlNFEE *TODO: replace this by an actual SVG when finalized*
Flaky trend analysis
Every triage, the team is in charge to take a look at the trends of failures. If a test got worse from the last triages, a bug to investigate the potential reasons of the failures will be filed. If a test went above 50% *TODO: does this value look correct?* a failures, a bug will be filed to remove it.
Transition plan
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:
- Reduction of the length of the automation report (see the new template)
- Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
- Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
- Split the jobs between flaky/non-flakies.