B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions

From MozillaWiki
< B2G‎ | QA‎ | Automation‎ | UI
Jump to navigation Jump to search
(Moved the new template part to B2G/QA/Automation/UI/Filing_Automation_Report)
(Modified per feedback given)
Line 1: Line 1:
'''''This document is currently in work in progress'''''
'''''This document is currently in work in progress'''''
= Current wording =
* urgent job === smoke
* non-urgent job === others


= Solid suites =
= Processing an automation report =
Objective: Keep the build green, even if there's a current bug in the product.
* As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
* Solid suites = Super sanity, sanity, smoketests
* Follow the  
* No more report investigation every day, the team (or more) is alerted by email sent by Jenkins.
* [[B2G/QA/Automation/UI/Filing_Automation_Report|Send out the email report]].
* Summarized workflow:
** Deactivate the test
** Check if this is either a manual, a general automation or a lab-only issue
** If possible, fix it. If not, call this issue out or nominate the test to the flaky suite.
** More detailed information: http://mzl.la/1vFYdTD *TODO: replace this by an actual SVG when finalized*


= Flaky suite =
= Finding a failure in an automation report in an urgent job =
Objectives: Make sure the features are working even when they depend from external services..
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority.
* A report is generated everyday and somebody is in charge of it.
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]].
* Summarized workflow:
* Add the failure bug to the next standup highlights, at [https://etherpad.mozilla.org/b2g-automation-daily-standup the standup notes]
** Check if this is either a manual, a general automation or a lab-only issue
** If possible, fix it. If not, call this issue out or nominate the test for removal.
** More information: http://mzl.la/1zlNFEE *TODO: replace this by an actual SVG when finalized*


== Flaky trend analysis ==  
= Finding a failure in an automation report in a non-urgent job =
Every triage, the team is in charge to take a look at the trends of failures. If a test got worse from the last triages, a bug to investigate the potential reasons of the failures will be filed. If a test went above 50% *TODO: does this value look correct?* a failures, a bug will be filed to remove it.
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority.
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]].
 
= Starting work on a bug =
* Assign yourself and fill the QA whiteboard to make it appear on the [[B2G/QA/Automation/UI/Scrum|sprint page]]
* When your patch is ready, check Treeherder report before going any further.
* Ask 2 reviewers from either the [[B2G/QA/Automation/UI#Core_Team|core team]] or [[B2G/QA/Automation/UI#Contributors|contributors]]. Only one is needed for enabling/disabling tests.
 
= Finishing work on a bug =
* Either the second reviewer or you could ask [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch#Easy_patch_submission_with_Autolander Autolander to merge your patch].
 
= Priorities =
* Any urgent test failing due to automation issues should be fixed ASAP
* Any non-urgent, non-intermittent automation bug should be fixed within a week
* Any test that proves to be newly intermittent should be reviewed at next triage
* Any test failing more than 50% should be disabled and put on backlog


= Transition plan =
= Transition plan =
Line 28: Line 38:
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
# Split the jobs between flaky/non-flakies.
# Split the jobs between flaky/non-flakies.
= TODO once Jenkins is able to send out emails =
== Receiving an urgent failure email ==

Revision as of 15:41, 18 March 2015

This document is currently in work in progress

Current wording

  • urgent job === smoke
  • non-urgent job === others

Processing an automation report

  • As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
  • Follow the
  • Send out the email report.

Finding a failure in an automation report in an urgent job

  • Immediately disable the test per the disable/xfail doc. This is the top priority.
  • File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
  • Add the failure bug to the next standup highlights, at the standup notes

Finding a failure in an automation report in a non-urgent job

  • Immediately disable the test per the disable/xfail doc. This is the top priority.
  • File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.

Starting work on a bug

  • Assign yourself and fill the QA whiteboard to make it appear on the sprint page
  • When your patch is ready, check Treeherder report before going any further.
  • Ask 2 reviewers from either the core team or contributors. Only one is needed for enabling/disabling tests.

Finishing work on a bug

Priorities

  • Any urgent test failing due to automation issues should be fixed ASAP
  • Any non-urgent, non-intermittent automation bug should be fixed within a week
  • Any test that proves to be newly intermittent should be reviewed at next triage
  • Any test failing more than 50% should be disabled and put on backlog

Transition plan

In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:

  1. Reduction of the length of the automation report (see the new template)
  2. Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
  3. Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
  4. Split the jobs between flaky/non-flakies.

TODO once Jenkins is able to send out emails

Receiving an urgent failure email

Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Automation/UI/Minimized_Acceptance_Execution&oldid=1062736"