B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions

Jump to navigation Jump to search

B2G/QA/Automation/UI/Minimized Acceptance Execution (view source)

Revision as of 15:41, 18 March 2015

885 bytes added ,  18 March 2015
Modified per feedback given
(Moved the new template part to B2G/QA/Automation/UI/Filing_Automation_Report)
(Modified per feedback given)
Line 1: Line 1:
'''''This document is currently in work in progress'''''
'''''This document is currently in work in progress'''''
= Current wording =
* urgent job === smoke
* non-urgent job === others


= Solid suites =
= Processing an automation report =
Objective: Keep the build green, even if there's a current bug in the product.
* As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
* Solid suites = Super sanity, sanity, smoketests
* Follow the  
* No more report investigation every day, the team (or more) is alerted by email sent by Jenkins.
* [[B2G/QA/Automation/UI/Filing_Automation_Report|Send out the email report]].
* Summarized workflow:
** Deactivate the test
** Check if this is either a manual, a general automation or a lab-only issue
** If possible, fix it. If not, call this issue out or nominate the test to the flaky suite.
** More detailed information: http://mzl.la/1vFYdTD *TODO: replace this by an actual SVG when finalized*


= Flaky suite =
= Finding a failure in an automation report in an urgent job =
Objectives: Make sure the features are working even when they depend from external services..
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority.
* A report is generated everyday and somebody is in charge of it.
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]].
* Summarized workflow:
* Add the failure bug to the next standup highlights, at [https://etherpad.mozilla.org/b2g-automation-daily-standup the standup notes]
** Check if this is either a manual, a general automation or a lab-only issue
** If possible, fix it. If not, call this issue out or nominate the test for removal.
** More information: http://mzl.la/1zlNFEE *TODO: replace this by an actual SVG when finalized*


== Flaky trend analysis ==  
= Finding a failure in an automation report in a non-urgent job =
Every triage, the team is in charge to take a look at the trends of failures. If a test got worse from the last triages, a bug to investigate the potential reasons of the failures will be filed. If a test went above 50% *TODO: does this value look correct?* a failures, a bug will be filed to remove it.
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority.
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]].
 
= Starting work on a bug =
* Assign yourself and fill the QA whiteboard to make it appear on the [[B2G/QA/Automation/UI/Scrum|sprint page]]
* When your patch is ready, check Treeherder report before going any further.
* Ask 2 reviewers from either the [[B2G/QA/Automation/UI#Core_Team|core team]] or [[B2G/QA/Automation/UI#Contributors|contributors]]. Only one is needed for enabling/disabling tests.
 
= Finishing work on a bug =
* Either the second reviewer or you could ask [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch#Easy_patch_submission_with_Autolander Autolander to merge your patch].
 
= Priorities =
* Any urgent test failing due to automation issues should be fixed ASAP
* Any non-urgent, non-intermittent automation bug should be fixed within a week
* Any test that proves to be newly intermittent should be reviewed at next triage
* Any test failing more than 50% should be disabled and put on backlog


= Transition plan =
= Transition plan =
Line 28: Line 38:
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
# Split the jobs between flaky/non-flakies.
# Split the jobs between flaky/non-flakies.
= TODO once Jenkins is able to send out emails =
== Receiving an urgent failure email ==
Jlorenzo
Confirmed users
213

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1062736"

Navigation menu