B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions

From MozillaWiki
< B2G‎ | QA‎ | Automation‎ | UI
Jump to navigation Jump to search
(Minor indentation fix)
 
(10 intermediate revisions by 3 users not shown)
Line 1: Line 1:
'''''This document is currently in work in progress'''''
'''''This document is currently in work in progress'''''
= Current wording =
* urgent job === smoke
* non-urgent job === others


= Solid suites =
= Processing an automation report =
Objective: Keep the build green, even if there's a current bug in the product.
* As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
* Solid suites = Super sanity, sanity, smoketests
* Follow the  
* No more report investigation every day, the team (or more) is alerted by email sent by Jenkins.
* [[B2G/QA/Automation/UI/Filing_Automation_Report|Send out the email report]].
* Summarized workflow:
** Deactivate the test
** Check if this is either a manual, a general automation or a lab-only issue
** If possible, fix it. If not, call this issue out or nominate the test to the flaky suite.
** More detailed information: http://mzl.la/1vFYdTD *TODO: replace this by an actual SVG when finalized*


= Flaky suite =
= Finding a failure in an automation report in an urgent job =
Objectives: Make sure the features are working even when they depend from external services..
== General steps ==
* A report is generated everyday and somebody is in charge of it.
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority.
* Summarized workflow:
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]].
** Check if this is either a manual, a general automation or a lab-only issue
* Add the failure bug to the next standup highlights, at [https://etherpad.mozilla.org/b2g-automation-daily-standup the standup notes]
** If possible, fix it. If not, call this issue out or nominate the test for removal.
** More information: http://mzl.la/1zlNFEE *TODO: replace this by an actual SVG when finalized*


== Flaky trend analysis ==  
== In details ==
Every triage, the team is in charge to take a look at the trends of failures. If a test got worse from the last triages, a bug to investigate the potential reasons of the failures will be filed. If a test went above 50% *TODO: does this value look correct?* a failures, a bug will be filed to remove it.
[[File:The_test_report_shows_a_test_failure.svg|Detailed flow chart|]]


= Triages =
= Finding a failure in an automation report in a non-urgent job =
On bugzilla, QA Whiteboard: [fxosqa-auto-suite-triage?]
* Immediately disable the test per the [[B2G/QA/Automation/UI/Xfail_and_Disable|disable/xfail doc]]. This is the top priority.
* File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the [[B2G/QA/Automation/UI/Filing_Bugs_Against_Automation_Errors|bug filing doc]].


Event: Every week, either during the automation roundtable or another event of half an hour
= Starting work on a bug =
* Assign yourself and fill the QA whiteboard to make it appear on the [[B2G/QA/Automation/UI/Scrum|sprint page]]
* Investigate if it's either an automation issue or an infrastructure one.
* If the issue can be fixed, when your patch is ready, check Treeherder report before going any further.
* If not, nominate it for the flaky suite
* Ask 2 reviewers from either the [[B2G/QA/Automation/UI#Core_Team|core team]] or [[B2G/QA/Automation/UI#Contributors|contributors]]. Only one is needed for enabling/disabling tests.


== In details ==
[[File:automation_failure.png|800px|Detailed flow chart 2]]
= Finishing work on a bug =
* Either the second reviewer or you could ask [https://developer.mozilla.org/en-US/Firefox_OS/Developing_Gaia/Submitting_a_Gaia_patch#Easy_patch_submission_with_Autolander Autolander to merge your patch].
= Priorities =
* Any urgent test failing due to automation issues should be fixed ASAP
* Any non-urgent, non-intermittent automation bug should be fixed within a week
* Any test that proves to be newly intermittent should be reviewed at next triage
* Any test failing more than 50% should be disabled and put on backlog


= Transition plan =
= Transition plan =
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:
# Reduction of the lenth of the automation report (see the new template)
# Reduction of the length of the automation report (see the [[B2G/QA/Automation/UI/Filing_Automation_Report|new template]])
# Configure Jenkins to send out email on every single failures (use [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin this plugin] to configure the content of the email.
# Configure Jenkins to send out email on every single failures (use [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin this plugin] to configure the content of the email.
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
# Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
# Split the jobs between flaky/non-flakies.
# Split the jobs between flaky/non-flakies.


== New report template ==  
= TODO once Jenkins is able to send out emails =
    Subject: 02/18/2014 Acceptance tests results - Smoketests passed - Non-smoketests 1 new failure, 3 remaining.
== Receiving an urgent failure email ==
 
    Smoketests: Passed. No product failure to call out. Automation failures: *insert a mzl.la link to a bugzilla request*
    Non-smoketests: One new product failure.
        * Bug 1134035 - [Dialer] Tapping the top call log entry, selects and deselects the 2nd call entry in edit mode
        3 remaining failures: *insert a mzl.la link to a bugzilla request*
    Build under test: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-flame-kk/2015/02/2015-02-17-16-02-34/

Latest revision as of 20:35, 19 March 2015

This document is currently in work in progress

Current wording

  • urgent job === smoke
  • non-urgent job === others

Processing an automation report

  • As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
  • Follow the
  • Send out the email report.

Finding a failure in an automation report in an urgent job

General steps

  • Immediately disable the test per the disable/xfail doc. This is the top priority.
  • File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
  • Add the failure bug to the next standup highlights, at the standup notes

In details

The test report shows a test failure.svg

Finding a failure in an automation report in a non-urgent job

  • Immediately disable the test per the disable/xfail doc. This is the top priority.
  • File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.

Starting work on a bug

  • Assign yourself and fill the QA whiteboard to make it appear on the sprint page
  • Investigate if it's either an automation issue or an infrastructure one.
  • If the issue can be fixed, when your patch is ready, check Treeherder report before going any further.
  • If not, nominate it for the flaky suite
  • Ask 2 reviewers from either the core team or contributors. Only one is needed for enabling/disabling tests.

In details

Detailed flow chart 2

Finishing work on a bug

Priorities

  • Any urgent test failing due to automation issues should be fixed ASAP
  • Any non-urgent, non-intermittent automation bug should be fixed within a week
  • Any test that proves to be newly intermittent should be reviewed at next triage
  • Any test failing more than 50% should be disabled and put on backlog

Transition plan

In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:

  1. Reduction of the length of the automation report (see the new template)
  2. Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
  3. Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
  4. Split the jobs between flaky/non-flakies.

TODO once Jenkins is able to send out emails

Receiving an urgent failure email

Retrieved from "https://wiki.mozilla.org/index.php?title=B2G/QA/Automation/UI/Minimized_Acceptance_Execution&oldid=1063177"