B2G/QA/Automation/UI/Minimized Acceptance Execution

This document is currently in work in progress

Solid suites

Objective: Keep the build green, even if there's a current bug in the product.

• Solid suites = Super sanity, sanity, smoketests
• No more report investigation every day, the team (or more) is alerted by email sent by Jenkins.
• Summarized workflow:
  • Deactivate the test
  • Check if this is either a manual, a general automation or a lab-only issue
  • If possible, fix it. If not, call this issue out or nominate the test to the flaky suite.
  • More detailed information: http://mzl.la/1vFYdTD *TODO: replace this by an actual SVG when finalized*

Flaky suite

Objectives: Make sure the features are working even when they depend from external services..

• A report is generated everyday and somebody is in charge of it.
• Summarized workflow:
  • Check if this is either a manual, a general automation or a lab-only issue
  • If possible, fix it. If not, call this issue out or nominate the test for removal.
  • More information: http://mzl.la/1zlNFEE *TODO: replace this by an actual SVG when finalized*

Flaky trend analysis

Every triage, the team is in charge to take a look at the trends of failures. If a test got worse from the last triages, a bug to investigate the potential reasons of the failures will be filed. If a test went above 50% *TODO: does this value look correct?* a failures, a bug will be filed to remove it.

Transition plan

In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:

1. Reduction of the length of the automation report (see the new template)
2. Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
3. Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
4. Split the jobs between flaky/non-flakies.

New report template

   Subject: 02/18/2014 Acceptance tests results - Smoketests passed - Non-smoketests 1 new failure, 3 remaining.

   Smoketests: Passed. No product failure to call out. Automation failures: *insert a mzl.la link to a bugzilla request*
   Non-smoketests: One new product failure.
       * Bug 1134035 - [Dialer] Tapping the top call log entry, selects and deselects the 2nd call entry in edit mode
       3 remaining failures: *insert a mzl.la link to a bugzilla request*
   Build under test: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-flame-kk/2015/02/2015-02-17-16-02-34/