Confirmed users
213
edits
B2G/QA/Automation/UI/Minimized Acceptance Execution: Difference between revisions
Jump to navigation
Jump to search
B2G/QA/Automation/UI/Minimized Acceptance Execution (view source)
Revision as of 14:55, 5 March 2015
, 5 March 2015Added Geo's first feedback.
(Small fixes on Geo) |
(Added Geo's first feedback.) |
||
| Line 5: | Line 5: | ||
* Solid suites = Super sanity, sanity, smoketests | * Solid suites = Super sanity, sanity, smoketests | ||
* No more report investigation every day, the team (or more) is alerted by email sent by Jenkins. | * No more report investigation every day, the team (or more) is alerted by email sent by Jenkins. | ||
* | * Summarized workflow: | ||
** Deactivate the test | |||
** Check if this is either a manual, a general automation or a lab-only issue | |||
** If possible, fix it. If not, call this issue out or nominate the test to the flaky suite. | |||
** More detailed information: http://mzl.la/1vFYdTD *TODO: replace this by an actual SVG when finalized* | |||
= Flaky suite = | = Flaky suite = | ||
Objectives: Make sure the features are working even when they depend from external services.. | Objectives: Make sure the features are working even when they depend from external services.. | ||
* A report is generated everyday and somebody is in charge of it. | * A report is generated everyday and somebody is in charge of it. | ||
* | * Summarized workflow: | ||
** Check if this is either a manual, a general automation or a lab-only issue | |||
** If possible, fix it. If not, call this issue out or nominate the test for removal. | |||
** More information: http://mzl.la/1zlNFEE *TODO: replace this by an actual SVG when finalized* | |||
== Flaky trend analysis == | == Flaky trend analysis == | ||
| Line 19: | Line 26: | ||
Event: Every week, either during the automation roundtable or another event of half an hour | Event: Every week, either during the automation roundtable or another event of half an hour | ||
= Transition plan = | |||
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention: | |||
1. Reduction of the lenth of the automation report (see the new template) | |||
2. Configure Jenkins to send out email on every single failures (use [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin this plugin] to configure the content of the email. | |||
3. Stop to send out the manually written automation report. Product bugs will be covered in the general daily report. | |||
4. Split the jobs between flaky/non-flakies. | |||
== New report template == | |||
Subject: 02/18/2014 Acceptance tests results - Smoketests passed - Non-smoketests 1 new failure, 3 remaining. | |||
Smoketests: Passed. No product failure to call out. Automation failures: *insert a mzl.la link to a bugzilla request* | |||
Non-smoketests: One new product failure. | |||
* Bug 1134035 - [Dialer] Tapping the top call log entry, selects and deselects the 2nd call entry in edit mode | |||
3 remaining failures: *insert a mzl.la link to a bugzilla request* | |||
Build under test: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-flame-kk/2015/02/2015-02-17-16-02-34/ | |||