QA/Automation/Projects/Mozmill Automation/Mozmill CI: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Marking pages as obsolete)
 
(17 intermediate revisions by 5 users not shown)
Line 1: Line 1:
__NOTOC__
__NOTOC__
'''OBSOLETE - The system is no longer running. See https://bugzilla.mozilla.org/show_bug.cgi?id=1386628'''
=Overview=
=Overview=
The Mozmill Automation team currently performs update testing triggered by a daily cron job. This approach has some weaknesses:
The Automation Development team is working on a Mozmill CI solution which triggers Mozmill tests immediately when a new build of Firefox has been made available.
* If no build lands, we still attempt to perform update testing.
* If a build lands off schedule, we attempt to update test it at the wrong time.
* There's no way to use this style of trigger to pre-qualify a build (or even promptly post-qualify it) so results have limited impact.
* Because we update build-over-build, using the same installation, anomalies in testing (or bugs in a given build) can cause the process to get "off-track" and poison subsequent update tests.


Instead, we'd like to move towards performing update testing as soon as the build lands. We'd also like to independently download and verify each day's build, rather than relying on using the same install day over day.
The ultimate goal is to build a system within QA, then hand off the process to Release Engineering for ongoing maintenance.


The ultimate goal is to build a prototype within QA walls, then hand off the process to Release Engineering for ongoing maintenance.
{| style="width: 90%; margin: 0 0 1em 1em" |
 
  | style="width: 20%" |
{|  
  | style="width: 80%" |
  | style="width: 33%" |
|- valign="top"
  | style="width: 33%" |
| '''Name:'''
| Triggered Daily Test-runs
  |- valign="top"
  |- valign="top"
  | '''Leads:'''
  | '''Leads:'''
  | [mailto:gmealer@mozilla.com Geo Mealer]<br/>[mailto:hskupin@mozilla.com Henrik Skupin]<br/>
  | [mailto:hskupin@mozilla.com Henrik Skupin]
|- valign="top"
| '''Contributors:'''
| [mailto:andreea.matei@softvisioninc.eu Andreea Matei], [mailto:cosmin.malutan@softvision.ro Cosmin Malutan]
|- valign="top"
| '''Repository:'''
| https://github.com/mozilla/mozmill-ci
  |- valign="top"
  |- valign="top"
  | '''Team Assistance:'''
  | '''Etherpad:'''
  | n/a
  | https://etherpad.mozilla.org/mozmill-ci
|}
|}


=Proposed Design=
=Documentation=
==Process==
A complete set of documents can be found on the pages linked below:
# Pulse Trigger received on build landing, contains current build id
 
# Trigger polling script on VM catches trigger
* [[/Duties|Duties]]
# Trigger polling script adds build to queue (directory full of action snippets or a single flat file) for processing.
* [[/Requirements|Requirements]]
#* Initial Proof of Concept will not include a queue.
* [[/Design Documents|Design Documents]]
# Queue polling script pulls the job off the queue, does two things:
* [[/FAQ|FAQ]]
## Downloads current build and saves it off for next trigger.
## Build saved off from last trigger will be updated and compared to expected *build id.
* Initial reporting to brasstacks, possibly email on failures
* Initial version for trunk, en_US only. Implementing OS X version first, other platforms to come very shortly thereafter.


==Resources already present==
== Roadmap ==
* Script to install a build
{| class="fullwidth-table sortable"
* Script to update a build and do some basic heuristics for testing exists.
|-
* Agreement from RelEng to send Pulse trigger, interim trigger already in place based on FTP scraping
| style="background:#EFEFEF; width:20%" | '''Date'''
* Sample code for Pulse interaction that can be adapted to our purpose.
| style="background:#EFEFEF; width:14%" | '''Version'''
==Things needed now==
| style="background:#EFEFEF; width:66%" | '''Details'''
* Update test script needs to be able to take a particular build id for verification
|- valign="top"
* Final details on Trigger information to be decided.
| Sep 30th, 2014
* Polling script
| [https://github.com/mozilla/mozmill-ci/issues/270 Issue #270]
==Things needed eventually==
| Get full coverage of all locales for Firefox beta and release testing
* Queuing mechanism (this will primarily be needed for L10N testing, where all locales land simultaneously and need to be queued on platform)
|}
* RelEng wants to do this in their infrastructure eventually if at all possible.
** If not possible, we'll need a way to report directly back to RelEng
** Not there yet, though. For now want to run it on our side to make sure of robustness before putting in critical path.

Latest revision as of 10:46, 14 May 2018


OBSOLETE - The system is no longer running. See https://bugzilla.mozilla.org/show_bug.cgi?id=1386628

Overview

The Automation Development team is working on a Mozmill CI solution which triggers Mozmill tests immediately when a new build of Firefox has been made available.

The ultimate goal is to build a system within QA, then hand off the process to Release Engineering for ongoing maintenance.

Name: Triggered Daily Test-runs
Leads: Henrik Skupin
Contributors: Andreea Matei, Cosmin Malutan
Repository: https://github.com/mozilla/mozmill-ci
Etherpad: https://etherpad.mozilla.org/mozmill-ci

Documentation

A complete set of documents can be found on the pages linked below:

Roadmap

Date Version Details
Sep 30th, 2014 Issue #270 Get full coverage of all locales for Firefox beta and release testing
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Automation/Projects/Mozmill_Automation/Mozmill_CI&oldid=1193758"