canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,747
edits
QA/Automation/Projects/Mozmill Automation/On Demand Test Framework: Difference between revisions
Jump to navigation
Jump to search
m
QA/Automation/Projects/Mozmill Automation/On Demand Test Framework (view source)
Revision as of 10:40, 13 January 2014
, 13 January 2014Whimboo moved page Auto-tools/Automation Development/Projects/Mozmill Automation/On Demand Test Framework to QA/Automation/Projects/Mozmill Automation/On Demand Test Framework: Team reorganization
m (Whimboo moved page Auto-tools/Automation Development/Projects/Mozmill Automation/On Demand Test Framework to QA/Automation/Projects/Mozmill Automation/On Demand Test Framework: Team reorganization) |
|||
| (39 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
==Overview== | ==Overview== | ||
(need to expand this, this is not at all final wording) | |||
The On-Demand Test Framework will be a full-service execution framework for Mozmill and perhaps eventually other tests. It will allow execution of tests across a grid of machines. Eventually, it will include auto-provisioning that grid and features to allow more granular parallel execution. | |||
{| style="width: 90%; margin: 0 0 1em 1em" | | {| style="width: 90%; margin: 0 0 1em 1em" | | ||
| Line 10: | Line 12: | ||
|- valign="top" | |- valign="top" | ||
| '''Leads:''' | | '''Leads:''' | ||
| [mailto:gmealer@mozilla.com Geo Mealer], [mailto: | | [mailto:gmealer@mozilla.com Geo Mealer], [mailto:dhunt@mozilla.com Dave Hunt] | ||
|- valign="top" | |- valign="top" | ||
| '''Contributors:''' | | '''Contributors:''' | ||
| TBD | | TBD | ||
|- valign="top" | |- valign="top" | ||
| ''' | | '''Pivotal:''' | ||
| | | https://www.pivotaltracker.com/projects/404111 | ||
|- valign="top" | |- valign="top" | ||
| ''' | | '''Bugzilla:''' | ||
| | | https://bugzilla.mozilla.org/show_bug.cgi?id=698654 | ||
|- valign="top" | |- valign="top" | ||
| ''' | | '''Other Docs:''' | ||
| | | [[/Docs|Index]] | ||
|} | |} | ||
== | ==Goal History== | ||
{| class="fullwidth-table" | | {| class="fullwidth-table" | | ||
| style="background:#EFEFEF; width: | | style="background:#EFEFEF; width: 20%" | '''Period''' | ||
| style="background:#EFEFEF; width: | | style="background:#EFEFEF; width: 20%" | '''Status''' | ||
| style="background:#EFEFEF; width: 60%" | | | style="background:#EFEFEF; width: 60%" | '''Goal''' | ||
|- valign="top" | |- valign="top" | ||
| | | 2011 Q4 | ||
| | | {{ok|}} | ||
| Implementation of a remote mechanism to trigger Mozmill functional and update tests for Firefox releases | |||
| | |||
|} | |} | ||
==Requirements== | |||
* We want to be able to kick off test runs with as little interaction and micromanaging as we can. | |||
* A test run equates to a script with one or more parameters, which will be run on one or more environments. These parameters correspond mostly to the test matrix, and will vary per run type. | |||
* The test run is capable of, within a given environment, doing everything from app deployment to testing to app cleanup. Or we can split these up if more convenient. | |||
* We'll want the system to spread testing among a number of available machines per environment, and ultimately create the machines as needed. | |||
* Ultimately, we'd like the ability to slice the test run on a parameter value to run in parallel. If we can get it, understanding how to slice the test run per-test would be even better. | |||
* We'd like to clearly monitor run progress, and watchdog for hung machines. | |||
* We'd like to see understandable test results, preferably consolidated across a whole run (at least per-env, if not cross-env). | |||
==Project Milestones== | ==Project Milestones== | ||
{| class="fullwidth-table" | | {| class="fullwidth-table" | | ||
| style="background:#EFEFEF; width: | | style="background:#EFEFEF; width: 20%" | '''Milestone''' | ||
| style="background:#EFEFEF; width: | | style="background:#EFEFEF; width: 20%" | '''Status''' | ||
| style="background:#EFEFEF; width: 60%" | Description | | style="background:#EFEFEF; width: 60%" | '''Description''' | ||
|- valign="top" | |||
| Basic Execution | |||
| Design Stage | |||
{{ok|}} | |||
| System allows execution of test-run demands against standing machines. | |||
|- valign="top" | |||
| Auto-Provisioning | |||
| Not started | |||
| System automatically provisions and/or resets virtual machines | |||
|- valign="top" | |- valign="top" | ||
| Parallelism | |||
| Not started | | Not started | ||
| | | System knows how to slice up a test run more finely than per-platform | ||
|} | |} | ||
== | ==Architecture== | ||
[[Image:On_Demand_Arch.png|On-Demand Test Framework Architecture]] | [[Image:On_Demand_Arch.png|On-Demand Test Framework Architecture]] | ||
The Trigger and Config Generator components are a proposed part of integrating the CI goal into this system by automatically generating a Demand, and will not be covered in this project. | The Trigger and Config Generator components are a proposed part of integrating the CI goal into this system by automatically generating a Demand, and will not be covered in this project. | ||
{| class="fullwidth-table" | | {| class="fullwidth-table" | | ||
| style="background:#EFEFEF; width: 25%" | Component | | style="background:#EFEFEF; width: 25%" | '''Component''' | ||
| style="background:#EFEFEF; width: | | style="background:#EFEFEF; width: 75%" | '''Description''' | ||
|- valign="top" | |- valign="top" | ||
| [[/Demand|Demand]] | | [[/Demand|Demand]] | ||
| Line 78: | Line 88: | ||
|- valign="top" | |- valign="top" | ||
| [[/Executor|Executor]] | | [[/Executor|Executor]] | ||
| Kicks off and centrally | | Kicks off and centrally monitors the test run | ||
|- valign="top" | |- valign="top" | ||
| [[/Provisioner|Provisioner]] | | [[/Provisioner|Provisioner]] | ||
| | | Initializes appropriate machines for execution | ||
|- valign="top" | |- valign="top" | ||
| [[/Machine|Machine]] | | [[/Machine|Machine]] | ||
| | | Environment on target machines for remote execution | ||
|} | |} | ||
If an off-the-shelf CI tool like Jenkins turns out be appropriate for our needs, it may subsume the Dashboard and Executor components. | |||
The question of which module controls/watchdogs the individual Test Machines performing the run has not been answered yet. Further analysis will show whether that's better as part of the Executor, Provisioner, whether the machine can run relatively autonomously once it's been initialized, or whether a new component is needed. | |||
In addition to these, Status formats must be defined where they exist in the architecture. It is unknown yet as to which module will define these, or whether they will be significant enough to require subprojects. | In addition to these, Status formats must be defined where they exist in the architecture. It is unknown yet as to which module will define these, or whether they will be significant enough to require subprojects. | ||
==Resources== | ==Resources== | ||
* [[/CI Documentation|Continuous Integration Details]] | |||
* [[/Public|Public Posts and Discussions]] | * [[/Public|Public Posts and Discussions]] | ||
* [[/Presentations|Presentations]] | * [[/Presentations|Presentations]] | ||
* [[/FAQ|FAQ]] | * [[/FAQ|FAQ]] | ||
* [[/Contribution|How to Contribute]] | * [[/Contribution|How to Contribute]] | ||