Confirmed users
955
edits
Mobile/Testing/Architectural Overview: Difference between revisions
Mobile/Testing/Architectural Overview (view source)
Revision as of 21:37, 20 August 2012
, 20 August 2012no edit summary
(Created page with "= High-level overview = Automated testing of Firefox for Android involves two components: a device to actually run the tests on (currently this is only Tegra or Panda developmen...") |
No edit summary |
||
| Line 4: | Line 4: | ||
This is contrast with desktop firefox, where the software to control running the tests (the "test harness") is the same as the machine running the actual tests. | This is contrast with desktop firefox, where the software to control running the tests (the "test harness") is the same as the machine running the actual tests. | ||
The highest-level software we use to control testing and hand out jobs is called BuildBot. Discussion of this is beyond the scope of this document, but the [[ReleaseEngineering|Release Engineering]] home page has lots of links. | |||
= Details = | = Details = | ||
| Line 9: | Line 11: | ||
== Test Devices (a.k.a. "Tegras" or "Pandas") == | == Test Devices (a.k.a. "Tegras" or "Pandas") == | ||
The general principle here (arrived at through iteration) is to keep the logic on the hardware as simple as possible, as it's generally much easier to change things at the level of the test controller. The said, there are | The general principle here (arrived at through iteration) is to keep the logic on the hardware as simple as possible, as it's generally much easier to change things at the level of the test controller. The said, there are two basic layers to the test devices: | ||
* Hardware | * Hardware and Operating System Image | ||
* Device Application Software | * Device Application Software | ||
| Line 47: | Line 48: | ||
=== Device Application Software === | === Device Application Software === | ||
''' | This is additional software written by us to help control or otherwise manage test devices. It is installed on top of the base operating system image. | ||
This software is written entirely in Java. | |||
'''Maintainer''': [[Auto-tools| Automation and Tools]] | |||
==== SUTAgent ==== | |||
The SUTAgent is an Android application which we install on all devices. It is the main mechanism by which our test controller can use any modify our system. It contains functions needed to push/pull files from the device, run programs, get information on the device's state, and a few other things. | |||
For more information (including up-to-date links to source code), see the [[Auto-tools/Projects/SUTAgent|SUTAgent Project Page]]. | |||
==== Watcher ==== | ==== Watcher ==== | ||
The watcher is a small android application | The watcher is a small android application which we install on all devices. It has just a few functions: | ||
* Ensure the Android lock screen never comes up | * Ensure the Android lock screen never comes up | ||
| Line 57: | Line 68: | ||
* ''FIXME'': There's other stuff that the watcher does too, but I can't remember offhand what it does. Maybe it allows updating the agent software? | * ''FIXME'': There's other stuff that the watcher does too, but I can't remember offhand what it does. Maybe it allows updating the agent software? | ||
Source code: The watcher source code is currently stored in mozilla-central in the | Source code: The watcher source code is currently stored in mozilla-central in the <code>build/mobile/sutagent/android/watcher</code> subdirectory ([https://hg.mozilla.org/mozilla-central/file/tip/build/mobile/sutagent/android/watcher web link]). | ||
== Test Controller (a.k.a. "Foopies") == | == Test Controller (a.k.a. "Foopies") == | ||
As mentioned above, most of the complexity in running automated testing is in this part of the automation, which controls the devices just described. There are six layers to this: | |||
* Mac Mini hardware and Linux/Mac OS image | |||
* Device management libraries | |||
* Build and maintenance scripts | |||
* Test harnesses (and actual tests) | |||
With the exception of the tests themselves, all this software is written in Python. | |||
=== Hardware / OS Image === | |||
'''Maintainers''': | |||
* Hardware: IT / [[ReleaseEngineering|Release Engineering]] (''FIXME: verify this'') | |||
* OS Image: [[ReleaseEngineering|Release Engineering]] | |||
The system we use to run all this software. | |||
We use two operating systems to run these: Linux and MacOS X. This mainly has historical reasons, and there will be no new test controllers built using the Mac OS X operating system. | |||
==== Linux Operating System ==== | |||
''FIXME'': Describe OS revs, installed versions of python. Include links to any other resources. | |||
==== Mac Operating System ==== | |||
''FIXME'': Describe OS revs, installed versions of python. Include links to any other resources. | |||
=== Device Management Libraries === | |||
'''Maintainer''': [[Auto-tools|Automation and Tools]] | |||
To interface with the test devices, we use a python library called mozdevice (part of the [[Auto-tools/Projects/MozBase|MozBase]] project) which allows us to interface with the SUTAgent application running on the devices. | |||
Both the build and maintenance scripts and the test harnesses themselves build on top of this basic component. | |||
Historically, the files that comprise mozdevice have been copied adhoc whereever they have been needed. To ease future maintenance, the goal is to centralize on a single maintained copy inside the mozbase project (with a regularly updated mirror inside mozilla-central to make things easier for contributors to run tests). | |||
Source code: The mozdevice source code is currently stored in mozilla-central in the <code>build/mobile</code> subdirectory ([https://hg.mozilla.org/mozilla-central/file/tip/build/mobile web link]) as <code>devicemanager.py</code> and <code>devicemanagerSUT.py</code>. | |||
Other copies of mozdevice are strewn through our various codebases. For example, old copies of this library are included and used directly by the build and maintenance scripts described below. | |||
=== Build and Maintenance Scripts === | |||
'''Maintainer''': [[ReleaseEngineering|Release Engineering]] | |||
These scripts are run by buildbot when a job is allocated to a test controller (foopy). It is their job both to setup a device for testing, run a suite of tests, then clean up the device for their next job. | |||
As of this writing, the eventual plan is to replace these scripts with mozharness scripts, which should prove to be easier to debug and maintain. | |||
Source code: This is stored in mercurial under the build/tools project under <code>sut_tools</code> ([https://hg.mozilla.org/build/tools/file/tip/sut_tools/ web link]) | |||
''FIXME'': I (wlach) have heard about all sorts of other tools that release engineering uses to maintain farms of tegras. For example, a graphical view of which tegras are failing, working, etc. We should (very briefly) describe these tools, and link to both them and their source (if they are not encompassed by the sut_tools link above). | |||
=== Test harnesses (and actual tests) === | |||
'''Maintainers''': | |||
* Test harnesses: [[Auto-Tools|Automation and Tools]] | |||
* Tests: All developers | |||
The purpose of test harnesses is to run actual tests. On mobile, a test harness running on a test controller takes a clean/working device as well as the test to be run as input and gives test results as output. | |||
You can find lots more detail on all the different types of test harnesses we use on mobile in [https://wiki.mozilla.org/Mobile/Fennec/Android#Testing testing section of the Android wiki page]. | |||