Mobile/Testing/Architectural Overview
High-level overview
Automated testing of Firefox for Android involves two components: a device to actually run the tests on (currently this is only Tegra or Panda development boards, so we call them "Tegras" or "Pandas") and a machine (a.k.a. a "Foopie") that actually takes care of setting up these devices and running the test.
This is contrast with desktop firefox, where the software to control running the tests (the "test harness") is the same as the machine running the actual tests.
The highest-level software we use to control testing and hand out jobs is called BuildBot. Discussion of this is beyond the scope of this document, but the Release Engineering home page has lots of links.
Details
Test Devices (a.k.a. "Tegras" or "Pandas")
The general principle here (arrived at through iteration) is to keep the logic on the hardware as simple as possible, as it's generally much easier to change things at the level of the test controller. The said, there are two basic layers to the test devices:
- Hardware and Operating System Image
- Device Application Software
Hardware / OS Image
Hardware and operating system images are really two different things, but we group them together because typically we build/qualify them as a unit (and there's very little overlap between the work we do to get an operating system running on one platform versus another).
Maintainers:
- Hardware: IT / Release Engineering (FIXME: verify this)
- OS Image: Automation and Tools
Tegra Platform
As of this writing, all automated Android testing is done on this hardware. This is an older board created by nvidia, and is no longer supported or sold. For this reason (as well as the fact that it only supports Android 2.2), it is being replaced by the Pandaboard platform.
The operating system is based on proprietary software provided to us by nVidia, with additional extensions/modifications added by us:
(FIXME: describe these).
Further info: ???
Panda Platform
This is the next generation of Android testing. It is based on a commodity board which can be freely ordered from Texas instruments.
The operating system is based on Linaro's flavour of the Android Open Source Project, with various modifications by us to make it work better as an automation target for Fennec (consistent resolution setting, applications built-in, etc.).
Further info:
Device Application Software
This is additional software written by us to help control or otherwise manage test devices. It is installed on top of the base operating system image.
This software is written entirely in Java.
Maintainer: Automation and Tools
SUTAgent
The SUTAgent is an Android application which we install on all devices. It is the main mechanism by which our test controller can use any modify our system. It contains functions needed to push/pull files from the device, run programs, get information on the device's state, and a few other things.
For more information (including up-to-date links to source code), see the SUTAgent Project Page.
Watcher
The watcher is a small android application which we install on all devices. It has just a few functions:
- Ensure the Android lock screen never comes up
- Periodically ping an external server and reboot if it is unreachable for too long (we currently disable this behavior as this is already taken care of by the foopies: we manually disable it in older versions and newer versions don't do this at all)
- FIXME: There's other stuff that the watcher does too, but I can't remember offhand what it does. Maybe it allows updating the agent software?
Source code: The watcher source code is currently stored in mozilla-central in the build/mobile/sutagent/android/watcher subdirectory (web link).
Test Controller (a.k.a. "Foopies")
As mentioned above, most of the complexity in running automated testing is in this part of the automation, which controls the devices just described. There are six layers to this:
- Mac Mini hardware and Linux/Mac OS image
- Device management libraries
- Build and maintenance scripts
- Test harnesses (and actual tests)
With the exception of the tests themselves, all this software is written in Python.
Hardware / OS Image
Maintainers:
- Hardware: IT / Release Engineering (FIXME: verify this)
- OS Image: Release Engineering
The system we use to run all this software.
We use two operating systems to run these: Linux and MacOS X. This mainly has historical reasons, and there will be no new test controllers built using the Mac OS X operating system.
Linux Operating System
FIXME: Describe OS revs, installed versions of python. Include links to any other resources.
Mac Operating System
FIXME: Describe OS revs, installed versions of python. Include links to any other resources.
Device Management Libraries
Maintainer: Automation and Tools
To interface with the test devices, we use a python library called mozdevice (part of the MozBase project) which allows us to interface with the SUTAgent application running on the devices.
Both the build and maintenance scripts and the test harnesses themselves build on top of this basic component.
Historically, the files that comprise mozdevice have been copied adhoc whereever they have been needed. To ease future maintenance, the goal is to centralize on a single maintained copy inside the mozbase project (with a regularly updated mirror inside mozilla-central to make things easier for contributors to run tests).
Source code: The mozdevice source code is currently stored in mozilla-central in the build/mobile subdirectory (web link) as devicemanager.py and devicemanagerSUT.py.
Other copies of mozdevice are strewn through our various codebases. For example, old copies of this library are included and used directly by the build and maintenance scripts described below.
Build and Maintenance Scripts
Maintainer: Release Engineering
These scripts are run by buildbot when a job is allocated to a test controller (foopy). It is their job both to setup a device for testing, run a suite of tests, then clean up the device for their next job.
As of this writing, the eventual plan is to replace these scripts with mozharness scripts, which should prove to be easier to debug and maintain.
Source code: This is stored in mercurial under the build/tools project under sut_tools (web link)
FIXME: I (wlach) have heard about all sorts of other tools that release engineering uses to maintain farms of tegras. For example, a graphical view of which tegras are failing, working, etc. We should (very briefly) describe these tools, and link to both them and their source (if they are not encompassed by the sut_tools link above).
Test harnesses (and actual tests)
Maintainers:
- Test harnesses: Automation and Tools
- Tests: All developers
The purpose of test harnesses is to run actual tests. On mobile, a test harness running on a test controller takes a clean/working device as well as the test to be run as input and gives test results as output.
You can find lots more detail on all the different types of test harnesses we use on mobile in testing section of the Android wiki page.