Mobile/Testing/Architectural Overview

High-level overview

Automated testing of Firefox for Android involves two components: a device to actually run the tests on and a machine (a.k.a. a host or "Foopie") that actually takes care of setting up these devices and running the test. This is contrast with desktop firefox, where the software to control running the tests (the "test harness") is the same as the machine running the actual tests.

The highest-level software we use to control testing and hand out jobs is called BuildBot. Discussion of this is beyond the scope of this document, but the Release Engineering home page has lots of links.

Details

Test Devices (Emulators)

The general principle here (arrived at through iteration) is to keep the logic on the hardware as simple as possible, as it's generally much easier to change things at the level of the test controller. The said, there are two basic layers to the test devices:

• Hardware and Operating System Image
• Device Application Software

Hardware / OS Image

Hardware and operating system images are really two different things, but we group them together because typically we build/qualify them as a unit.

Maintainers:

• Hardware: IT / Release Engineering (FIXME: verify this)
• OS Image: Automation and Tools

Emulators

Instead of running on a physical device, we can run the Android emulator (a fork of qemu) to simulate a device.

We couldn't find a suitable analog for testing our x86 builds of Firefox for Android, so we run in the Android x86 emulator, running an Android 4.2 image. These tests run on an ix slave, where we run up to 4 jobs at a time, in 4 separate emulators.

We use the Android ARM emulator, using various AOSP-derived Android images. For these jobs, we run on AWS, rather than on in-house machines.

Device Application Software

This is additional software written by us to help control or otherwise manage test devices. It is installed on top of the base operating system image.

This software is written entirely in Java.

Maintainer: Automation and Tools

SUTAgent

The SUTAgent is an Android application which we install on all devices. It is the main mechanism by which our test controller can use any modify our system. It contains functions needed to push/pull files from the device, run programs, get information on the device's state, and a few other things.

For more information (including up-to-date links to source code), see the SUTAgent Project Page.

Watcher

The watcher is a small android application which we install on all devices. It has just a few functions:

• Ensure the Android lock screen never comes up
• Periodically ping an external server and reboot if it is unreachable for too long (we currently disable this behavior as this is already taken care of by the foopies: we manually disable it in older versions and newer versions don't do this at all)
• Ensures the SUTAgent is running at startup and if it crashes for some reason.

Source code: The watcher source code is currently stored in mozilla-central in the build/mobile/sutagent/android/watcher subdirectory (web link).

Test Controller (a.k.a. "Foopies")

As mentioned above, most of the complexity in running automated testing is in this part of the automation, which controls the devices just described. There are six layers to this:

• Mac Mini hardware and Linux/Mac OS image
• Device management libraries
• Build and maintenance scripts
• Test harnesses (and actual tests)

With the exception of the tests themselves, all this software is written in Python.

Hardware / OS Image

Maintainers:

• Hardware: IT / Release Engineering (FIXME: verify this)
• OS Image: Release Engineering

The system we use to run all this software.

We use two operating systems to run these: Linux and MacOS X. This mainly has historical reasons, and there will be no new test controllers built using the Mac OS X operating system.

Linux Operating System

FIXME: Describe OS revs, installed versions of python. Include links to any other resources.

Mac Operating System

FIXME: Describe OS revs, installed versions of python. Include links to any other resources.

Device Management Libraries

Maintainer: Automation and Tools

To interface with the test devices, we use a python library called mozdevice (part of the MozBase project) which allows us to interface with the SUTAgent application running on the devices.

Both the build and maintenance scripts and the test harnesses themselves build on top of this basic component.

Historically, the files that comprise mozdevice have been copied adhoc whereever they have been needed. To ease future maintenance, the goal is to centralize on a single maintained copy inside the mozbase project (with a regularly updated mirror inside mozilla-central to make things easier for contributors to run tests).

Source code: The mozdevice source code is currently stored in mozilla-central in the build/mobile subdirectory as devicemanager.py and devicemanagerSUT.py.

Other copies of mozdevice are strewn through our various codebases. For example, old copies of this library are included and used directly by the build and maintenance scripts described below.

Build and Maintenance Scripts

Maintainer: Release Engineering

These scripts are run by buildbot when a job is allocated to a test controller (foopy). It is their job both to setup a device for testing, run a suite of tests, then clean up the device for their next job.

As of this writing, the eventual plan is to replace these scripts with mozharness scripts, which should prove to be easier to debug and maintain.

Source code: This is stored in mercurial under the build/tools project under sut_tools (web link)

FIXME: I (wlach) have heard about all sorts of other tools that release engineering uses to maintain farms of tegras. For example, a graphical view of which tegras are failing, working, etc. We should (very briefly) describe these tools, and link to both them and their source (if they are not encompassed by the sut_tools link above).

Test harnesses (and actual tests)

Maintainers:

• Test harnesses: Automation and Tools
• Tests: All developers

The purpose of test harnesses is to run actual tests. On mobile, a test harness running on a test controller takes a clean/working device as well as the test to be run as input and gives test results as output.

You can find lots more detail on all the different types of test harnesses we use on mobile in testing section of the Android wiki page.

Other Frameworks

There are two other mobile-testing frameworks that are run independently from the above, BuildBot-coupled systems:

• Autophone, which performs performance and a bit of correctness testing on end-user Android phones.
• Eideticker, a system that measures performance on real Android devices in a decoupled manner by analyzing video output via HDMI and camera.