Robocop provides UI-level testing for Firefox for Android. It is based on Google's Robotium test automation framework for Android. Robocop uses Robotium to inject events into the Java front-end of Firefox for Android and to monitor and verify Firefox events and UI views.
The source code for the Robocop harness, and the test source code, is maintained on mozilla-central in mobile/android/tests/browser/robocop.
Robocop-based UI functionality tests are run with the mochitest test harness and appear on Treeherder under the symbols rc1 and rc2.
Robocop-based performance tests are run with the Talos test harness and appear on Treeherder under the symbols rck, rck2, rp, and rpr.
Step one: Build Fennec
Follow the instructions at https://wiki.mozilla.org/Mobile/Fennec/Android#Building_Fennec to build, package, and install Fennec on your device.
Running all the tests
From the root directory:
mach build mobile/android/tests/browser/robocop mach robocop
Some notes for running tests:
- Your device screen should be turned on — otherwise some tests may fail.
- To run tests that load a page from the desktop host without timing out, both the test device and desktop host must be on the same network (e.g., device can't be on 3G).
Running a single test
From the root directory:
./mach robocop <test-name>
where <test-name> would be a test name such as "testLoad":
./mach robocop testLoad
See also https://wiki.mozilla.org/Mobile/Fennec/Android/Testing for advanced options.
Updating your test directory
It is not always necessary to rebuild Fennec to use new tests. If you are updating the build with your own test, ensure that your test has been added to the manifest in mobile/android/tests/browser/robocop/robocop.ini.
Otherwise, you can retrieve tests from the repository by executing:
hg pull hg update
Now you can build the robocop part of Fennec by executing:
mach build mobile/android/tests/browser/robocop
Frequently found errors
If you do not see your specific error below, it can sometimes help to uninstall and reinstall. To do this (and run the tests):
adb uninstall org.mozilla.roboexample.test mach build mobile/android/tests/browser/robocop mach robocop testLoad
Infrequent Runtime Errors
Two have been known to occur: One, where Fennec freezes when loading a page, the other where the page loads, but has 0 content. These are worked around by restarting the test.
What happens when a test is run
Our tests are run from the runrobocop.py script, which relies on devicemanager and other scripts/tools in the harness directory. Here is what happens:
- create a [fresh profile] and copy it to the device (/mnt/sdcard/tests/profile)
- we generate robotium.config and push it to the device
$ adb shell cat /mnt/sdcard/tests/robotium.config profile=/mnt/sdcard/tests/profile # from the first step logfile=/mnt/sdcard/tests/logs/mochitest.log # where robocop will write logs to host=http://mochi.test:8888/tests # mochi.test is defined in the profile proxy auto config rawhost=http://192.168.1.73:8888/tests # used for some tests
- for each test case, in [robocop.ini] we launch fennec. Here is an example of the first test case testallpagesTab:
am instrument -w -e deviceroot /mnt/sdcard/tests -e class org.mozilla.fennec.tests.testAllPagesTab org.mozilla.roboexample.test/org.mozilla.fennec.FennecInstrumentationTestRunner
- If you are writing tests for Robocop, see Writing Tests for more general API information and UITest for details on using the current testing framework
- We are using [NativeUI Messages] to talk between Robocop and gecko
- There has been interest in using a record/replay tool similar to [testdroid]. Most likely this will allow for people to record something and then we can hand edit it into a proper test case.
- adb shell am instrument -w org.mozilla.roboexample.test/android.test.InstrumentationTestRunner
- Trevor's Presentation on it can be found in [This zip]
- For documentation of the Robocop Talos performance tests, see Buildbot/Talos/Tests#Robocop