Auto-tools/Projects/Robocop/WritingTests

Robotium is a test framework created to make it easy to write powerful and robust automatic black-box test cases for Android applications. Robocop provides a wrapper around Robotium making it even easier to write and execute UI tests for native Fennec.

New tests should probably use the UITest base class - see the documentation at UITest. Most of the information below will also be relevant to users of UITest (though the information should probably be curated).

Creating a new test

These tests will extend BaseTest.

The best way to create a new Robocop test is to copy and modify an existing one -- see mobile/android/base/tests.

(We are currently upgrading our tests from Robotium 3.6 to 4.2 [bug 912590], so the APIs used in the tests may be out of date. It might be helpful to take a look at the Robotium changelog to see some of the new API methods.)

Create a new file in the test directory named test[YourFeature].java.in.

The top of each test file must have:

#filter substitution
package @ANDROID_PACKAGE_NAME@.tests

Then define a test class extending the Robotium ActivityInstrumentationTestCase2:

public class testMyFeature extends ActivityInstrumentationTestCase2

For convenience, you can also extend one of the provided abstract classes, like BaseTest, PixelTest, or AboutHomeTest. BaseTest derives from ActivityInstrumentationTestCase2 but also provides a bunch of boilerplate and utility functions.

public class testMyFeature extends BaseTest

Each test class must have three methods:

protected void setUp() // Starts Fennec and sets up commonly used member variables. This is usually very similar for every test class.
public void test[YourFeature]() // Your test code goes here. Use the Robocop API to access Fennec elements, click, send keys, and assert conditions.
public void tearDown() // Clean up. This is usually very similar for every test class.

If you extend one of the provided abstract classes, the setUp() and tearDown() are provided for you, but you can still override them and do additional setup/teardown if needed; just make sure you call super.setUp() and super.tearDown() in your overiddes as necessary.

Finally, add your new test file to the test manifest: mobile/android/base/tests/robocop.ini:

[testAllPagesTab]
[testHistoryTab]
...
[testMasterPassword]
[testDeviceSearchEngine]
[testMyFeature] # NEW!

Creating a new Content Provider test

We have infrastructure for testing content providers in an isolated environment. This means that we ensure that any updates to the databases behind content providers will not affect or be affected by Firefox.

To write a content provider test, follow the same file naming and Java packaging guidelines described for UI tests, but create a class that inherits from ContentProviderTest instead of BaseTest. Each Content Provider test class must have the same three methods as UI tests (setUp(), test[YourFeature]() and tearDown) with one important difference: the setUp() method should call the following parent method:

protected void setUp(String className, String authorityField);

Where className is the class of your content provider and authorityField is the name of the property in BrowserContract containing your content provider AUTHORITY string. For example, for BrowserProvider, the setUp() method would look something like:

public setUp() throws Exception {
    super.setUp("@ANDROID_PACKAGE_NAME@.db.BrowserProvider", "AUTHORITY");
    // Then other setup operations...
}

After this is called, mProvider will point to an isolated instance of your content provider and you can make insert/query/update/delete calls on it as expected. For more details and sample code on how to implement Content Provider tests, see testBrowserProvider.java.in.

APIs

Robotium itself provides a rich API through the Solo class - javadocs [1] for Solo are available.

"Robocop" provides an additional API to make common tasks easier. The main interfaces are Actions, Elements, and Driver.

Actions provides commonly used non-element specific actions that can be taken on the application, such as dragging and sending key events.

Actions
  //This will cause this process to spin until the gecko fires a specific JSON event, such as DOMContentLoaded
  void waitForGeckoEvent(String geckoEvent);
  //Clicks the given Key (Actions.SpecialKey.[DOWN|UP|LEFT|RIGHT|ENTER])
  void sendSpecialKey(SpecialKey button)
  //Sends a string of characters to the system. (most have been implemented but not all)
  void sendKeys(String keysToSend);
  //Sends a drag action across the screen
  void drag(int startingX, int endingX, int startingY, int endingY)
  // Run a sql query on the specified database
  public Cursor querySql(String dbPath, String sql);

Element represents each of the available UI objects in Fennec including the Awesomebar, the 'tabs' button, and different lists and menus.

Element
  //To click the element.
  void click()
  //Returns true if the element is currently displayed
  boolean isDisplayed();
  //Returns the text currently displayed on the element, or direct sub-elements.
  String getText();

Driver finds elements and provides info about the UI.

Driver
  //This is used to find elements given their id's name.
  Element findElement(String name);
  //This is used for getting information on scrolls. NOTE: It must be used for the next three methods to return useful information
  void setupScrollHandling();
  int getPageHeight(); //The total height of the page.
  int getScrollHeight(); //How far down the screen the client has scrolled.
  int getHeight(); //The height of the client's view.

  //The following are used to give information on the graphical location of the Gecko view on the screen.
  int getGeckoTop();
  int getGeckoLeft();
  int getGeckoHeight();
  int getGeckoWidth();

See <objdir>/mobile/android/base/R.java to find ids that can be used with Driver.findElement()

Finally, an evolving set of test base classes - BaseTest, PixelTest, etc - can be leveraged for some types of tests.

Tips

Event timing

A recurring issue when writing UI tests is the timing of events. To enter a URL, you need to click on the awesome bar and then send the key events for the text. If you click() and then immediately sendKeys(), the text probably won't get to the awesome bar. If you sleep() briefly before and after clicking, the task will probably succeed...but how long are those sleep() calls? Will the test still work on other devices, reliably? Avoid the temptation to scatter sleep() throughout your test. Whenever possible, wait for events or other feedback to verify the UI is in the required state before proceeding. For example, many tests will want to wait for startup before starting the test: driver.waitForGeckoEvent("Gecko:Ready").

Use waitForCondition instead of sleep()

If you cannot find an event to wait for and you are still tempted to sleep(), consider using waitForCondition() instead. waitForConditon() gives you a simple way to poll for a condition for a specified maximum amount of time. This provides the advantage that the test can proceed as soon as the condition is met, reducing test time.

Logging

Robocop logs to both logcat and a file log via Assert.dumpLog (or FennecNativeDriver.log). Test code can use these, or log as a side-effect of Assert.is, Assert.ok, etc. Do not call android.util.Log.i/w/e/etc directly.

Robocat is a command line tool to transform the JSON robocop dumps into logcat into a human-readable format: https://github.com/pocmo/Robocat

Indirect Database Operations

Robocop UI operations (eg click on a certain button) may initiate database operations (eg create a new record in a database table). When performing these actions manually, the effect appears to occur instantly, but of course the database operation takes some time, and often occurs on a background thread, independent of the UI. Robocop tests that indirectly initiate database operations and then immediately check for the effect of that operation (eg search for text in a table) may fail, or fail intermittently. It is generally better to loop, periodically checking for the effect to occur. See testBookmark for an example.

Screenshots

Robocop takes a screenshot when a test fails. See http://gbrownmozilla.wordpress.com/2013/03/26/screenshots-for-robocop/ for details.

Do not rely on view indices

Some Robotium APIs allow a test to specify a view by index; for instance, Solo.clickOnButton(int index). Selecting a button with a constant index may work for a broad set of circumstances only to fail under a different version of Android, or when a new button is added, or views are simply organized differently. In most cases, using something like Solo.clickOnButton(String text) has proven to be more reliable.

Check and report return values; use BaseTest.waitForText

Intermittent test failures can be very difficult to debug and often need to be debugged based entirely on the log from the failed test. Ignoring robotium return codes can make the task more complicated. For example, consider boolean Solo.waitForText(String). waitForText blocks until the specified text is displayed, or until a maximum timeout is reached. If the text is found, waitForText returns true; if it times out, false. Tests rarely assert based on waitForText, but use it to wait until a certain view is displayed:

 clickOnSomething()
 Solo.waitForText(someTitle)
 clickOnSomethingElse()
 mAsserter.ok(...)

If clickOnSomething fails to bring up the expected view, the test might only fail on the mAsserter.ok(), even though the problem is much earlier. So it's better to use something like:

 clickOnSomething()
 mAsserter.ok(Solo.waitForText(someTitle), ...)
 clickOnSomethingElse()
 mAsserter.ok(...)

or:

 clickOnSomething()
 if (!Solo.waitForText(someTitle))
     mAsserter.dumpLog(...)
 clickOnSomethingElse()
 mAsserter.ok(...)

or use BaseTest.waitForText, which conveniently logs on failure:

 clickOnSomething()
 waitForText(someTitle)
 clickOnSomethingElse()
 mAsserter.ok(...)

Be specific: Use clickOnButton or clickOnView in preference to clickOnText

We have seen several intermittent failures when using Solo.clickOnText because the specified text appears in more than one place. It's really easy to write a test that says clickOnText("OK"), intending of course to click on the OK button and not the prompt that says "Click OK to proceed"! Worse yet, that test might work sometimes but not others, since Robotium will simply click on the first view with the specified text. Using view-specific functions, like clickOnButton(String text) addresses many of these issues; look over the screen very carefully to catch the others! (You can also enumerate views to identify the view intended, then use clickOnView(View).)

Be careful when relying on scrolling in waitForText

Solo.waitForText (and some other Robotium functions) will conveniently scroll down in a list to find text that is not displayed. But it will not scroll up again! Consider a test like this:

 <open a menu or display a list>
 mAsserter.ok(waitForText("Item 1"), ...
 mAsserter.ok(waitForText("Item 2"), ...

• If both Item 1 and Item 2 are visible, this test passes.
• If Item 1 occurs first in the list, with Item 2 later, and Item 2 is off the bottom of the screen, then the first waitForText succeeds immediately, the second waitForText scrolls down, then succeeds, and the test passes.
• If Item 2 occurs first in the list, with Item 1 later, and Item 1 is off the bottom of the screen, then the first waitForText scrolls down to find Item 1. If that operation scrolls Item 2 off the top of the screen, then the second waitForText will fail!

Note that the test is therefore dependent on screen size and orientation, as well as Robotium scrolling behavior.

Useful links

1. Our UITest base class documentation
2. Latest Robotium javadocs: http://robotium.googlecode.com/svn/doc/index.html
3. Robotium project page: http://code.google.com/p/robotium/
4. Robotium changelog for versions: http://code.google.com/p/robotium/wiki/Changelog
5. Robotium GitHub repo: https://github.com/jayway/robotium/tree/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo