Getting Started

• You can find information about setting up the enviroment for running the tests here
• Please make sure to look over this page for some general information about Robocop tests.
• Robocop is a Java API based on the Robotium API used for automation testing on the Android OS
• The existing test cases can be found in the sorce directory under the folder mobile/android/base/tests
• The class that will cover the text will have to be named the same as the test file
• The test file name should be test[Feature_to_be_tested].java.in
• The test must be included in the same package as all other Robocop tests

How to start when writing a new test

• Setup the enviroment. Follow the instructions here.
• Download sources. More info can be found here about how to do it.
• If you already have the sources downloaded update to the lates version by running the command in the sources directory:

hg pull -u

• Make the build by running:

make -f client.mk

• Create a new file named "test" followed by the test name and with the extention "java.in" : for e.g. testBookmarks.java.in
• Copy the basic structure of the test presented in the next section to the test making sure to match the test name and the class and constructure names.
• Create a new mercurial queue patch on the repository by running in the source folder the command:

hg qnew <patch_name>

• Add the new test file to the queue by running the command:

hg add mobile/android/base/tests/<test_file_complete_name>

• Open robocop.ini from mobile/android/base/tests and add after the last Robocop test a new line with the test name between sqare brackets ( "[" and "]")
• Write the test, compile robocop (here is how) and run the test in order to test that it works
• After the test works follow the instruction to create a patch so the test can be uploaded to a bug and integrated in mozilla.
• Before updating the sources or after the patch is completed make sure to return the sources to the unmodified version by running the command in the source folder until there are no patches in the queue:

hg qpop

Basic structure of the test

#filter substitution
package @ANDROID_PACKAGE_NAME@.tests;

import @ANDROID_PACKAGE_NAME@.*;
 
/*
Insert info about the test here - a basic description of what the test does
 */
public class test[Feature_to_be_tested] extends BaseTest {
    @Override
    // This method is needed in the BaseTest class
    protected int getTestType() {
        return TEST_MOCHITEST; // Test type should remain Mochitest
    }
    /* 
    * This is the class constructor 
    * This method will contatin the test that will be ran
    */
    public void testLoad() {
   
    }
}

• This is a basic structure of a Robocop test
• Each test class must have three methods:
  • protected void setUp() // Starts Fennec and sets up commonly used member variables. This is usually very similar for every test class
  • public void test[YourFeature]() // Your test code goes here. Use the Robocop API to access Fennec elements, click, send keys, and assert conditions
  • public void tearDown() // Clean up
• If your tests extends BaseTest then you will not need to add the methods setUp() and tearDown() as they are already defined in BaseTest
• If you need to do extra clean-up, like removing entries you added in a database for e.g., the tearDown() method can be overwrittern but make sure to call super.tearDown() as needed in the method. The same is applicable for setUp().

APIs

Robotium itself provides a rich API through the Solo class - javadocs for Solo are available at [1].

"Robocop" provides an additional API to make common tasks easier. The main interfaces are Actions, Elements, and Driver.

Actions provides commonly used non-element specific actions that can be taken on the application, such as dragging and sending key events.

Actions
  //This will cause this process to spin until the gecko fires a specific JSON event, such as DOMContentLoaded
  void waitForGeckoEvent(String geckoEvent);
  //Clicks the given Key (Actions.SpecialKey.[DOWN|UP|LEFT|RIGHT|ENTER])
  void sendSpecialKey(SpecialKey button)
  //Sends a string of characters to the system. (most have been implemented but not all)
  void sendKeys(String keysToSend);
  //Sends a drag action across the screen
  void drag(int startingX, int endingX, int startingY, int endingY)
  // Run a sql query on the specified database
  public Cursor querySql(String dbPath, String sql);

Element represents each of the available UI objects in Fennec including the Awesomebar, the 'tabs' button, and different lists and menus.

Element
  //To click the element.
  void click()
  //Returns true if the element is currently displayed
  boolean isDisplayed();
  //Returns the text currently displayed on the element, or direct sub-elements.
  String getText();

Driver finds elements and provides info about the UI.

Driver
  //This is used to find elements given their id's name.
  Element findElement(String name);
  //This is used for getting information on scrolls. NOTE: It must be used for the next three methods to return useful information
  void setupScrollHandling();
  int getPageHeight(); //The total height of the page.
  int getScrollHeight(); //How far down the screen the client has scrolled.
  int getHeight(); //The height of the client's view.

  //The following are used to give information on the graphical location of the Gecko view on the screen.
  int getGeckoTop();
  int getGeckoLeft();
  int getGeckoHeight();
  int getGeckoWidth();

Finally, an evolving set of test base classes - BaseTest, PixelTest, etc - can be leveraged for some types of tests.

Usable IDs

The following is a list of ids that can be used with Driver.findElement(). Most of the following ids have not been tested, so might have unexpeced results, or require increased APIs for them. To know how a given object is used, in mobile/android/base, grep R.id.[id-name] * (from Objdir/mobile/android/base/R.java#id)

abouthome_content
add_tab
addons
address_bar
agent_mode
all_pages_list
awesome_bar
awesome_screen
awesomebar_button
awesomebar_tabs
awesomebar_text
background
bookmark
bookmark_icon
bookmark_title
bookmark_url
bookmarks_list
browser_toolbar
close
container
doorhanger_choices
doorhanger_container
doorhanger_title
favicon
forward
gecko_layout
grid
history_list
info
list
main_layout
notification_image
notification_progressbar
notification_text
notification_title
outline
plugin_container
preferences
quit
recommended_addon_list
reload
save_as_pdf
screenshot
select_list
share
site_security
stop
tabs
tabs_count
title
url

Gecko Events you can wait for

• There are a set of messages sent from Gecko to Java to signal different actions
• Also Java can register observers to catch changes to deferent actions
• Here is a list of Gecko Events on which you can create EventExpectors and block until the message is received

Addons:Change
Addons:All - shown when sending all addons to java as a json
Addons:FetchAll - get the state of all addons
Browser:Quit
Browser:ZoomToRect - zoom to element (viewport if the element is to large)
Browser:ZoomToPageWidth
Campaign:Set - Update the prefs for this session - related to distribution
CharEncoding:Data
CharEncoding:Get
CharEncoding:Set
CharEncoding:State
Content:LoadError
Content:PageShow
Content:ReaderEnabled
Content:StateChange - change in network state
Content:LocationChange - change of location
Content:SecurityChange
DesktopMode:Change
Distribution:Set
Distribution:Set:OK
DOMFullScreen:Start
DOMFullScreen:Stop
DOMContentLoaded
DOMLinkAdded
DOMMetaAdded
DOMModalDialogClosed
DOMUpdatePageReport
DOMTitleChanged
DOMWindowClose
DOMWillOpenModalDialog
Doorhanger:Add
Doorhanger:Remove
Doorhanger:Reply
Feedback:LastUrl
Feedback:OpenPlayStore
Feedback:MaybeLater
FilePicker:Show
FormAssist:Hide - shown when hide request given
FormAssist:AutoComplete - shown when autocomplete is displayed
FormAssist:Blocklisted 
FormAssist:Hidden - shown when the form assist has been hidden
FormAssist:ValidationMessage - shown when a vaildation message is displayed
FormHistory:Init - initialize form history db on update
FullScreen:Exit
Gecko:Ready
Gesture:CancelTouch
Gesture:DoubleTap
Gesture:LongPress
Gesture:Scroll
Gesture:ScrollAck
Gesture:SingleTap
Link:Favicon
Link:Feed
Menu:Add
Menu:Clicked
Menu:Update
Menu:Remove
MozApplicationManifest
MozMagnifyGesture
MozScrolledAreaChanged
nsPref:changed
OrderedBroadcast:Send
pageshow
Panning:Override
Panning:CancelOverride
PanZoom:StateChange
Passwords:Init - initialize passwords db on update
Permissions:Data
Prompt:Show
PluginBindingAttached
Pref:Change
Preferences:Get
Preferences:Set
Preferences:Observe
Preferences:RemoveObservers
PrivateBrowsing:Data
Reader:Add
Reader:Added
Reader:FaviconRequest
Reader:GoToReadingList
Reader:Remove
Reader:Share
robocop:scroll
SaveAs:PDF
Sanitize:ClearData
Sanitize:ClearHistory
Sanitize:Finished
scroll
ScrollTo:FocusedInput
SearchEngines:Data
SearchEngines:Get
Session:Back
Session:Forward
Session:Reload
Session:RestoreEnd
Session:ShowHistory
Session:StatePurged
Session:Stop
SessionHistory
Share:Text
SharedPreferences:Set
SharedPreferences:Observe
Shortcut:Remove
Tab:Close
Tab:Closed
Tab:HasTouchListener
Tab:Load
Tab:Select
Tab:Selected
Tab:ViewportMetadata
Telemetry:Add
Telemetry:Gather
TextSelection:HideHandles
TextSelection:PositionHandles
TextSelection:ShowHandles
Toast:Show
ToggleChrome:Focus
ToggleChrome:Show
ToggleChrome:Hide
Viewport:Change
Viewport:Flush
Viewport:FixedMarginsChanged
Wallpaper:Set
WebApps:Open
WebApps:PreInstall
WebApps:PostInstall
Window:Resize

Basic code sequences to do different actions

Load page

• Note that the tests need to use local pages in order to run on the mozilla setup. Outside pages are not accessible
• Using an URL

String url = "<insert_link_here>";
loadURL(url);

• Using the robocop tests blank page

String url = getAbsoluteUrl("/robocop/robocop_blank_01.html");
loadUrl(url);

Check Title

Element awesomebar = mDriver.findElement(getActivity(), "awesome_bar_title"); // search the awesomebar title
mAsserter.isnot(awesomebar, null, "Got the awesomebar"); // log "Got the awesomebar" in the logs
assertMatches(awesomebar.getText(), "<insert_title_here", "page title match"); // do a match between expected and actual title

Open the app menu (Custom Menu)

mActions.sendSpecialKey(Actions.SpecialKey.MENU);
//Open the More menu for devices with old style menues
if (mSolo.waitForText("^More$")) { 
mSolo.clickOnText("^More$");
}

Open Menu item

mSolo.waitForText("^Settings$");
mSolo.clickOnText("^Settings$");

Use the Back key

mActions.sendSpecialKey(Actions.SpecialKey.BACK);

Set up listeners to catch the page load

Actions.EventExpecter tabEventExpecter = mActions.expectGeckoEvent("Tab:Added");
Actions.EventExpecter contentEventExpecter = mActions.expectGeckoEvent("DOMContentLoaded");

Wait for the new tab and page to load

tabEventExpecter.blockForEvent();
contentEventExpecter.blockForEvent();

Wait for Gecko to load

mActions.expectGeckoEvent("Gecko:Ready").blockForEvent();

Open the Bookmarks tab - method

private ListView openBookmarksList() {
    Activity awesomeBarActivity = clickOnAwesomeBar();
    // Click the "Bookmarks" tab to switch to bookmarks list
    mSolo.clickOnText("Bookmarks");

    TabHost tabHost = (TabHost)mSolo.getView(TabHost.class, 0);
    return (ListView)tabHost.getCurrentView();
   }

• Similarly you can get the History and Top Sites tabs

Open Bookmark context menu

View child = list.getChildAt(<nr>); //bookmark nr- numbering starts at 0 
mSolo.clickLongOnView(child); //long tap to open context menu

• Similarly you can open the context menu for History and Top Sites items

Open a new tab and check the tab count

expectedTabCount = <nr_of_expected_tabs>;
tabCountText = tabCount.getText();
tabCountInt = Integer.parseInt(tabCountText);
mAsserter.is(tabCountInt, expectedTabCount, "Number of tabs increased");

Logging info in the Android Logs to help debug

• The Assert class can be found in the source folder in the subfolder /obj-android/build/mobile/robocop
• Please see the java code for the Assert methods for more info about how they work

Test an object is not the same as a second object

• Method signature:

public void isnot(Object a, Object b, String name)

• Example for checking that there is a title set for the current page:

Element awesomebar = mDriver.findElement(getActivity(), "awesome_bar_title");
mAsserter.isnot(awesomebar, null, "Got the awesomebar"); // Checks that the awesomebar title
 is not null and posts in the logs

Test an object is the same as a second object

• Method signature:

public void is(Object a, Object b, String name)

• Example for checking the default bookmarks:

ListView list = getAllPagesList(url);
mSolo.waitForText(url);

mAsserter.is(list.getChildCount(), 5, "all pages list has 5 children (the default bookmarks)"); // Checks that 
there are 5 entries in the list and posts in the logs

Write text in logs

• Method signature:

public void dumpLog(String message) // writes text in the logs

Log info about test progression

• Method signature:

public void ok(boolean condition, String name, String diag) 

• Example

ListView list = getAllPagesList(url);
mSolo.waitForText(url);

mAsserter.ok(list != null, "checking that all pages list exists", list.toString()); // tests if 
the list is null and if it isn't it prints the list after printing the message of what the test does

Debugging tests

See the screenshot from the fail in the try run

• Open the link to the tryrun and view the full logs of the failed run
• Go the the position the fail you are interested in has happened
• Search for "base64" and copy the hole line from the log
• Remove the "SCREENSHOT" from the beginning of the string and paste it in the URL bar of the browser
• You can see the screenshot of the output the moment the test failed

Debugging using logs

• You can use mAsserter.dumpLog(Sting) or mAsserter.ok(true, Sting, String) to print any string or value of any variable
• Note that you will need to cast the variable to String in order to print it: String.valueOf(Variable)

Substituting mSolo.sleep

• Don't use sleep unless absolutely necessary
• It's ok to use sleep while creating the test but at the end try and substitute it
• Here are some alternatives:
  • waitForText(String waitText)
  • waitForEnabledText(String waitText)
  • blockForEvent/ blockForEventData - gecko event expecters
  • waitForCondition
  • waitForTest
  • waitForPaint
  • waitForView(View view)
  • waitForView(View view, int timeout, boolean scroll)
  • waitForDialogToOpen()
  • waitForDialogToClose()
  • waitForView(int id)
  • waitForView(int id, int minimumNumberOfMatches, int timeout)
  • waitForView(int id, int minimumNumberOfMatches, int timeout, boolean scroll)
  • waitForActivity(name)
  • waitForActivity(Class<? extends Activity> activityClass)
  • waitForActivity(Class<? extends Activity> activityClass, int timeout)

Updating your test directory

It is not always necessary to rebuild Fennec to use new tests. If you are updating the build with your own test, ensure that your test has been added to the manifest in /PATH/TO/FENNEC/SOURCE/mobile/android/base/tests/robocop.ini .

Otherwise, you can retrieve tests from the repository by executing:

hg pull
hg update

Now you can build the robocop part of Fennec by executing:

cd objdir
make -C build/mobile/robocop/
make package 

Creating a diff patch file

All commands need to be run on the main root source folder.

Steps to perform for new patches:

• Create a new patch
• Add or modify any files
• Refresh the repository
• Pop the patch to have a clean repository
• Push the patch back
• Export the patch
• Pop the patch to clean up the repository

Steps to perform for existing patches:

• Push the patch to the repository
• Add or modify any files
• Refresh the repository
• Pop the patch to have a clean repository
• Push the patch back
• Export the patch
• Pop the patch to clean up the repository

Create a new patch in the Mercurial Queue

Merurial can create patch queue (patch stack) so the user can keep his repository clean and can add or work on multiple patches on the same repository

hg pull -u
hg qnew <patch name>

You need to do hg pull to have the repository up to date in order to create a correct diff patch

Adding new files to be tracked by Mercurial

You need to add new files to be tracked otherwise the diff will be created only for the existing changed files Run the following command for each new file

hg add mobile/android/base/tests/<test_file>

Refresh the repository

When first creating a patch:

hg qrefresh -m "Commit message"

When editing patches:

hg qrefresh

The commit message has the general form: "Bug <number> - <Description>". The commit message should be set for a new patch and will appear in the patch file.

Poping the patch

Removes the patch from the patch stack (Queue)

hg qpop

Pushing the patch

Adds the last patch to the top of the queue. Alternatively you can specify a patch name to add.

hg push -v [<patch_name>]

Viewing the diff file

hg qdiff 

Exporting the patch

Exports the top patch in the stack to a file that can be submitted for review

hg export tip > <patch_name.patch>

Remove files from the current patch

In case you added an unwanted file to the patch it can be removed by running:

hg rm -f mobile/android/base/tests/<test_file>

See a list of all patches in the queue and changed files outside of patches

• If there are files displayed that means that the file has been changed outside of any patch

hg status

Apply a certain patch/patch tree

• This command will apply the patch or the sequence of patches until the selected patch

hg qgoto <patch_name>

Adding a patch to the tryserver

• In order to add a patch to the tryserver you need to get Commit access level 1
• Follow the steps listed here to get the access
• All patches must contain a trychooser patch on top of it in order to limit the tests only to what it is needed. Build the statement for it using this page

Create the try patch over the existing patch queue

hg qnew -m "try: -b o -p android -u robocop-1,robocop-2 -t none" try

Push the changes to the tryserver

hg push -f -rtip ssh://<user>@hg.mozilla.org/try/

• This will push the changes and will give you a Treeherder link to your push
• When the building process for the test build will start you will also receive an email with the push info

Unlock the patch queue

• From Mercurial 2.0 the queue will be locked if you pushed to try
• You can unlock the queue in order to remove the patches or change them

hg phase -f --draft qbase:tip

Removing the try patch from the queue

• After you are done with the push you can delete the try patch so you can add it to other patches for push

hg qremove try