MozillaWiki

Auto-tools/Projects/Robocop: Difference between revisions

Page Discussion

Auto-tools/Projects/Robocop (view source)

Revision as of 11:40, 16 March 2021

2,240 bytes removed ,  16 March 2021
Removed reference to Robocop performance tests, which no longer exist.
(Refer to robocop (rather than robotium) as much as possible.)
(Removed reference to Robocop performance tests, which no longer exist.)
 
(21 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Problem  ==
== Introduction ==


With advent of NativeFennec, a new framework was needed to test the UI of the Mobile Browser. [http://robotium.org/ Robotium] was identified as a useful framework to take advantage of, but is not complete. Robocop incorporates all of Activity, Instrumentation, and Robotium Classes to inject events into the new Java front-end of Fennec.
Robocop provides UI-level testing for Firefox for Android. It is based on Google's [http://robotium.org/ Robotium] test automation framework for Android.
Robocop uses Robotium to inject events into the Java front-end of Firefox for Android and to monitor and verify Firefox events and UI views.
 
The source code for the Robocop harness, and the test source code, is maintained on mozilla-central in <tt>mobile/android/tests/browser/robocop</tt>.
 
Robocop-based UI functionality tests are run with the mochitest test harness and appear on Treeherder under the symbol tc(rc).


== Setup  ==
== Setup  ==
==== Step one: Build Fennec ====
=== Step one: Build Fennec ===
* Follow the instructions at https://wiki.mozilla.org/Mobile/Fennec/Android#Building_Fennec to build, package, and install Fennec on your device.


==== Step two: Build Firefox (Necessary for XPCShell) ====
Follow the instructions at https://wiki.mozilla.org/Mobile/Fennec/Android#Building_Fennec to build, package, and install Fennec on your device.
* Follow [https://developer.mozilla.org/En/Simple_Firefox_build these instructions] to build Firefox for desktop, just so that you can build xpcshell. As an alternative, you can download a "tests" zip from a current build for your platform, such as ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-23.0a1.en-US.linux-i686.tests.zip -- unzip and find xpcshell.


== Running tests ==
== Running tests ==


==== Set up the MOZ_HOST_BIN environment variable ====
==== Running all the tests ====
The MOZ_HOST_BIN variable will need to be set to the directory with XPCShell on your machine.
From the root directory:
You can execute the following command in the terminal, or it can be saved into .bashrc / .bash_profile for convinence:
<pre>export MOZ_HOST_BIN=/ABSOLUTE/PATH/TO/FIREFOX/SOURCE/{objdir}/dist/bin</pre>
If you're having difficulty finding the absolute path of XPCShell's directory, simply locate XPCShell in your {objdir}/dist/bin and execute the command pwd .


Again, {objdir} is a placeholder.
  mach build mobile/android/tests/browser/robocop
  mach robocop


==== Running all the tests ====
Some notes for running tests:
From the Fennec's objdir:
* Your device screen should be turned on — otherwise some tests may fail.
<pre>make mochitest-robocop</pre>
* To run tests that load a page from the desktop host without timing out, both the test device and desktop host must be on the same network (e.g., device can't be on 3G).


==== Running a single test ====
==== Running a single test ====
From the Fennec's objdir (using testLoad as an example):
From the root directory:
<pre>TEST_PATH=testLoad make mochitest-robocop</pre>
<pre>./mach robocop <test-name></pre>
where <test-name> would be a test name such as "testLoad":
<pre>./mach robocop testLoad</pre>


See also https://wiki.mozilla.org/Mobile/Fennec/Android#Testing and https://wiki.mozilla.org/Mobile/Fennec/Android#Robocop for advanced options.
See also https://wiki.mozilla.org/Mobile/Fennec/Android/Testing for advanced options.


== Updating your test directory ==
== Updating your test directory ==
It is not always necessary to rebuild Fennec to use new tests. If you are updating the build with your own test, ensure that your test has been added to the manifest in /PATH/TO/FENNEC/SOURCE/mobile/android/base/tests/robocop.ini .
 
It is not always necessary to rebuild Fennec to use new tests. If you are updating the build with your own test, ensure that your test has been added to the manifest in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt>.


Otherwise, you can retrieve tests from the repository by executing:
Otherwise, you can retrieve tests from the repository by executing:
<pre>
 
hg pull
  hg pull
hg update</pre>
  hg update


Now you can build the robocop part of Fennec by executing:
Now you can build the robocop part of Fennec by executing:
<pre>
cd objdir
make -C build/mobile/robocop/
make package </pre>
== Frequently found Errors  ==


OOM Error. This is an error found after use of Robocop frequently, but once it has happened, Fennec itself continues to crash. To fix it, simply uninstall and reinstall. This does not occur of frequent use of Robocop, as I have run 30 iterations of just robocop without it throwing this error.
mach build mobile/android/tests/browser/robocop


InvocationTargetException Error. This is an issue arising when using a Tegra.It looks like the following:<br>
== Frequently found errors  ==
<pre>W/System.err( 2610): java.lang.reflect.InvocationTargetException
If you do not see your specific error below, it can sometimes help to uninstall and reinstall. To do this (and run the tests):
W/System.err( 2610): at org.mozilla.gecko.GeckoAppShell.unregisterGeckoEventListener(GeckoAppShell.java:1565)
...
</pre>


Build Error. This is resolved by doing a clean build.
  adb uninstall org.mozilla.roboexample.test
<pre>
  mach build mobile/android/tests/browser/robocop
make[7]: Entering directory `/media/sdan/birch/objdir-native-droid/mobile/android/base/locales'
  mach robocop testLoad
make[7]: Leaving directory `/media/sdan/birch/objdir-native-droid/mobile/android/base/locales'
res/values/strings.xml:1: error: Error parsing XML: no element found
res/layout/awesomebar_search.xml:9: error: Error: No resource found that matches the given name (at 'hint' with value '@string/awesomebar_default_text').
res/layout-v11/awesomebar_search.xml:9: error: Error: No resource found that matches the given name (at 'hint' with value '@string/awesomebar_default_text').
res/layout/browser_toolbar.xml:7: error: Error: No resource found that matches the given name (at 'hint' with value '@string/awesomebar_default_text').
res/layout/crash_reporter.xml:6: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_message').
res/layout/crash_reporter.xml:14: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_help_message').
res/layout/crash_reporter.xml:21: error: Error: No resource found that matches the given name (at 'text' with value '@string/crash_send_report_message').
...
</pre>


Infrequent Runtime Errors. Two have been known to occur: One, where Fennec freezes when loading a page, the other where the page loads, but has 0 content. These are worked around by restarting the test.
===Infrequent Runtime Errors===
Two have been known to occur: One, where Fennec freezes when loading a page, the other where the page loads, but has 0 content. These are worked around by restarting the test.


== What happens when a test is run ==
== What happens when a test is run ==


Our tests are run from the talos and mochitest scripts. These rely on devicemanager and other scripts/tools in the harness directory.  Here is what happens:
Our tests are run from the runrobocop.py script, which relies on devicemanager and other scripts/tools in the harness directory.  Here is what happens:


* create a [[http://people.mozilla.org/~jmaher/robocop_profile.zip fresh profile]] and copy it to the device (/mnt/sdcard/tests/profile)
* create a [[http://people.mozilla.org/~jmaher/robocop_profile.zip fresh profile]] and copy it to the device (/mnt/sdcard/tests/profile)
Line 81: Line 67:
  host=http://mochi.test:8888/tests            # mochi.test is defined in the profile proxy auto config
  host=http://mochi.test:8888/tests            # mochi.test is defined in the profile proxy auto config
  rawhost=http://192.168.1.73:8888/tests      # used for some tests
  rawhost=http://192.168.1.73:8888/tests      # used for some tests
* we push [https://wiki.mozilla.org/Auto-tools/Projects/Robocop/FennecIDs fennec_ids.txt] to the device (/mnt/sdcard/tests/fennec_ids.txt)
* for each test case, in [[https://wiki.mozilla.org/Auto-tools/Projects/Robocop/robocop_ini robocop.ini]] we launch fennec.  Here is an example of the first test case testallpagesTab:
* for each test case, in [[https://wiki.mozilla.org/Auto-tools/Projects/Robocop/robocop_ini robocop.ini]] we launch fennec.  Here is an example of the first test case testallpagesTab:
  am instrument -w -e deviceroot /mnt/sdcard/tests -e class org.mozilla.fennec.tests.testAllPagesTab org.mozilla.roboexample.test/org.mozilla.fennec.FennecInstrumentationTestRunner
  am instrument -w -e deviceroot /mnt/sdcard/tests -e class org.mozilla.fennec.tests.testAllPagesTab org.mozilla.roboexample.test/org.mozilla.fennec.FennecInstrumentationTestRunner
Line 87: Line 72:
== Notes ==
== Notes ==


*If you are writing tests for Robocop, see [[https://wiki.mozilla.org/Auto-tools/Projects/Robocop/WritingTests Writing Tests]] for more API information
*If you are writing tests for Robocop, see [[Auto-tools/Projects/Robocop/WritingTests|Writing Tests]] for more general API information and [[Mobile/Fennec/Android/UITest|UITest]] for details on using the current testing framework
*We are using [[https://wiki.mozilla.org/Fennec/NativeUI/Messages NativeUI Messages]] to talk between Robocop and gecko  
*We are using [[https://wiki.mozilla.org/Fennec/NativeUI/Messages NativeUI Messages]] to talk between Robocop and gecko  
*There has been interest in using a record/replay tool similar to [[http://testdroid.com/testdroid/125/testdroid-recorder-100beta-out testdroid]]. Most likely this will allow for people to record something and then we can hand edit it into a proper test case.  
*There has been interest in using a record/replay tool similar to [[http://testdroid.com/testdroid/125/testdroid-recorder-100beta-out testdroid]]. Most likely this will allow for people to record something and then we can hand edit it into a proper test case.  
*adb shell am instrument -w org.mozilla.roboexample.test/android.test.InstrumentationTestRunner
*adb shell am instrument -w org.mozilla.roboexample.test/android.test.InstrumentationTestRunner
*Trevor's Presentation on it can be found in [[https://wiki.mozilla.org/File:Robocop_final_presentation.zip.gz This zip]]
*Trevor's Presentation on it can be found in [[https://wiki.mozilla.org/File:Robocop_final_presentation.zip.gz This zip]]
*For documentation of the Robocop Talos performance tests, see [[Buildbot/Talos/Tests#Robocop]]
Davehunt
Confirmed users
2,197

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/659863...1234439"