MozillaWiki

Mobile/Fennec/Android/Testing: Difference between revisions

Page Discussion

Mobile/Fennec/Android/Testing (view source)

Revision as of 20:49, 8 February 2019

6,929 bytes removed ,  8 February 2019
→‎Advanced setup
 
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.
This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.


Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.
Having trouble? Ping :gbrown on #mobile.


== For the impatient... ==
== For the impatient... ==
Line 13: Line 13:
   mach xpcshell-test
   mach xpcshell-test
   mach cppunittest
   mach cppunittest
  mach geckoview-junit
  mach web-platform-test
  mach marionette-test
  mach test


They will all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.


=== Quick reference ===
=== Quick reference ===
Line 111: Line 115:
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).
* Run specific tests from the IDE: select the "Build Variants" menu (bottom left), change "Test Artifact" to "Unit Tests", right-click on the test class you want to run, and select the "Run <test-class>" option.
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.
* Some tests will fail with encryption errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html
 
* There is a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).
 
'''Troubleshooting'''
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).


=== integration tests ===
=== integration tests ===
Line 138: Line 145:


   mach robocop <test-name>
   mach robocop <test-name>
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.
  mach build mobile/android/tests/browser/robocop
  mach robocop
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)


Notes:
Notes:
* Mach is self documenting!  For help, try <tt>mach help robocop</tt>.
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.
* A rooted device is NOT required.
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]


=== Static analysis ===
=== Static analysis ===
Line 224: Line 223:


Notes:
Notes:
* A rooted device IS required.
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.


Line 279: Line 278:


* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).
== browser-chrome ==
Before you run tests, you will need to make sure you have packaged the tests in your object dir:
  make -C <objdir-droid> package-tests
There is currently no special make command to build and run browser-chrome tests, but it should be possible to make them run by calling:
  cd <objdir-droid>/_tests/testing/mochitest
  python runtestsremote.py --dm_trans=adb --test-path=mobile --browser-chrome --deviceIP=1.2.3.4
                          --app=org.mozilla.fennec_$USER --xre-path=<objdir_x86>/dist/bin/


== Running tests on the Android emulator ==
== Running tests on the Android emulator ==


The "Android 2.3 API9 opt", "Android 4.3 API11+ opt" and "Android 4.3 API11+ Debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''. You can also borrow a slave: https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_a_slave.
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 7.0 x86_64 opt/debug" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.
 
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is now quite simple:


  ./mach android-emulator
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:


By default, the 'android-emulator' command will download the Android 4.3 API11+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)
  ./mach android-emulator --version 4.3


To use the Android 2.3 API9 image:
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)


  ./mach android-emulator --version 2.3
To use the Android 7.0 x86_64 image:


To use the Android 4.2 x86 image:
  ./mach android-emulator --version x86-7.0


  ./mach android-emulator --version x86
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is '''much''' faster than the arm emulator.)


The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.
Line 322: Line 307:
   ./mach android-emulator && ./mach install && ./mach mochitest
   ./mach android-emulator && ./mach install && ./mach mochitest


For the '''very''' lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:
For your convenience, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:


   $ ./mach mochitest testing/mochitest/tests/Harness_sanity
   $ ./mach mochitest testing/mochitest/tests/Harness_sanity
Line 341: Line 326:
   ...
   ...


=== Testing with stock AVD images (Android 4.x+ only) ===
=== Multiple emulators/devices ===
 
Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.
 
# From the Android Virtual Device (AVD) Manager, click 'Create....'
# Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).
# Choose a system image with API level 15 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.
# Next, check 'Use Host GPU' if possible. See [http://developer.android.com/tools/devices/emulator.html#acceleration Using Hardware Acceleration].
# Hit Finish and that should be it. If you have setup Firefox for Android to [[Mobile/Fennec/Android/IDEs|run on your IDE]], you should be able to launch the emulator straight from there as well.
 
=== Multiple emulators ===


It's also possible to test with multiple emulators by correctly setting the ANDROID_SERIAL environment variable to the device ID seen in ''adb devices'':
It is possible to test with multiple emulators or devices using the --deviceSerial argument:
     $ adb devices
     $ adb devices
     List of devices attached
     List of devices attached
Line 360: Line 335:


To make Robocop (for example) run on ''emulator-5556'':
To make Robocop (for example) run on ''emulator-5556'':
     $ export ANDROID_SERIAL=emulator-5556
     $ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal
    $ mach robocop # Runs your tests as normal
 
== Device Managers ==
Most test suites - mochitests, reftests, xpcshell tests, and others - use a "device manager" module to communicate with the remote device. There are two device manager implementations: ADB and SUT.
 
The ADB device manager uses the adb command from the Android SDK to communicate with the remote device. To use the ADB device manager:
* ensure the adb command is in your shell's PATH
* mach test commands use adb by default; if running tests with make, set environment variable DM_TRANS=adb
 
The SUT device manager uses TCP to communicate with a remote agent, which must be installed on the device. To use the SUT device manager:
* ensure TCP connectivity between the local host and the remote device: check that they are on the same network and you can ping each from the other
* ensure the SUT agent is installed and started on the remote device
** the SUT agent APK is built alongside Firefox for Android; just install <objdir-droid>/build/mobile/sutagent/android/sutAgentAndroid.apk
** The agent should be configured to start automatically with your phone when it boots. To start it immediately from an adb shell execute: `am start -n com.mozilla.SUTAgentAndroid/.SUTAgentAndroid -a android.intent.action.MAIN`
* if running tests with make, set environment variable DM_TRANS=sut and set TEST_DEVICE=<ip address of device -- displayed by SUT agent>


== Host Builds (MOZ_HOST_BIN) ==
== Host Builds (MOZ_HOST_BIN) ==
Line 381: Line 341:
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.


'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you.
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.
'''
'''
=== Quick setup ===
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].
 
You can download prebuilt host utilities:
 
{| class="wikitable"
|-
! Host architecture !! Size !! Download link
|-
| Mac OS X || 76MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
|-
| Linux (64-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-x86_64.tar.xz
|-
| Linux (32-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-i686.tar.xz
|}
 
Extract the archives using tar (or tar and xzip), like:
 
  cd ~/.mozbuild
  wget http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
  tar Jxf host-utils-37.0a2.en-US.mac.tar.xz
  export MOZ_HOST_BIN=~/.mozbuild/host-utils.37.0a2.en-US.mac
 
Packaging the host utility archives listed above is easy: follow [[Packaging Android host utilities|these instructions]].


=== Advanced setup ===
=== Advanced setup ===
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.
  wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py
  python getxre.py -d $DIR
  export MOZ_HOST_BIN=$DIR/bin


Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:
Line 442: Line 374:


   --remoteTestRoot=<remote-directory>
   --remoteTestRoot=<remote-directory>
== talos ==
*NOTE: Talos will be retired and replaced by new tests on autophone before the end of Q2 2016.
*NOTE: this requires a fix for {{bug|1122701}}
*NOTE: this requires python 2.7
See also https://wiki.mozilla.org/Buildbot/Talos
  hg clone http://hg.mozilla.org/build/talos talos
  cd talos
  python INSTALL.py
  . bin/activate
  cd talos
  <obtain your fennec apk and put it in the current directory>
  <obtain your robocop apk and put it in the current directory>
  <obtain your fennec_ids.txt and put it in the current directory>
  <install your fennec apk>
  <if running a robocop-based test, install robocop.apk>
  <start sutagent on the device>
  python remotePerfConfigurator.py --apkPath=fennec-38.0a1.en-US.android-arm.apk
    -v -e org.mozilla.fennec
    --webServer=<your local desktop IP>:8080
    --noChrome
    --remoteDevice=<sutagent-IP>
    --sampleConfig=remote.config --output=local.yml --browserWait=60
    --activeTests=tcheck2
    --fennecIDs=fennec_ids.txt
  python run_tests.py --apkPath=fennec-38.0a1.en-US.android-arm.apk --noisy local.yml
For Robocop based tests (tcheck2, tprovider, etc...), we need to use the --fennecIDs flag to pass in the generated fennec_ids.txt file from the build you are testing.  This file is generated during build time and has to match the fennec.apk and robocop.apk file.  When this flag is used, we copy fennec_ids.txt and robotium.config (generated during configuration time) to the device and use those to run Robocop.  If you are running ts, tp4, tsvg, or other traditional talos tests, there is no need for the --fennecIDs flag.
Aside: For a quick-and-dirty hacky way to run robocop-talos tests locally, see [[Mobile/Fennec/Android/LocalRoboTalos]]
== S1/S2 Automation ==
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times.
S1/S2 graphs can be viewed at:  http://phonedash.mozilla.org/
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests
== Eideticker ==
Eideticker measures perceived Firefox performance by video capturing automated browser interactions.
Eideticker is no longer running Android tests. Historical data can be viewed at: http://eideticker.mozilla.org/
See also:
** http://wrla.ch/blog/2011/11/measuring-what-the-user-sees/
** http://wrla.ch/blog/2012/03/announcing-the-eideticker-mobile-performance-dashboard/
** https://github.com/mozilla/eideticker


== Trouble-shooting testing problems ==
== Trouble-shooting testing problems ==
Line 510: Line 390:
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
* Is your device rooted? Many test suites require root permissions. Don't want to root your device? Consider using an emulator with 'mach android-emulator'.
Gbrown
Confirmed users
1,759

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1134275...1207442"