MozillaWiki

Mobile/Fennec/Android/Testing: Difference between revisions

Page Discussion

Mobile/Fennec/Android/Testing (view source)

Revision as of 20:49, 8 February 2019

6,982 bytes removed ,  8 February 2019
→‎Advanced setup
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.
This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.


Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.
Having trouble? Ping :gbrown on #mobile.


== For the impatient... ==
== For the impatient... ==
Line 13: Line 13:
   mach xpcshell-test
   mach xpcshell-test
   mach cppunittest
   mach cppunittest
  mach geckoview-junit
  mach web-platform-test
  mach marionette-test
  mach test


They will all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.


=== Quick reference ===
=== Quick reference ===
Line 141: Line 145:


   mach robocop <test-name>
   mach robocop <test-name>
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.
  mach build mobile/android/tests/browser/robocop
  mach robocop
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)


Notes:
Notes:
* Mach is self documenting!  For help, try <tt>mach help robocop</tt>.
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.
* A rooted device is NOT required.
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]


=== Static analysis ===
=== Static analysis ===
Line 227: Line 223:


Notes:
Notes:
* A rooted device IS required.
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.


Line 283: Line 279:
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).


== browser-chrome ==
== Running tests on the Android emulator ==


Before you run tests, you will need to make sure you have packaged the tests in your object dir:
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 7.0 x86_64 opt/debug" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.


  make -C <objdir-droid> package-tests
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:


There is currently no special make command to build and run browser-chrome tests, but it should be possible to make them run by calling:
  ./mach android-emulator --version 4.3


  cd <objdir-droid>/_tests/testing/mochitest
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)
  python runtestsremote.py --dm_trans=adb --test-path=mobile --browser-chrome --deviceIP=1.2.3.4
                          --app=org.mozilla.fennec_$USER --xre-path=<objdir_x86>/dist/bin/


== Running tests on the Android emulator ==
To use the Android 7.0 x86_64 image:


The "Android 4.3 API15+ opt" and "Android 4.3 API15+ debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.
  ./mach android-emulator --version x86-7.0


Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is now quite simple:
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is '''much''' faster than the arm emulator.)
 
  ./mach android-emulator
 
By default, the 'android-emulator' command will download the Android 4.3 API15+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)
 
To use the Android 4.2 x86 image:
 
  ./mach android-emulator --version x86


The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.
Line 321: Line 307:
   ./mach android-emulator && ./mach install && ./mach mochitest
   ./mach android-emulator && ./mach install && ./mach mochitest


For the '''very''' lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:
For your convenience, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:


   $ ./mach mochitest testing/mochitest/tests/Harness_sanity
   $ ./mach mochitest testing/mochitest/tests/Harness_sanity
Line 340: Line 326:
   ...
   ...


=== Testing with stock AVD images (Android 4.x+ only) ===
=== Multiple emulators/devices ===


Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.
It is possible to test with multiple emulators or devices using the --deviceSerial argument:
 
# From the Android Virtual Device (AVD) Manager, click 'Create....'
# Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).
# Choose a system image with API level 15 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.
# Next, check 'Use Host GPU' if possible. See [http://developer.android.com/tools/devices/emulator.html#acceleration Using Hardware Acceleration].
# Hit Finish and that should be it. If you have setup Firefox for Android to [[Mobile/Fennec/Android/IDEs|run on your IDE]], you should be able to launch the emulator straight from there as well.
 
=== Multiple emulators ===
 
It's also possible to test with multiple emulators by correctly setting the ANDROID_SERIAL environment variable to the device ID seen in ''adb devices'':
     $ adb devices
     $ adb devices
     List of devices attached
     List of devices attached
Line 359: Line 335:


To make Robocop (for example) run on ''emulator-5556'':
To make Robocop (for example) run on ''emulator-5556'':
     $ export ANDROID_SERIAL=emulator-5556
     $ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal
    $ mach robocop # Runs your tests as normal
 
== Device Managers ==
Most test suites - mochitests, reftests, xpcshell tests, and others - use a "device manager" module to communicate with the remote device. There are two device manager implementations: ADB and SUT.
 
The ADB device manager uses the adb command from the Android SDK to communicate with the remote device. To use the ADB device manager:
* ensure the adb command is in your shell's PATH
* mach test commands use adb by default; if running tests with make, set environment variable DM_TRANS=adb
 
The SUT device manager uses TCP to communicate with a remote agent, which must be installed on the device. To use the SUT device manager:
* ensure TCP connectivity between the local host and the remote device: check that they are on the same network and you can ping each from the other
* ensure the SUT agent is installed and started on the remote device
** the SUT agent APK is built alongside Firefox for Android; just install <objdir-droid>/build/mobile/sutagent/android/sutAgentAndroid.apk
** The agent should be configured to start automatically with your phone when it boots. To start it immediately from an adb shell execute: `am start -n com.mozilla.SUTAgentAndroid/.SUTAgentAndroid -a android.intent.action.MAIN`
* if running tests with make, set environment variable DM_TRANS=sut and set TEST_DEVICE=<ip address of device -- displayed by SUT agent>


== Host Builds (MOZ_HOST_BIN) ==
== Host Builds (MOZ_HOST_BIN) ==
Line 380: Line 341:
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.


'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you.
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.
'''
'''
=== Quick setup ===
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].
 
You can download prebuilt host utilities:
 
{| class="wikitable"
|-
! Host architecture !! Size !! Download link
|-
| Mac OS X || 76MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
|-
| Linux (64-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-x86_64.tar.xz
|-
| Linux (32-bit) || 46MB || http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.linux-i686.tar.xz
|}
 
Extract the archives using tar (or tar and xzip), like:
 
  cd ~/.mozbuild
  wget http://people.mozilla.org/~nalexander/host-utils/host-utils-37.0a2.en-US.mac.tar.xz
  tar Jxf host-utils-37.0a2.en-US.mac.tar.xz
  export MOZ_HOST_BIN=~/.mozbuild/host-utils.37.0a2.en-US.mac
 
Packaging the host utility archives listed above is easy: follow [[Packaging Android host utilities|these instructions]].


=== Advanced setup ===
=== Advanced setup ===
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.
  wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py
  python getxre.py -d $DIR
  export MOZ_HOST_BIN=$DIR/bin


Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:
Line 441: Line 374:


   --remoteTestRoot=<remote-directory>
   --remoteTestRoot=<remote-directory>
== talos ==
*NOTE: Talos will be retired and replaced by new tests on autophone before the end of Q2 2016.
*NOTE: this requires a fix for {{bug|1122701}}
*NOTE: this requires python 2.7
See also https://wiki.mozilla.org/Buildbot/Talos
  hg clone http://hg.mozilla.org/build/talos talos
  cd talos
  python INSTALL.py
  . bin/activate
  cd talos
  <obtain your fennec apk and put it in the current directory>
  <obtain your robocop apk and put it in the current directory>
  <obtain your fennec_ids.txt and put it in the current directory>
  <install your fennec apk>
  <if running a robocop-based test, install robocop.apk>
  <start sutagent on the device>
  python remotePerfConfigurator.py --apkPath=fennec-38.0a1.en-US.android-arm.apk
    -v -e org.mozilla.fennec
    --webServer=<your local desktop IP>:8080
    --noChrome
    --remoteDevice=<sutagent-IP>
    --sampleConfig=remote.config --output=local.yml --browserWait=60
    --activeTests=tcheck2
    --fennecIDs=fennec_ids.txt
  python run_tests.py --apkPath=fennec-38.0a1.en-US.android-arm.apk --noisy local.yml
For Robocop based tests (tcheck2, tprovider, etc...), we need to use the --fennecIDs flag to pass in the generated fennec_ids.txt file from the build you are testing.  This file is generated during build time and has to match the fennec.apk and robocop.apk file.  When this flag is used, we copy fennec_ids.txt and robotium.config (generated during configuration time) to the device and use those to run Robocop.  If you are running ts, tp4, tsvg, or other traditional talos tests, there is no need for the --fennecIDs flag.
Aside: For a quick-and-dirty hacky way to run robocop-talos tests locally, see [[Mobile/Fennec/Android/LocalRoboTalos]]
== S1/S2 Automation ==
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times.
S1/S2 graphs can be viewed at:  http://phonedash.mozilla.org/
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests
== Eideticker ==
Eideticker measures perceived Firefox performance by video capturing automated browser interactions.
Eideticker is no longer running Android tests. Historical data can be viewed at: http://eideticker.mozilla.org/
See also:
** http://wrla.ch/blog/2011/11/measuring-what-the-user-sees/
** http://wrla.ch/blog/2012/03/announcing-the-eideticker-mobile-performance-dashboard/
** https://github.com/mozilla/eideticker


== Trouble-shooting testing problems ==
== Trouble-shooting testing problems ==
Line 509: Line 390:
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.
* Is your device rooted? Many test suites require root permissions. Don't want to root your device? Consider using an emulator with 'mach android-emulator'.
Gbrown
Confirmed users
1,759

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1163653...1207442"