https://wiki.mozilla.org/api.php?action=feedcontributions&user=Gbrown&feedformat=atomMozillaWiki - User contributions [en]2024-03-29T14:03:40ZUser contributionsMediaWiki 1.27.4https://wiki.mozilla.org/index.php?title=Template:NextReleaseDate&diff=1249252Template:NextReleaseDate2023-12-19T15:07:09Z<p>Gbrown: </p>
<hr />
<div>2024-01-23</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/release/current&diff=1249251Template:Version/Gecko/release/current2023-12-19T15:05:52Z<p>Gbrown: </p>
<hr />
<div>121</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/beta/current&diff=1249250Template:Version/Gecko/beta/current2023-12-19T15:05:26Z<p>Gbrown: </p>
<hr />
<div>122</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/central/current&diff=1249249Template:Version/Gecko/central/current2023-12-19T15:05:08Z<p>Gbrown: </p>
<hr />
<div>123</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/release/next&diff=1249248Template:Version/Gecko/release/next2023-12-19T15:04:43Z<p>Gbrown: </p>
<hr />
<div>124</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:NextReleaseDate&diff=1245792Template:NextReleaseDate2023-03-13T16:38:30Z<p>Gbrown: </p>
<hr />
<div>2023-04-11</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/release/current&diff=1245791Template:Version/Gecko/release/current2023-03-13T16:37:08Z<p>Gbrown: </p>
<hr />
<div>111</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/beta/current&diff=1245790Template:Version/Gecko/beta/current2023-03-13T16:36:49Z<p>Gbrown: </p>
<hr />
<div>112</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/central/current&diff=1245789Template:Version/Gecko/central/current2023-03-13T16:36:22Z<p>Gbrown: </p>
<hr />
<div>113</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/release/next&diff=1245788Template:Version/Gecko/release/next2023-03-13T16:35:50Z<p>Gbrown: </p>
<hr />
<div>114</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Packaging_Android_host_utilities&diff=1242000Packaging Android host utilities2022-04-20T19:57:16Z<p>Gbrown: /* Packaging Android host utilities */</p>
<hr />
<div>== Packaging Android host utilities ==<br />
<br />
Host utilities are executable files that run on a *host* machine. These utilities provide services to an Android *target* device, such as running a web server or a certificate authority. Host utilities must be periodically updated with a good build from mozilla-central or autoland due to addition of new tests targeted at Android and/or changing of component behavior.<br />
<br />
This page will provide instruction on how to package new versions of the host utilities.<br />
<br />
=== Generate new archive ===<br />
<br />
On Treeherder, identify a target build, preferably from mozilla-central or autoland. Check the test output to ensure a reasonably good build (one without too many failures).<br />
<br />
example: for the version 66.0 update, [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c9fa642e9e3aecaa8852f5c647904eecddaa4450&selectedJob=216676591 this build] was selected. For more details, please see [https://bugzilla.mozilla.org/show_bug.cgi?id=1514075 bug 1514075].<br />
<br />
==== Linux ====<br />
<br />
For Linux, the steps are fairly straightforward. Follow these steps to locate and download the necessary files.<br />
<br />
# From the selected treeherder build, identify '''Linux x64 opt'''.<br />
# Click on the green B icon, which will bring up a pane covering bottom third of the window.<br />
# On the bottom left pane is a header named '''Task'''. Click on the hash.<br />
# A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".<br />
# Download target.common.tests.tar.gz.<br />
# Download target.tar.bz2.<br />
# Follow the contents of the script below:<br />
<br />
<pre><br />
tar xvf target.tar.bz2<br />
tar xvf target.common.tests.tar.gz -C 'temp_common'<br />
rm firefox/firefox*<br />
rm -r firefox/browser<br />
mv 'temp_common'/bin/* firefox<br />
mv firefox host-utils-66.0a1.en-US.linux-x86_64<br />
tar cvf host-utils-66.0a1.en-US.linux-x86_64.tar host-utils-66.0a1.en-US.linux-x86_64<br />
gzip host-utils-66.0a1.en-US.linux-x86_64.tar<br />
</pre><br />
<br />
Repeat for Linux 32bit, substituting x86 for x86_64 where necessary.<br />
<br />
==== macOS ====<br />
<br />
Preparing for macOS is not quite so simple. Due to requirements of macOS, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If binaries are unsigned, users will be prompted at every invocation to "allow network connections". For more information on signing macOS binaries, see [http://apple.stackexchange.com/a/150711 here].<br />
<br />
# Navigate to [https://ftp.mozilla.org/pub/firefox/nightly/ Mozilla FTP].<br />
# Drill down to the appropriate year, month. Select a date that is not l10n.<br />
# Ensure files in the directory are prefixed with the version of Firefox that is desired.<br />
# Download firefox-66.0a1.en-US.mac.common.tests.tar.gz.<br />
# Download firefox-66.0a1.en-US.mac.dmg.<br />
# Follow the contents of the script below:<br />
<br />
<pre><br />
tar xvf firefox-66.0a1.en-US.mac.common.tests.tar.gz 'bin/*'<br />
open firefox-66.0a1.en-US.mac.dmg<br />
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/MacOS/* bin<br />
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/Resources/* bin<br />
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -<br />
mv bin host-utils-66.0a1.en-US.mac<br />
tar cvf host-utils-66.0a1.en-US.mac.tar host-utils-66.0a1.en-US.mac/*<br />
gzip host-utils-66.0a1.en-US.mac.tar<br />
</pre><br />
<br />
==== Windows (experimental) ====<br />
Similar to Linux. Follow these steps to locate and download the necessary files on Linux (or mozilla-build on Windows)<br />
<br />
# From the selected treeherder build, identify '''Windows 2012 opt'''.<br />
# Click on the green B icon, which will bring up a pane covering bottom third of the window.<br />
# On the bottom left pane is a header named '''Task'''. Click on the hash.<br />
# A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".<br />
# Download target.common.tests.tar.gz.<br />
# Download target.zip<br />
# Follow the contents of the script below:<br />
<br />
<pre><br />
unzip target.zip<br />
mkdir temp_common<br />
tar xvf target.common.tests.tar.gz -C 'temp_common'<br />
rm firefox/firefox*<br />
rm -r firefox/browser<br />
mv 'temp_common'/bin/* firefox<br />
mv firefox host-utils-66.0a1.en-US.win32<br />
tar cvf host-utils-66.0a1.en-US.win32.tar host-utils-66.0a1.en-US.win32<br />
gzip host-utils-66.0a1.en-US.win32.tar<br />
</pre><br />
<br />
==== Uploading to ToolTool ====<br />
<br />
# Compare contents of current archive to the new archive. For instructions on existing archive, see the section below.<br />
# Ensure uploading user has a valid token at [https://mozilla-releng.net/tokens/ Mozilla Releng].<br />
# Add file to temporary manifest<br />
<pre><br />
python tooltool.py add --unpack --visibility public [file]<br />
</pre><br />
# Upload the archive:<br />
<pre><br />
python tooltool.py upload --authentication-file=[token_location] --message [commit_message]<br />
</pre><br />
# Update the manifest in testing/config/tooltool-manifests/macosx64/hostutils.manifest.<br />
<br />
If updating host utilities for Linux, repeat using 32bit/x86 archives.<br />
<br />
Do the same for Windows. testing/config/tooltool-manifests/win32/hostutils.manifest<br />
<br />
=== Verification ===<br />
<br />
ToolTool will return an updated manifest once the archives are uploaded and processed. This new manifest must first be tested, follow the steps below:<br />
<br />
==== Linux ====<br />
<br />
Changes to host utilities for Linux can be tested on the try server.<br />
<br />
# Update to current tip by hg pull and hg update -r tip -C<br />
# Copy and paste the updated manifest to the hostutils.manifest file for Linux 64 and Linux 32.<br />
# Commit change and run moz-phab.<br />
# Initiate a try run of all Android tests:<br />
<br />
<pre><br />
# all (use this by default)<br />
./mach try fuzzy --no-artifact -q='test-android-'<br />
<br />
# only hardware<br />
# ./mach try fuzzy --no-artifact -q=test-android-hw-<br />
# only emulator<br />
# ./mach try fuzzy --no-artifact -q=test-android-em-<br />
</pre><br />
<br />
==== macOS ====<br />
<br />
Changes to host utilities for macOS cannot be tested on try server, therefore the following steps are taken:<br />
<br />
# Update to current tip by hg pull and hg update -r tip -C<br />
# Copy and paste the updated manifest to the hostutils.manifest file for Mac.<br />
# Commit change and run moz-phab.<br />
# Ensure the currently selected MOZCONFIG designates an Android build target. <br />
# Sanitize and rebuild with the following:<br />
<pre><br />
./mach clobber<br />
./mach build<br />
</pre><br />
# Once build is complete, package the apk using ./mach package<br />
# Install on device with ./mach install<br />
# Run a short test with ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
# Accept offer to update host utilities.<br />
<br />
If tests complete without issues, the generated host utilities are deemed to be good.<br />
<br />
== Download existing archive ==<br />
<br />
It is possible to download existing host utilities.<br />
<br />
# Locate the existing manifest file using [https://searchfox.org/mozilla-central/search?q=hostutils.manifest&path=SearchFox].<br />
# Using ToolTool, download the host utilities:<br />
<pre><br />
python tooltool.py fetch -m old_hostutils_manifest.tt<br />
</pre></div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:NextReleaseDate&diff=1238261Template:NextReleaseDate2021-10-04T20:32:05Z<p>Gbrown: </p>
<hr />
<div>2021-11-02</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/release/current&diff=1238260Template:Version/Gecko/release/current2021-10-04T20:30:43Z<p>Gbrown: </p>
<hr />
<div>93.0</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/beta/current&diff=1238259Template:Version/Gecko/beta/current2021-10-04T20:30:08Z<p>Gbrown: </p>
<hr />
<div>94</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/central/current&diff=1238258Template:Version/Gecko/central/current2021-10-04T20:29:08Z<p>Gbrown: </p>
<hr />
<div>95</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:Version/Gecko/release/next&diff=1238257Template:Version/Gecko/release/next2021-10-04T20:28:46Z<p>Gbrown: </p>
<hr />
<div>94</div>Gbrownhttps://wiki.mozilla.org/index.php?title=ReleaseEngineering/TryServer&diff=1224683ReleaseEngineering/TryServer2020-03-05T22:21:54Z<p>Gbrown: /* Re-running tasks with custom parameters from Treeherder */</p>
<hr />
<div>= Try Server =<br />
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.<br />
<br />
== Getting access to the Try Server ==<br />
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]<br />
<br />
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/auth.html this guide] for instructions.<br />
<br />
== Configuration ==<br />
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:<br />
<br />
$ ./mach vcs-setup<br />
<br />
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:<br />
<br />
[paths]<br />
try = ssh://hg.mozilla.org/try<br />
<br />
Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.<br />
<br />
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].<br />
<br />
== How to push to try ==<br />
There are a few ways to schedule jobs on try.<br />
<br />
=== Attach new jobs from the review ===<br />
<br />
For every patch submitted for review in Phabricator, a new Try run is automatically created. This run is created for static analysis, linting and other tasks. <br />
As described below, attaching new jobs to the run is easy and doesn't require more actions from the developer.<br />
Just click on the ''Treeherder Jobs''.<br />
<br />
[[File:Screenshot 2019-10-07 ⚙ D48316 Bug 1585593 - Enable Fission for Raptor.png]]<br />
<br />
For more details, see:<br />
https://firefox-source-docs.mozilla.org/tools/try/index.html#attaching-new-jobs-from-a-review<br />
<br />
=== Scheduling jobs with Mach Try ===<br />
<br />
The recommended way to [https://firefox-source-docs.mozilla.org/tools/try/index.html interact with try] is via the command line tool |mach try|. Since try can be used for a wide range of use cases, by developers with varying levels of experience, there is no one size fits all solution when it comes to selecting which tasks to run. Therefore |mach try| offers a range of subcommands called "selectors" for a variety of purposes. To see a list of available selectors, run:<br />
<br />
$ mach try --help<br />
<br />
For deeper information on a given selector, run:<br />
<br />
$ mach try <selector> --help<br />
<br />
If you don't specify a selector/subcommand to try, it will default to using the `syntax` selector. You can change which selector gets used by default by creating ~/.mozbuild/machrc and adding:<br />
<br />
[try]<br />
default=<selector><br />
<br />
See [https://firefox-source-docs.mozilla.org/tools/try/selectors/index.html this documentation] for more information on all the available selectors, and [https://firefox-source-docs.mozilla.org/tools/try/presets.html this document] for a list of available presets (derived from [https://searchfox.org/mozilla-central/source/tools/tryselect/try_presets.yml try_presets.yml]).<br />
<br />
<br />
=== Scheduling jobs with Treeherder ===<br />
<br />
You can push to try with:<br />
<br />
$ ./mach try empty<br />
<br />
Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.<br />
<br />
Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png<br />
<br />
If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed. Click the job symbols you wish to schedule. <br />
<br />
If you select a test job, the required build will automatically be scheduled:<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png<br />
<br />
Finally, click "Trigger New Jobs" near the top right of your push.<br />
<br />
NOTE: An action task gets scheduled which will schedule all the tasks you chose.<br />
<br />
=== Scheduling jobs with Treeherder (Search) ===<br />
<br />
Similar to the above method, there is now an additional way to add jobs to an existing push via Treeherder. Open the push in Treeherder and then choose "Add New Jobs (Search)" from the push's menu:<br />
<br />
[[File:Add New Jobs Search.png]]<br />
<br />
Treeherder will then fetch the list of runnable jobs in the background, then pop up a dialog window listing those jobs:<br />
<br />
[[File:Addnewjobssearchinterface.png]]<br />
<br />
At the top of the dialog, there is a search bar. Type part of your desired job name into the search bar and press 'Enter' to filter the list of jobs. (This uses a fuzzy search library, so some minimal typos are allowed). From the filtered list of jobs, select as many of the results as you want (ctrl/shift modifiers work on this list for selecting multiple jobs):<br />
<br />
[[File:Filteredaddnewjobssearch.png]]<br />
<br />
Then click the green "Add Selected" button to add the selected jobs to the list of jobs to be run (or click the green "Add All" button to add the entire list of jobs currently in view.<br />
<br />
Repeat the filter and selection process until you have all of your desired jobs added to the list. You can remove jobs from the list to be run using the red buttons lower on the dialog if you selected them in error:<br />
<br />
[[File:Addnewjobssearchselectedjobs.png]]<br />
<br />
When you are happy with your selection, click the blue "Trigger (X) Selected Jobs" button at the bottom of the dialog. Treeherder will send out a request for those jobs to be run, and an action task will be spawned to do all of the behind the scenes magic to make that happen.<br />
<br />
=== Pushing Directly ===<br />
<br />
It's also possible to push to try directly from hg (for now). This might be required if you're using [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. In this case, the only way to select tasks is by manually adding try syntax to your commit message:<br />
<br />
$ hg commit -m "try: -b o -p linux64 -u mochitest"<br />
$ hg push -f try<br />
<br />
Using |mach try| is recommended as that will automatically create and cleanup a temporary commit no your behalf, whereas with this method you are responsible for cleaning up any extra commits you make or commit messages you modify.<br />
<br />
== Viewing the results ==<br />
You can see the results of your tryserver build in a number of ways:<br />
<br />
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results<br />
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax<br />
* The link to treeherder will be printed on the command line.<br />
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.<br />
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].<br />
<br />
== Using a custom mozconfig ==<br />
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.<br />
<br />
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.<br />
<br />
Android mozconfigs are in mobile/android/config/mozconfigs.<br />
<br />
If you modify a mozconfig file named <tt>nightly</tt>, please make sure you also change any mozconfig file named <tt>beta</tt> and/or <tt>release</tt> in the same directory, otherwise the test_compare_mozconfigs test will fail.<br />
<br />
Note:<br />
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.<br />
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.<br />
<br />
== Re-running tasks with custom parameters from Treeherder ==<br />
<br />
It is possible to run a task again, with certain types of changes to its run time environment and parameters. This feature is commonly used to enable debug logging, or run a test repeatedly to reproduce test failures.<br />
<br />
In treeherder, select the job you want to re-run (Note: it currently only works for [https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/actions/retrigger_mochitest.py#30-31 some jobs]). The details panel should open. Look for the 3 dots icon ("Other job details") and select "Custom actions...".<br />
<br />
[[File:Custom-action.png|thumb|Selecting "Custom Action..." in treeherder.]]<br />
<br />
The <tt>retrigger-custom</tt> action is the one selected in the drop-down menu. Modify the task parameters as desired and select "Trigger". <br />
<br />
[[File:Custom Taskcluster Job Actions.png|thumb|Treeherder modal to request an action associated to a task]]<br />
<br />
This will schedule the task with your specified parameters. The new task will be labeled as <tt>custom-<original_task_name></tt>.<br />
<br />
Some of the parameters that you can change are (see screenshot above):<br />
<br />
* Environment variables (e.g. <tt>MOZ_LOG</tt>)<br />
* Modify Python's logging level<br />
* Path of the test to execute<br />
* Gecko preferences (think of <tt>about:config</tt>)<br />
* Run a test repeatedly<br />
* Run until the test fails<br />
<br />
Common problems:<br />
* By default the entire task is re-run, running all the tests in the original task. If repeat/runUntilFailure is specified, it may take a long time to repeat all the tests, and the log file may be huge. Consider using the path: parameter to limit the run to the test(s) of interest.<br />
<br />
== Getting debug symbols ==<br />
<br />
NOTE: You might be able to accomplish this by re-running a build task from Treeherder by adding the env variable to the new task.<br />
<br />
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.<br />
<br />
== Adding new jobs ==<br />
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].<br />
<br />
== Desktop l10n jobs ==<br />
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.<br />
<br />
== Desktop l10n jobs (on Taskcluster) ==<br />
<br />
To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Android Single-Locale l10n jobs (on Taskcluster) ==<br />
<br />
(NOTE: Only supported on Gecko 51a1 and above)<br />
<br />
To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Server Status ==<br />
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending<br />
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running<br />
<br />
== Other Notes ==<br />
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion<br />
* Treeherder data is purged after 4 months.<br />
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]<br />
<br />
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}<br />
<br />
== Other Mozilla Try Servers ==<br />
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository<br />
<br />
== Problem Diagnosis ==<br />
=== Can not access try server ===<br />
Test your account & configuration<br />
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"<br />
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.<br />
<br />
<div id="long_try_push"></div><br />
=== Pushes to try take a very long time ===<br />
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)<br />
<br />
If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.<br />
<br />
=== Waiting for Lock ===<br />
If you get a message similar to:<br />
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'<br />
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549<br />
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.<br />
<br />
=== Waiting for Lock multiple times with the same pid ===<br />
Similar to the above case, but with the same pid when you retry over and over again.<br />
<br />
Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].<br />
<br />
== CiDuty issues == <br />
=== How do I trigger additional talos/test runs for a given try build? ===<br />
If your syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.<br />
<br />
=== How do I cancel existing jobs? ===<br />
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.<br />
<br />
== See Also ==<br />
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories<br />
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices<br />
* https://wiki.mozilla.org/ReleaseEngineering:Autoland<br />
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]<br />
* [https://catlee.github.io/highscores/highscores.html Scoreboard]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=ReleaseEngineering/TryServer&diff=1224428ReleaseEngineering/TryServer2020-03-02T15:29:02Z<p>Gbrown: /* Re-running tasks with custom parameters from Treeherder */</p>
<hr />
<div>= Try Server =<br />
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.<br />
<br />
== Getting access to the Try Server ==<br />
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]<br />
<br />
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/auth.html this guide] for instructions.<br />
<br />
== Configuration ==<br />
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:<br />
<br />
$ ./mach vcs-setup<br />
<br />
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:<br />
<br />
[paths]<br />
try = ssh://hg.mozilla.org/try<br />
<br />
Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.<br />
<br />
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].<br />
<br />
== How to push to try ==<br />
There are a few ways to schedule jobs on try.<br />
<br />
=== Attach new jobs from the review ===<br />
<br />
For every patch submitted for review in Phabricator, a new Try run is automatically created. This run is created for static analysis, linting and other tasks. <br />
As described below, attaching new jobs to the run is easy and doesn't require more actions from the developer.<br />
Just click on the ''Treeherder Jobs''.<br />
<br />
[[File:Screenshot 2019-10-07 ⚙ D48316 Bug 1585593 - Enable Fission for Raptor.png]]<br />
<br />
For more details, see:<br />
https://firefox-source-docs.mozilla.org/tools/try/index.html#attaching-new-jobs-from-a-review<br />
<br />
=== Scheduling jobs with Mach Try ===<br />
<br />
The recommended way to [https://firefox-source-docs.mozilla.org/tools/try/index.html interact with try] is via the command line tool |mach try|. Since try can be used for a wide range of use cases, by developers with varying levels of experience, there is no one size fits all solution when it comes to selecting which tasks to run. Therefore |mach try| offers a range of subcommands called "selectors" for a variety of purposes. To see a list of available selectors, run:<br />
<br />
$ mach try --help<br />
<br />
For deeper information on a given selector, run:<br />
<br />
$ mach try <selector> --help<br />
<br />
If you don't specify a selector/subcommand to try, it will default to using the `syntax` selector. You can change which selector gets used by default by creating ~/.mozbuild/machrc and adding:<br />
<br />
[try]<br />
default=<selector><br />
<br />
See [https://firefox-source-docs.mozilla.org/tools/try/selectors/index.html this documentation] for more information on all the available selectors, and [https://firefox-source-docs.mozilla.org/tools/try/presets.html this document] for a list of available presets (derived from [https://searchfox.org/mozilla-central/source/tools/tryselect/try_presets.yml try_presets.yml]).<br />
<br />
<br />
=== Scheduling jobs with Treeherder ===<br />
<br />
You can push to try with:<br />
<br />
$ ./mach try empty<br />
<br />
Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.<br />
<br />
Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png<br />
<br />
If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed. Click the job symbols you wish to schedule. <br />
<br />
If you select a test job, the required build will automatically be scheduled:<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png<br />
<br />
Finally, click "Trigger New Jobs" near the top right of your push.<br />
<br />
NOTE: An action task gets scheduled which will schedule all the tasks you chose.<br />
<br />
=== Scheduling jobs with Treeherder (Search) ===<br />
<br />
Similar to the above method, there is now an additional way to add jobs to an existing push via Treeherder. Open the push in Treeherder and then choose "Add New Jobs (Search)" from the push's menu:<br />
<br />
[[File:Add New Jobs Search.png]]<br />
<br />
Treeherder will then fetch the list of runnable jobs in the background, then pop up a dialog window listing those jobs:<br />
<br />
[[File:Addnewjobssearchinterface.png]]<br />
<br />
At the top of the dialog, there is a search bar. Type part of your desired job name into the search bar and press 'Enter' to filter the list of jobs. (This uses a fuzzy search library, so some minimal typos are allowed). From the filtered list of jobs, select as many of the results as you want (ctrl/shift modifiers work on this list for selecting multiple jobs):<br />
<br />
[[File:Filteredaddnewjobssearch.png]]<br />
<br />
Then click the green "Add Selected" button to add the selected jobs to the list of jobs to be run (or click the green "Add All" button to add the entire list of jobs currently in view.<br />
<br />
Repeat the filter and selection process until you have all of your desired jobs added to the list. You can remove jobs from the list to be run using the red buttons lower on the dialog if you selected them in error:<br />
<br />
[[File:Addnewjobssearchselectedjobs.png]]<br />
<br />
When you are happy with your selection, click the blue "Trigger (X) Selected Jobs" button at the bottom of the dialog. Treeherder will send out a request for those jobs to be run, and an action task will be spawned to do all of the behind the scenes magic to make that happen.<br />
<br />
=== Pushing Directly ===<br />
<br />
It's also possible to push to try directly from hg (for now). This might be required if you're using [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. In this case, the only way to select tasks is by manually adding try syntax to your commit message:<br />
<br />
$ hg commit -m "try: -b o -p linux64 -u mochitest"<br />
$ hg push -f try<br />
<br />
Using |mach try| is recommended as that will automatically create and cleanup a temporary commit no your behalf, whereas with this method you are responsible for cleaning up any extra commits you make or commit messages you modify.<br />
<br />
== Viewing the results ==<br />
You can see the results of your tryserver build in a number of ways:<br />
<br />
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results<br />
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax<br />
* The link to treeherder will be printed on the command line.<br />
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.<br />
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].<br />
<br />
== Using a custom mozconfig ==<br />
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.<br />
<br />
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.<br />
<br />
Android mozconfigs are in mobile/android/config/mozconfigs.<br />
<br />
If you modify a mozconfig file named <tt>nightly</tt>, please make sure you also change any mozconfig file named <tt>beta</tt> and/or <tt>release</tt> in the same directory, otherwise the test_compare_mozconfigs test will fail.<br />
<br />
Note:<br />
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.<br />
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.<br />
<br />
== Re-running tasks with custom parameters from Treeherder ==<br />
<br />
It is possible to run a task again, with certain types of changes to its run time environment and parameters. This feature is commonly used to enable debug logging, or run a test repeatedly to reproduce test failures.<br />
<br />
In treeherder, select the job you want to re-run (Note: it currently only works for [https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/actions/retrigger_mochitest.py#30-31 some jobs]). The details panel should open. Look for the 3 dots icon ("Other job details") and select "Custom actions...".<br />
<br />
[[File:Custom-action.png|thumb|Selecting "Custom Action..." in treeherder.]]<br />
<br />
The <tt>retrigger-mochitest</tt> action (TODO: A better name is needed) is the one selected in the drop-down menu. Modify the task parameters as desired and select "Trigger". <br />
<br />
[[File:Custom Taskcluster Job Actions.png|thumb|Treeherder modal to request an action associated to a task]]<br />
<br />
This will schedule the task with your specified parameters. The new task will be labeled as <tt>custom-<original_task_name></tt>.<br />
<br />
Some of the parameters that you can change are (see screenshot above):<br />
<br />
* Environment variables (e.g. <tt>MOZ_LOG</tt>)<br />
* Modify Python's logging level<br />
* Path of the test to execute<br />
* Gecko preferences (think of <tt>about:config</tt>)<br />
* Run a test repeatedly<br />
* Run until the test fails<br />
<br />
Common problems:<br />
* By default the entire task is re-run, running all the tests in the original task. If repeat/runUntilFailure is specified, it may take a long time to repeat all the tests, and the log file may be huge. Consider using the path: parameter to limit the run to the test(s) of interest.<br />
<br />
== Getting debug symbols ==<br />
<br />
NOTE: You might be able to accomplish this by re-running a build task from Treeherder by adding the env variable to the new task.<br />
<br />
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.<br />
<br />
== Adding new jobs ==<br />
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].<br />
<br />
== Desktop l10n jobs ==<br />
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.<br />
<br />
== Desktop l10n jobs (on Taskcluster) ==<br />
<br />
To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Android Single-Locale l10n jobs (on Taskcluster) ==<br />
<br />
(NOTE: Only supported on Gecko 51a1 and above)<br />
<br />
To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Server Status ==<br />
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending<br />
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running<br />
<br />
== Other Notes ==<br />
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion<br />
* Treeherder data is purged after 4 months.<br />
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]<br />
<br />
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}<br />
<br />
== Other Mozilla Try Servers ==<br />
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository<br />
<br />
== Problem Diagnosis ==<br />
=== Can not access try server ===<br />
Test your account & configuration<br />
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"<br />
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.<br />
<br />
<div id="long_try_push"></div><br />
=== Pushes to try take a very long time ===<br />
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)<br />
<br />
If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.<br />
<br />
=== Waiting for Lock ===<br />
If you get a message similar to:<br />
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'<br />
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549<br />
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.<br />
<br />
=== Waiting for Lock multiple times with the same pid ===<br />
Similar to the above case, but with the same pid when you retry over and over again.<br />
<br />
Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].<br />
<br />
== CiDuty issues == <br />
=== How do I trigger additional talos/test runs for a given try build? ===<br />
If your syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.<br />
<br />
=== How do I cancel existing jobs? ===<br />
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.<br />
<br />
== See Also ==<br />
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories<br />
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices<br />
* https://wiki.mozilla.org/ReleaseEngineering:Autoland<br />
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]<br />
* [https://catlee.github.io/highscores/highscores.html Scoreboard]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=ReleaseEngineering/TryServer&diff=1223661ReleaseEngineering/TryServer2020-02-12T15:24:00Z<p>Gbrown: /* Re-running tasks with custom parameters from Treeherder */</p>
<hr />
<div>= Try Server =<br />
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.<br />
<br />
== Getting access to the Try Server ==<br />
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]<br />
<br />
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/auth.html this guide] for instructions.<br />
<br />
== Configuration ==<br />
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:<br />
<br />
$ ./mach vcs-setup<br />
<br />
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:<br />
<br />
[paths]<br />
try = ssh://hg.mozilla.org/try<br />
<br />
Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.<br />
<br />
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].<br />
<br />
== How to push to try ==<br />
There are a few ways to schedule jobs on try.<br />
<br />
=== Attach new jobs from the review ===<br />
<br />
For every patch submitted for review in Phabricator, a new Try run is automatically created. This run is created for static analysis, linting and other tasks. <br />
As described below, attaching new jobs to the run is easy and doesn't require more actions from the developer.<br />
Just click on the ''Treeherder Jobs''.<br />
<br />
[[File:Screenshot 2019-10-07 ⚙ D48316 Bug 1585593 - Enable Fission for Raptor.png]]<br />
<br />
For more details, see:<br />
https://firefox-source-docs.mozilla.org/tools/try/index.html#attaching-new-jobs-from-a-review<br />
<br />
=== Scheduling jobs with Mach Try ===<br />
<br />
The recommended way to [https://firefox-source-docs.mozilla.org/tools/try/index.html interact with try] is via the command line tool |mach try|. Since try can be used for a wide range of use cases, by developers with varying levels of experience, there is no one size fits all solution when it comes to selecting which tasks to run. Therefore |mach try| offers a range of subcommands called "selectors" for a variety of purposes. To see a list of available selectors, run:<br />
<br />
$ mach try --help<br />
<br />
For deeper information on a given selector, run:<br />
<br />
$ mach try <selector> --help<br />
<br />
If you don't specify a selector/subcommand to try, it will default to using the `syntax` selector. You can change which selector gets used by default by creating ~/.mozbuild/machrc and adding:<br />
<br />
[try]<br />
default=<selector><br />
<br />
See [https://firefox-source-docs.mozilla.org/tools/try/selectors/index.html this documentation] for more information on all the available selectors.<br />
<br />
<br />
=== Scheduling jobs with Treeherder ===<br />
<br />
You can push to try with:<br />
<br />
$ ./mach try empty<br />
<br />
Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.<br />
<br />
Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png<br />
<br />
If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed. Click the job symbols you wish to schedule. <br />
<br />
If you select a test job, the required build will automatically be scheduled:<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png<br />
<br />
Finally, click "Trigger New Jobs" near the top right of your push.<br />
<br />
NOTE: An action task gets scheduled which will schedule all the tasks you chose.<br />
<br />
=== Scheduling jobs with Treeherder (Search) ===<br />
<br />
Similar to the above method, there is now an additional way to add jobs to an existing push via Treeherder. Open the push in Treeherder and then choose "Add New Jobs (Search)" from the push's menu:<br />
<br />
[[File:Add New Jobs Search.png]]<br />
<br />
Treeherder will then fetch the list of runnable jobs in the background, then pop up a dialog window listing those jobs:<br />
<br />
[[File:Addnewjobssearchinterface.png]]<br />
<br />
At the top of the dialog, there is a search bar. Type part of your desired job name into the search bar and press 'Enter' to filter the list of jobs. (This uses a fuzzy search library, so some minimal typos are allowed). From the filtered list of jobs, select as many of the results as you want (ctrl/shift modifiers work on this list for selecting multiple jobs):<br />
<br />
[[File:Filteredaddnewjobssearch.png]]<br />
<br />
Then click the green "Add Selected" button to add the selected jobs to the list of jobs to be run (or click the green "Add All" button to add the entire list of jobs currently in view.<br />
<br />
Repeat the filter and selection process until you have all of your desired jobs added to the list. You can remove jobs from the list to be run using the red buttons lower on the dialog if you selected them in error:<br />
<br />
[[File:Addnewjobssearchselectedjobs.png]]<br />
<br />
When you are happy with your selection, click the blue "Trigger (X) Selected Jobs" button at the bottom of the dialog. Treeherder will send out a request for those jobs to be run, and an action task will be spawned to do all of the behind the scenes magic to make that happen.<br />
<br />
=== Pushing Directly ===<br />
<br />
It's also possible to push to try directly from hg (for now). This might be required if you're using [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. In this case, the only way to select tasks is by manually adding try syntax to your commit message:<br />
<br />
$ hg commit -m "try: -b o -p linux64 -u mochitest"<br />
$ hg push -f try<br />
<br />
Using |mach try| is recommended as that will automatically create and cleanup a temporary commit no your behalf, whereas with this method you are responsible for cleaning up any extra commits you make or commit messages you modify.<br />
<br />
== Viewing the results ==<br />
You can see the results of your tryserver build in a number of ways:<br />
<br />
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results<br />
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax<br />
* The link to treeherder will be printed on the command line.<br />
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.<br />
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].<br />
<br />
== Using a custom mozconfig ==<br />
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.<br />
<br />
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.<br />
<br />
Android mozconfigs are in mobile/android/config/mozconfigs.<br />
<br />
If you modify a mozconfig file named <tt>nightly</tt>, please make sure you also change any mozconfig file named <tt>beta</tt> and/or <tt>release</tt> in the same directory, otherwise the test_compare_mozconfigs test will fail.<br />
<br />
Note:<br />
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.<br />
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.<br />
<br />
== Re-running tasks with custom parameters from Treeherder ==<br />
<br />
In treeherder, select the job you want to re-run (Note: it currently only works for [https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/actions/retrigger_mochitest.py#30-31 some jobs]). The details panel should open. Look for the 3 dots icon ("Other job details") and select "Custom actions...".<br />
<br />
[[File:Custom-action.png|thumb|Selecting "Custom Action..." in treeherder.]]<br />
<br />
The <tt>retrigger-mochitest</tt> action (TODO: A better name is needed) is the one selected in the drop-down menu. Modify the task parameters as desired and select "Trigger". <br />
<br />
[[File:Custom Taskcluster Job Actions.png|thumb|Treeherder modal to request an action associated to a task]]<br />
<br />
This will schedule the task with your specified parameters. The new task will be labeled as <tt>custom-<original_task_name></tt>.<br />
<br />
Some of the parameters that you can change are (see screenshot above):<br />
<br />
* Environment variables (e.g. <tt>MOZ_LOG</tt>)<br />
* Modify Python's logging level<br />
* Path of the test to execute<br />
* Gecko preferences (think of <tt>about:config</tt>)<br />
* Run a test repeatedly<br />
* Run until the test fails<br />
<br />
== Getting debug symbols ==<br />
<br />
NOTE: You might be able to accomplish this by re-running a build task from Treeherder by adding the env variable to the new task.<br />
<br />
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.<br />
<br />
== Adding new jobs ==<br />
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].<br />
<br />
== Desktop l10n jobs ==<br />
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.<br />
<br />
== Desktop l10n jobs (on Taskcluster) ==<br />
<br />
To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Android Single-Locale l10n jobs (on Taskcluster) ==<br />
<br />
(NOTE: Only supported on Gecko 51a1 and above)<br />
<br />
To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Server Status ==<br />
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending<br />
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running<br />
<br />
== Other Notes ==<br />
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion<br />
* Treeherder data is purged after 4 months.<br />
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]<br />
<br />
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}<br />
<br />
== Other Mozilla Try Servers ==<br />
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository<br />
<br />
== Problem Diagnosis ==<br />
=== Can not access try server ===<br />
Test your account & configuration<br />
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"<br />
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.<br />
<br />
<div id="long_try_push"></div><br />
=== Pushes to try take a very long time ===<br />
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)<br />
<br />
If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.<br />
<br />
=== Waiting for Lock ===<br />
If you get a message similar to:<br />
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'<br />
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549<br />
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.<br />
<br />
=== Waiting for Lock multiple times with the same pid ===<br />
Similar to the above case, but with the same pid when you retry over and over again.<br />
<br />
Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].<br />
<br />
== CiDuty issues == <br />
=== How do I trigger additional talos/test runs for a given try build? ===<br />
If your syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.<br />
<br />
=== How do I cancel existing jobs? ===<br />
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.<br />
<br />
== See Also ==<br />
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories<br />
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices<br />
* https://wiki.mozilla.org/ReleaseEngineering:Autoland<br />
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]<br />
* [https://catlee.github.io/highscores/highscores.html Scoreboard]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=ReleaseEngineering/TryServer&diff=1223660ReleaseEngineering/TryServer2020-02-12T15:23:23Z<p>Gbrown: /* Re-running tasks with custom parameters from Treeherder */</p>
<hr />
<div>= Try Server =<br />
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.<br />
<br />
== Getting access to the Try Server ==<br />
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]<br />
<br />
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/auth.html this guide] for instructions.<br />
<br />
== Configuration ==<br />
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:<br />
<br />
$ ./mach vcs-setup<br />
<br />
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:<br />
<br />
[paths]<br />
try = ssh://hg.mozilla.org/try<br />
<br />
Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.<br />
<br />
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].<br />
<br />
== How to push to try ==<br />
There are a few ways to schedule jobs on try.<br />
<br />
=== Attach new jobs from the review ===<br />
<br />
For every patch submitted for review in Phabricator, a new Try run is automatically created. This run is created for static analysis, linting and other tasks. <br />
As described below, attaching new jobs to the run is easy and doesn't require more actions from the developer.<br />
Just click on the ''Treeherder Jobs''.<br />
<br />
[[File:Screenshot 2019-10-07 ⚙ D48316 Bug 1585593 - Enable Fission for Raptor.png]]<br />
<br />
For more details, see:<br />
https://firefox-source-docs.mozilla.org/tools/try/index.html#attaching-new-jobs-from-a-review<br />
<br />
=== Scheduling jobs with Mach Try ===<br />
<br />
The recommended way to [https://firefox-source-docs.mozilla.org/tools/try/index.html interact with try] is via the command line tool |mach try|. Since try can be used for a wide range of use cases, by developers with varying levels of experience, there is no one size fits all solution when it comes to selecting which tasks to run. Therefore |mach try| offers a range of subcommands called "selectors" for a variety of purposes. To see a list of available selectors, run:<br />
<br />
$ mach try --help<br />
<br />
For deeper information on a given selector, run:<br />
<br />
$ mach try <selector> --help<br />
<br />
If you don't specify a selector/subcommand to try, it will default to using the `syntax` selector. You can change which selector gets used by default by creating ~/.mozbuild/machrc and adding:<br />
<br />
[try]<br />
default=<selector><br />
<br />
See [https://firefox-source-docs.mozilla.org/tools/try/selectors/index.html this documentation] for more information on all the available selectors.<br />
<br />
<br />
=== Scheduling jobs with Treeherder ===<br />
<br />
You can push to try with:<br />
<br />
$ ./mach try empty<br />
<br />
Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.<br />
<br />
Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png<br />
<br />
If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed. Click the job symbols you wish to schedule. <br />
<br />
If you select a test job, the required build will automatically be scheduled:<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png<br />
<br />
Finally, click "Trigger New Jobs" near the top right of your push.<br />
<br />
NOTE: An action task gets scheduled which will schedule all the tasks you chose.<br />
<br />
=== Scheduling jobs with Treeherder (Search) ===<br />
<br />
Similar to the above method, there is now an additional way to add jobs to an existing push via Treeherder. Open the push in Treeherder and then choose "Add New Jobs (Search)" from the push's menu:<br />
<br />
[[File:Add New Jobs Search.png]]<br />
<br />
Treeherder will then fetch the list of runnable jobs in the background, then pop up a dialog window listing those jobs:<br />
<br />
[[File:Addnewjobssearchinterface.png]]<br />
<br />
At the top of the dialog, there is a search bar. Type part of your desired job name into the search bar and press 'Enter' to filter the list of jobs. (This uses a fuzzy search library, so some minimal typos are allowed). From the filtered list of jobs, select as many of the results as you want (ctrl/shift modifiers work on this list for selecting multiple jobs):<br />
<br />
[[File:Filteredaddnewjobssearch.png]]<br />
<br />
Then click the green "Add Selected" button to add the selected jobs to the list of jobs to be run (or click the green "Add All" button to add the entire list of jobs currently in view.<br />
<br />
Repeat the filter and selection process until you have all of your desired jobs added to the list. You can remove jobs from the list to be run using the red buttons lower on the dialog if you selected them in error:<br />
<br />
[[File:Addnewjobssearchselectedjobs.png]]<br />
<br />
When you are happy with your selection, click the blue "Trigger (X) Selected Jobs" button at the bottom of the dialog. Treeherder will send out a request for those jobs to be run, and an action task will be spawned to do all of the behind the scenes magic to make that happen.<br />
<br />
=== Pushing Directly ===<br />
<br />
It's also possible to push to try directly from hg (for now). This might be required if you're using [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. In this case, the only way to select tasks is by manually adding try syntax to your commit message:<br />
<br />
$ hg commit -m "try: -b o -p linux64 -u mochitest"<br />
$ hg push -f try<br />
<br />
Using |mach try| is recommended as that will automatically create and cleanup a temporary commit no your behalf, whereas with this method you are responsible for cleaning up any extra commits you make or commit messages you modify.<br />
<br />
== Viewing the results ==<br />
You can see the results of your tryserver build in a number of ways:<br />
<br />
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results<br />
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax<br />
* The link to treeherder will be printed on the command line.<br />
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.<br />
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].<br />
<br />
== Using a custom mozconfig ==<br />
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.<br />
<br />
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.<br />
<br />
Android mozconfigs are in mobile/android/config/mozconfigs.<br />
<br />
If you modify a mozconfig file named <tt>nightly</tt>, please make sure you also change any mozconfig file named <tt>beta</tt> and/or <tt>release</tt> in the same directory, otherwise the test_compare_mozconfigs test will fail.<br />
<br />
Note:<br />
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.<br />
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.<br />
<br />
== Re-running tasks with custom parameters from Treeherder ==<br />
<br />
In treeherder, select the job you want to re-run (Note: it currently only works for [https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/actions/retrigger_mochitest.py#30-31 some jobs]). The details panel should open. Look for the 3 dots icon ("Other job details") and select "Custom actions...".<br />
<br />
[[File:Custom-action.png|frame|Selecting "Custom Action..." in treeherder.]]<br />
<br />
The <tt>retrigger-mochitest</tt> action (TODO: A better name is needed) is the one selected in the drop-down menu. Modify the task parameters as desired and select "Trigger". <br />
<br />
[[File:Custom Taskcluster Job Actions.png|thumb|Treeherder modal to request an action associated to a task]]<br />
<br />
This will schedule the task with your specified parameters. The new task will be labeled as <tt>custom-<original_task_name></tt>.<br />
<br />
Some of the parameters that you can change are (see screenshot above):<br />
<br />
* Environment variables (e.g. <tt>MOZ_LOG</tt>)<br />
* Modify Python's logging level<br />
* Path of the test to execute<br />
* Gecko preferences (think of <tt>about:config</tt>)<br />
* Run a test repeatedly<br />
* Run until the test fails<br />
<br />
== Getting debug symbols ==<br />
<br />
NOTE: You might be able to accomplish this by re-running a build task from Treeherder by adding the env variable to the new task.<br />
<br />
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.<br />
<br />
== Adding new jobs ==<br />
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].<br />
<br />
== Desktop l10n jobs ==<br />
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.<br />
<br />
== Desktop l10n jobs (on Taskcluster) ==<br />
<br />
To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Android Single-Locale l10n jobs (on Taskcluster) ==<br />
<br />
(NOTE: Only supported on Gecko 51a1 and above)<br />
<br />
To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Server Status ==<br />
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending<br />
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running<br />
<br />
== Other Notes ==<br />
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion<br />
* Treeherder data is purged after 4 months.<br />
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]<br />
<br />
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}<br />
<br />
== Other Mozilla Try Servers ==<br />
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository<br />
<br />
== Problem Diagnosis ==<br />
=== Can not access try server ===<br />
Test your account & configuration<br />
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"<br />
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.<br />
<br />
<div id="long_try_push"></div><br />
=== Pushes to try take a very long time ===<br />
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)<br />
<br />
If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.<br />
<br />
=== Waiting for Lock ===<br />
If you get a message similar to:<br />
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'<br />
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549<br />
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.<br />
<br />
=== Waiting for Lock multiple times with the same pid ===<br />
Similar to the above case, but with the same pid when you retry over and over again.<br />
<br />
Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].<br />
<br />
== CiDuty issues == <br />
=== How do I trigger additional talos/test runs for a given try build? ===<br />
If your syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.<br />
<br />
=== How do I cancel existing jobs? ===<br />
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.<br />
<br />
== See Also ==<br />
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories<br />
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices<br />
* https://wiki.mozilla.org/ReleaseEngineering:Autoland<br />
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]<br />
* [https://catlee.github.io/highscores/highscores.html Scoreboard]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=File:Custom-action.png&diff=1223659File:Custom-action.png2020-02-12T15:19:10Z<p>Gbrown: </p>
<hr />
<div>Selecting "Custom Action..." in treeherder.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=ReleaseEngineering/TryServer&diff=1223657ReleaseEngineering/TryServer2020-02-12T15:05:35Z<p>Gbrown: /* Re-running tasks with custom parameters from Treeherder */</p>
<hr />
<div>= Try Server =<br />
The [https://treeherder.mozilla.org/#/jobs?repo=try try server] is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.<br />
<br />
== Getting access to the Try Server ==<br />
To use the try server, you'll need [http://www.mozilla.org/hacking/commit-access-policy/ Level 1 Commit Access]. You can learn more about Mozilla's commit access policies and start the process of signing up for an account here: [http://www.mozilla.org/hacking/committer/ Becoming a Mozilla Contributor]<br />
<br />
After you have level 1 commit access, you'll need to do a bit of setup before you'll be able to push. See [https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/auth.html this guide] for instructions.<br />
<br />
== Configuration ==<br />
Before using try, there is some recommended configuration you should set up. This can be accomplished by running:<br />
<br />
$ ./mach vcs-setup<br />
<br />
Be sure to at least enable the [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html firefoxtree extension] and the push-to-try extension. The push-to-try extension is required if you wish to use the |mach try| command (see below). Firefoxtree will give you a handy 'try' alias you can use for pushing, and prevent you from accidentally pushing multiple heads. If for some reason you prefer not to use firefoxtree, you can set the same alias up manually in your hgrc:<br />
<br />
[paths]<br />
try = ssh://hg.mozilla.org/try<br />
<br />
Note: Never pull the whole try repo. You'll end up with hundreds of heads containing everyone's half baked and broken pushes.<br />
<br />
For more information, see [http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html mercurial for mozillians].<br />
<br />
== How to push to try ==<br />
There are a few ways to schedule jobs on try.<br />
<br />
=== Attach new jobs from the review ===<br />
<br />
For every patch submitted for review in Phabricator, a new Try run is automatically created. This run is created for static analysis, linting and other tasks. <br />
As described below, attaching new jobs to the run is easy and doesn't require more actions from the developer.<br />
Just click on the ''Treeherder Jobs''.<br />
<br />
[[File:Screenshot 2019-10-07 ⚙ D48316 Bug 1585593 - Enable Fission for Raptor.png]]<br />
<br />
For more details, see:<br />
https://firefox-source-docs.mozilla.org/tools/try/index.html#attaching-new-jobs-from-a-review<br />
<br />
=== Scheduling jobs with Mach Try ===<br />
<br />
The recommended way to [https://firefox-source-docs.mozilla.org/tools/try/index.html interact with try] is via the command line tool |mach try|. Since try can be used for a wide range of use cases, by developers with varying levels of experience, there is no one size fits all solution when it comes to selecting which tasks to run. Therefore |mach try| offers a range of subcommands called "selectors" for a variety of purposes. To see a list of available selectors, run:<br />
<br />
$ mach try --help<br />
<br />
For deeper information on a given selector, run:<br />
<br />
$ mach try <selector> --help<br />
<br />
If you don't specify a selector/subcommand to try, it will default to using the `syntax` selector. You can change which selector gets used by default by creating ~/.mozbuild/machrc and adding:<br />
<br />
[try]<br />
default=<selector><br />
<br />
See [https://firefox-source-docs.mozilla.org/tools/try/selectors/index.html this documentation] for more information on all the available selectors.<br />
<br />
<br />
=== Scheduling jobs with Treeherder ===<br />
<br />
You can push to try with:<br />
<br />
$ ./mach try empty<br />
<br />
Once the push succeeds, you can open it in Treeherder and then sign in to schedule jobs manually.<br />
<br />
Select the drop-down arrow at the top right of your push, and choose "Add new Jobs". It might take several seconds for the jobs to load ({{bug|1288028}}) :<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042414/TH_dropdown.png<br />
<br />
If it helps, you can use the filter box in the site header to restrict the list of runnable jobs being displayed. Click the job symbols you wish to schedule. <br />
<br />
If you select a test job, the required build will automatically be scheduled:<br />
<br />
https://s3.amazonaws.com/media-p.slid.es/uploads/185196/images/2042426/TH_with_jobs.png<br />
<br />
Finally, click "Trigger New Jobs" near the top right of your push.<br />
<br />
NOTE: An action task gets scheduled which will schedule all the tasks you chose.<br />
<br />
=== Scheduling jobs with Treeherder (Search) ===<br />
<br />
Similar to the above method, there is now an additional way to add jobs to an existing push via Treeherder. Open the push in Treeherder and then choose "Add New Jobs (Search)" from the push's menu:<br />
<br />
[[File:Add New Jobs Search.png]]<br />
<br />
Treeherder will then fetch the list of runnable jobs in the background, then pop up a dialog window listing those jobs:<br />
<br />
[[File:Addnewjobssearchinterface.png]]<br />
<br />
At the top of the dialog, there is a search bar. Type part of your desired job name into the search bar and press 'Enter' to filter the list of jobs. (This uses a fuzzy search library, so some minimal typos are allowed). From the filtered list of jobs, select as many of the results as you want (ctrl/shift modifiers work on this list for selecting multiple jobs):<br />
<br />
[[File:Filteredaddnewjobssearch.png]]<br />
<br />
Then click the green "Add Selected" button to add the selected jobs to the list of jobs to be run (or click the green "Add All" button to add the entire list of jobs currently in view.<br />
<br />
Repeat the filter and selection process until you have all of your desired jobs added to the list. You can remove jobs from the list to be run using the red buttons lower on the dialog if you selected them in error:<br />
<br />
[[File:Addnewjobssearchselectedjobs.png]]<br />
<br />
When you are happy with your selection, click the blue "Trigger (X) Selected Jobs" button at the bottom of the dialog. Treeherder will send out a request for those jobs to be run, and an action task will be spawned to do all of the behind the scenes magic to make that happen.<br />
<br />
=== Pushing Directly ===<br />
<br />
It's also possible to push to try directly from hg (for now). This might be required if you're using [https://www.mercurial-scm.org/wiki/MqExtension mercurial queues]. In this case, the only way to select tasks is by manually adding try syntax to your commit message:<br />
<br />
$ hg commit -m "try: -b o -p linux64 -u mochitest"<br />
$ hg push -f try<br />
<br />
Using |mach try| is recommended as that will automatically create and cleanup a temporary commit no your behalf, whereas with this method you are responsible for cleaning up any extra commits you make or commit messages you modify.<br />
<br />
== Viewing the results ==<br />
You can see the results of your tryserver build in a number of ways:<br />
<br />
* You'll get an email on a successful push with a link to treeherder for your revision as well as optional emails on any non-successful build/test/talos results<br />
* You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax<br />
* The link to treeherder will be printed on the command line.<br />
* Look for your changeset on [https://treeherder.mozilla.org/#/jobs?repo=try Treeherder]. You can add '''&author=YOUR.EMAIL''' to only see your pushes.<br />
* Download your completed builds from [http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/?C=M;O=D firefox/tryserver-builds on ftp.m.o].<br />
<br />
== Using a custom mozconfig ==<br />
The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.<br />
<br />
If you want to apply the same mozconfig changes to multiple platforms, you can edit <tt>build/mozconfig.common.override</tt> instead. This file is included at the end of each of the in-tree mozconfig files.<br />
<br />
Android mozconfigs are in mobile/android/config/mozconfigs.<br />
<br />
If you modify a mozconfig file named <tt>nightly</tt>, please make sure you also change any mozconfig file named <tt>beta</tt> and/or <tt>release</tt> in the same directory, otherwise the test_compare_mozconfigs test will fail.<br />
<br />
Note:<br />
* TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.<br />
** Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.<br />
<br />
== Re-running tasks with custom parameters from Treeherder ==<br />
<br />
In treeherder, select the job you want to re-run (Note: it currently only works for [https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/actions/retrigger_mochitest.py#30-31 some jobs]). The details panel should open. Look for the 3 dots icon ("Other job details") and select "Custom actions...".<br />
<br />
[[File:Custom Taskcluster Job Actions.png|thumb|Treeherder modal to request an action associated to a task]]<br />
<br />
The <tt>retrigger-mochitest</tt> action (TODO: A better name is needed) is the one selected in the drop-down menu. Modify the task parameters as desired and select "Trigger". This will schedule the task with your specified parameters. The new task will be labeled as <tt>custom-<original_task_name></tt>.<br />
<br />
Some of the parameters that you can change are (see screenshot above):<br />
<br />
* Environment variables (e.g. <tt>MOZ_LOG</tt>)<br />
* Modify Python's logging level<br />
* Path of the test to execute<br />
* Gecko preferences (think of <tt>about:config</tt>)<br />
* Run a test repeatedly<br />
* Run until the test fails<br />
<br />
== Getting debug symbols ==<br />
<br />
NOTE: You might be able to accomplish this by re-running a build task from Treeherder by adding the env variable to the new task.<br />
<br />
By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add <tt>MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1</tt> to the in-tree mozconfigs. You can do this for all platforms by importing [http://hg.mozilla.org/users/tmielczarek_mozilla.com/mq/raw-file/b44faf6cd177/enable-full-symbols this patch] into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.<br />
<br />
== Adding new jobs ==<br />
It's possible to create new jobs (or modify existing ones) directly *in* your try push, provided you use taskcluster. Just edit the relevant configuration in testing/taskcluster. For more information on creating jobs, see the [http://docs.taskcluster.net/ taskcluster docs].<br />
<br />
== Desktop l10n jobs ==<br />
You can use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW). Leaving a full list of locales is likely to hit a timeout on Mac and Windows<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded to the same sub-directory as en-US builds, eg try-linux/ for 32bit linux.<br />
<br />
== Desktop l10n jobs (on Taskcluster) ==<br />
<br />
To get Desktop l10n jobs on taskcluster on try, simply pass `-b o -p linux64-l10n,linux-l10n` with your try push. You'll also get these jobs automatically with `-p linux64,linux` if you are touching one of the files known to be heavily involved in l10n jobs. (Taskcluster support for other platforms is not yet available)<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>browser/locales/all-locales</tt> (eg top-locales like de fr ja ja-JP-mac ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try.py#4 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Android Single-Locale l10n jobs (on Taskcluster) ==<br />
<br />
(NOTE: Only supported on Gecko 51a1 and above)<br />
<br />
To get Android l10n jobs on taskcluster on try, simply pass `-p android-api-15-l10n` with your try push. You'll also get these jobs automatically with `-p android-api-15` if you are touching one of the files known to be heavily involved in l10n jobs.<br />
<br />
You can also use the steps in [[ReleaseEngineering/TryServer#Scheduling jobs with Treeherder|Scheduling jobs with Treeherder]] to add localized desktop builds to your try push, regardless of whether you used try syntax at first. Filtering with 'l10n' helps to find the jobs amongst the many possibilities.<br />
<br />
The jobs can be customized by modifying files prior to pushing:<br />
* reducing the number of locales by limiting <tt>mobile/android/locales/all-locales</tt> (eg top-locales like de fr ru zh-TW).<br />
* use a different en-US build by modifying [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/single_locale/try_android-api-15.py#3 en_us_binary_url], but note that the building en-US and then l10n in one push is not a tested scenario<br />
<br />
The resulting builds are uploaded as a task artifact, and are not yet signed.<br />
<br />
== Server Status ==<br />
* Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending<br />
* In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running<br />
<br />
== Other Notes ==<br />
* Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion<br />
* Treeherder data is purged after 4 months.<br />
* If you have any problems please [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Release%20Engineering&status_whiteboard=tryserver file a bug]<br />
<br />
* Suggestions for the future can be made [[Build:TryServer:Suggestions|here]] or file a blocking bug against {{bug|try_enhancements}}<br />
<br />
== Other Mozilla Try Servers ==<br />
* [[ReleaseEngineering/ThunderbirdTryServer|Thunderbird Try Server]] for the comm-central repository<br />
<br />
== Problem Diagnosis ==<br />
=== Can not access try server ===<br />
Test your account & configuration<br />
* <code>ssh hg.mozilla.org</code>, response: "No Interactive shells allowed here!"<br />
* <code>ssh hg.mozilla.org clone invalid_sandbox</code>, response: menu display and interactive prompting.<br />
<br />
<div id="long_try_push"></div><br />
=== Pushes to try take a very long time ===<br />
Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See [http://dtor.com/halfire/2014/07/02/2014_06_try_server_update.html#caching caching] details.)<br />
<br />
If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using [https://bugzilla.mozilla.org/enter_bug.cgi?comment=please%20schedule%20a%20reset%20of%20the%20try%20repository%20ASAP%20with%20sheriffs%20%26%20releng.&component=WebOps%3A%20Source%20Control&op_sys=All&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Push%20to%20try%20taking%20XXX%20minutes this template] (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.<br />
<br />
=== Waiting for Lock ===<br />
If you get a message similar to:<br />
remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'<br />
remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549<br />
It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.<br />
<br />
=== Waiting for Lock multiple times with the same pid ===<br />
Similar to the above case, but with the same pid when you retry over and over again.<br />
<br />
Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as [[#long_try_push|above]].<br />
<br />
== CiDuty issues == <br />
=== How do I trigger additional talos/test runs for a given try build? ===<br />
If your syntax included the tests you'd like more of, then select the job you want on Treeherder and use the + button. For test suites you didn't request originally you can use the "Add New Jobs" interface in Treeherder.<br />
<br />
=== How do I cancel existing jobs? ===<br />
For individual jobs, select the relevant one on Treeherder and use the cancel button. To cancel all jobs, use the menu arrow shown on the header row for each push, and then the "Cancel all" option.<br />
<br />
== See Also ==<br />
* https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories<br />
* https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices<br />
* https://wiki.mozilla.org/ReleaseEngineering:Autoland<br />
* [http://build.mozilla.org/buildapi/self-serve Manage Submissions] [[https://build.mozilla.org/buildapi/self-serve/try try]]<br />
* [https://catlee.github.io/highscores/highscores.html Scoreboard]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Stockwell/Robot&diff=1213100Auto-tools/Projects/Stockwell/Robot2019-05-31T17:08:56Z<p>Gbrown: /* Intermittent Failures Robot */</p>
<hr />
<div>= Intermittent Failures Robot =<br />
<br />
Comments and other changes to intermittent-failure bugs by the "Intermittent Failures Robot" are made by the code at<br />
<br />
https://github.com/mozilla/treeherder/blob/master/treeherder/intermittents_commenter/commenter.py<br />
<br />
The commenter runs in two modes, "daily" and "weekly". The commenter runs in weekly mode on Sunday and in daily mode every other day of the week. The commenter may add a comment to a bug and/or update its whiteboard and/or priority.<br />
<br />
Bugs should be filed in Tree Management : Intermittent Failures View.<br />
<br />
== Comments ==<br />
=== Weekly ===<br />
* A comment is posted to every bug with 1 or more failures in the last week.<br />
<br />
* If there were 150 failures in the last 21 days, and the whiteboard does not contain [stockwell disabled], [stockwell disable-recommended], or [stockwell infra] an extra comment is added:<br />
<br />
** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **<br />
<br />
* Otherwise, if there were 75 or more failures in the last week, an extra comment is added:<br />
<br />
** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **<br />
** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **<br />
<br />
* Otherwise, if there were 30 or more failures in the last week, an extra comment is added:<br />
<br />
** This failure happened more than 30 times this week! Resolving this bug is a high priority. **<br />
** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **<br />
<br />
=== Daily ===<br />
* A comment is posted to every bug with 15 or more failures in the last day.<br />
<br />
== Whiteboard ==<br />
* If the "150 failures in the last 21 days" comment was made, the whiteboard is updated to [stockwell disable-recommended].<br />
* If the bug is in a bugzilla component that is participating in the stockwell owner-triage trial, and the whiteboard does not contain [stockwell disabled], [stockwell disable-recommended], [stockwell fixed], or [stockwell infra], and a frequent-failure comment has been made, the whiteboard is updated to [stockwell needswork:owner].<br />
* It appears that [stockwell needswork] is no longer set by the robot.<br />
=== Weekly ===<br />
* If a bug already has [stockwell needswork] or [stockwell needswork:owner] and the weekly failure count is less than 20, the whiteboard is updated to [stockwell unknown].<br />
<br />
== Priority ==<br />
* If the whiteboard was updated to [stockwell needswork:owner] and the priority is not one of ("--", "P1", "P2", "P3"), the priority is set to "--" to indicate this bug needs triage.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Stockwell/Robot&diff=1213099Auto-tools/Projects/Stockwell/Robot2019-05-31T16:56:08Z<p>Gbrown: /* OrangeFactor Robot */</p>
<hr />
<div>= Intermittent Failures Robot =<br />
<br />
Comments and other changes to intermittent-failure bugs by the "Intermittent Failures Robot" are made by the code at<br />
<br />
https://github.com/mozilla/treeherder/blob/master/treeherder/intermittents_commenter/commenter.py<br />
<br />
The commenter runs in two modes, "daily" and "weekly". The commenter runs in weekly mode on Sunday and in daily mode every other day of the week. The commenter may add a comment to a bug and/or update its whiteboard and/or priority.<br />
<br />
== Comments ==<br />
=== Weekly ===<br />
* A comment is posted to every bug with 1 or more failures in the last week.<br />
<br />
* If there were 150 failures in the last 21 days, and the whiteboard does not contain [stockwell disabled], [stockwell disable-recommended], or [stockwell infra] an extra comment is added:<br />
<br />
** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **<br />
<br />
* Otherwise, if there were 75 or more failures in the last week, an extra comment is added:<br />
<br />
** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **<br />
** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **<br />
<br />
* Otherwise, if there were 30 or more failures in the last week, an extra comment is added:<br />
<br />
** This failure happened more than 30 times this week! Resolving this bug is a high priority. **<br />
** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **<br />
<br />
=== Daily ===<br />
* A comment is posted to every bug with 15 or more failures in the last day.<br />
<br />
== Whiteboard ==<br />
* If the "150 failures in the last 21 days" comment was made, the whiteboard is updated to [stockwell disable-recommended].<br />
* If the bug is in a bugzilla component that is participating in the stockwell owner-triage trial, and the whiteboard does not contain [stockwell disabled], [stockwell disable-recommended], [stockwell fixed], or [stockwell infra], and a frequent-failure comment has been made, the whiteboard is updated to [stockwell needswork:owner].<br />
* It appears that [stockwell needswork] is no longer set by the robot.<br />
=== Weekly ===<br />
* If a bug already has [stockwell needswork] or [stockwell needswork:owner] and the weekly failure count is less than 20, the whiteboard is updated to [stockwell unknown].<br />
<br />
== Priority ==<br />
* If the whiteboard was updated to [stockwell needswork:owner] and the priority is not one of ("--", "P1", "P2", "P3"), the priority is set to "--" to indicate this bug needs triage.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=CI_Automation/windows10_aarch64&diff=1208191CI Automation/windows10 aarch642019-02-26T17:16:09Z<p>Gbrown: /* Run tests Locally */</p>
<hr />
<div>=Overview=<br />
Since mid-January 2019 the CI-A team has been working to enable existing test harnesses, continuous integration tests and other tools to run on Windows 10 ARM64.<br />
<br />
= General Information =<br />
<br />
== Hardware ==<br />
<br />
* Make: Lenovo<br />
* Model: C630 YOGA<br />
* Processor: Qualcomm Snapdragon 850 3.0GHz<br />
* Cores: 8<br />
* Memory: 8GB<br />
* Disk: 128GB SSD<br />
<br />
== Hosting ==<br />
<br />
Currently an array of 9 machines are hosted at [https://bitbar.com/ Bitbar] in the United States.<br />
<br />
= Setup =<br />
<br />
Tests that are run against windows10-aarch64 execute using [https://github.com/taskcluster/generic-worker Taskcluster Generic-Worker]. These are installed as a service on the Windows 10 ARM64 manually or via [https://github.com/mozilla-releng/OpenCloudConfig OpenCloudConfig].<br />
<br />
A brief walkthrough of the steps to have Taskcluster Generic-Worker running on Windows 10 ARM64 will be provided.<br />
<br />
== Using only generic-worker ==<br />
<br />
Follow this step to install Taskcluster Generic-Worker on the hardware, and have it launch as a service. After following these steps, the hardware should be ready to accept any tasks started on Taskcluster. <br />
<br />
Instruction originally from [https://bugzilla.mozilla.org/show_bug.cgi?id=1522997#c2 1522997].<br />
<br />
=== Prerequisites ===<br />
* disable Windows S mode<br />
* disable User Account Control<br />
* disable Windows Firewall<br />
* download NSSM to C:\nssm-2.24\<br />
* create "Remote Desktop Users" group:<br />
net localgroup "Remote Desktop Users" /add<br />
* log in to Taskcluster<br />
* request scope `assume:project:taskcluster:generic-worker-tester` <br />
<br />
=== Steps ===<br />
<br />
# download the current 386 release of `generic-worker-windows-386.exe` from [https://github.com/taskcluster/generic-worker/releases taskcluster generic-worker].<br />
# download the latest 386 version of livelog.exe and taskcluster-proxy.exe.<br />
# create new directory C:\generic-worker.<br />
# move the three executable files under C:\generic-worker.<br />
# rename generic-worker-windows-386.exe to generic-worker.exe.<br />
# generate two signing keys:<br />
generic-worker new-openpgp-keypair --file <unique_file_name><br />
generic-worker new-ed25519-keypair --file <unique_file_name><br />
# create generic-worker.config and include the following:<br />
"accessToken": "<access token tied to taskcluster>",<br />
"clientId": "<client ID tied to taskcluster>",<br />
"ed25519SigningKeyLocation": "<file location you wrote ed25519 private key in step 6>",<br />
"livelogSecret": "<any text>",<br />
"openpgpSigningKeyLocation": "<file location you wrote gpg private key kn step 6>",<br />
"provisionerId": "test-provisioner",<br />
"publicIP": "<ideally an IP address of one of your network interfaces>",<br />
"rootURL": "https://taskcluster.net",<br />
"workerGroup": "test-worker-group",<br />
"workerId": "test-worker-id",<br />
"workerType": "<a unique string that only you will use for your test worker(s)>"<br />
# launch cmd.exe with Administrator rights.<br />
# cd c:\generic-worker<br />
# generic-worker.exe install service --config generic-worker.config --nssm c:\nssm-2.24\win32\nssm.exe<br />
# reboot once installed.<br />
# launch cmd.exe with Administrator rights.<br />
# sc query "Generic Worker"<br />
<br />
== Using OpenCloudConfig ==<br />
<br />
This is the method that is used in production.<br />
<br />
Steps originally taken from [https://bugzilla.mozilla.org/show_bug.cgi?id=1520432#c2 1520432].<br />
<br />
# Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/aarch64/userdata/rundsc.ps1')<br />
<br />
= Currently Running =<br />
<br />
Currently supported list of tests include:<br />
<br />
* awsy<br />
* mochitest (all flavors, including e10s)<br />
* web-platform-tests (all flavors)<br />
* reftests (including crashtest, jsreftest)<br />
* xpcshell<br />
<br />
Supported, requires non-artifact build:<br />
<br />
* jittest<br />
* gtests<br />
* cppunittest<br />
<br />
There is remaining work needed to get these test suites running:<br />
* talos<br />
* raptor<br />
* marionette<br />
<br />
For an up-to-date list of tests, please refer to [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#222 this file].<br />
<br />
= Run tests Locally =<br />
Theoretically, you can run tests locally with mach from a local build environment. However, since our aarch64 builds are usually cross-compiled in an x86 environment, you probably don't have a local build environment!<br />
<br />
The recommended alternative is to use mozharness to download, install, and test a build from try or continuous integration. A handy script is provided as an attachment to {{bug|1520867}} that greatly simplifies running tests from mozharness; let's call that script 'moztest'.<br />
<br />
Run moztest from a MozillaBuild shell. You need only a few parameters:<br />
* The task-id of the Windows-aarch64 build that you want to test: Click on the aarch64 build in treeherder, and copy the "Task" shown in the treeherder detail pane; it might look like "Q-CE8DFvSAWmc08vw6bd6A".<br />
* The name of the test suite you want to run: one of (cppunit, gtest, xpcshell, mochitest, mochitest-chrome, mochitest-clipboard, mochitest-dt, mochitest-gpu, mochitest-media, crashtest, jsreftest, reftest, jittest, web-platform, web-platform-reftest, web-platform-wdspec, raptor-speedometer, raptor-tp6, talos-g5, talos-chromez)<br />
* Optionally, the test "chunk" number to run and the number of test chunks to split the suite into.<br />
<br />
For example:<br />
* moztest Q-CE8DFvSAWmc08vw6bd6A cppunit<br />
* moztest Q-CE8DFvSAWmc08vw6bd6A xpcshell 1 3<br />
<br />
= Run tests on Try =<br />
<br />
This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available. Note, the number of hardware is limited so please exercise caution when scheduling tests.<br />
<br />
== Overview ==<br />
<br />
Follow these steps to be able to enable windows10-aarch64 tests for the try server. These steps are required as of 2019-02-25; it will become obsolete when windows10-aarch64 tests are released to the general public.<br />
<br />
=== Prerequisites ===<br />
<br />
* try access (commit access level 1)<br />
* up-to-date mozilla-central codebase<br />
<br />
=== Steps ===<br />
<br />
# open the file at taskcluster/ci/test/test-platforms.yml<br />
# search for 'windows10-aarch64/opt'<br />
# uncomment all or some of the items under 'test-sets'<br />
# make changes to the local codebase that needs testing<br />
# ./mach try fuzzy<br />
# select tests that need to be run (e.g. 'windows10-aarch64 xpcshell')<br />
# enter<br />
<br />
Tests will appear in Treeherder under the heading ''windows10-aarch64 opt''.<br />
<br />
= Greening tests =<br />
<br />
Since Windows on ARM64 is a new platform/architecture combination, failures unique to this combination is to be expected. It will be necessary to fix, correct or update the tests in order to obtain a green run.<br />
<br />
== Example 1 ==<br />
<br />
As part of [https://bugzilla.mozilla.org/show_bug.cgi?id=1525743 1525743], the timeout for mochitest-browser-chrome was extended to 4x the default value if the platform combination of Windows and ARM64 is detected.<br />
<br />
See change: https://phabricator.services.mozilla.com/D19882<br />
<br />
This change greened the test that was previously failing due to a timeout.<br />
<br />
== Example 2 ==<br />
<br />
<TODO add instructions for reftest manifest, manifest parser, and wpt manifest for skip-if><br />
<br />
= Bugs =<br />
<br />
These are the top-level tracking bugs; the recommended view is [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1522997&hide_resolved=0 tree] (login required).<br />
<br />
<bugzilla><br />
{<br />
"blocks": "1522997"<br />
}<br />
</bugzilla></div>Gbrownhttps://wiki.mozilla.org/index.php?title=CI_Automation/windows10_aarch64&diff=1208190CI Automation/windows10 aarch642019-02-26T16:47:06Z<p>Gbrown: /* Run tests Locally */</p>
<hr />
<div>=Overview=<br />
Since mid-January 2019 the CI-A team has been working to enable existing test harnesses, continuous integration tests and other tools to run on Windows 10 ARM64.<br />
<br />
= General Information =<br />
<br />
== Hardware ==<br />
<br />
* Make: Lenovo<br />
* Model: C630 YOGA<br />
* Processor: Qualcomm Snapdragon 850 3.0GHz<br />
* Cores: 8<br />
* Memory: 8GB<br />
* Disk: 128GB SSD<br />
<br />
== Hosting ==<br />
<br />
Currently an array of 9 machines are hosted at [https://bitbar.com/ Bitbar] in the United States.<br />
<br />
= Setup =<br />
<br />
Tests that are run against windows10-aarch64 execute using [https://github.com/taskcluster/generic-worker Taskcluster Generic-Worker]. These are installed as a service on the Windows 10 ARM64 manually or via [https://github.com/mozilla-releng/OpenCloudConfig OpenCloudConfig].<br />
<br />
A brief walkthrough of the steps to have Taskcluster Generic-Worker running on Windows 10 ARM64 will be provided.<br />
<br />
== Using only generic-worker ==<br />
<br />
Follow this step to install Taskcluster Generic-Worker on the hardware, and have it launch as a service. After following these steps, the hardware should be ready to accept any tasks started on Taskcluster. <br />
<br />
Instruction originally from [https://bugzilla.mozilla.org/show_bug.cgi?id=1522997#c2 1522997].<br />
<br />
=== Prerequisites ===<br />
* disable Windows S mode<br />
* disable User Account Control<br />
* disable Windows Firewall<br />
* download NSSM to C:\nssm-2.24\<br />
* create "Remote Desktop Users" group:<br />
net localgroup "Remote Desktop Users" /add<br />
* log in to Taskcluster<br />
* request scope `assume:project:taskcluster:generic-worker-tester` <br />
<br />
=== Steps ===<br />
<br />
# download the current 386 release of `generic-worker-windows-386.exe` from [https://github.com/taskcluster/generic-worker/releases taskcluster generic-worker].<br />
# download the latest 386 version of livelog.exe and taskcluster-proxy.exe.<br />
# create new directory C:\generic-worker.<br />
# move the three executable files under C:\generic-worker.<br />
# rename generic-worker-windows-386.exe to generic-worker.exe.<br />
# generate two signing keys:<br />
generic-worker new-openpgp-keypair --file <unique_file_name><br />
generic-worker new-ed25519-keypair --file <unique_file_name><br />
# create generic-worker.config and include the following:<br />
"accessToken": "<access token tied to taskcluster>",<br />
"clientId": "<client ID tied to taskcluster>",<br />
"ed25519SigningKeyLocation": "<file location you wrote ed25519 private key in step 6>",<br />
"livelogSecret": "<any text>",<br />
"openpgpSigningKeyLocation": "<file location you wrote gpg private key kn step 6>",<br />
"provisionerId": "test-provisioner",<br />
"publicIP": "<ideally an IP address of one of your network interfaces>",<br />
"rootURL": "https://taskcluster.net",<br />
"workerGroup": "test-worker-group",<br />
"workerId": "test-worker-id",<br />
"workerType": "<a unique string that only you will use for your test worker(s)>"<br />
# launch cmd.exe with Administrator rights.<br />
# cd c:\generic-worker<br />
# generic-worker.exe install service --config generic-worker.config --nssm c:\nssm-2.24\win32\nssm.exe<br />
# reboot once installed.<br />
# launch cmd.exe with Administrator rights.<br />
# sc query "Generic Worker"<br />
<br />
== Using OpenCloudConfig ==<br />
<br />
This is the method that is used in production.<br />
<br />
Steps originally taken from [https://bugzilla.mozilla.org/show_bug.cgi?id=1520432#c2 1520432].<br />
<br />
# Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/aarch64/userdata/rundsc.ps1')<br />
<br />
= Currently Running =<br />
<br />
Currently supported list of tests include:<br />
<br />
* awsy<br />
* mochitest (all flavors, including e10s)<br />
* web-platform-tests (all flavors)<br />
* reftests (including crashtest, jsreftest)<br />
* xpcshell<br />
<br />
Supported, requires non-artifact build:<br />
<br />
* jittest<br />
* gtests<br />
* cppunittest<br />
<br />
There is remaining work needed to get these test suites running:<br />
* talos<br />
* raptor<br />
* marionette<br />
<br />
For an up-to-date list of tests, please refer to [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#222 this file].<br />
<br />
= Run tests Locally =<br />
Theoretically, you can run tests locally with mach from a local build environment. However, since our aarch64 builds are usually cross-compiled in an x86 environment, you probably don't have a local build environment!<br />
<br />
Instead...<br />
<br />
= Run tests on Try =<br />
<br />
This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available. Note, the number of hardware is limited so please exercise caution when scheduling tests.<br />
<br />
== Overview ==<br />
<br />
Follow these steps to be able to enable windows10-aarch64 tests for the try server. These steps are required as of 2019-02-25; it will become obsolete when windows10-aarch64 tests are released to the general public.<br />
<br />
=== Prerequisites ===<br />
<br />
* try access (commit access level 1)<br />
* up-to-date mozilla-central codebase<br />
<br />
=== Steps ===<br />
<br />
# open the file at taskcluster/ci/test/test-platforms.yml<br />
# search for 'windows10-aarch64/opt'<br />
# uncomment all or some of the items under 'test-sets'<br />
# make changes to the local codebase that needs testing<br />
# ./mach try fuzzy<br />
# select tests that need to be run (e.g. 'windows10-aarch64 xpcshell')<br />
# enter<br />
<br />
Tests will appear in Treeherder under the heading ''windows10-aarch64 opt''.<br />
<br />
= Greening tests =<br />
<br />
Since Windows on ARM64 is a new platform/architecture combination, failures unique to this combination is to be expected. It will be necessary to fix, correct or update the tests in order to obtain a green run.<br />
<br />
== Example 1 ==<br />
<br />
As part of [https://bugzilla.mozilla.org/show_bug.cgi?id=1525743 1525743], the timeout for mochitest-browser-chrome was extended to 4x the default value if the platform combination of Windows and ARM64 is detected.<br />
<br />
See change: https://phabricator.services.mozilla.com/D19882<br />
<br />
This change greened the test that was previously failing due to a timeout.<br />
<br />
== Example 2 ==<br />
<br />
<TODO add instructions for reftest manifest, manifest parser, and wpt manifest for skip-if><br />
<br />
= Bugs =<br />
<br />
These are the top-level tracking bugs; the recommended view is [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1522997&hide_resolved=0 tree] (login required).<br />
<br />
<bugzilla><br />
{<br />
"blocks": "1522997"<br />
}<br />
</bugzilla></div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1207442Mobile/Fennec/Android/Testing2019-02-08T20:49:11Z<p>Gbrown: /* Advanced setup */</p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
mach web-platform-test<br />
mach marionette-test<br />
mach test<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
Notes:<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 7.0 x86_64 opt/debug" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 7.0 x86_64 image:<br />
<br />
./mach android-emulator --version x86-7.0<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is '''much''' faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For your convenience, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Multiple emulators/devices ===<br />
<br />
It is possible to test with multiple emulators or devices using the --deviceSerial argument:<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.<br />
* Is your device rooted? Many test suites require root permissions. Don't want to root your device? Consider using an emulator with 'mach android-emulator'.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1207441Mobile/Fennec/Android/Testing2019-02-08T20:46:51Z<p>Gbrown: </p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
mach web-platform-test<br />
mach marionette-test<br />
mach test<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
Notes:<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 7.0 x86_64 opt/debug" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 7.0 x86_64 image:<br />
<br />
./mach android-emulator --version x86-7.0<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is '''much''' faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For your convenience, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Multiple emulators/devices ===<br />
<br />
It is possible to test with multiple emulators or devices using the --deviceSerial argument:<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.<br />
<br />
wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py<br />
python getxre.py -d $DIR<br />
export MOZ_HOST_BIN=$DIR/bin<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.<br />
* Is your device rooted? Many test suites require root permissions. Don't want to root your device? Consider using an emulator with 'mach android-emulator'.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1207440Mobile/Fennec/Android/Testing2019-02-08T20:40:28Z<p>Gbrown: /* Running tests on the Android emulator */</p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.<br />
<br />
mach build mobile/android/tests/browser/robocop<br />
mach robocop<br />
<br />
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)<br />
<br />
Notes:<br />
* Mach is self documenting! For help, try <tt>mach help robocop</tt>.<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 7.0 x86_64 opt/debug" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 7.0 x86_64 image:<br />
<br />
./mach android-emulator --version x86-7.0<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is '''much''' faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For your convenience, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Multiple emulators/devices ===<br />
<br />
It is possible to test with multiple emulators or devices using the --deviceSerial argument:<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.<br />
<br />
wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py<br />
python getxre.py -d $DIR<br />
export MOZ_HOST_BIN=$DIR/bin<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== S1/S2 Automation ==<br />
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times. <br />
<br />
S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/<br />
<br />
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2<br />
<br />
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests<br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1207439Mobile/Fennec/Android/Testing2019-02-08T20:35:41Z<p>Gbrown: /* robocop UI tests */</p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.<br />
<br />
mach build mobile/android/tests/browser/robocop<br />
mach robocop<br />
<br />
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)<br />
<br />
Notes:<br />
* Mach is self documenting! For help, try <tt>mach help robocop</tt>.<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 4.2 x86 image:<br />
<br />
./mach android-emulator --version x86<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is much faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For the '''very''' lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Testing with stock AVD images (Android 4.x+ only) ===<br />
<br />
Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.<br />
<br />
# From the Android Virtual Device (AVD) Manager, click 'Create....'<br />
# Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).<br />
# Choose a system image with API level 16 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.<br />
# Next, check 'Use Host GPU' if possible. See [http://developer.android.com/tools/devices/emulator.html#acceleration Using Hardware Acceleration].<br />
# Hit Finish and that should be it. If you have setup Firefox for Android to [[Mobile/Fennec/Android/IDEs|run on your IDE]], you should be able to launch the emulator straight from there as well.<br />
<br />
=== Multiple emulators/devices ===<br />
<br />
It's also possible to test with multiple emulators or devices using the --deviceSerial argument:<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.<br />
<br />
wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py<br />
python getxre.py -d $DIR<br />
export MOZ_HOST_BIN=$DIR/bin<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== S1/S2 Automation ==<br />
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times. <br />
<br />
S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/<br />
<br />
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2<br />
<br />
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests<br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1206642Mobile/Fennec/Android/Testing2019-01-24T21:16:46Z<p>Gbrown: /* Multiple emulators */</p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.<br />
<br />
mach build mobile/android/tests/browser/robocop<br />
mach robocop<br />
<br />
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)<br />
<br />
Notes:<br />
* Mach is self documenting! For help, try <tt>mach help robocop</tt>.<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 4.2 x86 image:<br />
<br />
./mach android-emulator --version x86<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is much faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For the '''very''' lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Testing with stock AVD images (Android 4.x+ only) ===<br />
<br />
Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.<br />
<br />
# From the Android Virtual Device (AVD) Manager, click 'Create....'<br />
# Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).<br />
# Choose a system image with API level 16 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.<br />
# Next, check 'Use Host GPU' if possible. See [http://developer.android.com/tools/devices/emulator.html#acceleration Using Hardware Acceleration].<br />
# Hit Finish and that should be it. If you have setup Firefox for Android to [[Mobile/Fennec/Android/IDEs|run on your IDE]], you should be able to launch the emulator straight from there as well.<br />
<br />
=== Multiple emulators/devices ===<br />
<br />
It's also possible to test with multiple emulators or devices using the --deviceSerial argument:<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.<br />
<br />
wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py<br />
python getxre.py -d $DIR<br />
export MOZ_HOST_BIN=$DIR/bin<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== S1/S2 Automation ==<br />
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times. <br />
<br />
S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/<br />
<br />
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2<br />
<br />
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests<br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Packaging_Android_host_utilities&diff=1202866Packaging Android host utilities2018-10-24T19:21:00Z<p>Gbrown: /* Generating new archive - removed elfhack reference - see bug 1499915 */</p>
<hr />
<div>== Packaging Android host utilities ==<br />
<br />
The host utilities are executable files that run on a *host* machine. The utilities provide services to an Android *target* device, including a web server and certificate authority. For historical reason, the host utilities include an essentially complete version of Firefox. Here's how to package new versions of the host utilities.<br />
<br />
First, identify the target build. Generally, prefer Beta channel builds to Nightly channel builds.<br />
<br />
For the version 60 update, host-utils was based on builds from https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=bb369804a51e7665c0b44d3778681ca132cb1c2c. See bug 1433279. <br />
<br />
=== Linux ===<br />
<br />
==== Generating new archive ====<br />
<br />
Follow these steps to locate and download the necessary files.<br />
<br />
# From the selected treeherder build, identify "Linux opt" and "Linux x64 opt". <br />
# Click on the B icon, which will bring up a pane below. <br />
# Identify and click on the hash on the left hand pane called "Task".<br />
# In the newly opened TaskCluster tab, click on tab named "Run Artifacts".<br />
# Download target.common.tests.tar.gz & target.tar.bz2.<br />
# Follow the contents of the script below:<br />
<br />
<pre><br />
tar xvf target.tar.bz2<br />
tar xvf target.common.tests.tar.gz "bin/*"<br />
rm firefox/firefox*<br />
rm -r firefox/browser<br />
mv bin/* firefox<br />
mv firefox host-utils-60.0a1.en-US.linux-x86_64<br />
</pre><br />
<br />
==== Uploading to ToolTool ====<br />
<br />
1. Prepare new archive for upload:<br />
<br />
<pre><br />
tar cvf host-utils-60.0a1.en-US.linux-x86_64.tar host-utils-60.0a1.en-US.linux-x86_64<br />
gzip host-utils-60.0a1.en-US.linux-x86_64.tar<br />
</pre><br />
<br />
''replace the version numbers as appropriate.''<br />
<br />
2. Compare contents of current archive to the new archive. For instructions on existing archive, see the section below.<br />
<br />
3. Ensure uploading user has a valid token at [https://mozilla-releng.net/tokens/ Mozilla Releng].<br />
<br />
4. Upload the archive:<br />
<br />
<pre><br />
python tooltool.py upload [name_of_archive] --authentication-file=[token_location] --message [commit_message]<br />
</pre><br />
<br />
5. Update the manifest in testing/config/tooltool-manifests/linux64/hostutils.manifest.<br />
<br />
6. Repeat, using "Linux opt" archives, for 32 bit.<br />
<br />
=== Mac OS X ===<br />
<br />
Preparing for Mac OS X is not quite so simple. Due to changes in the way that Mac OS X codesigns binaries, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If you don't codesign them, you will be prompted every invocation to "allow network connections". See http://apple.stackexchange.com/a/150711 for a discussion of the issue and the codesigning invocation used below. You will be prompted for your password (by <tt>sudo</tt>), but it's possible super user permissions are not needed.<br />
<br />
<pre><br />
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-37.0a2.en-US.mac.tests.zip<br />
unzip firefox-37.0a2.en-US.linux-x86_64.tests.zip "bin/*"<br />
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-37.0a2.en-US.mac.dmg<br />
open firefox-37.0a2.en-US.mac.dmg<br />
cp -R /Volumes/FirefoxDeveloperEdition/FirefoxDeveloperEdition.app/Contents/MacOS/* bin<br />
cp -R /Volumes/FirefoxDeveloperEdition/FirefoxDeveloperEdition.app/Contents/Resources/* bin<br />
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -<br />
mv bin host-utils-37.0a2.en-US.mac<br />
</pre><br />
<br />
=== Download existing archive ===<br />
<br />
It is possible to download existing host utilities.<br />
<br />
1. locate the existing manifest file using [https://searchfox.org/mozilla-central/source/testing/config/tooltool-manifests/linux64/hostutils.manifest SearchFox] for both 32bit and 64bit.<br />
<br />
2. using ToolTool, download the host utilities:<br />
<pre><br />
python tooltool.py fetch -m old_hostutils_manifest.tt<br />
</pre></div>Gbrownhttps://wiki.mozilla.org/index.php?title=Sheriffing/Job_Visibility_Policy&diff=1202861Sheriffing/Job Visibility Policy2018-10-24T17:03:30Z<p>Gbrown: /* Adding a new test task, or a new test platform? */</p>
<hr />
<div>This page exists to clarify the policy towards how jobs reporting to [https://treeherder.mozilla.org/ Treeherder] are managed. Common sense will apply in cases where some of the requirements are not applicable for a particular platform/build/test type.<br />
<br />
To propose changes to this policy, please speak to the sheriffs and/or send a message to [https://lists.mozilla.org/listinfo/dev-tree-management dev.tree-management].<br />
<br />
== Overview of the Job Visibility Tiers ==<br />
Jobs reporting to Treeherder can fall into three tiers.<br />
* <b>Tier 1:</b> Jobs that run on a [https://developer.mozilla.org/docs/Supported_build_configurations Tier-1 platform], are shown by default on Treeherder, and are sheriff-managed. Bustage will cause a tree closure and is expected to result in a quick follow-up push or a backout (at the discretion of the sheriff on duty). Bugs will be filed for new intermittent test failures and are subject to the [https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy Test Disabling Policy] if not addressed in a timely fashion.<br />
* <b>Tier 2:</b> Jobs are shown by default on Treeherder, but are not sheriff-managed. Results will be shown on Treeherder "for information only". New test failures/bustage will not result in a backout, but a tracking bug will be filed when observed.<br />
* <b>Tier 3:</b> Jobs are not shown by default on Treeherder. All responsibilities for monitoring the results will fall upon the owner of the job.<br />
<br />
== Requirements for jobs shown in the default Treeherder view ==<br />
The below section applies to <b>both Tier 1 and Tier 2 jobs</b>. Owners of non-sheriff managed project/disposable repos do not need to meet these requirements. However, they must be satisfied prior to being enabled in production.<br />
<br />
==== Has an active owner ====<br />
* Who is committed to ensuring the other requirements are met not just initially, but over the long term.<br />
* Who will ensure the new job type is switched off to save resources should we stop finding it useful in the future.<br />
<br />
==== Usable job logs ====<br />
* Full logs should be available for both successful and failed runs in either raw or structured formats.<br />
* The crash reporter should be enabled, mini-dumps processed correctly (ie: with symbols available) & the resultant valid crash stack visible in the log (it is recommended to use mozcrash to avoid reinventing the wheel).<br />
* Failures must appear in the Treeherder failure summary in order to avoid having to open the full log for every failure.<br />
* Failure output must be in the format expected by the Treeherder's [https://github.com/mozilla/treeherder-service/blob/master/treeherder/model/error_summary.py bug suggestion generator] (otherwise sheriffs have to manually search Bugzilla when classifying/annotation intermittent failures):<br />
** For in-tree/product issues (eg: test failures, crashes):<br />
*** Delimeter: ' | '<br />
*** 1st token: One of {TEST-UNEXPECTED-FAIL, TEST-UNEXPECTED-PASS, PROCESS-CRASH}.<br />
*** 2nd token: A unique test name/filepath (not a generic test loader that runs 100s of other test files, since otherwise bug suggestions will return too many results).<br />
*** 3rd token: The specific failure message (eg: the test part that failed, the top frame of a crash or the leaked objects list for a leak).<br />
** For non test-specific issues (eg: infra/automation/harness):<br />
*** Treeherder falls back to searching Bugzilla for the entire failure line (excluding mozharness logging prefix), so it should be both unique to that failure type & repeatable (ie: no use of process IDs or timestamps, for which there will rarely be a repeat match against a bug summary).<br />
** Exceptions & timeouts must be handled with appropriate log output (eg: the failure line must state in which test the timeout occurred, not just that the entire run has timed out).<br />
* The sheriffs will be happy to advise regarding the above.<br />
<br />
==== Has sufficient documentation ====<br />
* Has a wiki page with:<br />
** An overview of the test-suite.<br />
** Instructions for running locally.<br />
** How to disable an individual failing test.<br />
** The current owner/who to contact for help.<br />
** The Bugzilla product/component where bugs should be filed (Github issues is not discoverable enough and prevents the use of bug dependencies within the rest of the project).<br />
* That wiki page is linked to from https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing<br />
<br />
== Additional requirements for Tier 1 jobs ==<br />
<br />
==== Breakage is expected to be followed by tree closure or backout ====<br />
* Failures visible in the default view (other than those that are known intermittents/transient), must have their cause backed out in a timely fashion or else the tree closed until diagnosed.<br />
* Sheriffs will generally ping in #developers on irc.mozilla.org when such a situation arises. If sufficient time passes without acknowledgement (typically ~5min), the regressing patch(es) will be backed out in order to minimize the length of the closure for other developers.<br />
* If acknowledged, sheriffs will decide in conjunction with the developer whether backing out or fixing in-place is the most reasonable resolution. The sheriff maintains the right to backout if necessary, however.<br />
<br />
==== Runs on mozilla-central and all trees that merge into it ====<br />
* Necessary because job failures when tree X merges into mozilla-central will not be attributable to a single changeset, resulting in either tree closure or backout of the entire merge (see the previous requirement).<br />
* When filing the release engineering bug to enable your job on all the required trees, ask to enable it on "mozilla-central based trees" and release engineering will enable it in the default config from which all trunk trees inherit (unless the various tree owners have explicitly opted out). As a rough guide, mozilla-central based trees include mozilla-inbound, autoland, as well as many of the other project/disposable repositories.<br />
<br />
==== Scheduled on every push ====<br />
* Otherwise job failures will not be attributable to a single changeset, resulting in either tree closure or backout of multiple pushes (see requirement #2).<br />
* An exception is made for nightly builds with an virtually equivalent non-nightly variant that is built on every push & for tests run on PGO builds (given that PGO builds take an inordinate amount of time, we still schedule them every 3/6 hours depending on tree, and relatively speaking there are not too many PGO-only test failures). Periodic builds have also been granted an exception as they don't run tests and have sufficient coverage on other platforms such that the odds of unique bustage are small and relatively easy to diagnose.<br />
* Note also that coalescing (buildbot queue collapsing when there is more than one queued job of the exact same tree/type) may mean that not all scheduled jobs actually get run. Whilst coalescing makes sheriffing harder, it's a necessary evil given that automation infrastructure demand frequently outstrips supply.<br />
<br />
==== Must avoid patterns known to cause non deterministic failures ====<br />
* Must avoid pulling the tip of external repositories as part of the build - since landings there can cause non-obvious failures. If an external repository is absolutely necessary, instead reference the desired changeset from a manifest in mozilla-central (like talos or gaia do).<br />
* Must not rely on resources from sites whose content we do not control/have no SLA:<br />
** Since these will cause failures when the external site is unavailable, as well as impacting end to end times & adding noise to performance tests.<br />
** eg: Emulator/driver binaries direct from a vendor's site, package downloads from PyPi or page assets for unit/performance tests.<br />
** Ensure MOZ_DISABLE_NONLOCAL_CONNECTIONS is defined in the automation environment (see {{bug|995417}}) & use a list of automation prefs for switching off undesirable behaviour (eg automatic updates, telemetry pings; see {{bug|1023483}} for where these are set).<br />
* Must not contain time bombs, e.g. tests that will fail after a certain date or when run at certain times (e.g., the day summer time starts or ends, or when the test starts before midnight and finishes after midnight).<br />
* See the [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges best practices for avoiding intermittent failures (oranges)].<br />
<br />
==== Low intermittent failure rate ====<br />
* A high failure rate:<br />
** Causes unnecessary sheriff workload.<br />
** Affects the ability to sheriff the trees as a whole, particularly during times of heavy coalescing.<br />
** Undermines devs confidence in the platform/test-suite - which as demonstrated by Firefox for Android, permanently affects their willingness to believe any future failures, even once the intermittent-failure rate is lowered.<br />
* A mozilla-central push results in ~400 jobs. The typical OrangeFactor across all trunk trees is normally (excluding the recent spike) 3-4, ie: a failure rate of ~1%. <br />
* Therefore as a rough guide a new platform/testsuite must have at most a 5% per job failure rate initially, and ideally <1% longer term.<br />
* However, sheriffs will make the final determination of whether a job type has too many intermittent failures. This will be a based on a combination of factors including failure rate, length of time the failures have been occurring, owner interest in fixing them & whether Treeherder is able to make bug suggestions.<br />
<br />
==== Easily run on try server ====<br />
* Needed so that developers who have had their landing backed out for breaking the job type are able to debug the failures/test the fix, particularly if they only reproduce on our infrastructure.<br />
* Developers should not be expected to guess try chooser options, so http://trychooser.pub.build.mozilla.org/ should be updated if appropriate.<br />
<br />
== Optional, but helpful ==<br />
<br />
==== Easy for a dev to run locally ====<br />
* Supported by mach (if appropriate).<br />
* Ideally part of mozilla-central (legacy exceptions being Talos, gaia).<br />
<br />
==== Supports the disabling of individual tests ====<br />
* It must be possible for sheriffs to disable an individual test per platform or entirely, by either annotating the test or editing a manifest/moz.build/Makefile in the relevant gecko repository.<br />
<br />
== Requesting changes in visibility ==<br />
* Jobs that are marked as tier 3 will be hidden in Treeherder by default.<br />
* To adjust the tier for a Taskcluster job, use a bug either in the Taskcluster Task Graph component, or else a component related to the type of task being adjusted, then edit the in-tree task definiton.<br />
* For legacy buildbot jobs, the tier is set via a hardcoded whitelist of job signatures. File a bug in the Treeherder Data ingestion component and follow the steps here: https://treeherder.readthedocs.io/common_tasks.html#hide-jobs-with-tiers<br />
* CC :sheriffs when adjusting a job's tier, so they are aware of the change and can confirm the criteria have been met.<br />
<br />
== Adding a new test task, or a new test platform? ==<br />
* Be sure to demonstrate an acceptable intermittent failure rate for your new test tasks on try, and include the try links in the bug which adds the new tasks. Usually that means repeating each new test task at least 10 times (try: --rebuild 10). <br />
* For each known intermittent failure, check the expected frequency from recent comments in the bug, or by looking up the failure in treeherder's Intermittent Failures view; if you see higher failure rates in your try push, consider fixing or disabling the test(s) before enabling your new task(s).<br />
<br />
== My platform/test-suite does not meet the base requirements, what now? ==<br />
* Your platform/test-suite will still be being run, just not shown on the default view. This model has worked well for many projects/build types (eg jetpack, xulrunner, spidermonkey).<br />
* To see it, click the "show/hide hidden jobs" checkbox to the left of the quick filter input field in the Treeherder UI. Alternatively, |&exclusion_profile=false| can be added to the URL to show all hidden jobs.<br />
* To filter the jobs displayed, under the 'Filters' menu use the 'job name' field.<br />
* For Try specifically, you can request that the job type by made non-default (ie requires explicit opt-in when using trychooser syntax, and won't be scheduled with '-u all' or similar), in order to be shown in the default view on Try - [http://hg.mozilla.org/build/buildbot-configs/file/27dff21cb799/mozilla/config.py#l332 example].</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing&diff=1200195Mobile/Testing2018-08-28T23:40:51Z<p>Gbrown: /* Archive of Notes */</p>
<hr />
<div>= About =<br />
<br />
We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).<br />
<br />
Note that the scope of this page is restricted to Firefox for Android (Fennec).<br />
<br />
== Documentation ==<br />
<br />
* [[Mobile/Testing/Architectural_Overview|Architectural Overview]] of our mobile automated testing systems.<br />
* For information on running the same tests we run in automation on your desktop, see the [https://wiki.mozilla.org/Mobile/Fennec/Android/Testing testing] for Android wiki page.<br />
<br />
= Status Meetings =<br />
Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.<br />
Details<br />
* Every other Wednesday @ 11:00am PST/PDT<br />
* Vidyo in Mobile (x9998, or 99998# from phone)<br />
* #mobile for back channel<br />
* Notes: https://docs.google.com/document/d/1hxZ0xLFbtXA3AKZltYwL21lRA8qRvrwVjSlQO4dfv0U/edit?usp=sharing<br />
<br />
== Archive of Notes ==<br />
[[Template:MobileTesting|template]]<br />
<br />
Create a new weekly agenda from the [[Template:MobileTesting|template]]:<br />
<br />
<createbox><br />
align=left<br />
type=create<br />
preload=Template:MobileTesting<br />
default={{#time: m_d_y | wednesday}}<br />
prefix=Mobile/Testing/<br />
<br />
</createbox><br />
# NOW USING GOOGLE DOC, ABOVE<br />
# [[Mobile/Testing/08_15_18 | 08/15/18]]<br />
# [[Mobile/Testing/08_01_18 | 08/01/18]]<br />
# [[Mobile/Testing/07_18_18 | 07/18/18]]<br />
# [[Mobile/Testing/07_04_18 | 07/04/18]] Cancelled!<br />
# [[Mobile/Testing/06_20_18 | 06/20/18]]<br />
# [[Mobile/Testing/06_06_18 | 06/06/18]]<br />
# [[Mobile/Testing/05_23_18 | 05/23/18]]<br />
# [[Mobile/Testing/05_09_18 | 05/09/18]]<br />
# [[Mobile/Testing/04_25_18 | 04/25/18]]<br />
# [[Mobile/Testing/04_11_18 | 04/11/18]]<br />
# [[Mobile/Testing/03_28_18 | 03/28/18]]<br />
# [[Mobile/Testing/03_14_18 | 03/14/18]]<br />
# [[Mobile/Testing/02_28_18 | 02/28/18]]<br />
# [[Mobile/Testing/02_14_18 | 02/14/18]]<br />
# [[Mobile/Testing/01_31_18 | 01/31/18]]<br />
# [[Mobile/Testing/01_17_18 | 01/17/18]]<br />
<br />
Older<br />
# [[Mobile/Testing/Archive | Archive of notes from 2011 - 2017]]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing&diff=1200194Mobile/Testing2018-08-28T23:35:38Z<p>Gbrown: /* Status Meetings */</p>
<hr />
<div>= About =<br />
<br />
We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).<br />
<br />
Note that the scope of this page is restricted to Firefox for Android (Fennec).<br />
<br />
== Documentation ==<br />
<br />
* [[Mobile/Testing/Architectural_Overview|Architectural Overview]] of our mobile automated testing systems.<br />
* For information on running the same tests we run in automation on your desktop, see the [https://wiki.mozilla.org/Mobile/Fennec/Android/Testing testing] for Android wiki page.<br />
<br />
= Status Meetings =<br />
Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.<br />
Details<br />
* Every other Wednesday @ 11:00am PST/PDT<br />
* Vidyo in Mobile (x9998, or 99998# from phone)<br />
* #mobile for back channel<br />
* Notes: https://docs.google.com/document/d/1hxZ0xLFbtXA3AKZltYwL21lRA8qRvrwVjSlQO4dfv0U/edit?usp=sharing<br />
<br />
== Archive of Notes ==<br />
[[Template:MobileTesting|template]]<br />
<br />
Create a new weekly agenda from the [[Template:MobileTesting|template]]:<br />
<br />
<createbox><br />
align=left<br />
type=create<br />
preload=Template:MobileTesting<br />
default={{#time: m_d_y | wednesday}}<br />
prefix=Mobile/Testing/<br />
<br />
</createbox><br />
# [[Mobile/Testing/08_15_18 | 08/15/18]]<br />
# [[Mobile/Testing/08_01_18 | 08/01/18]]<br />
# [[Mobile/Testing/07_18_18 | 07/18/18]]<br />
# [[Mobile/Testing/07_04_18 | 07/04/18]] Cancelled!<br />
# [[Mobile/Testing/06_20_18 | 06/20/18]]<br />
# [[Mobile/Testing/06_06_18 | 06/06/18]]<br />
# [[Mobile/Testing/05_23_18 | 05/23/18]]<br />
# [[Mobile/Testing/05_09_18 | 05/09/18]]<br />
# [[Mobile/Testing/04_25_18 | 04/25/18]]<br />
# [[Mobile/Testing/04_11_18 | 04/11/18]]<br />
# [[Mobile/Testing/03_28_18 | 03/28/18]]<br />
# [[Mobile/Testing/03_14_18 | 03/14/18]]<br />
# [[Mobile/Testing/02_28_18 | 02/28/18]]<br />
# [[Mobile/Testing/02_14_18 | 02/14/18]]<br />
# [[Mobile/Testing/01_31_18 | 01/31/18]]<br />
# [[Mobile/Testing/01_17_18 | 01/17/18]]<br />
<br />
Older<br />
# [[Mobile/Testing/Archive | Archive of notes from 2011 - 2017]]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing&diff=1200193Mobile/Testing2018-08-28T23:34:16Z<p>Gbrown: /* Status Meetings */</p>
<hr />
<div>= About =<br />
<br />
We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).<br />
<br />
Note that the scope of this page is restricted to Firefox for Android (Fennec).<br />
<br />
== Documentation ==<br />
<br />
* [[Mobile/Testing/Architectural_Overview|Architectural Overview]] of our mobile automated testing systems.<br />
* For information on running the same tests we run in automation on your desktop, see the [https://wiki.mozilla.org/Mobile/Fennec/Android/Testing testing] for Android wiki page.<br />
<br />
= Status Meetings =<br />
Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.<br />
Details<br />
* Every other Wednesday @ 10:15am PST/PDT<br />
* Vidyo in Mobile (x9998, or 99998# from phone)<br />
* #mobile for back channel<br />
* Notes: https://docs.google.com/document/d/1hxZ0xLFbtXA3AKZltYwL21lRA8qRvrwVjSlQO4dfv0U/edit?usp=sharing<br />
<br />
== Archive of Notes ==<br />
[[Template:MobileTesting|template]]<br />
<br />
Create a new weekly agenda from the [[Template:MobileTesting|template]]:<br />
<br />
<createbox><br />
align=left<br />
type=create<br />
preload=Template:MobileTesting<br />
default={{#time: m_d_y | wednesday}}<br />
prefix=Mobile/Testing/<br />
<br />
</createbox><br />
# [[Mobile/Testing/08_15_18 | 08/15/18]]<br />
# [[Mobile/Testing/08_01_18 | 08/01/18]]<br />
# [[Mobile/Testing/07_18_18 | 07/18/18]]<br />
# [[Mobile/Testing/07_04_18 | 07/04/18]] Cancelled!<br />
# [[Mobile/Testing/06_20_18 | 06/20/18]]<br />
# [[Mobile/Testing/06_06_18 | 06/06/18]]<br />
# [[Mobile/Testing/05_23_18 | 05/23/18]]<br />
# [[Mobile/Testing/05_09_18 | 05/09/18]]<br />
# [[Mobile/Testing/04_25_18 | 04/25/18]]<br />
# [[Mobile/Testing/04_11_18 | 04/11/18]]<br />
# [[Mobile/Testing/03_28_18 | 03/28/18]]<br />
# [[Mobile/Testing/03_14_18 | 03/14/18]]<br />
# [[Mobile/Testing/02_28_18 | 02/28/18]]<br />
# [[Mobile/Testing/02_14_18 | 02/14/18]]<br />
# [[Mobile/Testing/01_31_18 | 01/31/18]]<br />
# [[Mobile/Testing/01_17_18 | 01/17/18]]<br />
<br />
Older<br />
# [[Mobile/Testing/Archive | Archive of notes from 2011 - 2017]]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing&diff=1199562Mobile/Testing2018-08-15T20:19:16Z<p>Gbrown: /* Notes */</p>
<hr />
<div>= About =<br />
<br />
We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).<br />
<br />
Note that the scope of this page is restricted to Firefox for Android (Fennec).<br />
<br />
== Documentation ==<br />
<br />
* [[Mobile/Testing/Architectural_Overview|Architectural Overview]] of our mobile automated testing systems.<br />
* For information on running the same tests we run in automation on your desktop, see the [https://wiki.mozilla.org/Mobile/Fennec/Android/Testing testing] for Android wiki page.<br />
<br />
= Status Meetings =<br />
Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.<br />
Details<br />
* Every other Wednesday @ 10:15am PST/PDT<br />
* Vidyo in Mobile (x9998, or 99998# from phone)<br />
* #mobile for back channel<br />
== Notes ==<br />
[[Template:MobileTesting|template]]<br />
<br />
Create a new weekly agenda from the [[Template:MobileTesting|template]]:<br />
<br />
<createbox><br />
align=left<br />
type=create<br />
preload=Template:MobileTesting<br />
default={{#time: m_d_y | wednesday}}<br />
prefix=Mobile/Testing/<br />
<br />
:snorp suggested using a google doc going forward...<br />
</createbox><br />
# [[Mobile/Testing/08_15_18 | 08/15/18]]<br />
# [[Mobile/Testing/08_01_18 | 08/01/18]]<br />
# [[Mobile/Testing/07_18_18 | 07/18/18]]<br />
# [[Mobile/Testing/07_04_18 | 07/04/18]] Cancelled!<br />
# [[Mobile/Testing/06_20_18 | 06/20/18]]<br />
# [[Mobile/Testing/06_06_18 | 06/06/18]]<br />
# [[Mobile/Testing/05_23_18 | 05/23/18]]<br />
# [[Mobile/Testing/05_09_18 | 05/09/18]]<br />
# [[Mobile/Testing/04_25_18 | 04/25/18]]<br />
# [[Mobile/Testing/04_11_18 | 04/11/18]]<br />
# [[Mobile/Testing/03_28_18 | 03/28/18]]<br />
# [[Mobile/Testing/03_14_18 | 03/14/18]]<br />
# [[Mobile/Testing/02_28_18 | 02/28/18]]<br />
# [[Mobile/Testing/02_14_18 | 02/14/18]]<br />
# [[Mobile/Testing/01_31_18 | 01/31/18]]<br />
# [[Mobile/Testing/01_17_18 | 01/17/18]]<br />
<br />
Older<br />
# [[Mobile/Testing/Archive | Archive of notes from 2011 - 2017]]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Template:MobileTesting&diff=1199561Template:MobileTesting2018-08-15T20:18:02Z<p>Gbrown: /* Bitbar */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Bitbar ==<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_15_18&diff=1199559Mobile/Testing/08 15 182018-08-15T17:58:45Z<p>Gbrown: /* Bitbar */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
* ...is dead: {{bug|1483346}}<br />
== Bitbar ==<br />
<br />
* {{Bug|1482957}} - Android hardware tests @ Bitbar no longer downloading minidump_stackwalk.<br />
* bc Investigating Taskcluster exceptions for mochitest-1 on recent remote automation changes.<br />
* {{bug|1483600}} - Raptor webext content script is not injected (blocking speedometer)<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** select test suites now running on mozilla-central only, tier 3:<br />
** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=android%20x86%207.0<br />
* can we enable marionette on release builds? {{bug|1426822}}<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_15_18&diff=1199527Mobile/Testing/08 15 182018-08-15T00:19:08Z<p>Gbrown: /* Autophone */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
* ...is dead: {{bug|1483346}}<br />
== Bitbar ==<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** select test suites now running on mozilla-central only, tier 3:<br />
** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=android%20x86%207.0<br />
* can we enable marionette on release builds? {{bug|1426822}}<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_15_18&diff=1199519Mobile/Testing/08 15 182018-08-14T20:58:23Z<p>Gbrown: /* Product Integrity */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
* {{bug|1483346}}<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** select test suites now running on mozilla-central only, tier 3:<br />
** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=android%20x86%207.0<br />
* can we enable marionette on release builds? {{bug|1426822}}<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_15_18&diff=1199518Mobile/Testing/08 15 182018-08-14T20:56:54Z<p>Gbrown: /* Product Integrity */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
* {{bug|1483346}}<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** select test suites now running on mozilla-central only, tier 3:<br />
** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=android%20x86%207.0<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_15_18&diff=1199517Mobile/Testing/08 15 182018-08-14T20:56:14Z<p>Gbrown: Created page with "= Previous Action Items = = Status reports = == Autophone == * {{bug|1483346}} == Dev team == == Product Integrity == * packet.net work in progress in {{bug|1460411}} ** s..."</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
* {{bug|1483346}}<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** select test suites now running on mozilla-central only, tier 3:<br />
** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=android%20x86%207.0<br />
** all plain mochitests and reftests failing: {{bug|1479584}}<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing&diff=1199516Mobile/Testing2018-08-14T20:52:12Z<p>Gbrown: /* Notes */</p>
<hr />
<div>= About =<br />
<br />
We use automated testing of our mobile browser to ensure that new releases get better, not worse. As with Firefox for Desktop, we are concerned both with testing the correctness of the browser (no crashes, pages should render correctly) and performance (new releases should be faster than old ones).<br />
<br />
Note that the scope of this page is restricted to Firefox for Android (Fennec).<br />
<br />
== Documentation ==<br />
<br />
* [[Mobile/Testing/Architectural_Overview|Architectural Overview]] of our mobile automated testing systems.<br />
* For information on running the same tests we run in automation on your desktop, see the [https://wiki.mozilla.org/Mobile/Fennec/Android/Testing testing] for Android wiki page.<br />
<br />
= Status Meetings =<br />
Interested parties meet every other week to discuss the current status of testing on Mobile and coordinate the required work between teams.<br />
Details<br />
* Every other Wednesday @ 10:15am PST/PDT<br />
* Vidyo in Mobile (x9998, or 99998# from phone)<br />
* #mobile for back channel<br />
== Notes ==<br />
[[Template:MobileTesting|template]]<br />
<br />
Create a new weekly agenda from the [[Template:MobileTesting|template]]:<br />
<br />
<createbox><br />
align=left<br />
type=create<br />
preload=Template:MobileTesting<br />
default={{#time: m_d_y | wednesday}}<br />
prefix=Mobile/Testing/<br />
</createbox><br />
# [[Mobile/Testing/08_15_18 | 08/15/18]]<br />
# [[Mobile/Testing/08_01_18 | 08/01/18]]<br />
# [[Mobile/Testing/07_18_18 | 07/18/18]]<br />
# [[Mobile/Testing/07_04_18 | 07/04/18]] Cancelled!<br />
# [[Mobile/Testing/06_20_18 | 06/20/18]]<br />
# [[Mobile/Testing/06_06_18 | 06/06/18]]<br />
# [[Mobile/Testing/05_23_18 | 05/23/18]]<br />
# [[Mobile/Testing/05_09_18 | 05/09/18]]<br />
# [[Mobile/Testing/04_25_18 | 04/25/18]]<br />
# [[Mobile/Testing/04_11_18 | 04/11/18]]<br />
# [[Mobile/Testing/03_28_18 | 03/28/18]]<br />
# [[Mobile/Testing/03_14_18 | 03/14/18]]<br />
# [[Mobile/Testing/02_28_18 | 02/28/18]]<br />
# [[Mobile/Testing/02_14_18 | 02/14/18]]<br />
# [[Mobile/Testing/01_31_18 | 01/31/18]]<br />
# [[Mobile/Testing/01_17_18 | 01/17/18]]<br />
<br />
Older<br />
# [[Mobile/Testing/Archive | Archive of notes from 2011 - 2017]]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/CommonTips&diff=1198337Mobile/Fennec/Android/CommonTips2018-08-01T16:53:08Z<p>Gbrown: /* Lint */</p>
<hr />
<div>== General coding tips ==<br />
<br />
=== Finding relevant code ===<br />
<br />
Almost all of the Fennec-specific code is in the mozilla-central source tree under:<br />
<br />
mobile/android/...<br />
<br />
Of particular interest to most new contributors will be:<br />
<br />
mobile/android/base/java/org/mozilla/gecko/BrowserApp.java # the main Android activity that starts when you open Fennec<br />
mobile/android/chrome/content/browser.js # the main JavaScript file that controls Gecko to make it do what we want<br />
<br />
If you are looking for something specific, use the code-search tool at http://dxr.mozilla.org/ to search for relevant pieces of code.<br />
<br />
=== Debugging ===<br />
For the JavaScript parts of the code base, [https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Firefox_for_Android remote debugging] with a desktop version of Firefox is the most effective way to see what is going on, add breakpoints, etc.<br />
<br />
For Android Java code, if you have IntelliJ set up, the debugger works well and allows you to set breakpoints, inspect variables and objects, etc.<br />
See http://developer.android.com/tools/debugging/ddms.html<br />
<br />
Debugging without an IDE is much more primitive, and you can try using Android Log statements and inspecting logcat.<br />
<br />
[http://developer.android.com/guide/developing/tools/logcat.html logcat] is a tool that is going to show you some logs prompted by the device. It might be a good help if you don't want to or can't run gdb. You can use it by running this command: <br />
<br />
adb logcat -v time<br />
<br />
You can make things appear in logcat using printf_stderr. With debug builds, NS_WARNING, NS_ERROR and NS_ASSERTIONS will show up in logcat. If you're trying to debug something, you may wish to pipe the logcat output through grep to filter out irrelevant things (most Fennec-related output will be viewable by <br />
<br />
adb logcat -v time | grep Gecko<br />
<br />
but remember that when attaching log output to a bug you should include unfiltered output as there may be relevant log entries under other tags.<br />
<br />
If you want more debugging tools, [[Mobile/Fennec/Android/AdvancedTopics#Advanced_Debugging|Advanced Debugging]] is a good place to look.<br />
<br />
=== Coding Style ===<br />
<br />
The Java style follow the [https://developer.mozilla.org/en-US/docs/Developer_Guide/Coding_Style#Java_practices Mozilla Coding Style].<br />
<br />
You can run `./mach gradle checkstyle` to automatically catch some style issues locally.<br />
<br />
==== Common nits in Reviews ====<br />
Check for these nits before asking for review for a patch:<br />
* Remove trailing whitespace<br />
* Spacing<br />
** Four spaces for Java indent (for main Fennec code - [https://github.com/mozilla-services/android-sync android-sync github project] has different spacing)<br />
** Around operators (+, -, etc.) and :<br />
** Between comment "//" and text<br />
* Braces for Java if statements, even if they are one line<br />
* Comments should be full sentences (capitalization, punctuation)<br />
** Good comments are useful and clear even for someone reading a particular area of code for the first time<br />
* Avoid using single-letter variables in almost all cases - it makes code harder to read<br />
<br />
== Strings==<br />
<br />
=== Add an Android string resource ===<br />
<br />
# Add an XML ''string entity'' to <tt>mobile/android/base/locales/en-US/android_strings.dtd</tt> with the English string you want to add.<br />
# Add a <tt>&lt;string&gt;</tt> element to <tt>mobile/android/base/strings.xml.in</tt>.<br />
# Build using <tt>mach build mobile/android/base</tt> or your IDE.<br />
<br />
Your new string should appear in <tt>org.mozilla.gecko.R.string</tt>. Here's an [https://hg.mozilla.org/mozilla-central/rev/861e4bd9e7fe example patch] that adds the single string ''pref_private_data_syncedTabs''.<br />
<br />
Why is this necessary? Mozilla's main localization process is based on XML entities. Localization teams localize the XML entity definitions using existing tools, but they do not see <tt>strings.xml</tt> itself. Building prepares the final <tt>strings.xml</tt> for use. You can be sure your changes are in place by finding your entity and string in <tt>$OBJDIR/mobile/android/base/res/values/strings.xml</tt>.<br />
<br />
=== Modify an existing Android string resource ===<br />
<br />
# Find the relevant <tt>&lt;string&gt;</tt> element in <tt>mobile/android/base/strings.xml.in</tt>.<br />
# Find the underlying English ''string entity'' in <tt>mobile/android/base/locales/en-US/android_strings.dtd</tt>. Usually, you'll see <tt>&lt;string&gt;&amp;string_entity;&lt;string&gt;</tt>; the string entity is ''string_entity''.<br />
# If your change is just fixing a typo (spelling error, capitalization, whitespace), just update the <tt>android_strings.dtd</tt>.<br />
# If you are ''really'' changing the string, you '''also need to change the entity name'''. It's traditional to add or increment a trailing integer, like ''string_entity2''.<br />
# Build using <tt>mach build mobile/android/base</tt> or your IDE.<br />
<br />
Your updated string should appear. Here's an [https://hg.mozilla.org/mozilla-central/rev/1e8c1b79132f example patch] that changes several strings, including renaming ''tab_queue_notification_text_singular'' to ''tab_queue_notification_text_singular2''.<br />
<br />
Why is this necessary? Mozilla's localization process only recognizes new string entities, not modified string entities. (The old, unused entity is automatically ignored and eventually deleted.)<br />
<br />
== Testing ==<br />
<br />
=== Add a new Robocop test ===<br />
<br />
# Add a Java test file named like <tt>mobile/android/tests/browser/robocop/testMyThing.java</tt>. This will get your test compiled into the Robocop APK.<br />
# Add your test file/name section like ''[testMyThing]'' to <tt>mobile/android/tests/browser/robocop/robocop.ini</tt>. Without this, the Robocop test harness will not know about your test!<br />
# Optionally add HTML, JS, and CSS resources to folder <tt>mobile/android/tests/browser/robocop</tt>.<br />
# Run <tt>mach build mobile/android</tt> to get them built/installed before running your test.<br />
# Make sure you have host binaries downloaded and configured, following [[Mobile/Fennec/Android#Host_Builds_.28MOZ_HOST_BIN.29|Host Builds (MOZ_HOST_BIN)]].<br />
# Run <tt>mach robocop testMyThing</tt> to run your new test on your device.<br />
# Iterate!<br />
<br />
Here's an [https://hg.mozilla.org/mozilla-central/rev/101feffdaed8 example patch] that adds a fairly complicated new ''testSelectionCarets'' test.<br />
<br />
See also some tips on writing [[Mobile/Fennec/Android/UITest|UITest]]s, or you can read about [[Mobile/Fennec/Android/Testing|Testing on Firefox for Android]].<br />
<br />
== Other ==<br />
<br />
=== Copy a profile from your phone ===<br />
<br />
E.g., to explore with <tt>sqlite3</tt>. Use [https://addons.mozilla.org/en-US/android/addon/copy-profile/ Copy Profile].<br />
<br />
=== Add new Android ids that identify resources that are only included in some builds ===<br />
<br />
If you add a resource that you refer to by ID in code, your build will fail if that resource isn't present in a build that excludes that resource. You'll typically see this when adding a resource that's only used in v11+ builds, which aren't included in resource-constrained builds for Gingerbread. <br />
<br />
For example, you'll see log output like:<br />
<br />
<pre><br />
14:15:27 INFO - /builds/slave/…/build/src/mobile/android/base/preferences/GeckoPreferenceFragment.java:122: error: cannot find symbol<br />
14:15:27 INFO - return R.id.pref_header_general;<br />
</pre><br />
<br />
To fix this, you should should add a stub for the new id in <tt>mobile/android/base/resources/values/ids.xml</tt>.<br />
<br />
You should also make sure that code that needs a valid resource id is behind a version check.<br />
<br />
=== Add a new table to browser.db ===<br />
To follow along with code, see [https://bugzilla.mozilla.org/show_bug.cgi?id=1250707 Bug 1250707].<br />
==== Prepare migration tests ====<br />
*Capture a current version of the database (before your changes!) with some browsing data from your device using the Copy Profile add-on. Put this in <tt>mobile/android/tests/browser/robocop/assets/browser_db_upgrade/v<#>.db</tt>. It is used to test your upgrade changes in <tt>testBrowserDatabaseHelperUpgrades</tt>. The test should automatically pick up your new version.<br />
<br />
====Create the table and query & insert via BrowserProvider====<br />
*Add your table name & schema as a new <tt>static</tt> class in <tt>BrowserContract</tt>. It will likely need the <tt>@RobocopTarget</tt> annotation to be accessible for testing<br />
*Increment the <tt>DATABASE_VERSION</tt> in <tt>BrowserDatabaseHelper</tt> and update the related bug # comment<br />
*Implement a create method to create your table<br />
**Add a call to it in <tt>BrowserDatabaseHelper.onCreate</tt><br />
**Add a method to upgrade the database from the old version to the new version (which likely calls your create table method) and add it to <tt>BrowserDatabaseHelper.onUpgrade</tt><br />
*Register your new table with <tt>BrowserProvider</tt> by adding your table to the <tt>URI_MATCHER</tt> in the <tt>BrowserProvider</tt> constructor (pattern match the other values). Note the <tt>TABLE_*</tt> constant, the constants assigned to <tt>int</tt>s, and the <tt>*_PROJECTION_MAP</tt> constant.<br />
*Add to the <tt>BrowserProvider.query</tt> method (pattern match!) by setting the projection map and table in the <tt>SQLiteQueryBuilder</tt> object.<br />
*Add to the <tt>BrowserProvider.insertInTransaction</tt> method (pattern match!) by calling out to your own method to insert into the DB<br />
<br />
====Test insertions via BrowserProvider====<br />
*Add a <tt>TestInsert* extends TestCase</tt> class with a <tt>test</tt> method. Insert a value, query, and make sure you receive what you think you should! <tt>TestInsertUrlAnnotations</tt> should provide a guide on how to write an extensible test that allows you to more easily test additions later on in this guide.<br />
*Register your test to be run in <tt>TestBrowserProvider.setUp</tt><br />
<br />
====Add a class to abstract querying your table from within Fennec====<br />
So instead of accessing the content provider directly, you can call, <tt>YourTable.insert(url, title);</tt>.<br />
<br />
Alternatively, add these methods directly to <tt>LocalBrowserDB</tt> & friends if your queries overlap the queries there or share data (e.g. table join). This method won't be discussed here.<br />
*Add a new interface for your class in <tt>omg.db</tt>. It should contain an <tt>insert*</tt> method with a <tt>ContentResolver</tt> arg and whatever else you want to insert. This method will likely need <tt>@RobocopTarget</tt><br />
*Add a class implementing this interface. The current pattern for the class name is <tt>Local*</tt> (like <tt>LocalBrowserDB</tt>). The insert method should use the <tt>ContentResolver</tt> to talk insert via the <tt>BrowserProvider</tt> methods you previously implemented.<br />
*Add a <tt>getTableName</tt> method to the <tt>BrowserDB</tt> interface. It will like need <tt>@RobocopTarget</tt>.<br />
*Construct your class in the constructor of <tt>LocalBrowserDB</tt> and store it in a member variable. Add the implementation of your getter from above (also <tt>@RobocopTarget</tt>).<br />
*Add <tt>Stub* implements <your-interface></tt> to <tt>StubBrowserDB</tt> and have the implementation do nothing. Construct your <tt>Stub*</tt> with your member variable declaration and add the getter.<br />
*Don't forget to add your new files to <tt>moz.build</tt><br />
<br />
====Test the class to query via the ContentResolver====<br />
*Add to the test method you previously wrote in <tt>testBrowserProvider</tt>. Follow <tt>TestInsertUrlAnnotations</tt> for an extensible pattern.<br />
<br />
====Next steps====<br />
You probably want an <tt>update</tt> and <tt>delete</tt> method too... get to it! :P<br />
<br />
=== Updating search engine icons ===<br />
[[Mobile/Fennec/Android/Updating search engine icons]]<br />
<br />
=== Updating the SDK on the builders ===<br />
[[Mobile/Fennec/Android/Updating SDK on builders]]</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android&diff=1198336Mobile/Fennec/Android2018-08-01T16:49:49Z<p>Gbrown: /* Test results */</p>
<hr />
<div>{{Admon/important|The Fennec build documentation has moved!|See the most up to date documentation at https://developer.mozilla.org/en-US/docs/Simple_Firefox_for_Android_build}}<br />
<br />
Here is a table of contents of all the in-depth information you might need to find about Firefox for Android development.<br />
<br />
New to the community? Welcome! [[Mobile/Get_Involved|Start here!]]<br />
<br />
== Develop ==<br />
*[[Mobile/Get Involved|New contributor page]]<br />
**[[Mobile/Fennec/Android/Suggested workflow|Suggested workflow]]<br />
*[https://developer.mozilla.org/en-US/docs/Simple_Firefox_for_Android_build Build documentation]<br />
*[[Mobile/Fennec/Android/Testing|Testing]]<br />
*[[Mobile/Fennec/Android/CommonTips|Common tips & how-to's]]<br />
*[[Mobile/Fennec/Android/Multilocale_Builds|Multilocale Builds]] - how to build an apk containing multiple languages (instead of just en-US by default).<br />
*[[Mobile/Fennec/Android/Development/Addons|Useful addons for development (e.g. copy profile)]]<br />
*[[Mobile/Fennec/Android/AdvancedTopics|Advanced topics (e.g. special build configs & advanced debugging)]]<br />
*[https://gecko.readthedocs.org/en/latest/mobile/android/fennec/index.html In-tree Firefox for Android documentation]<br />
*[[Mobile/Triage|Triage]]<br />
*[[Mobile/Metrics|Metrics]]<br />
*[[Mobile/Fennec/Android/png_optimisation|png optimisation]] - a guide on how to prepare png and raster images for landing in the tree, in order to minimise their size. This includes details on webp conversion, when applicable.<br />
<br />
=== App and development overview ===<br />
* [[Mobile/Fennec/Android/App_Structure|App Structure]]: Fennec is a combination of Java frontend code, Javascript glue and display code, and C++ rendering code. Here's a brief 9000m (30'000ft) overview of what each of those parts does.<br />
** [[Mobile/Fennec/Android/Build_Systems|Build Systems]]: Our app structure results in a complex build process. Moreover, we are in the process of migrating our builds from a custom (Makefile based) build, to using Gradle for the frontend/Java portions of our app. This is a brief description of our parallel build systems and what we're trying to achieve in the future.<br />
<br />
=== Feature development ===<br />
* [[Mobile/Distribution_Files|Distribution files]]<br />
* [[Mobile/Fennec/Android/Java_telemetry|Java telemetry]]<br />
* [[Mobile/Fennec/Android/Switchboard|Switchboard]]<br />
* [[Mobile/Fennec/Android/Downloadable_Content|Downloadable Content]]<br />
<br />
=== Test results ===<br />
*[https://scan.coverity.com/projects/firefox-mobile Coverity static analysis]<br />
<br />
=== Build infrastructure ===<br />
*[[Mobile/Fennec/Android/Task Cluster notes|Task Cluster notes]]<br />
*[[Mobile/Fennec/Android/mach_bootstrap_SDK_dependencies|`./mach bootstrap` SDK dependencies]]<br />
<br />
=== General developer resources ===<br />
*[https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html Mercurial for Mozillians]<br />
*[https://mozilla-version-control-tools.readthedocs.org/en/latest/mozreview.html MozReview]<br />
<br />
=== Crash Stats ===<br />
* The crash-stats page lives at https://crash-stats.mozilla.com/home/product/FennecAndroid<br />
** Be careful to select "'''FennecAndroid'''" under the product dropdown to see Firefox on Android crashes.<br />
*** Nightly has the name XX.0a1 (e.g. 52.0a1)<br />
*** Aurora has the name XX.0a2 (e.g. 51.0a1 - the number is one lower than nightly)<br />
*** Beta is XX.0bN (e.g. 50.0b12). N is increased every time a new beta is released (usually weekly).<br />
**** '''Note:''' Multiple beta versions can be listed under the versions dropdown, the first one listed might not be the currently released beta.<br />
*** Release is XX.0.N (e.g. 49.0.2). N is increased every time there is a dot release, we usually try to avoid dot releases.<br />
** Beware: a single device (which potentially has a hardware issue and/or a user who has done something strange with their configuration) can result in a crash-spike on nightly, or even aurora - not every crash is something significant.<br />
** You can view devices that are affected by selecting a crash-signature, then going to "Aggregations", followed by clicking on the "Aggregate on" dropdown and selecting "Android device". Some issues might be device or manufacturer specific.<br />
** To create a bugzilla entry for a given crash, open a crash report (if you are viewing a signature, go to "reports" and click on one of the items there). From the crash report search for "Bugzilla - Report this bug in" and select the appropriate module.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1198335Mobile/Fennec/Android/Testing2018-08-01T16:48:16Z<p>Gbrown: /* Eideticker */</p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.<br />
<br />
mach build mobile/android/tests/browser/robocop<br />
mach robocop<br />
<br />
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)<br />
<br />
Notes:<br />
* Mach is self documenting! For help, try <tt>mach help robocop</tt>.<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 4.2 x86 image:<br />
<br />
./mach android-emulator --version x86<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is much faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For the '''very''' lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Testing with stock AVD images (Android 4.x+ only) ===<br />
<br />
Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.<br />
<br />
# From the Android Virtual Device (AVD) Manager, click 'Create....'<br />
# Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).<br />
# Choose a system image with API level 16 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.<br />
# Next, check 'Use Host GPU' if possible. See [http://developer.android.com/tools/devices/emulator.html#acceleration Using Hardware Acceleration].<br />
# Hit Finish and that should be it. If you have setup Firefox for Android to [[Mobile/Fennec/Android/IDEs|run on your IDE]], you should be able to launch the emulator straight from there as well.<br />
<br />
=== Multiple emulators ===<br />
<br />
It's also possible to test with multiple emulators by correctly setting the ANDROID_SERIAL environment variable to the device ID seen in ''adb devices'':<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.<br />
<br />
wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py<br />
python getxre.py -d $DIR<br />
export MOZ_HOST_BIN=$DIR/bin<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== S1/S2 Automation ==<br />
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times. <br />
<br />
S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/<br />
<br />
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2<br />
<br />
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests<br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Fennec/Android/Testing&diff=1198334Mobile/Fennec/Android/Testing2018-08-01T16:44:40Z<p>Gbrown: /* For the impatient... */</p>
<hr />
<div>This page has instructions for running Firefox tests locally on a device (Android phone, tablet, or emulator) of your choice.<br />
<br />
Having trouble? Ping :gbrown on #mobile, or ask for help on #ateam.<br />
<br />
== For the impatient... ==<br />
mach commands allow most test suites to be run easily on Android, just like on desktop. These commands explicitly support Firefox for Android:<br />
<br />
mach robocop<br />
mach mochitest<br />
mach reftest<br />
mach crashtest<br />
mach jstestbrowser<br />
mach xpcshell-test<br />
mach cppunittest<br />
mach geckoview-junit<br />
<br />
They all run against a connected Android device using your Firefox for Android build. Don't have a device? These commands will offer to start an emulator.<br />
<br />
=== Quick reference ===<br />
As a front-end dev, the following tests will be regularly useful:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Robocop<br />
| rc*<br />
| N<br />
| <tt>./mach robocop</tt><br />
| On-device UI tests<br />
| [[#robocop UI tests|link]]<br />
|-<br />
| JUnit4 tests<br />
| test (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:test</tt> [0]<br />
| Java unit test suite<br />
| [[#JUnit4 tests|link]]<br />
|-<br />
| integration<br />
| N/A<br />
| N/A<br />
| via IDE<br />
| On-device integration tests<br />
| [[#integration tests|link]]<br />
|}<br />
<br />
As well as the following static analysis tools:<br />
{| class="wikitable sortable"<br />
|-<br />
! Name<br />
! Results name (TH)<br />
! Auto?<br />
! Local command<br />
! Description<br />
! More<br />
|-<br />
| Checkstyle<br />
| checkstyle (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:checkstyle</tt><br />
| Reports style violation in Java code<br />
| [[#checkstyle|link]]<br />
|-<br />
| Android Lint<br />
| lint (tier 2)<br />
| Y<br />
| <tt>./mach gradle app:lint</tt><br />
| Detects common errors in Android code & resources<br />
| [[#Android Lint|link]]<br />
|-<br />
| eslint<br />
| ES (lint opt: tier 2)<br />
| Y<br />
| <tt>./mach eslint mobile/android</tt> [1]<br />
| Checks for JS errors (e.g. syntax errors)<br />
| [[#eslint|link]]<br />
|}<br />
<br />
Key:<br />
* <b>TH</b> stands for Treeherder<br />
* <b>Auto?</b> refers to jobs that run automatically on Treeherder when the associated files change. For more info, see [[Mobile/Fennec/Testing/Understanding_treeherder#Automatic_tasks|the docs on automatic tasks]].<br />
* <b>More</b> links to more information regarding a test<br />
<br />
[0]: (5/24/16): There are packages to install to get most of the tests to pass locally. After that, there is still one local test failure that does not appear in automation. See [[#JUnit4 tests]].<br><br />
[1]: You must first run a setup command – see [[#eslint]]<br />
<br />
== Test Environment ==<br />
When testing Firefox for Android with mach on your local computer, tests run on an Android device, but are controlled by a test harness running on your computer. The test environment usually consists of:<br />
* a "host" computer, running Linux or OSX<br />
* a usb-connected Android device, such as a phone or tablet, or an Android emulator running on your computer<br />
* a Firefox for Android build, including an apk<br />
* "host utilities" -- xpcshell, ssltunnel, and like binaries built for the host platform<br />
* a "device manager" to communicate with the Android device<br />
* a TCP/IP network connection between host and device<br />
<br />
A test harness (typically written in python) runs on the host computer. The harness uses the device manager to communicate with the Android device (by default, the adb device manager is used, which uses the adb command from the Android SDK). "Browser tests" like robocop, mochitest, and reftest run in the browser, so Firefox for Android must be installed on the device before starting the test. To serve remote content to browser tests, the harness runs xpcshell and other utilities on the host while the tests are running. Tests may load content from the host, so a network connection between host and device is essential.<br />
<br />
Running tests with mach simplifies environment concerns significantly:<br />
* If a single phone or tablet is connected to your computer and visible with "adb devices", that device will be used automatically. If an emulator is running and visible with "adb devices", that device will be used automatically. If no device is visible to "adb devices", mach will offer to start an emulator.<br />
* If Firefox for Android is not installed on the device, mach will offer to install Firefox.<br />
* If the MOZ_HOST_BIN environment variable points to a directory containing xpcshell, that directory will be used for host utilities; otherwise, mach will offer to download and setup host utilities for you.<br />
<br />
== Front-end-centric ==<br />
(In the interest of removing duplication) For basic details & run instructions, see [[#Quick reference]].<br />
<br />
=== JUnit4 tests ===<br />
* Runs on [http://robolectric.org/ Robolectric], which mocks various Android libraries so you can write unit tests for Android libraries that would ordinarily have to be run on device<br />
* Supports [http://mockito.org/ Mockito] for custom mocking (see [http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/background/junit4/src/org/mozilla/gecko/dlc/TestVerifyAction.java TestVerifyAction for a sample]).<br />
* Run specific tests from the IDE: right-click on the test class you want to run, and select the "Run <test-class>" option.<br />
<br />
<br />
'''Troubleshooting'''<br />
* Some tests will fail with encryption & connection errors if you do not install the appropriate packages: https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-September/001499.html Note that this package is also available in brew cask: `jce-unlimited-strength-policy`. You may need to wait/restart after installing for it to take effect. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1271000#c8 bug 1271000 comment 8] for further discussion.<br />
* After installing the policies, there is still a local test failure in TestJSONWebTokenUtils.testDSAGeneration that does not appear in automation ([https://bugzilla.mozilla.org/show_bug.cgi?id=1275423 bug 1275423]).<br />
<br />
=== integration tests ===<br />
* Run from the IDE: You can run specific tests locally in the IDE by selecting the "Build Variants" menu (bottom left), changing "Test Artifact" to "Android Instrumentation Tests", right-clicking on the test class you want to run, and selecting the "Run <test-class>" option.<br />
* There is no way to run these tests from the command line<br />
* These do not run in automation so junit tests (via robolectric) or robocop tests (which are integration tests with a UI component) are generally preferred.<br />
<br />
=== robocop UI tests ===<br />
[[Auto-tools/Projects/Robocop|General Robocop information]] and http://mxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/README.rst.<br />
<br />
The Robocop test suite verifies UI behavior in Firefox for Android by pointing-and-clicking through the UI on a running device or emulator. It is built on the Robotium testing framework. To run tests locally, a separate Robocop test APK also needs to be installed.<br />
<br />
To run robocop tests, first build and install Firefox for Android, <br />
<br />
mach build<br />
mach package<br />
mach install<br />
<br />
Next, execute <tt>mach robocop</tt> which installs the Robocop APK to the device and starts testing the entire test suite. <br />
<br />
mach robocop<br />
<br />
or<br />
<br />
mach robocop <test-name><br />
<br />
If you make changes to the tests and want to see them run on a device, you need to build the tests again and reinstall.<br />
<br />
mach build mobile/android/tests/browser/robocop<br />
mach robocop<br />
<br />
This builds the tests in <tt>mobile/android/tests/browser/robocop</tt> and then installs the debug-signed Robocop APK onto the device. (Building <tt>mobile/android</tt> also builds the tests within <tt>mobile/android/tests</tt>.)<br />
<br />
Notes:<br />
* Mach is self documenting! For help, try <tt>mach help robocop</tt>.<br />
* To run one test at a time, find the test name (like "testLoad") in <tt>mobile/android/tests/browser/robocop/robocop.ini</tt> and pass it as an argument, like: <tt>mach robocop testLoad</tt>.<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
*Additional tips at [[Auto-tools/Projects/Robocop#Frequently_found_errors]]<br />
<br />
=== Static analysis ===<br />
There are some other tools to be found at the [[Mobile/Fennec/Android/Testing/Not_yet_in_common_use|Not yet in common use page]].<br />
<br />
==== checkstyle ====<br />
* Run it in Intellij/Android Studio with [[Mobile/Fennec/Android/Static_analysis/checkstyle-idea|checkstyle-idea]].<br />
* '''example checks?''' [http://checkstyle.sourceforge.net/google_style.html Google's style guide]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/checkstyle.xml mobile/android/app/checkstyle.xml]<br />
* '''open issues?''' Issues we care about are blocking the meta<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
<br />
==== Android Lint ====<br />
* '''example checks?''' [https://sites.google.com/a/android.com/tools/tips/lint-checks `lint --show`]<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml mobile/android/app/lint.xml]<br />
* '''open issues?''' Run locally for a complete list.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=1170283 meta bug]<br />
* If you fix all issues for a warning, modify [http://mxr.mozilla.org/mozilla-central/source/mobile/android/app/lint.xml lint.xml] to change your warning into an error!<br />
<br />
==== eslint ====<br />
To use eslint, you must first set it up:<br />
./mach eslint --setup # run once, or if the command breaks for some reason<br />
./mach eslint mobile/android<br />
<br />
* '''example checks?''' http://eslint.org/docs/rules/<br />
* '''config?''' [http://mxr.mozilla.org/mozilla-central/find?string=.eslint&tree=mozilla-central&hint=mobile%2F mobile/*.eslintrc] & [http://mxr.mozilla.org/mozilla-central/source/.eslintignore root .eslintignore]<br />
* '''open issues?''' Open a .eslintrc and enable checks.<br />
* '''meta bug?''' [https://bugzilla.mozilla.org/show_bug.cgi?id=939329 meta bug]<br />
* If you fix all issues for a warning, modify the appropriate .eslintrc to change your warning into an error!<br />
<br />
=== Other automation tasks ===<br />
==== android-api-15-gradle-dependencies ====<br />
This job is used to get our gradle and application dependencies in a format usable by the builders. See [https://gecko.readthedocs.io/en/latest/build/buildsystem/toolchains.html#firefox-for-android-with-gradle readthedocs] for more info.<br />
<br />
== mochitest (plain and chrome) ==<br />
[https://developer.mozilla.org/en/docs/Mochitest General mochitest info]<br />
[https://developer.mozilla.org/en-US/docs/Chrome_tests General mochitest-chrome info]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running tests:<br />
mach mochitest --flavor plain|chrome<br />
OR mach mochitest <test-dir><br />
OR mach mochitest <test-dir>/<test-name><br />
<br />
Use "find -name mochitest.ini" to find valid test directories for mochitest plain.<br />
<br />
Use "find -name chrome.ini" to find valid test directories for mochitest chrome.<br />
<br />
== xpcshell ==<br />
[https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests General xpcshell test information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. xpcshell tests do not require Firefox for Android to be installed, but the APK must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run all tests referenced by the master xpcshell manifest:<br />
<br />
mach xpcshell-test<br />
<br />
To run a subset of tests in the specified directory:<br />
<br />
mach xpcshell-test <test-directory><br />
OR mach xpcshell-test <test-directory>/<test-name><br />
<br />
Once either of the xpcshell-test commands has completed successfully, all test files have been copied to device, and it is then possible to repeat a single test quickly without setup:<br />
<br />
mach xpcshell-test <test-directory> --no-setup<br />
OR mach xpcshell-test <test-directory>/<test-name> --no-setup<br />
<br />
Notes:<br />
* A rooted device is recommended, but may not be required. Test harnesses may need to kill processes, copy and delete files, or perform other operations which may require special permissions on some devices.<br />
* Setup can take several minutes! Setup is faster if unzip is available on the remote device; if your device does not have unzip, try installing busybox.<br />
<br />
== cppunittests ==<br />
[https://developer.mozilla.org/en/docs/Compiled-code_automated_tests General cppunit test information]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built: mach build && mach package. cppunit tests do not require Firefox for Android to be installed, but the unit test executables must exist on the host.<br />
* Ensure your device is connected and visible with "adb devices".<br />
<br />
To run a single compiled code test:<br />
<br />
mach cppunittest <test><br />
<br />
For example,<br />
<br />
mach cppunittest <absolute-path-to-objdir>/xpcom/tests/TestTimers<br />
<br />
To run all the compiled code tests listed in the master cppunittests.ini manifest:<br />
<br />
mach cppunittest<br />
<br />
Notes:<br />
* Specifying the test by relative path is difficult; specify an absolute path to the test binary.<br />
* It is not possible to run all tests in a directory.<br />
* All files are copied to /data/local/tests by default. On some devices, you may need to create /data/local/tests and make it world writable.<br />
<br />
== reftests (and crashtests and js-reftests) ==<br />
[https://developer.mozilla.org/en/docs/Creating_reftest-based_unit_tests General reftest information.]<br />
<br />
Pre-requisites:<br />
* Ensure that Firefox for Android has been built and installed: mach build && mach package && mach install<br />
* Ensure your device is connected and visible with "adb devices".<br />
* '''Ensure that the device and host machine are on the same network''' <-- This is important. The device and the host need to communicate over the network to run the tests. Having the device connected to the host via USB is '''not''' sufficient.<br />
<br />
Running reftests:<br />
mach reftest<br />
OR mach reftest <test-dir><br />
OR mach reftest <test-dir>/<test-name><br />
<br />
Use "find -name reftest.list" to find valid test directories.<br />
<br />
Running crashtests:<br />
mach crashtest<br />
OR mach crashtest <test-dir><br />
OR mach crashtest <test-dir>/<test-name><br />
<br />
Use "find -name crashtest.list" to find valid test directories.<br />
<br />
Running js-reftests:<br />
mach jstestbrowser<br />
<br />
Notes:<br />
<br />
* There are many reftests; trying to run them all at once is not recommended (takes a long time, may exhaust memory).<br />
<br />
== Running tests on the Android emulator ==<br />
<br />
The "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests on treeherder run in an Android ARM emulator. "Android 4.2 x86 opt" tests run in an Android x86 emulator. For best results reproducing test failures, try server is recommended: '''Running the same tests on the same emulator on different host hardware may produce different results'''.<br />
<br />
Still, if you want to run the emulator locally, using the same Android image used for tests on treeherder, it is simple:<br />
<br />
./mach android-emulator --version 4.3<br />
<br />
That 'android-emulator' command will download the Android 4.3 API16+ Android image from tooltool, install it, and launch the Android emulator using all the same parameters used for tests on treeherder. (The Android SDK must be installed locally. mach will try to find the emulator binary in your $PATH environment variable, via the $ANDROID_SDK_ROOT environment variable, through your Android build configuration, and finally in the default location used by 'mach bootstrap'.)<br />
<br />
To use the Android 4.2 x86 image:<br />
<br />
./mach android-emulator --version x86<br />
<br />
(On Linux, the x86 emulator requires that kvm is installed. The resulting emulator is much faster than the arm emulator.)<br />
<br />
The first time you run an emulator with any particular version, it may take several minutes to download and install the image; subsequent runs will be '''much''' faster.<br />
<br />
If you want to "reset" an image (throw away any installed apks and/or settings changes):<br />
<br />
./mach android-emulator --force-update<br />
<br />
will discard the previous image and download a new one.<br />
<br />
Once an emulator is running, you can run tests against it just like any other Android device. For example:<br />
<br />
./mach android-emulator && ./mach install && ./mach mochitest<br />
<br />
For the '''very''' lazy, most mach test commands check that an Android device is connected; if not, they suggest running an emulator. Similarly, if tests are requested on a device that doesn't have Firefox installed, mach will offer to install it. So if you just run "mach mochitest" without a phone connected and without an emulator running, you might get:<br />
<br />
$ ./mach mochitest testing/mochitest/tests/Harness_sanity<br />
No Android devices connected. Start an emulator? (Y/n) y<br />
Starting emulator running Android 4.3...<br />
It looks like Firefox is not installed on this device.<br />
Install Firefox? (Y/n) y<br />
Installing Firefox. This may take a while...<br />
From _tests: Kept 36271 existing; Added/updated 0; Removed 0 files and 0 directories.<br />
######<br />
### Now running mochitest-plain.<br />
######<br />
0:01.36 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests<br />
0:01.67 LOG: MainThread INFO Checking for orphan ssltunnel processes...<br />
0:01.76 LOG: MainThread INFO Checking for orphan xpcshell processes...<br />
0:01.82 SUITE_START: MainThread 23<br />
0:01.82 TEST_START: MainThread testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPermissions.html<br />
...<br />
<br />
=== Testing with stock AVD images (Android 4.x+ only) ===<br />
<br />
Testing with the above configurations is ideal, but in some cases you may prefer to use stock AVD device images from the Nexus profiles available.<br />
<br />
# From the Android Virtual Device (AVD) Manager, click 'Create....'<br />
# Choose a Nexus phone or tablet device configuration. (Nexus 9 for tablets, Nexus 5 for phones is recommended).<br />
# Choose a system image with API level 16 or 18. (If you're not sure if you want arm or x86 then you probably want arm). If no suitable image is available, one can be installed with the Android SDK Manager.<br />
# Next, check 'Use Host GPU' if possible. See [http://developer.android.com/tools/devices/emulator.html#acceleration Using Hardware Acceleration].<br />
# Hit Finish and that should be it. If you have setup Firefox for Android to [[Mobile/Fennec/Android/IDEs|run on your IDE]], you should be able to launch the emulator straight from there as well.<br />
<br />
=== Multiple emulators ===<br />
<br />
It's also possible to test with multiple emulators by correctly setting the ANDROID_SERIAL environment variable to the device ID seen in ''adb devices'':<br />
$ adb devices<br />
List of devices attached<br />
emulator-5554 device<br />
emulator-5556 device<br />
<br />
To make Robocop (for example) run on ''emulator-5556'':<br />
$ mach robocop --deviceSerial=emulator-5556 # Runs your tests as normal<br />
<br />
== Host Builds (MOZ_HOST_BIN) ==<br />
<br />
Android mochitests and reftests are driven by test suites on a ''host'' machine running [https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell xpcshell]. The Android device being driven is referred to as the ''target'' device. The test suite locates xpcshell on the host machine via the environment variable '''MOZ_HOST_BIN''', which must point to the directory that contains the <tt>xpcshell</tt> binary (executable on the host machine), its associated executables (<tt>certutil</tt>, <tt>pk12util</tt>, <tt>ssltunnel</tt>, etc), and its shared libraries.<br />
<br />
'''When mach is used to run tests and MOZ_HOST_BIN has not been set, mach will download and setup host utilities for you. It is best to use that: don't set MOZ_HOST_BIN and let mach take of it.<br />
'''<br />
If a change to the host utilities is required, follow [[Packaging Android host utilities|these instructions]].<br />
<br />
=== Advanced setup ===<br />
<br />
Alternatively, if one of the prebuilt host utilities packages is not appropriate for you or does not work, you can fetch the host utils by using the '''getxre''' utility that comes with [[Mobile/Fennec/Android/GDB|JimDB]]. This utility will download the correct binaries for your host machine, and set the appropriate permissions. You should prefer the prebuilt host utilities because they are more likely to be tested and are significantly smaller downloads than the intermediate packages downloaded by <tt>getxre</tt>. (<tt>getxre</tt> does what the [[Packaging_Android_host_utilities|packaging instructions]] describe, except it has not been updated for current Firefox package structure on Mac OS X.) In the following stanza, replace <tt>$DIR</tt> with an output directory, such as <tt>~/.mozbuild/host-utils</tt>.<br />
<br />
wget https://github.com/darchons/android-gdbutils/raw/master/python/getxre.py<br />
python getxre.py -d $DIR<br />
export MOZ_HOST_BIN=$DIR/bin<br />
<br />
Alternatively, you can build desktop Firefox with a mozconfig which might be as simple as:<br />
<br />
ac_add_options --enable-application=browser<br />
mk_add_options MOZ_OBJDIR=./objdir-desktop<br />
<br />
Then execute (note that MOZ_HOST_BIN must specify an absolute path):<br />
<br />
MOZCONFIG=mozconfig.desktop ./mach build<br />
export MOZ_HOST_BIN=/path/to/objdir-desktop/dist/bin<br />
<br />
On Linux, the path to that build may also need to be in your LD_LIBRARY_PATH, unless your LD_LIBRARY_PATH contains ".":<br />
<br />
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.<br />
<br />
== Test Directory ==<br />
All Android tests require a remote test directory: A place to store pre-configured profiles, support binaries and libraries, test files, etc. Most tests use a directory in /mnt/sdcard by default; xpcshell and cppunittests use /data/local by default (because it is usually not possible to set execute permission on files on /mnt/sdcard).<br />
<br />
The default remote test directory is usually correct and sufficient, but sometimes the default is not appropriate for a device:<br />
* the device may not contain an SD card, or the SD card may not be mounted<br />
* there may not be enough free space on the default location's partition<br />
* the default location may not be writable by the ADB shell and/or SUT agent<br />
<br />
(If you are using a Nexus S, the trick to making your device mountable is to not allow USB Storage between your computer and your device. When you plug in your device to your computer, simply don't click the button to allow this on your device and you should be able to run your tests.)<br />
<br />
If necessary, the default remote test directory may be changed with:<br />
<br />
--remoteTestRoot=<remote-directory><br />
<br />
== S1/S2 Automation ==<br />
These tests start Firefox for Android with a URL and measure the time to throbber start, time to throbber stop, and drawing end times. <br />
<br />
S1/S2 graphs can be viewed at: http://phonedash.mozilla.org/<br />
<br />
Manual run instructions can be found at: https://etherpad.mozilla.org/fennec-perf-ts-take2<br />
<br />
See also: https://wiki.mozilla.org/Mobile/Performance/S1S2-Tests<br />
<br />
== Eideticker ==<br />
Eideticker measures perceived Firefox performance by video capturing automated browser interactions.<br />
<br />
Eideticker is no longer running Android tests. Historical data can be viewed at: http://eideticker.mozilla.org/<br />
<br />
See also: <br />
<br />
** http://wrla.ch/blog/2011/11/measuring-what-the-user-sees/ <br />
** http://wrla.ch/blog/2012/03/announcing-the-eideticker-mobile-performance-dashboard/<br />
** https://github.com/mozilla/eideticker<br />
<br />
== Trouble-shooting testing problems ==<br />
<br />
* Does your mozconfig contain "ac_add_options --disable-tests"?<br />
* Is adb in your $PATH?<br />
* Is your device connected? Does it appear in the output from "adb devices"?<br />
* Can you run adb shell?<br />
* Ensure the device's screen is on.<br />
* Ensure that the device and host machine are on the same network.<br />
** Can you ping the device from the host? Can you ping the host from the device?<br />
** Are the phone and the desktop both using wifi? (wifi vs ethernet??)<br />
** Are the phone and the desktop both using the same wifi network? (Mozilla vs Mozilla Guest??)<br />
** Is the desktop environment running in a VM? If so, you likely want a "Bridged" connection -- not NAT or Host-only.<br />
* Ensure the test harness has the correct IP address for your machine<br />
** Make sure _SERVER_ADDR in the test output is the same as your machine's IP address.<br />
* If using MOZ_HOST_BIN, ensure the binaries in your MOZ_HOST_BIN folder are executable, in case you pull them down from the FTP site. You might see "OSError: [Errno 13] Permission denied" if they are not executable. Use chmod to fix them. Check certutil, pk12util and ssltunnel.</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_01_18&diff=1198322Mobile/Testing/08 01 182018-08-01T16:05:09Z<p>Gbrown: /* Product Integrity */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** all mochitests and reftests failing: {{bug|1479584}}, :jchen on the case<br />
** wcosta recently provided docker-worker support for packet.net -- looking good<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrownhttps://wiki.mozilla.org/index.php?title=Mobile/Testing/08_01_18&diff=1198272Mobile/Testing/08 01 182018-07-31T20:48:58Z<p>Gbrown: /* Product Integrity */</p>
<hr />
<div>= Previous Action Items =<br />
<br />
= Status reports =<br />
<br />
== Autophone ==<br />
<br />
== Dev team ==<br />
<br />
== Product Integrity ==<br />
* packet.net work in progress in {{bug|1460411}}<br />
** wcosta can run tasks in docker-worker on packet.net ... but gbrown cannot (scope problem)<br />
** all mochitests and reftests failing: {{bug|1479584}}<br />
<br />
= Round Table =<br />
<br />
= Action Items =</div>Gbrown