MozillaWiki

Auto-tools/Projects/Marionette update tests: Difference between revisions

Page Discussion

Auto-tools/Projects/Marionette update tests (view source)

Revision as of 13:48, 14 August 2015

7,219 bytes added ,  14 August 2015
m
small change
m (adding new section)
m (small change)
 
(28 intermediate revisions by the same user not shown)
Line 1: Line 1:
Mozmill tests where re-written using Marionette ([[Auto-tools/Projects/m21s|m21s project)]].
Mozmill tests were re-written using Marionette ([[Auto-tools/Projects/m21s|m21s project)]].
The update testing was also ported and it is being integrated as part of the automated release process ({{bug|1148546}}).
The current code lives in [https://github.com/mozilla/firefox-ui-tests|firefox_ui_tests] repository.
The update tests were also ported and are to be run as part of the automated release process ({{bug|1148546}}).


The tests are being integrated inside of [https://github.com/mozilla/build-tools/blob/master/release/updates/verify.sh verify.sh] (not yet).
The harness has been developed as a mozharness script [https://github.com/armenzg/build-mozharness/blob/update_testing/scripts/firefox_ui_updates.py firefox_ui_updates.py].
 
To see any remaining work before this update tests can substitute manual work is described in {{bug|1182796}}.


= Support =
= Support =
Line 8: Line 11:


= Testing matrix =
= Testing matrix =
'''NOTE''': We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug.
The following block needs to be reviewed and represent what is agreed in here:
https://github.com/mozilla/mozmill-ci/issues/535
<i>
There's been some discussions about:
There's been some discussions about:
* testing all locales for the last X betas (maybe 3)
* testing all locales for the last X betas (maybe 3)
Line 19: Line 28:
*** To make things easy we might just test the last beta of the specific gecko version
*** To make things easy we might just test the last beta of the specific gecko version
** 38.0.latest_esr
** 38.0.latest_esr
</i>
== Beta users on RC ==
When we're creating the RC builds, we update *beta* users to it (even though an RC, at the end, is meant for the release users).
This means that after we're done with the RC, we get them back into the betas rather than leave them with the release users.
Here's an example on how to test the update path for a *beta* user on an RC:
python scripts/firefox_ui_updates.py \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg \
--firefox-ui-branch mozilla-beta \
--update-channel beta-localtest \
--cfg developer_config.py
If I didn't set --update-channel, it would update to 38.0.5 instead of the latest beta.
= Setup steps =
hg clone http://hg.mozilla.org/releases/mozilla-{beta,release}
cd testing/mozharness
== Specific tests ==
Currently we run two tests for each locale: direct update and fallback update.
You can use append --update-direct-only and --update-fallback-only to only run one of the two.
== Chunking ==
In the automation we run the scripts with --this-chunk/--total-chunks to parallelize the execution:
To see what would run on which chunk you can use --dry-run True.
== Configuration files ==
There is a generic_releng_config.py with basic values to run inside of Release Engineering's infrastructure.
There is also platform spefic config (think 32-bit vs 64-bit on the same host) to set the MINIDUMP_STACKWALK variable for crash dumps (see [https://hg.mozilla.org/releases/mozilla-aurora/rev/e48ae21ed414 patch]).
There is also the developer_config.py which allows you to run the harness on your local machine.
= Status summary of failures =
You can look in the logs and look for:
SUMMARY - Firefox UI update tests failed locales:
= Harness options =
You can see available options by running this:
python scripts/firefox_ui_updates.py --help
= Internally =
The mozharness script takes care of setting up the environment, checking out repositories, updating to the right branches, setting the right environment variables and execute the tests.
Internally, we call the [http://hg.mozilla.org/build/mozharness/file/2a5172fb2335/scripts/firefox_ui_updates.py#l247|firefox_ui_update] binary that gets generated from the firefox ui tests repository (the runner is defined in [https://github.com/mozilla/firefox-ui-tests/blob/master/firefox_ui_harness/runners/update.py#L53|here]).
A basic call to the binary looks similar to this:
firefox-ui-update --installer /path/to/installer/Firefox%2038.0.dmg --gecko-log=/path/to/cwd/build/gecko.log
The actual tests are defined in [https://github.com/mozilla/firefox-ui-tests/tree/master/firefox_ui_tests/update|here].
= Run on your local machine =
'''NOTE:''' If you see a failure on the release logs, you will also have printed out '''the exact command you need to run'''.
Here's a sample command from one of the release jobs:
# This is a sample command for running Linux64 jobs
python scripts/scripts/firefox_ui_updates.py --cfg generic_releng_config.py \
--cfg generic_releng_linux64.py --firefox-ui-branch mozilla-beta \
--update-verify-config mozBeta-firefox-linux64.cfg \
--tools-tag FIREFOX_40_0b7_RELEASE_RUNTIME --total-chunks 6 \
--this-chunk 1 --build-number 1
All you would have to do:
* copy a command from a log (use sample command above)
* append --cfg developer_config.py to run it locally on your machine
* remove one of the two "scripts/" in the command
The code has been optimized to run on your local machine.


This is open to debate.
== Developer mode ==
To run anything on your local machine you will need to append --cfg developer_config.py.
The script will tell you if you don't.
It clears any hardcoded paths that are used for Release Engineering machinery.


= Run the tests =
== Without using releng configs ==
The script that is used by Release Engineering can be run like this:
Instead of using the Release Engineering update verify configuration files, you can also run the harness in these different ways:
hg clone http://hg.mozilla.org/build/tools
cd release/updates
./verify.sh --marionette mozBeta-firefox-linux64.cfg # Config depends on your host OS


This script will create the following directories:
--installer-url
* cache - where downloaded installers are store to speed up future downloads
python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --cfg developer_config.py
* downloads - temporary dir where installers being tested on a specific iteration
* venv - where the dependencies of firefox-ui-tests are installed


== --keep-venv, --dont-clear-cache and --developer-mode ==
--installer-path
You can also use --keep-env and --dont-clear-cache to prevent blowing up the dirs every time.
python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-path `pwd`/Firefox%2038.0.dmg --cfg developer_config.py


Use --developer-mode when you run this script outside of the releng network.
== Running it more than once ==
It will fail if you don't since we don't put firefox-ui-tests python package on pypi.
The harness allows you to run specific steps in isolation.
Even if we did, if you use --developer-mode you will ensure to use the same versions of python packages as the ones used on a release automation job.
You can see the available actions like this:
<pre>
python scripts/firefox_ui_updates.py --list-actions
Actions available:
    * clobber
    * checkout
    * create-virtualenv
    * determine-testing-configuration
    * run-tests
</pre>


== Smaller subset of locales ==
If you run once the script, you can run it another time but only execute certain steps or skip certain ones.
To run a small set of tests, simply create a new config with only the lines from the original configs.
You can also reduce the list of locales in each line.


== Output of the script ==
For instance, you can say --no-clobber, which will not remove the repositories that the script has checked out.
Two text files are generated:
* short_log.txt - it contains the output of firefox-ui-update
* joint_output.txt - it contains the output of gecko intertwined withe firefox-ui-update
** This is dumped into verify.sh's output if a test does not pass


= Run by hand =
If you specify specific actions (e.g. --run-tests) it will *only* executed that action and ignore the others.
You don't need verify.sh to run a single locale.


If you have not run ./verify.sh you can create the venv like this:
= Test your changes on the Try server =
../common/setup_firefox_ui_updates.sh venv
The Firefox UI tests are jobs that run when a release is running.
Unfortunately, you can't land changes on the various repositories and easily test your new changes until the next release happens.
This makes the development and testing cycle very difficult.


You can use --keep-venv and --developer-mode for this script.
There is a hacky way to run these jobs on [https://treeherder.mozilla.org/ui/#/jobs?repo=try treeherder].
What we're going to do is hijack a job on buildbot to execute the firefox_ui_updates.py script and fake which arguments the script was called with.


After you run verify.sh or setup_firefox_ui_updates.sh, you can run the tests like this:
Follow these steps:
source venv/bin/activate (For Windows "Scripts" instead of "bin")
# have a checkout of mozilla-central
# cd mozilla-entral/testing/mozharness/scripts
# rm desktop_unittest.py; ln firefox_ui_updates.py desktop_unittest.py
#* You have to hard-link since archiver [http://mxr.mozilla.org/build/source/tools/buildfarm/utils/archiver_client.py#175 does not allow symlinks on Windows]
# apply [https://bugzilla.mozilla.org/attachment.cgi?id=8648050 this patch]
#* Just before calling [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_ui_updates.py#l104 super()] it modified the sys.argv values
#* You need to specify through ENABLE_BITS if to test 32-bit or 64-bit builds
# after that you can push to try with this commit message (try: -b o -p android-x86 -u none -t none)
#* The try message should contain at least one platform that would be triggered on buildbot, otherwise, we won't be able to use BuildApi since the revision won't exist unless there is at least one platform to schedule
# once you push to try, you will have to schedule a job with Mozilla CI tools
#* pip install -U mozci
#* export REVISION=your_try_revision_here
#* cancel on TH the running Android x86 build
#* See section below on how to trigger the tests


Download the binary you want to try:
== Selecting the right platform and the right command ==
wget ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-04-20-03-02-04-mozilla-central/firefox-40.0a1.en-US.win32.installer.exe
In order to set the right command on a job, you will have to grab it from these two sets of commands.
On a push we can either test the 64-bit platforms or the 32-bit ones.


Run the two tests:
Use one of these two:
  firefox-ui-update --installer firefox-40.0a1.en-US.win32.installer.exe
* Ubuntu 64-bit, Windows 8 64-bit and Mac OS X
  ENABLE_BITS=64
export REVISION=03abd4caa54e && \
mozci-trigger -b "Ubuntu VM 12.04 x64 try opt test mochitest-1,Rev5 MacOSX Yosemite 10.10 try opt test mochitest-1,Windows 8 64-bit try opt test mochitest-1" -r $REVISION --file http://dummy.com
* Ubuntu 32-bit and Windows 7 32-bit
ENABLE_BITS=32
export REVISION=20bbdfa06139 && \
mozci-trigger -b "Ubuntu VM 12.04 try opt test mochitest-1,Windows 7 32-bit try opt test mochitest-1" -r $REVISION --file http://dummy.com


NOTE: By default, a gecko.log file is generated to remove noise from running firefox-ui-update. Read the help message to know how to adjust it if need be.
NOTE: Using --file is a necessary hack that avoids checking if there is an existing build


NOTE: On Linux loaners you need to set DISPLAY=:2
== Using your own repositories ==
You can append to the command this values:
* --firefox-ui-repo <git_repo>
* --firefox-ui-branch <git_branch>
* --tools-repo <hg_repo>
* --tools-tag <hg_tag>


For beta and release builds which have not yet been released, you need the right update channel:
== Re-triggering jobs ==
firefox-ui-update --installer "Firefox Setup 38.0b6.exe" --update-channel=beta-localtest
You don't need to push more changes to try if all you require is further changes to a Firefox UI repo or a tools repo.
You can simply push to your non gecko repos and re-trigger the job on Treeherder.


= How to use a newer version of firefox-ui-tests =
= Extra notes =
Run these commands:
The update tests allow you to specify some more options. Please see all the --update-* options of the
git clone https://github.com/mozilla/firefox-ui-tests.git
"firefox-ui-update --help" command. What you definitely also need is
cd firefox-ui-tests
--update-allow-mar-channel, --update-target-version, and
python setup.py sdist
--update-target-buildid. The latter two are for final checks that Firefox
wget https://hg.mozilla.org/build/braindump/raw-file/default/utils/publish_package_our_pypi.sh
has been updated to the correct target version. The MAR channel option is
chmod +x publish_package_our_pypi.sh
for updating a release build to a beta build, which happens for RC builds on
# Connect to the VPN
the beta channel.
./publish_package_our_pypi.sh dist/firefox-ui-tests-*.tar.gz


= Failure modes =
Here are the two examples:
 
1. Ensure that the update has been applied correctly by checking the target
version and build id
 
./firefox-ui-update --installer %Firefox38.0b8% \
--update-target-version=38.0b9 --update-target-buildid=20150429135941.
 
Even if the tests pass without those options defined, it does not mean that
the update was successful. You really want to check that Firefox got updated
to the target build.
 
2. A beta user got updated to a RC build and has to be brought back to a
beta build. Therefore you need --update-allow-mar-channel
 
Lets say a beta user was on 38.0b9 and got updated to 38.0RC. This RC build
is on the release channel, but beta users have to be brought back to beta
channel. Given that we currently cannot test updates in multiple steps vXb9
> -> RC -> v(X+1)b1, we have to run vXb9 -> RC, and RC -> v(X+1)b1 separately.
Using the RC build as source build only has the release channel active. The
option above can also enable the beta channel via update-settings.ini.
 
./firefox-ui-update --installer %Firefox38.0% --update-target-version=39.0b1
--update-target-buildid=%whateveritwillbe%
--update-allow-mar-channel=firefox-mozilla-beta
 
= Archive - Failure modes =
The failures reported below are known issues when the infrastructure does not behave as expected.
The failures reported below are known issues when the infrastructure does not behave as expected.
All other harness issues have been dealt with.
All other harness issues have been dealt with.
Line 117: Line 250:
     self.assertTrue(update_available)
     self.assertTrue(update_available)
  AssertionError: False is not true
  AssertionError: False is not true
TODO: gather gecko log


== Socket 2828 is unavailable ==
== Socket 2828 is unavailable ==
Line 146: Line 277:
You can see the socket held by firefox you can run this command:
You can see the socket held by firefox you can run this command:
  $ netstat -anp | grep ':2828 '
  $ netstat -anp | grep ':2828 '
# TODO add output of an instance


You can fix it by running `killall -9 firefox`.
You can fix it by running `killall -9 firefox`.
Armenzg
Confirmed users
3,990

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1070924...1090226"