Changes

Jump to: navigation, search

Auto-tools/Projects/Marionette update tests

1,327 bytes removed, 21:58, 24 July 2015
Refactoring
The harness has been developed as a mozharness script [https://github.com/armenzg/build-mozharness/blob/update_testing/scripts/firefox_ui_updates.py firefox_ui_updates.py].
 
To see any remaining work before this update tests can substitute manual work is described in {{bug|1182796}}.
= Support =
= Testing matrix =
'''NOTE''': We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug (XXX: To be filed).
The following block needs to be reviewed and represent what is agreed in here:
--cfg developer_config.py
If I didn't set --update-channel, it would update to 38.0.5 instead of the latest beta on 39.
= Run the tests Setup steps = hg clone http://hg.mozilla.org/build/mozharness cd mozharness python scripts/firefox_ui_updates.py --cfg generic_releng_config.py --cfg update_testsreleases/mozilla-{beta,release}.py cd testing/mozharness
== Specific tests ==
Currently we run two tests for each locale: direct update and fallback update.
XXX: This still needs to be implemented.You can use append --update-direct-only and --update-fallback-only to only run one of the two.
== Chunking ==
In the automation we run the scripts with --this-chunk/--total-chunks to parallelize the execution:
python scripts/firefox_ui_updates.py --cfg generic_releng_config.py --cfg update_tests/mozilla-{beta,release}.py --this-chunk=1 --total-chunks=10 To see what will would run on which chunk you can use --dry-run True. Remember to read about how to run locally in this doc.
== Configuration files ==
There is a config per release branchgeneric_releng_config.You can see them in [http://hg.mozillapy with basic values to run inside of Release Engineering's infrastructure.org/build/mozharness/file/default/configs/update_tests|configs/update_tests]. The only two values that are specified are:* the tools update verify There is also platform spefic config (e.g. mozBetathink 32-firefoxbit vs 64-linux64.cfgbit on the same host)** You can to set the MINIDUMP_STACKWALK variable for crash dumps (see them all in [https://hg.mozilla.org/buildreleases/tools/filemozilla-aurora/defaultrev/release/updates|heree48ae21ed414 patch]).* There is also the developer_config.py which [https://github.com/mozilla/firefox-ui-tests|firefox_ui_tests] branch allows you to check out (erun the harness on your local machine.g. mozilla-beta)
= Harness options =
You can see available options by running this:
python scripts/firefox_ui_updates.py --help
 
Here's a list of some of the options that are important
--firefox-ui-repo=FIREFOX_UI_REPO
which firefox_ui_tests repo to use
--firefox-ui-branch=FIREFOX_UI_BRANCH
which branch to use for firefox_ui_tests
--tools-repo=TOOLS_REPO
Which tools repo to check out
--tools-tag=TOOLS_TAG
Which revision/tag to use for the tools repository.
--update-verify-config=UPDATE_VERIFY_CONFIG
Which update verify config file to use.
--this-chunk=THIS_CHUNK
What chunk of locales to process.
--total-chunks=TOTAL_CHUNKS
Total chunks to dive the locales into.
--dry-run=DRY_RUN Only show what was going to be tested.
--installer-url=INSTALLER_URL
Point to an installer to download and test against.
--installer-path=INSTALLER_PATH
Point to an installer to test against.
--update-allow-mar-channel=UPDATE_ALLOW_MAR_CHANNEL
Additional MAR channel to be allowed for updates, e.g.
"firefox-mozilla-beta" for updating a release build to
the latest beta build.
--update-channel=UPDATE_CHANNEL
Update channel to use.
--update-direct-only
Only perform a direct update
--update-fallback-only
Only perform a fallback update
--update-target-version=UPDATE_TARGET_VERSION
Version of the updated build.
--update-target-buildid=UPDATE_TARGET_BUILDID
Build ID of the updated build
= Internally =
The mozharness script takes care of setting up the environment, checking out repositories, updating to the right branches , setting the right environment variables and execute the tests.
Internally, we call the [http://hg.mozilla.org/build/mozharness/file/2a5172fb2335/scripts/firefox_ui_updates.py#l247|firefox_ui_update] binary that gets generated from the firefox ui tests repository (the runner is defined in [https://github.com/mozilla/firefox-ui-tests/blob/master/firefox_ui_harness/runners/update.py#L53|here]).
The A basic call to the binary looks similar to this:XXX: Double check this
firefox-ui-update --installer /path/to/installer/Firefox%2038.0.dmg --gecko-log=/path/to/cwd/build/gecko.log
= Run on your local machine =
'''NOTE:''' If any tests fail in the jobs running for you see a failure on the releasesrelease logs, you will be able to see also have printed out '''the exact command you need to execute on your local machinerun'''. Here's a sample command from one of the release jobs: # This is a sample command for running Linux64 jobs python scripts/scripts/firefox_ui_updates.py --cfg generic_releng_config.py \ --cfg generic_releng_linux64.py --firefox-ui-branch mozilla-beta \ --update-verify-config mozBeta-firefox-linux64. It will ask cfg \ --tools-tag FIREFOX_40_0b7_RELEASE_RUNTIME --total-chunks 6 \ --this-chunk 1 --build-number 1 All you would have to do:* copy a command from a log (use sample command above)* append --installer-url and have cfg developer_config.py to run it locally on your machine* remove one of the two "scripts/" in the right options command The code has been optimized to userun on your local machine.
== Developer mode ==
It clears any hardcoded paths that are used for Release Engineering machinery.
== --installer-url/--installer-path Without using releng configs ==
Instead of using the Release Engineering update verify configuration files, you can also run the harness in these different ways:
--installer-url:
python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --cfg developer_config.py
For instance, you can say --no-clobber, which will not remove the repositories that the script has checked out.
 
If you specify specific actions (e.g. --run-tests) it will *only* executed that action and ignore the others.
= Extra notes =
 The update tests allow you tospecify some more options. Please see all the --update-* options of the
"firefox-ui-update --help" command. What you definitely also need is
--update-allow-mar-channel, --update-target-version, and
self.assertTrue(update_available)
AssertionError: False is not true
 
TODO: gather gecko log
== Socket 2828 is unavailable ==
You can see the socket held by firefox you can run this command:
$ netstat -anp | grep ':2828 '
# TODO add output of an instance
You can fix it by running `killall -9 firefox`.
Confirm
3,990
edits

Navigation menu