MozillaWiki

Auto-tools/Projects/Marionette update tests: Difference between revisions

Page Discussion

Auto-tools/Projects/Marionette update tests (view source)

Revision as of 13:48, 14 August 2015

1,850 bytes added ,  14 August 2015
m
small change
m (add italics)
m (small change)
 
(11 intermediate revisions by the same user not shown)
Line 4: Line 4:


The harness has been developed as a mozharness script [https://github.com/armenzg/build-mozharness/blob/update_testing/scripts/firefox_ui_updates.py firefox_ui_updates.py].
The harness has been developed as a mozharness script [https://github.com/armenzg/build-mozharness/blob/update_testing/scripts/firefox_ui_updates.py firefox_ui_updates.py].
To see any remaining work before this update tests can substitute manual work is described in {{bug|1182796}}.


= Support =
= Support =
Line 9: Line 11:


= Testing matrix =
= Testing matrix =
'''NOTE''': We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug (XXX: To be filed)
'''NOTE''': We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug.


The following block needs to be reviewed and represent what is agreed in here:
The following block needs to be reviewed and represent what is agreed in here:
Line 40: Line 42:
  --cfg developer_config.py
  --cfg developer_config.py


If I didn't set --update-channel, it would update to 38.0.5 instead of the latest beta on 39.
If I didn't set --update-channel, it would update to 38.0.5 instead of the latest beta.


= Run the tests =
= Setup steps =
  hg clone http://hg.mozilla.org/build/mozharness
  hg clone http://hg.mozilla.org/releases/mozilla-{beta,release}
cd mozharness
cd testing/mozharness
python scripts/firefox_ui_updates.py --cfg generic_releng_config.py --cfg update_tests/mozilla-{beta,release}.py


== Specific tests ==
== Specific tests ==
Currently we run two tests for each locale: direct update and fallback update.
Currently we run two tests for each locale: direct update and fallback update.


XXX: This still needs to be implemented.
You can use append --update-direct-only and --update-fallback-only to only run one of the two.
You can use --update-direct-only and --update-fallback-only to only run one of the two.


== Chunking ==
== Chunking ==
In the automation we run the scripts with --this-chunk/--total-chunks to parallelize the execution:
In the automation we run the scripts with --this-chunk/--total-chunks to parallelize the execution:
python scripts/firefox_ui_updates.py --cfg generic_releng_config.py --cfg update_tests/mozilla-{beta,release}.py --this-chunk=1 --total-chunks=10
To see what would run on which chunk you can use --dry-run True.
 
To see what will run on which chunk you can use --dry-run True.
 
Remember to read about how to run locally in this doc.


== Configuration files ==
== Configuration files ==
There is a config per release branch.
There is a generic_releng_config.py with basic values to run inside of Release Engineering's infrastructure.
You can see them in [http://hg.mozilla.org/build/mozharness/file/default/configs/update_tests|configs/update_tests].
There is also platform spefic config (think 32-bit vs 64-bit on the same host) to set the MINIDUMP_STACKWALK variable for crash dumps (see [https://hg.mozilla.org/releases/mozilla-aurora/rev/e48ae21ed414 patch]).
There is also the developer_config.py which allows you to run the harness on your local machine.


The only two values that are specified are:
= Status summary of failures =
* the tools update verify config (e.g. mozBeta-firefox-linux64.cfg)
You can look in the logs and look for:
** You can see them all in [https://hg.mozilla.org/build/tools/file/default/release/updates|here]
SUMMARY - Firefox UI update tests failed locales:
* which [https://github.com/mozilla/firefox-ui-tests|firefox_ui_tests] branch to check out (e.g. mozilla-beta)


= Harness options =
= Harness options =
You can see available options by running this:
You can see available options by running this:
  python scripts/firefox_ui_updates.py --help
  python scripts/firefox_ui_updates.py --help
Here's a list of some of the options that are important
  --firefox-ui-repo=FIREFOX_UI_REPO
                        which firefox_ui_tests repo to use
  --firefox-ui-branch=FIREFOX_UI_BRANCH
                        which branch to use for firefox_ui_tests
  --tools-repo=TOOLS_REPO
                        Which tools repo to check out
  --tools-tag=TOOLS_TAG
                        Which revision/tag to use for the tools repository.
  --update-verify-config=UPDATE_VERIFY_CONFIG
                        Which update verify config file to use.
  --this-chunk=THIS_CHUNK
                        What chunk of locales to process.
  --total-chunks=TOTAL_CHUNKS
                        Total chunks to dive the locales into.
  --dry-run=DRY_RUN    Only show what was going to be tested.
  --installer-url=INSTALLER_URL
                        Point to an installer to download and test against.
  --installer-path=INSTALLER_PATH
                        Point to an installer to test against.
  --update-allow-mar-channel=UPDATE_ALLOW_MAR_CHANNEL
                        Additional MAR channel to be allowed for updates, e.g.
                        "firefox-mozilla-beta" for updating a release build to
                        the latest beta build.
  --update-channel=UPDATE_CHANNEL
                        Update channel to use.
  --update-direct-only
                        Only perform a direct update
  --update-fallback-only
                        Only perform a fallback update
  --update-target-version=UPDATE_TARGET_VERSION
                        Version of the updated build.
  --update-target-buildid=UPDATE_TARGET_BUILDID
                        Build ID of the updated build


= Internally =
= Internally =
The mozharness script takes care of setting up the environment, checking out repositories, updating to the right branches and execute the tests.
The mozharness script takes care of setting up the environment, checking out repositories, updating to the right branches, setting the right environment variables and execute the tests.


Internally, we call the [http://hg.mozilla.org/build/mozharness/file/2a5172fb2335/scripts/firefox_ui_updates.py#l247|firefox_ui_update] that gets generated from the firefox ui tests repository (the runner is defined in [https://github.com/mozilla/firefox-ui-tests/blob/master/firefox_ui_harness/runners/update.py#L53|here]).
Internally, we call the [http://hg.mozilla.org/build/mozharness/file/2a5172fb2335/scripts/firefox_ui_updates.py#l247|firefox_ui_update] binary that gets generated from the firefox ui tests repository (the runner is defined in [https://github.com/mozilla/firefox-ui-tests/blob/master/firefox_ui_harness/runners/update.py#L53|here]).


The call to the binary looks similar to this:
A basic call to the binary looks similar to this:
XXX: Double check this
  firefox-ui-update --installer /path/to/installer/Firefox%2038.0.dmg --gecko-log=/path/to/cwd/build/gecko.log
  firefox-ui-update --installer /path/to/installer/Firefox%2038.0.dmg --gecko-log=/path/to/cwd/build/gecko.log


Line 121: Line 81:


= Run on your local machine =
= Run on your local machine =
If any tests fail in the jobs running for the releases, you will be able to see the exact command to execute on your local machine. It will ask you to use --installer-url and have the right options to use.
'''NOTE:''' If you see a failure on the release logs, you will also have printed out '''the exact command you need to run'''.
 
Here's a sample command from one of the release jobs:
# This is a sample command for running Linux64 jobs
python scripts/scripts/firefox_ui_updates.py --cfg generic_releng_config.py \
--cfg generic_releng_linux64.py --firefox-ui-branch mozilla-beta \
--update-verify-config mozBeta-firefox-linux64.cfg \
--tools-tag FIREFOX_40_0b7_RELEASE_RUNTIME --total-chunks 6 \
--this-chunk 1 --build-number 1
 
All you would have to do:
* copy a command from a log (use sample command above)
* append --cfg developer_config.py to run it locally on your machine
* remove one of the two "scripts/" in the command
 
The code has been optimized to run on your local machine.


== Developer mode ==
== Developer mode ==
Line 128: Line 103:
It clears any hardcoded paths that are used for Release Engineering machinery.
It clears any hardcoded paths that are used for Release Engineering machinery.


== --installer-url/--installer-path ==
== Without using releng configs ==
Instead of using the Release Engineering update verify configuration files, you can also run the harness in these different ways:
Instead of using the Release Engineering update verify configuration files, you can also run the harness in these different ways:


--installer-url:
--installer-url
  python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --cfg developer_config.py
  python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --cfg developer_config.py


Line 153: Line 128:


For instance, you can say --no-clobber, which will not remove the repositories that the script has checked out.
For instance, you can say --no-clobber, which will not remove the repositories that the script has checked out.
If you specify specific actions (e.g. --run-tests) it will *only* executed that action and ignore the others.
= Test your changes on the Try server =
The Firefox UI tests are jobs that run when a release is running.
Unfortunately, you can't land changes on the various repositories and easily test your new changes until the next release happens.
This makes the development and testing cycle very difficult.
There is a hacky way to run these jobs on [https://treeherder.mozilla.org/ui/#/jobs?repo=try treeherder].
What we're going to do is hijack a job on buildbot to execute the firefox_ui_updates.py script and fake which arguments the script was called with.
Follow these steps:
# have a checkout of mozilla-central
# cd mozilla-entral/testing/mozharness/scripts
# rm desktop_unittest.py; ln firefox_ui_updates.py desktop_unittest.py
#* You have to hard-link since archiver [http://mxr.mozilla.org/build/source/tools/buildfarm/utils/archiver_client.py#175 does not allow symlinks on Windows]
# apply [https://bugzilla.mozilla.org/attachment.cgi?id=8648050 this patch]
#* Just before calling [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_ui_updates.py#l104 super()] it modified the sys.argv values
#* You need to specify through ENABLE_BITS if to test 32-bit or 64-bit builds
# after that you can push to try with this commit message (try: -b o -p android-x86 -u none -t none)
#* The try message should contain at least one platform that would be triggered on buildbot, otherwise, we won't be able to use BuildApi since the revision won't exist unless there is at least one platform to schedule
# once you push to try, you will have to schedule a job with Mozilla CI tools
#* pip install -U mozci
#* export REVISION=your_try_revision_here
#* cancel on TH the running Android x86 build
#* See section below on how to trigger the tests
== Selecting the right platform and the right command ==
In order to set the right command on a job, you will have to grab it from these two sets of commands.
On a push we can either test the 64-bit platforms or the 32-bit ones.
Use one of these two:
* Ubuntu 64-bit, Windows 8 64-bit and Mac OS X
ENABLE_BITS=64
export REVISION=03abd4caa54e && \
mozci-trigger -b "Ubuntu VM 12.04 x64 try opt test mochitest-1,Rev5 MacOSX Yosemite 10.10 try opt test mochitest-1,Windows 8 64-bit try opt test mochitest-1" -r $REVISION --file http://dummy.com
* Ubuntu 32-bit and Windows 7 32-bit
ENABLE_BITS=32
export REVISION=20bbdfa06139 && \
mozci-trigger -b "Ubuntu VM 12.04 try opt test mochitest-1,Windows 7 32-bit try opt test mochitest-1" -r $REVISION --file http://dummy.com
NOTE: Using --file is a necessary hack that avoids checking if there is an existing build
== Using your own repositories ==
You can append to the command this values:
* --firefox-ui-repo <git_repo>
* --firefox-ui-branch <git_branch>
* --tools-repo <hg_repo>
* --tools-tag <hg_tag>
== Re-triggering jobs ==
You don't need to push more changes to try if all you require is further changes to a Firefox UI repo or a tools repo.
You can simply push to your non gecko repos and re-trigger the job on Treeherder.


= Extra notes =
= Extra notes =
 
The update tests allow you to specify some more options. Please see all the --update-* options of the
The update tests allow to
specify some more options. Please see all the --update-* options of the
"firefox-ui-update --help" command. What you definitely also need is
"firefox-ui-update --help" command. What you definitely also need is
--update-allow-mar-channel, --update-target-version, and
--update-allow-mar-channel, --update-target-version, and
Line 224: Line 250:
     self.assertTrue(update_available)
     self.assertTrue(update_available)
  AssertionError: False is not true
  AssertionError: False is not true
TODO: gather gecko log


== Socket 2828 is unavailable ==
== Socket 2828 is unavailable ==
Line 253: Line 277:
You can see the socket held by firefox you can run this command:
You can see the socket held by firefox you can run this command:
  $ netstat -anp | grep ':2828 '
  $ netstat -anp | grep ':2828 '
# TODO add output of an instance


You can fix it by running `killall -9 firefox`.
You can fix it by running `killall -9 firefox`.
Armenzg
Confirmed users
3,990

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1079790...1090226"