Auto-tools/Projects/Marionette update tests: Difference between revisions

Jump to navigation Jump to search

Auto-tools/Projects/Marionette update tests (view source)

Revision as of 13:48, 14 August 2015

7,763 bytes added ,  14 August 2015
m
small change
(add more notes)
m (small change)
 
(22 intermediate revisions by the same user not shown)
Line 4: Line 4:


The harness has been developed as a mozharness script [https://github.com/armenzg/build-mozharness/blob/update_testing/scripts/firefox_ui_updates.py firefox_ui_updates.py].
The harness has been developed as a mozharness script [https://github.com/armenzg/build-mozharness/blob/update_testing/scripts/firefox_ui_updates.py firefox_ui_updates.py].
To see any remaining work before this update tests can substitute manual work is described in {{bug|1182796}}.


= Support =
= Support =
Line 9: Line 11:


= Testing matrix =
= Testing matrix =
'''NOTE''': We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug (XXX: To be filed)
'''NOTE''': We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug.


The following block needs to be reviewed and represent what is agreed in here:
https://github.com/mozilla/mozmill-ci/issues/535
<i>
There's been some discussions about:
There's been some discussions about:
* testing all locales for the last X betas (maybe 3)
* testing all locales for the last X betas (maybe 3)
Line 22: Line 28:
*** To make things easy we might just test the last beta of the specific gecko version
*** To make things easy we might just test the last beta of the specific gecko version
** 38.0.latest_esr
** 38.0.latest_esr
</i>
== Beta users on RC ==
When we're creating the RC builds, we update *beta* users to it (even though an RC, at the end, is meant for the release users).
This means that after we're done with the RC, we get them back into the betas rather than leave them with the release users.
Here's an example on how to test the update path for a *beta* user on an RC:
python scripts/firefox_ui_updates.py \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg \
--firefox-ui-branch mozilla-beta \
--update-channel beta-localtest \
--cfg developer_config.py
If I didn't set --update-channel, it would update to 38.0.5 instead of the latest beta.
= Setup steps =
hg clone http://hg.mozilla.org/releases/mozilla-{beta,release}
cd testing/mozharness
== Specific tests ==
Currently we run two tests for each locale: direct update and fallback update.
You can use append --update-direct-only and --update-fallback-only to only run one of the two.
== Chunking ==
In the automation we run the scripts with --this-chunk/--total-chunks to parallelize the execution:
To see what would run on which chunk you can use --dry-run True.


= Run the tests =
== Configuration files ==
hg clone http://hg.mozilla.org/build/mozharness
There is a generic_releng_config.py with basic values to run inside of Release Engineering's infrastructure.
cd mozharness
There is also platform spefic config (think 32-bit vs 64-bit on the same host) to set the MINIDUMP_STACKWALK variable for crash dumps (see [https://hg.mozilla.org/releases/mozilla-aurora/rev/e48ae21ed414 patch]).
python scripts/firefox_ui_updates.py --cfg generic_releng_config.py --cfg update_tests/mozilla-{beta,release}.py
There is also the developer_config.py which allows you to run the harness on your local machine.


== Options ==
= Status summary of failures =
XXX: To be added
You can look in the logs and look for:
SUMMARY - Firefox UI update tests failed locales:


* chunking
= Harness options =
* installer-path
You can see available options by running this:
* installer-url
python scripts/firefox_ui_updates.py --help
* dry-run


== Run on your local machine ==
= Internally =
Append --cfg developer_config.py
The mozharness script takes care of setting up the environment, checking out repositories, updating to the right branches, setting the right environment variables and execute the tests.
It will clear any hardcoded paths that are used for Release Engineering machinery.
 
Internally, we call the [http://hg.mozilla.org/build/mozharness/file/2a5172fb2335/scripts/firefox_ui_updates.py#l247|firefox_ui_update] binary that gets generated from the firefox ui tests repository (the runner is defined in [https://github.com/mozilla/firefox-ui-tests/blob/master/firefox_ui_harness/runners/update.py#L53|here]).
 
A basic call to the binary looks similar to this:
firefox-ui-update --installer /path/to/installer/Firefox%2038.0.dmg --gecko-log=/path/to/cwd/build/gecko.log
 
The actual tests are defined in [https://github.com/mozilla/firefox-ui-tests/tree/master/firefox_ui_tests/update|here].
 
= Run on your local machine =
'''NOTE:''' If you see a failure on the release logs, you will also have printed out '''the exact command you need to run'''.
 
Here's a sample command from one of the release jobs:
# This is a sample command for running Linux64 jobs
python scripts/scripts/firefox_ui_updates.py --cfg generic_releng_config.py \
--cfg generic_releng_linux64.py --firefox-ui-branch mozilla-beta \
--update-verify-config mozBeta-firefox-linux64.cfg \
--tools-tag FIREFOX_40_0b7_RELEASE_RUNTIME --total-chunks 6 \
--this-chunk 1 --build-number 1
 
All you would have to do:
* copy a command from a log (use sample command above)
* append --cfg developer_config.py to run it locally on your machine
* remove one of the two "scripts/" in the command
 
The code has been optimized to run on your local machine.
 
== Developer mode ==
To run anything on your local machine you will need to append --cfg developer_config.py.
The script will tell you if you don't.
It clears any hardcoded paths that are used for Release Engineering machinery.
 
== Without using releng configs ==
Instead of using the Release Engineering update verify configuration files, you can also run the harness in these different ways:
 
--installer-url
python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0-candidates/build3/mac/en-US/Firefox%2038.0.dmg --cfg developer_config.py
 
--installer-path
python scripts/firefox_ui_updates.py --firefox-ui-branch mozilla-beta --installer-path `pwd`/Firefox%2038.0.dmg --cfg developer_config.py
 
== Running it more than once ==
The harness allows you to run specific steps in isolation.
You can see the available actions like this:
<pre>
python scripts/firefox_ui_updates.py --list-actions
Actions available:
    * clobber
    * checkout
    * create-virtualenv
    * determine-testing-configuration
    * run-tests
</pre>
 
If you run once the script, you can run it another time but only execute certain steps or skip certain ones.
 
For instance, you can say --no-clobber, which will not remove the repositories that the script has checked out.
 
If you specify specific actions (e.g. --run-tests) it will *only* executed that action and ignore the others.
 
= Test your changes on the Try server =
The Firefox UI tests are jobs that run when a release is running.
Unfortunately, you can't land changes on the various repositories and easily test your new changes until the next release happens.
This makes the development and testing cycle very difficult.
 
There is a hacky way to run these jobs on [https://treeherder.mozilla.org/ui/#/jobs?repo=try treeherder].
What we're going to do is hijack a job on buildbot to execute the firefox_ui_updates.py script and fake which arguments the script was called with.
 
Follow these steps:
# have a checkout of mozilla-central
# cd mozilla-entral/testing/mozharness/scripts
# rm desktop_unittest.py; ln firefox_ui_updates.py desktop_unittest.py
#* You have to hard-link since archiver [http://mxr.mozilla.org/build/source/tools/buildfarm/utils/archiver_client.py#175 does not allow symlinks on Windows]
# apply [https://bugzilla.mozilla.org/attachment.cgi?id=8648050 this patch]
#* Just before calling [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_ui_updates.py#l104 super()] it modified the sys.argv values
#* You need to specify through ENABLE_BITS if to test 32-bit or 64-bit builds
# after that you can push to try with this commit message (try: -b o -p android-x86 -u none -t none)
#* The try message should contain at least one platform that would be triggered on buildbot, otherwise, we won't be able to use BuildApi since the revision won't exist unless there is at least one platform to schedule
# once you push to try, you will have to schedule a job with Mozilla CI tools
#* pip install -U mozci
#* export REVISION=your_try_revision_here
#* cancel on TH the running Android x86 build
#* See section below on how to trigger the tests
 
== Selecting the right platform and the right command ==
In order to set the right command on a job, you will have to grab it from these two sets of commands.
On a push we can either test the 64-bit platforms or the 32-bit ones.
 
Use one of these two:
* Ubuntu 64-bit, Windows 8 64-bit and Mac OS X
ENABLE_BITS=64
export REVISION=03abd4caa54e && \
mozci-trigger -b "Ubuntu VM 12.04 x64 try opt test mochitest-1,Rev5 MacOSX Yosemite 10.10 try opt test mochitest-1,Windows 8 64-bit try opt test mochitest-1" -r $REVISION --file http://dummy.com
* Ubuntu 32-bit and Windows 7 32-bit
ENABLE_BITS=32
export REVISION=20bbdfa06139 && \
mozci-trigger -b "Ubuntu VM 12.04 try opt test mochitest-1,Windows 7 32-bit try opt test mochitest-1" -r $REVISION --file http://dummy.com
 
NOTE: Using --file is a necessary hack that avoids checking if there is an existing build
 
== Using your own repositories ==
You can append to the command this values:
* --firefox-ui-repo <git_repo>
* --firefox-ui-branch <git_branch>
* --tools-repo <hg_repo>
* --tools-tag <hg_tag>
 
== Re-triggering jobs ==
You don't need to push more changes to try if all you require is further changes to a Firefox UI repo or a tools repo.
You can simply push to your non gecko repos and re-trigger the job on Treeherder.


= Extra notes =
= Extra notes =
 
The update tests allow you to specify some more options. Please see all the --update-* options of the
The update tests allow to
specify some more options. Please see all the --update-* options of the
"firefox-ui-update --help" command. What you definitely also need is
"firefox-ui-update --help" command. What you definitely also need is
--update-allow-mar-channel, --update-target-version, and
--update-allow-mar-channel, --update-target-version, and
Line 56: Line 196:
version and build id
version and build id


> ./firefox-ui-update --installer %Firefox38.0b8% \
./firefox-ui-update --installer %Firefox38.0b8% \
> --update-target-version=38.0b9 --update-target-buildid=20150429135941.
--update-target-version=38.0b9 --update-target-buildid=20150429135941.


Even if the tests pass without those options defined, it does not mean that
Even if the tests pass without those options defined, it does not mean that
Line 73: Line 213:
option above can also enable the beta channel via update-settings.ini.
option above can also enable the beta channel via update-settings.ini.


> ./firefox-ui-update --installer %Firefox38.0% --update-target-version=39.0b1
./firefox-ui-update --installer %Firefox38.0% --update-target-version=39.0b1
> --update-target-buildid=%whateveritwillbe%
--update-target-buildid=%whateveritwillbe%
> --update-allow-mar-channel=firefox-mozilla-beta
--update-allow-mar-channel=firefox-mozilla-beta


= Archive - Failure modes =
= Archive - Failure modes =
Line 110: Line 250:
     self.assertTrue(update_available)
     self.assertTrue(update_available)
  AssertionError: False is not true
  AssertionError: False is not true
TODO: gather gecko log


== Socket 2828 is unavailable ==
== Socket 2828 is unavailable ==
Line 139: Line 277:
You can see the socket held by firefox you can run this command:
You can see the socket held by firefox you can run this command:
  $ netstat -anp | grep ':2828 '
  $ netstat -anp | grep ':2828 '
# TODO add output of an instance


You can fix it by running `killall -9 firefox`.
You can fix it by running `killall -9 firefox`.
Armenzg
Confirmed users
3,990

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1079183...1090226"

Navigation menu