Auto-tools/Projects/Marionette update tests

From MozillaWiki
< Auto-tools‎ | Projects
Revision as of 00:09, 29 April 2015 by Armenzg (talk | contribs) (adding new section)
Jump to navigation Jump to search

Mozmill tests where re-written using Marionette (m21s project). The update testing was also ported and it is being integrated as part of the automated release process (bug 1148546).

The tests are being integrated inside of verify.sh (not yet).

Support

Only Gecko 38 and higher is supported.

Testing matrix

There's been some discussions about:

  • testing all locales for the last X betas (maybe 3)
  • testing all locales for 1 beta for the last Y versions (maybe 3)
  • testing all locales for the latest esr

For example:

  • If on gecko 40, beta2:
    • 39.0b6, 39.0b7, 40.0b1
    • 35.0bX, 36.0bY, 37.0bZ (X, Y & Z belonging to the range of 1 to the latest beta version for that gecko version)
      • To make things easy we might just test the last beta of the specific gecko version
    • 38.0.latest_esr

This is open to debate.

Run the tests

The script that is used by Release Engineering can be run like this: 

hg clone http://hg.mozilla.org/build/tools
cd release/updates
./verify.sh --marionette mozBeta-firefox-linux64.cfg # Config depends on your host OS

This script will create the following directories:

  • cache - where downloaded installers are store to speed up future downloads
  • downloads - temporary dir where installers being tested on a specific iteration
  • venv - where the dependencies of firefox-ui-tests are installed

--keep-venv, --dont-clear-cache and --developer-mode

You can also use --keep-env and --dont-clear-cache to prevent blowing up the dirs every time.

Use --developer-mode when you run this script outside of the releng network. It will fail if you don't since we don't put firefox-ui-tests python package on pypi. Even if we did, if you use --developer-mode you will ensure to use the same versions of python packages as the ones used on a release automation job.

Smaller subset of locales

To run a small set of tests, simply create a new config with only the lines from the original configs. You can also reduce the list of locales in each line.

Output of the script

Two text files are generated:

  • short_log.txt - it contains the output of firefox-ui-update
  • joint_output.txt - it contains the output of gecko intertwined withe firefox-ui-update
    • This is dumped into verify.sh's output if a test does not pass

Run by hand

You don't need verify.sh to run a single locale.

If you have not run ./verify.sh you can create the venv like this: 

../common/setup_firefox_ui_updates.sh venv

You can use --keep-venv and --developer-mode for this script.

After you run verify.sh or setup_firefox_ui_updates.sh, you can run the tests like this: 

source venv/bin/activate (For Windows "Scripts" instead of "bin")

Download the binary you want to try: 

wget ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-04-20-03-02-04-mozilla-central/firefox-40.0a1.en-US.win32.installer.exe

Run the two tests: 

firefox-ui-update --installer firefox-40.0a1.en-US.win32.installer.exe

NOTE: By default, a gecko.log file is generated to remove noise from running firefox-ui-update. Read the help message to know how to adjust it if need be.

NOTE: On Linux loaners you need to set DISPLAY=:2

For beta and release builds which have not yet been released, you need the right update channel: 

firefox-ui-update --installer "Firefox Setup 38.0b6.exe" --update-channel=beta-localtest

How to use a newer version of firefox-ui-tests

Run these commands: 

git clone https://github.com/mozilla/firefox-ui-tests.git
cd firefox-ui-tests
python setup.py sdist
wget https://hg.mozilla.org/build/braindump/raw-file/default/utils/publish_package_our_pypi.sh
chmod +x publish_package_our_pypi.sh
# Connect to the VPN
./publish_package_our_pypi.sh dist/firefox-ui-tests-*.tar.gz

Failure modes

The failures reported below are known issues when the infrastructure does not behave as expected. All other harness issues have been dealt with.

AUS is temporarily unavailable

firefox-ui-update raises this exception: 

    self.assertTrue(update_available)
AssertionError: False is not true

In the gecko log you will see Error: Unexpected node name, expected: updates, got: parsererror: 

*** AUS:SVC Checker:checkForUpdates - sending request to: https://aus4.mozilla.org/update/3/Firefox/38.0/20150330154247/Linux_x86_64-gcc3/hu/beta-localtest/Linux%202.6.32-504.3.3.el6.x86_64%20(GTK%202.20.1)/default/default/update.xml?force=1
*** AUS:SVC Checker:onLoad - request completed downloading document
*** AUS:SVC Checker:_updates get - unexpected node name!
*** AUS:SVC Checker:onLoad - there was a problem checking for updates. Exception: Error: Unexpected node name, expected: updates, got: parsererror
*** AUS:SVC Checker:onLoad - request.status: 503
*** AUS:SVC getStatusTextFromCode - transfer error: A frissítés XML-fájlja nem található. (404), default code: 404
*** AUS:SVC recordInHealthReport - updateCheckFailed - 1503

The "unexpeced node name" comes from here: https://hg.mozilla.org/mozilla-central/annotate/50b95032152c/toolkit/mozapps/update/nsUpdateService.js#l3577

Originally discovered on bug 1152460.

Running out of space

If the machine running the tests runs out of space we can see this issue revealed on the Gecko log: 

*** AUS:SVC readStatusFile - status: failed: 7, path: /tmp/tmpAyKRdJ.binary-update-tests/updates/0/update.status

Originall discovered on bug 1154060.

FTP/stage/S3 is unavailable temporarily OR wrong update channel

firefox-ui-update will raise an exception: 

    self.assertTrue(update_available)
AssertionError: False is not true

TODO: gather gecko log

Socket 2828 is unavailable

The socket for Marionette was left open from a previous run. This is not an infra issue unlike the other ones. This bug needs to be fixed on Marionette bug 1141519 and a work around will be placed through bug 1156475.

Exception raised: 

   assert(self.wait_for_port()), "Timed out waiting for port!"
AssertionError: Timed out waiting for port!

You can see the socket by running this: 

$ netstat -anp | grep ':2828 ' | grep TIME_WAIT
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:2828          127.0.0.1:47185         TIME_WAIT   -

This happens when "the Python client shuts down the socket abruptly (due to a global timeout on the operation, for instance)" bug 1141519

The only way to get back to a good known state is by waiting for that socket to timeout on its own.

Socket 2828 is unavailable

If you're running the suite by hand and you have aborted a previous run, you might have left firefox running. If you 

MarionetteException: MarionetteException: localhost:2828 is unavailable.

You can see the socket held by firefox you can run this command: 

$ netstat -anp | grep ':2828 '
# TODO add output of an instance

You can fix it by running `killall -9 firefox`.

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Marionette_update_tests&oldid=1070924"