Auto-tools/Projects/Marionette update tests

Mozmill tests were re-written using Marionette (m21s project). The current code lives in [1] repository. The update tests were also ported and are to be run as part of the automated release process (bug 1148546).

The harness has been developed as a mozharness script firefox_ui_updates.py.

Support

Only Gecko 38 and higher is supported.

Testing matrix

NOTE: We currently test all releases for all locales starting from Gecko 38. This should be reduced in a follow up bug (XXX: To be filed)

There's been some discussions about:

• testing all locales for the last X betas (maybe 3)
• testing all locales for 1 beta for the last Y versions (maybe 3)
• testing all locales for the latest esr

For example:

• If on gecko 40, beta2:
  • 39.0b6, 39.0b7, 40.0b1
  • 35.0bX, 36.0bY, 37.0bZ (X, Y & Z belonging to the range of 1 to the latest beta version for that gecko version)
    • To make things easy we might just test the last beta of the specific gecko version
  • 38.0.latest_esr

Run the tests

hg clone http://hg.mozilla.org/build/mozharness
cd mozharness
python scripts/firefox_ui_updates.py --cfg generic_releng_config.py --cfg update_tests/mozilla-{beta,release}.py

Options

XXX: To be added

• chunking
• installer-path
• installer-url
• dry-run

Run on your local machine

Append --cfg developer_config.py It will clear any hardcoded paths that are used for Release Engineering machinery.

Extra notes

The update tests allow to specify some more options. Please see all the --update-* options of the "firefox-ui-update --help" command. What you definitely also need is --update-allow-mar-channel, --update-target-version, and --update-target-buildid. The latter two are for final checks that Firefox has been updated to the correct target version. The MAR channel option is for updating a release build to a beta build, which happens for RC builds on the beta channel.

Here are the two examples:

1. Ensure that the update has been applied correctly by checking the target version and build id

./firefox-ui-update --installer %Firefox38.0b8% \
--update-target-version=38.0b9 --update-target-buildid=20150429135941.

Even if the tests pass without those options defined, it does not mean that the update was successful. You really want to check that Firefox got updated to the target build.

2. A beta user got updated to a RC build and has to be brought back to a beta build. Therefore you need --update-allow-mar-channel

Lets say a beta user was on 38.0b9 and got updated to 38.0RC. This RC build is on the release channel, but beta users have to be brought back to beta channel. Given that we currently cannot test updates in multiple steps vXb9 > -> RC -> v(X+1)b1, we have to run vXb9 -> RC, and RC -> v(X+1)b1 separately. Using the RC build as source build only has the release channel active. The option above can also enable the beta channel via update-settings.ini.

./firefox-ui-update --installer %Firefox38.0% --update-target-version=39.0b1
--update-target-buildid=%whateveritwillbe%
--update-allow-mar-channel=firefox-mozilla-beta

Archive - Failure modes

The failures reported below are known issues when the infrastructure does not behave as expected. All other harness issues have been dealt with.

AUS is temporarily unavailable

firefox-ui-update raises this exception:

    self.assertTrue(update_available)
AssertionError: False is not true

In the gecko log you will see Error: Unexpected node name, expected: updates, got: parsererror:

*** AUS:SVC Checker:checkForUpdates - sending request to: https://aus4.mozilla.org/update/3/Firefox/38.0/20150330154247/Linux_x86_64-gcc3/hu/beta-localtest/Linux%202.6.32-504.3.3.el6.x86_64%20(GTK%202.20.1)/default/default/update.xml?force=1
*** AUS:SVC Checker:onLoad - request completed downloading document
*** AUS:SVC Checker:_updates get - unexpected node name!
*** AUS:SVC Checker:onLoad - there was a problem checking for updates. Exception: Error: Unexpected node name, expected: updates, got: parsererror
*** AUS:SVC Checker:onLoad - request.status: 503
*** AUS:SVC getStatusTextFromCode - transfer error: A frissítés XML-fájlja nem található. (404), default code: 404
*** AUS:SVC recordInHealthReport - updateCheckFailed - 1503

The "unexpeced node name" comes from here: https://hg.mozilla.org/mozilla-central/annotate/50b95032152c/toolkit/mozapps/update/nsUpdateService.js#l3577

Originally discovered on bug 1152460.

Running out of space

If the machine running the tests runs out of space we can see this issue revealed on the Gecko log:

*** AUS:SVC readStatusFile - status: failed: 7, path: /tmp/tmpAyKRdJ.binary-update-tests/updates/0/update.status

Originall discovered on bug 1154060.

FTP/stage/S3 is unavailable temporarily OR wrong update channel

firefox-ui-update will raise an exception:

    self.assertTrue(update_available)
AssertionError: False is not true

TODO: gather gecko log

Socket 2828 is unavailable

The socket for Marionette was left open from a previous run. This is not an infra issue unlike the other ones. This bug needs to be fixed on Marionette bug 1141519 and a work around will be placed through bug 1156475.

Exception raised:

   assert(self.wait_for_port()), "Timed out waiting for port!"
AssertionError: Timed out waiting for port!

You can see the socket by running this:

$ netstat -anp | grep ':2828 ' | grep TIME_WAIT
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:2828          127.0.0.1:47185         TIME_WAIT   -

This happens when "the Python client shuts down the socket abruptly (due to a global timeout on the operation, for instance)" bug 1141519

The only way to get back to a good known state is by waiting for that socket to timeout on its own.

Socket 2828 is unavailable

If you're running the suite by hand and you have aborted a previous run, you might have left firefox running. If you

MarionetteException: MarionetteException: localhost:2828 is unavailable.

You can see the socket held by firefox you can run this command:

$ netstat -anp | grep ':2828 '
# TODO add output of an instance

You can fix it by running `killall -9 firefox`.