QA/Mozmill Test Automation/On Demand Update Testing/Documentation

System Prep

Things in this section should be done to configure the system.

1. Check for a functioning heartbeat emitter. Run one if unsure.
2. Set up listeners on all desired platforms, grouped with a common cluster name.

Heartbeat Emitter

• Generates traffic to keep release listener Pulse connections from being automatically closed.
• Fine to run more than one. If you're not sure, start one.

Usage

./heartbeat_emitter.py

Options/Behavior

No other requirements. It will print out local time every time a heartbeat is sent, so you can diagnose time of any server failures or freezes.

Release Listener

• Listens for a release request and automatically runs the correct script
• Run one per platform. Make sure a heartbeat has been set up.

Usage

./release_listener.py --help
Usage: release_listener.py [options] cluster platform

Launches a listener for the on-demand testing system. The listener will only 
respond to requests for the specified cluster. The listener will respond to
requests for either the specified platform or for all platforms.

Options:
  -h, --help            show this help message and exit
  -u UPDATE_SCRIPT, --update=UPDATE_SCRIPT
                        update test script to run [default:
                        ./release_update.py]
  -f FUNCTIONAL_SCRIPT, --functional=FUNCTIONAL_SCRIPT
                        functional test script to run [default:
                        ./release_bft.py]
  -d, --debug           print out extra debug information on ignored messages

Sample Command Line

./release_listener.py qa-foobar mac

This runs a listener in the qa-foobar cluster, platform set to mac.

Options/Behavior

• Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs.
• Platform must exactly match one of platform specifiers used in staging (and thus, usually the config file, see below). Otherwise, it's an arbitrary string.
• --update will redefine which test script is called for update requests. The default accomodates our internal systems.
• --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems.
• --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding.
• The listeners always print output to the console regarding what they're testing. This can be useful to make sure they're working.
• If listeners freeze up, chances are they're not receiving a heartbeat.

Release Prep

Things in this section should be done on a per-release basis, prior to the release.

1. Create a configuration file for release staging.
2. Stage the release.

Configuration File

• Used by the staging script (below) to simplify downloading builds for a test matrix.
• The configuration file is not used during the actual release testing process.

Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the staging script creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places.

Sample File

; Example configuration for an update test-run

[testrun]
; Application to test
application=firefox

; Type of test, could be update or bft
script=update

; Destination folder builds will be saved to
directory=3.6.20

; One section per platform tests are being run on:
;
; Section name is the specific OS listener platform, one of:
;   win2000, winxp, vista, win7, win7_64, linux, linux_64, mac
;
; platform key refers to the binary's platform on ftp, one of:
;   linux, linux64, mac, mac64, win32, win64
;
; Rest of keys are test matrix for that OS, as:
;   version=locale1 locale2 locale3
;   3.6.18=en-US (release build of Firefox 3.6.18)
;   3.6.18#2=de  (candidate builds #2 of the upcoming Firefox 3.6.18)

[win7]
platform=win32
3.6.18=en-US de
3.6=en-US fi

[winxp]
platform=win32
3.6.18=en-US fr
3.6=en-US it

[mac]
platform=mac
3.6.19=en-US de
3.6.18=en-US ja-JP-mac
3.6=en-US fr

[linux]
platform=linux
3.6.18=en-US pl
3.6=en-US en-GB

Usage

Create the file as in the example above.

Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages.

The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the release listeners, above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners.

Staging Script

• Uses a config file to stage necessary release binaries in the proper directory hierarchy.
• Releases should be staged for testing as early as possible.
• For functional tests, that will be once the binary is available for testing.
• For update tests the binaries to be staged are previous versions and can be staged at any time.
• While we typically do so internally, you do not need to provide a section for every OS/listener. Unspecified OSes will not be tested against.

Usage

./scripts/testrun_release.py --help
Usage: testrun_release.py [options]

Script to download builds for Firefox and Thunderbird from the Mozilla server. 

Options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config=CONFIG_FILE
                        Config file with a download specification, see
                        configs/release_general.cfg.example

Sample Command Line

./scripts/testrun_release.py -c 3.6.20.cfg

Options

• -c is a required option (really, a parameter in disguise). It takes the configuration file as created above.
• If everything works well, you will see the output as the script downloads and places all the correct builds.

Running a Release

Once release prep is done and the binary is available, this is how to run the actual release.

1. Push a release to the listeners
2. If any errors and reruns are necessary, push a release to a single listener

Release Test Pusher

• Pushes a test request to release listeners.
• Can push to all listeners or just one, as needed.

Usage

./release_push.py --help
Usage: release_push.py [options] update cluster branch channel
       release_push.py [options] functional cluster branch

Pushes a request for an on-demand release test to the listener. Options will
be passed to the test script run by the listener. Only listeners for the
specified cluster will run tests. If platform is specified, only listeners for
that platform within the specified cluster will run tests.

Options:
  -h, --help            show this help message and exit
  -p PLATFORM, --platform=PLATFORM
                        platform to push to [default: all]

Sample Command Line

./release_push.py update qa-foobar 3.6.20 betatest

This runs update testing on all platforms in the qa-foobar cluster, using the 3.6.20 staging directory and looking for updates on betatest.

./release_push.py --platform=win7_64 update qa-foobar 3.6.20 betatest

As above, but only on the Windows 7 64-bit listener

./release_push.py functional qa-foobar 3.6.21

This runs functional testing on all platforms in the qa-foobar cluster, using the 3.6.21 staging directory

Options/Behavior

• update/functional refers to the type of test being run.
• The value of cluster must match the cluster used to configure the listeners, above.
• The value of branch must match the name of the directory builds are staged in. This is specified by the directory key in the config file above.
• For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release.
• If no --platform option is given, all listeners for the cluster will run tests. If a --platform value is given, only listeners for that platform will run tests.
• If you're unsure that listeners are working, you can push a test against a non-existent branch. This won't run anything significant, but you can check the output of a listener console to see if it received the push.