QA/Mozmill Test Automation/On Demand Update Testing/Documentation

System Prep

Things in this section should be done to configure the system.

1. Check for a functioning heartbeat emitter. Run one if unsure.
2. Set up listeners on all desired platforms, grouped with a common cluster name.

Heartbeat Emitter

• Generates traffic to keep Pulse connections from being automatically closed.
• Must be running -somewhere- on the internet. Can be anywhere.
• Fine to run more than one. If you're not sure, boot one.

Usage

./heartbeat_emitter.py

Options/Behavior

No other requirements. It will print out local time every time a heartbeat is sent, so you can diagnose time of any server failures or freezes.

Release Listener

• Listens for a release request and automatically runs the correct script
• Run one per platform. Make sure a heartbeat has been set up.

Usage

./release_listener.py --help
Usage: release_listener.py [options] cluster platform

Launches a listener for the on-demand testing system. The listener will only 
respond to requests for the specified cluster. The listener will respond to
requests for either the specified platform or for all platforms.

Options:
  -h, --help            show this help message and exit
  -u UPDATE_SCRIPT, --update=UPDATE_SCRIPT
                        update test script to run [default:
                        ./release_update.py]
  -f FUNCTIONAL_SCRIPT, --functional=FUNCTIONAL_SCRIPT
                        functional test script to run [default:
                        ./release_bft.py]
  -d, --debug           print out extra debug information on ignored messages

Sample command lines

./release_listener.py qa-foobar mac

This runs a listener in the qa-foobar cluster, platform set to mac.

Options/Behavior

• Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs.
• Platform must exactly match one of platform specifiers used in the staging (and thus, usually the config file, see below). Otherwise, it's an arbitrary string.
• --update will redefine which test script is called for update requests. The default accomodates our internal systems.
• --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems.
• --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding.

Release Prep

The things in this section should be done on a per-release basis, prior to the release.

1. Create a configuration file for release staging.
2. If update testing, pre-stage the release.

Configuration File

Sample File

; Example configuration for an update test-run

[testrun]
; Application to test
application=firefox

; Type of test, could be update or bft
script=update

; Destination folder builds will be saved to
directory=3.6.20

; One section per platform tests are being run on:
;
; Section name is the specific OS listener platform, one of:
;   win2000, winxp, vista, win7, win7_64, linux, linux_64, mac
;
; platform key refers to the binary's platform on ftp, one of:
;   linux, linux64, mac, mac64, win32, win64
;
; Rest of keys are test matrix for that OS, as:
;   version=locale1 locale2 locale3

[win7]
platform=win32
3.6.18=en-US de
3.6=en-US fi

[winxp]
platform=win32
3.6.18=en-US fr
3.6=en-US it

[mac]
platform=mac
3.6.19=en-US de
3.6.18=en-US ja-JP-mac
3.6=en-US fr

[linux]
platform=linux
3.6.18=en-US pl
3.6=en-US en-GB

Usage

Create the file as in template above.

Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages.

The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the release listeners, above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners.