QA/Mozmill Test Automation/On Demand Update Testing/Documentation
- 1 System Prep
- 2 Release Prep
- 3 Running a Release
Things in this section should be done to configure the system.
- Check for a functioning heartbeat emitter. Run one if unsure.
- Set up listeners on all desired platforms, grouped with a common cluster name.
- Generates traffic to keep release listener Pulse connections from being automatically closed.
- Fine to run more than one. If you're not sure, start one.
No other requirements. It will print out local time every time a heartbeat is sent, so you can diagnose time of any server failures or freezes.
- Listens for a release request and automatically runs the correct script
- Run one per platform. Make sure a heartbeat has been set up.
./release_listener.py --help Usage: release_listener.py [options] cluster platform Launches a listener for the on-demand testing system. The listener will only respond to requests for the specified cluster. The listener will respond to requests for either the specified platform or for all platforms. Options: -h, --help show this help message and exit -u UPDATE_SCRIPT, --update=UPDATE_SCRIPT update test script to run [default: ./release_update.py] -f FUNCTIONAL_SCRIPT, --functional=FUNCTIONAL_SCRIPT functional test script to run [default: ./release_bft.py] -d, --debug print out extra debug information on ignored messages
Sample Command Line
./release_listener.py qa-foobar mac
This runs a listener in the qa-foobar cluster, platform set to mac.
- Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs.
- Platform must exactly match one of platform specifiers used in staging (and thus, usually the config file, see below). Otherwise, it's an arbitrary string.
- --update will redefine which test script is called for update requests. The default accomodates our internal systems.
- --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems.
- --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding.
- The listeners always print output to the console regarding what they're testing. This can be useful to make sure they're working.
- If listeners freeze up, chances are they're not receiving a heartbeat.
Things in this section should be done on a per-release basis, prior to the release.
- Used by the staging script (below) to simplify downloading builds for a test matrix.
- The configuration file is not used during the actual release testing process.
Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the staging script creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places.
; Example configuration for an update test-run [testrun] ; Application to test application=firefox ; Type of test, could be update or bft script=update ; Destination folder builds will be saved to directory=3.6.20 ; One section per platform tests are being run on: ; ; Section name is the specific OS listener platform, one of: ; win2000, winxp, vista, win7, win7_64, linux, linux_64, mac ; ; platform key refers to the binary's platform on ftp, one of: ; linux, linux64, mac, mac64, win32, win64 ; ; Rest of keys are test matrix for that OS, as: ; version=locale1 locale2 locale3 ; 3.6.18=en-US (release build of Firefox 3.6.18) ; 3.6.18#2=de (candidate builds #2 of the upcoming Firefox 3.6.18) [win7] platform=win32 3.6.18=en-US de 3.6=en-US fi [winxp] platform=win32 3.6.18=en-US fr 3.6=en-US it [mac] platform=mac 3.6.19=en-US de 3.6.18=en-US ja-JP-mac 3.6=en-US fr [linux] platform=linux 3.6.18=en-US pl 3.6=en-US en-GB
Create the file as in the example above.
Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages.
The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the release listeners, above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners.
- Uses a config file to stage necessary release binaries in the proper directory hierarchy.
- Releases should be staged for testing as early as possible.
- For functional tests, that will be once the binary is available for testing.
- For update tests the binaries to be staged are previous versions and can be staged at any time.
- While we typically do so internally, you do not need to provide a section for every OS/listener. Unspecified OSes will not be tested against.
./scripts/testrun_release.py --help Usage: testrun_release.py [options] Script to download builds for Firefox and Thunderbird from the Mozilla server. Options: -h, --help show this help message and exit -c CONFIG_FILE, --config=CONFIG_FILE Config file with a download specification, see configs/release_general.cfg.example
Sample Command Line
./scripts/testrun_release.py -c 3.6.20.cfg
- -c is a required option (really, a parameter in disguise). It takes the configuration file as created above.
- If everything works well, you will see the output as the script downloads and places all the correct builds.
Running a Release
Once release prep is done and the binary is available, this is how to run the actual release.
- Push a release to the listeners
- If any errors and reruns are necessary, push a release to a single listener
Release Test Pusher
- Pushes a test request to release listeners.
- Can push to all listeners or just one, as needed.
./release_push.py --help Usage: release_push.py [options] update cluster branch channel release_push.py [options] functional cluster branch Pushes a request for an on-demand release test to the listener. Options will be passed to the test script run by the listener. Only listeners for the specified cluster will run tests. If platform is specified, only listeners for that platform within the specified cluster will run tests. Options: -h, --help show this help message and exit -p PLATFORM, --platform=PLATFORM platform to push to [default: all]
Sample Command Line
./release_push.py update qa-foobar 3.6.20 betatest
This runs update testing on all platforms in the qa-foobar cluster, using the 3.6.20 staging directory and looking for updates on betatest.
./release_push.py --platform=win7_64 update qa-foobar 3.6.20 betatest
As above, but only on the Windows 7 64-bit listener
./release_push.py functional qa-foobar 3.6.21
This runs functional testing on all platforms in the qa-foobar cluster, using the 3.6.21 staging directory
- update/functional refers to the type of test being run.
- The value of cluster must match the cluster used to configure the listeners, above.
- The value of branch must match the name of the directory builds are staged in. This is specified by the directory key in the config file above.
- For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release.
- If no --platform option is given, all listeners for the cluster will run tests. If a --platform value is given, only listeners for that platform will run tests.
- If you're unsure that listeners are working, you can push a test against a non-existent branch. This won't run anything significant, but you can check the output of a listener console to see if it received the push.