canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,747
edits
QA/Mozmill Test Automation/On Demand Update Testing/Documentation: Difference between revisions
QA/Mozmill Test Automation/On Demand Update Testing/Documentation (view source)
Revision as of 15:44, 16 August 2011
, 16 August 2011→Sample File
| (15 intermediate revisions by one other user not shown) | |||
| Line 3: | Line 3: | ||
Things in this section should be done to configure the system. | Things in this section should be done to configure the system. | ||
# Check for a functioning heartbeat emitter. Run one if unsure. | # Check for a functioning [[#Heartbeat_Emitter|heartbeat emitter]]. Run one if unsure. | ||
# Set up listeners on all desired platforms, grouped with a common cluster name. | # Set up [[#Release_Listener|listeners]] on all desired platforms, grouped with a common cluster name. | ||
== Heartbeat Emitter == | == Heartbeat Emitter == | ||
* Generates traffic to keep Pulse connections from being automatically closed. | * Generates traffic to keep [[#Release_Listener|release listener]] Pulse connections from being automatically closed. | ||
* Fine to run more than one. If you're not sure, start one. | * Fine to run more than one. If you're not sure, start one. | ||
| Line 20: | Line 20: | ||
== Release Listener == | == Release Listener == | ||
* Listens for a release request and automatically runs the correct script | * Listens for a release request and automatically runs the correct script | ||
* Run one per platform. Make sure a heartbeat has been set up. | * Run one per platform. Make sure a [[#Heartbeat_Emitter|heartbeat]] has been set up. | ||
=== Usage === | === Usage === | ||
| Line 41: | Line 41: | ||
-d, --debug print out extra debug information on ignored messages | -d, --debug print out extra debug information on ignored messages | ||
=== Sample | === Sample Command Line === | ||
./release_listener.py qa-foobar mac | ./release_listener.py qa-foobar mac | ||
This runs a listener in the qa-foobar cluster, platform set to mac. | This runs a listener in the qa-foobar cluster, platform set to mac. | ||
=== Options/Behavior === | === Options/Behavior === | ||
* Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs. | * Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs. | ||
* Platform must exactly match one of platform specifiers used in | * Platform must exactly match one of platform specifiers used in staging (and thus, usually the [[#Configuration_File|config file]], see below). Otherwise, it's an arbitrary string. | ||
* --update will redefine which test script is called for update requests. The default accomodates our internal systems. | * --update will redefine which test script is called for update requests. The default accomodates our internal systems. | ||
* --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems. | * --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems. | ||
* --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding. | * --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding. | ||
* The listeners always print output to the console regarding what they're testing. This can be useful to make sure they're working. | |||
* If listeners freeze up, chances are they're not receiving a [[#Heartbeat_Emitter|heartbeat]]. | |||
= Release Prep = | = Release Prep = | ||
Things in this section should be done on a per-release basis, prior to the release. | |||
# Create a configuration file for release staging. | # Create a [[#Configuration_File|configuration file]] for release staging. | ||
# | # [[#Staging_Script|Stage]] the release. | ||
== Configuration File == | == Configuration File == | ||
* Used by the [[#Staging_Script|staging script]] (below) to simplify downloading builds for a test matrix. | |||
* The configuration file is not used during the actual release testing process. | |||
Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the [[#Staging_Script|staging script]] creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places. | |||
Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the staging script creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places. | |||
=== Sample File === | === Sample File === | ||
| Line 89: | Line 92: | ||
; Rest of keys are test matrix for that OS, as: | ; Rest of keys are test matrix for that OS, as: | ||
; version=locale1 locale2 locale3 | ; version=locale1 locale2 locale3 | ||
; 3.6.18=en-US (release build of Firefox 3.6.18) | |||
; 3.6.18#2=de (candidate builds #2 of the upcoming Firefox 3.6.18) | |||
[win7] | [win7] | ||
| Line 113: | Line 118: | ||
=== Usage === | === Usage === | ||
Create the file as in | Create the file as in the example above. | ||
Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages. | Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages. | ||
The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the release listeners, above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners. | The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the [[#Release_Listener|release listeners]], above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners. | ||
== Staging Script == | |||
* Uses a [[#Configuration_File|config file]] to stage necessary release binaries in the proper directory hierarchy. | |||
* Releases should be staged for testing as early as possible. | |||
* For functional tests, that will be once the binary is available for testing. | |||
* For update tests the binaries to be staged are previous versions and can be staged at any time. | |||
* While we typically do so internally, you do not need to provide a section for every OS/[[#Release_Listener|listener]]. Unspecified OSes will not be tested against. | |||
=== Usage === | |||
./scripts/testrun_release.py --help | |||
Usage: testrun_release.py [options] | |||
Script to download builds for Firefox and Thunderbird from the Mozilla server. | |||
Options: | |||
-h, --help show this help message and exit | |||
-c CONFIG_FILE, --config=CONFIG_FILE | |||
Config file with a download specification, see | |||
configs/release_general.cfg.example | |||
=== Sample Command Line === | |||
./scripts/testrun_release.py -c 3.6.20.cfg | |||
=== Options === | |||
* -c is a required option (really, a parameter in disguise). It takes the [[#Configuration_File|configuration file]] as created above. | |||
* If everything works well, you will see the output as the script downloads and places all the correct builds. | |||
= Running a Release = | |||
Once release prep is done and the binary is available, this is how to run the actual release. | |||
# [[#Release_Test_Pusher|Push]] a release to the listeners | |||
# If any errors and reruns are necessary, [[#Release_Test_Pusher|push]] a release to a single listener | |||
== Release Test Pusher == | |||
* Pushes a test request to release listeners. | |||
* Can push to all listeners or just one, as needed. | |||
=== Usage === | |||
./release_push.py --help | |||
Usage: release_push.py [options] update cluster branch channel | |||
release_push.py [options] functional cluster branch | |||
Pushes a request for an on-demand release test to the listener. Options will | |||
be passed to the test script run by the listener. Only listeners for the | |||
specified cluster will run tests. If platform is specified, only listeners for | |||
that platform within the specified cluster will run tests. | |||
Options: | |||
-h, --help show this help message and exit | |||
-p PLATFORM, --platform=PLATFORM | |||
platform to push to [default: all] | |||
=== Sample Command Line === | |||
./release_push.py update qa-foobar 3.6.20 betatest | |||
This runs update testing on all platforms in the qa-foobar cluster, using the 3.6.20 staging directory and looking for updates on betatest. | |||
./release_push.py --platform=win7_64 update qa-foobar 3.6.20 betatest | |||
As above, but only on the Windows 7 64-bit listener | |||
./release_push.py functional qa-foobar 3.6.21 | |||
This runs functional testing on all platforms in the qa-foobar cluster, using the 3.6.21 staging directory | |||
=== Options/Behavior === | |||
* update/functional refers to the type of test being run. | |||
* The value of cluster must match the cluster used to configure the [[#Release_Listener|listeners]], above. | |||
* The value of branch must match the name of the directory builds are staged in. This is specified by the directory key in the [[#Configuration_File|config file]] above. | |||
* For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release. | |||
* If no --platform option is given, all [[#Release_Listener|listeners]] for the cluster will run tests. If a --platform value is given, only [[#Release_Listener|listeners]] for that platform will run tests. | |||
* If you're unsure that [[#Release_Listener|listeners]] are working, you can push a test against a non-existent branch. This won't run anything significant, but you can check the output of a [[#Release_Listener|listener]] console to see if it received the push. | |||