canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,747
edits
QA/Mozmill Test Automation/On Demand Update Testing/Documentation: Difference between revisions
QA/Mozmill Test Automation/On Demand Update Testing/Documentation (view source)
Revision as of 15:44, 16 August 2011
, 16 August 2011→Sample File
| (12 intermediate revisions by one other user not shown) | |||
| Line 7: | Line 7: | ||
== Heartbeat Emitter == | == Heartbeat Emitter == | ||
* Generates traffic to keep Pulse connections from being automatically closed. | * Generates traffic to keep [[#Release_Listener|release listener]] Pulse connections from being automatically closed. | ||
* Fine to run more than one. If you're not sure, start one. | * Fine to run more than one. If you're not sure, start one. | ||
| Line 41: | Line 41: | ||
-d, --debug print out extra debug information on ignored messages | -d, --debug print out extra debug information on ignored messages | ||
=== Sample | === Sample Command Line === | ||
./release_listener.py qa-foobar mac | ./release_listener.py qa-foobar mac | ||
| Line 52: | Line 52: | ||
* --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems. | * --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems. | ||
* --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding. | * --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding. | ||
* The listeners always print output to the console regarding what they're testing. This can be useful to make sure they're working. | |||
* If listeners freeze up, chances are they're not receiving a [[#Heartbeat_Emitter|heartbeat]]. | |||
= Release Prep = | = Release Prep = | ||
| Line 64: | Line 66: | ||
* The configuration file is not used during the actual release testing process. | * The configuration file is not used during the actual release testing process. | ||
Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the #Staging_Script staging script] creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places. | Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the [[#Staging_Script|staging script]] creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places. | ||
=== Sample File === | === Sample File === | ||
| Line 90: | Line 92: | ||
; Rest of keys are test matrix for that OS, as: | ; Rest of keys are test matrix for that OS, as: | ||
; version=locale1 locale2 locale3 | ; version=locale1 locale2 locale3 | ||
; 3.6.18=en-US (release build of Firefox 3.6.18) | |||
; 3.6.18#2=de (candidate builds #2 of the upcoming Firefox 3.6.18) | |||
[win7] | [win7] | ||
| Line 125: | Line 129: | ||
* For functional tests, that will be once the binary is available for testing. | * For functional tests, that will be once the binary is available for testing. | ||
* For update tests the binaries to be staged are previous versions and can be staged at any time. | * For update tests the binaries to be staged are previous versions and can be staged at any time. | ||
* While we typically do so internally, you do not need to provide a section for every OS/[[#Release_Listener|listener]]. Unspecified OSes will not be tested against. | |||
=== Usage === | === Usage === | ||
| Line 172: | Line 177: | ||
-p PLATFORM, --platform=PLATFORM | -p PLATFORM, --platform=PLATFORM | ||
platform to push to [default: all] | platform to push to [default: all] | ||
=== Sample Command Line === | |||
./release_push.py update qa-foobar 3.6.20 betatest | |||
This runs update testing on all platforms in the qa-foobar cluster, using the 3.6.20 staging directory and looking for updates on betatest. | |||
./release_push.py --platform=win7_64 update qa-foobar 3.6.20 betatest | |||
As above, but only on the Windows 7 64-bit listener | |||
./release_push.py functional qa-foobar 3.6.21 | |||
This runs functional testing on all platforms in the qa-foobar cluster, using the 3.6.21 staging directory | |||
=== Options/Behavior === | === Options/Behavior === | ||
| Line 180: | Line 198: | ||
* For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release. | * For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release. | ||
* If no --platform option is given, all [[#Release_Listener|listeners]] for the cluster will run tests. If a --platform value is given, only [[#Release_Listener|listeners]] for that platform will run tests. | * If no --platform option is given, all [[#Release_Listener|listeners]] for the cluster will run tests. If a --platform value is given, only [[#Release_Listener|listeners]] for that platform will run tests. | ||
* If you're unsure that [[#Release_Listener|listeners]] are working, you can push a test against a non-existent branch. This won't run anything significant, but you can check the output of a [[#Release_Listener|listener]] console to see if it received the push. | |||