canmove, Confirmed users
2,041
edits
QA/Mozmill Test Automation/On Demand Update Testing/Documentation: Difference between revisions
Jump to navigation
Jump to search
QA/Mozmill Test Automation/On Demand Update Testing/Documentation (view source)
Revision as of 00:12, 16 August 2011
, 16 August 2011no edit summary
(→Usage) |
No edit summary |
||
| Line 3: | Line 3: | ||
Things in this section should be done to configure the system. | Things in this section should be done to configure the system. | ||
# Check for a functioning heartbeat emitter. Run one if unsure. | # Check for a functioning [[#Heartbeat_Emitter|heartbeat emitter]]. Run one if unsure. | ||
# Set up listeners on all desired platforms, grouped with a common cluster name. | # Set up [[#Release_Listener|listeners]] on all desired platforms, grouped with a common cluster name. | ||
== Heartbeat Emitter == | == Heartbeat Emitter == | ||
| Line 20: | Line 20: | ||
== Release Listener == | == Release Listener == | ||
* Listens for a release request and automatically runs the correct script | * Listens for a release request and automatically runs the correct script | ||
* Run one per platform. Make sure a heartbeat has been set up. | * Run one per platform. Make sure a [[#Heartbeat_Emitter|heartbeat]] has been set up. | ||
=== Usage === | === Usage === | ||
| Line 47: | Line 47: | ||
=== Options/Behavior === | === Options/Behavior === | ||
* Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs. | * Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs. | ||
* Platform must exactly match one of platform specifiers used in | * Platform must exactly match one of platform specifiers used in staging (and thus, usually the [[#Configuration_File|config file]], see below). Otherwise, it's an arbitrary string. | ||
* --update will redefine which test script is called for update requests. The default accomodates our internal systems. | * --update will redefine which test script is called for update requests. The default accomodates our internal systems. | ||
* --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems. | * --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems. | ||
| Line 54: | Line 54: | ||
= Release Prep = | = Release Prep = | ||
Things in this section should be done on a per-release basis, prior to the release. | |||
# Create a configuration file for release staging. | # Create a [[#Configuration_File|configuration file]] for release staging. | ||
# | # [[#Staging_Script|Stage]] the release. | ||
== Configuration File == | == Configuration File == | ||
* Used by the [[#Staging_Script|staging script]] (below) to simplify downloading builds for a test matrix. | |||
* The configuration file is not used during the actual release testing process. | |||
Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the #Staging_Script staging script] creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places. | |||
Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the staging script creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places. | |||
=== Sample File === | === Sample File === | ||
| Line 117: | Line 117: | ||
Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages. | Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages. | ||
The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the release listeners, above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners. | The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the [[#Release_Listener|release listeners]], above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners. | ||
== Staging Script == | |||
* Uses a [[#Configuration_File|config file]] to stage necessary release binaries in the proper directory hierarchy. | |||
* Releases should be staged for testing as early as possible. | |||
* For functional tests, that will be once the binary is available for testing. | |||
* For update tests the binaries to be staged are previous versions and can be staged at any time. | |||
=== Usage === | |||
./scripts/testrun_release.py --help | |||
Usage: testrun_release.py [options] | |||
Script to download builds for Firefox and Thunderbird from the Mozilla server. | |||
Options: | |||
-h, --help show this help message and exit | |||
-c CONFIG_FILE, --config=CONFIG_FILE | |||
Config file with a download specification, see | |||
configs/release_general.cfg.example | |||
=== Sample Command Line === | |||
./scripts/testrun_release.py -c 3.6.20.cfg | |||
=== Options === | |||
* -c is a required option (really, a parameter in disguise). It takes the [[#Configuration_File|configuration file]] as created above. | |||
* If everything works well, you will see the output as the script downloads and places all the correct builds. | |||
= Running a Release = | |||
Once release prep is done and the binary is available, this is how to run the actual release. | |||
# [[#Release_Test_Pusher|Push]] a release to the listeners | |||
# If any errors and reruns are necessary, [[#Release_Test_Pusher|push]] a release to a single listener | |||
== Release Test Pusher == | |||
* Pushes a test request to release listeners. | |||
* Can push to all listeners or just one, as needed. | |||
=== Usage === | |||
./release_push.py --help | |||
Usage: release_push.py [options] update cluster branch channel | |||
release_push.py [options] functional cluster branch | |||
Pushes a request for an on-demand release test to the listener. Options will | |||
be passed to the test script run by the listener. Only listeners for the | |||
specified cluster will run tests. If platform is specified, only listeners for | |||
that platform within the specified cluster will run tests. | |||
Options: | |||
-h, --help show this help message and exit | |||
-p PLATFORM, --platform=PLATFORM | |||
platform to push to [default: all] | |||
=== Options/Behavior === | |||
* update/functional refers to the type of test being run. | |||
* The value of cluster must match the cluster used to configure the [[#Release_Listener|listeners]], above. | |||
* The value of branch must match the name of the directory builds are staged in. This is specified by the directory key in the [[#Configuration_File|config file]] above. | |||
* For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release. | |||
* If no --platform option is given, all [[#Release_Listener|listeners]] for the cluster will run tests. If a --platform value is given, only [[#Release_Listener|listeners]] for that platform will run tests. | |||