QA/Mozmill Test Automation/On Demand Update Testing/Documentation: Difference between revisions

← Older edit

QA/Mozmill Test Automation/On Demand Update Testing/Documentation (view source)

Revision as of 15:44, 16 August 2011

6,918 bytes added , 16 August 2011

→‎Sample File

Whimboo

canmove, Confirmed users, Bureaucrats and Sysops emeriti

4,747

edits

@@ Line 1: / Line 1: @@
-The following is just an initial dump of information to be used for future documentation.
+= System Prep =
-= Heartbeat Emitter =
+Things in this section should be done to configure the system.
-* Generates traffic to keep Pulse connections from being automatically closed.
-* Must be running -somewhere- on the internet. Can be anywhere.
-* Fine to run more than one. If you're not sure, boot one.
-== Usage ==
+# Check for a functioning [[#Heartbeat_Emitter|heartbeat emitter]]. Run one if unsure.
+# Set up [[#Release_Listener|listeners]] on all desired platforms, grouped with a common cluster name.
+== Heartbeat Emitter ==
+* Generates traffic to keep [[#Release_Listener|release listener]] Pulse connections from being automatically closed.
+* Fine to run more than one. If you're not sure, start one.
+=== Usage ===
   ./heartbeat_emitter.py
-== Options/Behavior ==
+=== Options/Behavior ===
 No other requirements. It will print out local time every time a heartbeat is sent, so you can diagnose time of any server failures or freezes.
-= Release Listener =
+== Release Listener ==
 * Listens for a release request and automatically runs the correct script
-* Run one per platform. Make sure a heartbeat has been set up.
+* Run one per platform. Make sure a [[#Heartbeat_Emitter|heartbeat]] has been set up.
-== Usage ==
+=== Usage ===
   ./release_listener.py --help
@@ Line 37: / Line 41: @@
     -d, --debug           print out extra debug information on ignored messages
-=== Sample command lines ===
+=== Sample Command Line ===
   ./release_listener.py qa-foobar mac
 This runs a listener in the qa-foobar cluster, platform set to mac.
-== Options/Behavior ==
-* Cluster is an additional arbitrary key to indicate which listeners should respond. Only listeners that match a push's cluster will respond.
+=== Options/Behavior ===
-* --update will redefine which test script is called for update requests
+* Internally, our clusters are the hostnames of the Mac Pro that groups the platform VMs.
-* --functional will redefine which test script is called for functional requests
+* Platform must exactly match one of platform specifiers used in staging (and thus, usually the [[#Configuration_File|config file]], see below). Otherwise, it's an arbitrary string.
+* --update will redefine which test script is called for update requests. The default accomodates our internal systems.
+* --functional will redefine which test script is called for functional requests. Again, the default accomodates our internal systems.
+* --debug lets you know when things are ignored due to platform/cluster mismatches. This can be very noisy, but it's useful when you have no idea why something isn't responding.
+* The listeners always print output to the console regarding what they're testing. This can be useful to make sure they're working.
+* If listeners freeze up, chances are they're not receiving a [[#Heartbeat_Emitter|heartbeat]].
+= Release Prep =
+Things in this section should be done on a per-release basis, prior to the release.
+# Create a [[#Configuration_File|configuration file]] for release staging.
+# [[#Staging_Script|Stage]] the release.
+== Configuration File ==
+* Used by the [[#Staging_Script|staging script]] (below) to simplify downloading builds for a test matrix.
+* The configuration file is not used during the actual release testing process.
+Our testing scripts know how to navigate a pre-created directory hierarchy to perform tests; the [[#Staging_Script|staging script]] creates that hierarchy by reading the configuration file and putting the binaries to be tested against in the appropriate places.
+=== Sample File ===
+ ; Example configuration for an update test-run
+ [testrun]
+ ; Application to test
+ application=firefox
+ ; Type of test, could be update or bft
+ script=update
+ ; Destination folder builds will be saved to
+ directory=3.6.20
+ ; One section per platform tests are being run on:
+ ;
+ ; Section name is the specific OS listener platform, one of:
+ ;   win2000, winxp, vista, win7, win7_64, linux, linux_64, mac
+ ;
+ ; platform key refers to the binary's platform on ftp, one of:
+ ;   linux, linux64, mac, mac64, win32, win64
+ ;
+ ; Rest of keys are test matrix for that OS, as:
+ ;   version=locale1 locale2 locale3
+ ;   3.6.18=en-US (release build of Firefox 3.6.18)
+ ;   3.6.18#2=de  (candidate builds #2 of the upcoming Firefox 3.6.18)
+ [win7]
+ platform=win32
+.6.18=en-US de
+.6=en-US fi
+ [winxp]
+ platform=win32
+.6.18=en-US fr
+.6=en-US it
+ [mac]
+ platform=mac
+.6.19=en-US de
+.6.18=en-US ja-JP-mac
+.6=en-US fr
+ [linux]
+ platform=linux
+.6.18=en-US pl
+.6=en-US en-GB
+=== Usage ===
+Create the file as in the example above.
+Do note the difference between the OS platform specifiers (section names) and the binary platform specifiers (platform keys). For example, win7_64 (Windows 7 64-bit OS) could be set up to run either win32 or win64 binary packages.
+The OS specifiers must correspond to the "platform" given as a command-line parameter when launching the [[#Release_Listener|release listeners]], above. While this is arbitrary, the ones given as possibilities in the example above are the ones we use for our release listeners.
+== Staging Script ==
+* Uses a [[#Configuration_File|config file]] to stage necessary release binaries in the proper directory hierarchy.
+* Releases should be staged for testing as early as possible.
+* For functional tests, that will be once the binary is available for testing.
+* For update tests the binaries to be staged are previous versions and can be staged at any time.
+* While we typically do so internally, you do not need to provide a section for every OS/[[#Release_Listener|listener]]. Unspecified OSes will not be tested against.
+=== Usage ===
+ ./scripts/testrun_release.py --help
+ Usage: testrun_release.py [options]
+ Script to download builds for Firefox and Thunderbird from the Mozilla server.
+ Options:
+   -h, --help            show this help message and exit
+   -c CONFIG_FILE, --config=CONFIG_FILE
+                         Config file with a download specification, see
+                         configs/release_general.cfg.example
+=== Sample Command Line ===
+ ./scripts/testrun_release.py -c 3.6.20.cfg
+=== Options ===
+* -c is a required option (really, a parameter in disguise). It takes the [[#Configuration_File|configuration file]] as created above.
+* If everything works well, you will see the output as the script downloads and places all the correct builds.
+= Running a Release =
+Once release prep is done and the binary is available, this is how to run the actual release.
+# [[#Release_Test_Pusher|Push]] a release to the listeners
+# If any errors and reruns are necessary, [[#Release_Test_Pusher|push]] a release to a single listener
+== Release Test Pusher ==
+* Pushes a test request to release listeners.
+* Can push to all listeners or just one, as needed.
+=== Usage ===
+ ./release_push.py --help
+ Usage: release_push.py [options] update cluster branch channel
+        release_push.py [options] functional cluster branch
+ Pushes a request for an on-demand release test to the listener. Options will
+ be passed to the test script run by the listener. Only listeners for the
+ specified cluster will run tests. If platform is specified, only listeners for
+ that platform within the specified cluster will run tests.
+ Options:
+   -h, --help            show this help message and exit
+   -p PLATFORM, --platform=PLATFORM
+                         platform to push to [default: all]
+=== Sample Command Line ===
+ ./release_push.py update qa-foobar 3.6.20 betatest
+This runs update testing on all platforms in the qa-foobar cluster, using the 3.6.20 staging directory and looking for updates on betatest.
+ ./release_push.py --platform=win7_64 update qa-foobar 3.6.20 betatest
+As above, but only on the Windows 7 64-bit listener
+ ./release_push.py functional qa-foobar 3.6.21
+This runs functional testing on all platforms in the qa-foobar cluster, using the 3.6.21 staging directory
+=== Options/Behavior ===
+* update/functional refers to the type of test being run.
+* The value of cluster must match the cluster used to configure the [[#Release_Listener|listeners]], above.
+* The value of branch must match the name of the directory builds are staged in. This is specified by the directory key in the [[#Configuration_File|config file]] above.
+* For update testing only, the value of channel is the channel builds are available on; this is typically betatest, beta, releasetest, or release.
+* If no --platform option is given, all [[#Release_Listener|listeners]] for the cluster will run tests. If a --platform value is given, only [[#Release_Listener|listeners]] for that platform will run tests.
+* If you're unsure that [[#Release_Listener|listeners]] are working, you can push a test against a non-existent branch. This won't run anything significant, but you can check the output of a [[#Release_Listener|listener]] console to see if it received the push.