QA/Automation/Projects/Mozmill Automation/On Demand Test Framework/Docs/Current Process

Let's look at this from the standpoint of release testing, since that's what I know, and the primary on-demand task. I'm going to include a functional configuration file and an update configuration file. I'm doing this w/ configuration file because it does a pretty good job of succinctly defining the test run, not because we -have- to use it in our solution:


FUNCTIONAL:

; Example configuration for a functional test-run
;
; Possible platforms: linux, linux64, mac, win32, win64
;

[testrun]
application=firefox
directory=7.0#2
script=functional

[linux]
platform=linux
7.0#2=en-US

[linux-64]
platform=linux
7.0#2=en-US

[mac]
platform=mac
7.0#2=en-US

[win2000]
platform=win32
7.0#2=en-US

[winxp]
platform=win32
7.0#2=en-US

[vista]
platform=win32
7.0#2=en-US

[win7]
platform=win32
7.0#2=en-US

[win7-64]
platform=win32
7.0#2=en-US

So this is pretty straightforward. At a high level, it's saying download the en-US 7.0 candidate 2 build for each listed platform (win32, linux, mac) and run the functional testrun on each of the requested environments (linux, linux-64, mac, win2000, etc.)

I'll also tell you how this currently works. If we can make it work something like this, we don't have to change much in terms of the Mozmill side:

Step 1)

The first step is to create the testrun directory tree in a share every environment can see. This is done once per test run (not per-env). The configuration file is fed to a script (confusingly) called testrun_release.py, which uses the same back end as download.py. That creates a testrun directory like this:

7.0#2/
    |
    |-- functional_linux/
    |-- functional_linux-64/
    |-- functional_mac/
    |-- functional_win2000/
    |-- functional_winxp/
    |-- functional_vista/
    |-- functional_win7/
    |-- functional_win7-64/
    |-- functional_logs/

Each subdirectory (except logs) corresponds with a section of the configuration file, and is preloaded with installers of the requested platform and locale as part of this process. So functional_winxp/ gets a win32 en-US installer in it.

If we'd listed more than one locale ("7.0#2=en-US fr", for example") installers for all listed locales would be put in that directory. The installers are renamed on download so that their name includes both the platform and the locale, to make them unique within the directory.

Step 2)

A script called testrun_functional.py is now run in each env. In real life, this actually runs by either calling a wrapper by name of release_bft.py, or via a trigger to my execution daemon. But either way, it ultimately runs that script.

The parameters to this script are the version being tested and the name of the current env.

The version maps to the top-level directory created in step 1. The env maps to a functional_[env] subdirectory. testrun_functional.py simply hits that resulting directory, and runs the functional test run once per each installer found within it. So in our case, it'd run just for US English, but in the other example I gave it would run once for US and once for French.

To sum up, this test run script currently uses the directory structure as the definition of what it's going to execute. Step 1 (and the configuration file) is used to create the directory structure, and thus only indirectly drives execution.

Results are sent to Brasstacks. We do not currently have the concept of a cohesive test run, so you get one set of results per locale, per environment. Execution logs land in functional_logs as linux.log, winxp.log, generally [env].log.

That's a functional test run. Here's an update run.

UPDATE:

; Example configuration for an update test-run
;
; Possible platforms: linux, linux64, mac, win32, win64
;

[testrun]
application=firefox
directory=7.0#2
script=update

[mac]
platform=mac
4.0=en-US ja-JP-mac
4.0.1=en-US de
5.0=en-US ja-JP-mac
5.0.1=en-US fr
6.0=en-US ja-JP-mac
6.0.1=en-US de
6.0.2=en-US ko

[win2000]
platform=win32
4.0=en-US fr
4.0.1=en-US it
5.0=en-US es-ES
5.0.1=en-US ru
6.0=en-US ko
6.0.1=en-US de
6.0.2=en-US zh-TW

[winxp]
platform=win32
4.0=en-US fr
4.0.1=en-US it
5.0=en-US es-ES
5.0.1=en-US ru
6.0=en-US ko
6.0.1=en-US de
6.0.2=en-US zh-TW

...etc. Clipped for length.

This is an update run for the same release. Most things work the same as above, but there an important difference. In an update test run, you don't download the build being tested; instead you download the build being updated from, and what you're testing is that you end up on the build being released after updating.

We also do a -ton- more locale testing for updates. I'm unsure why. Probably because any individual build's test run is overall quicker, so can be repeated more without much loss in elapsed time.

Step 1)

Run testrun_release.py on a shared volume, same as above. Now you'll get this directory:

7.0#2/
    |
    |-- update_mac/
    |-- update_win2000/
    |-- update_winxp/
    |-- update_logs/
    |-- ...

The only difference here is the prefix (which by now you might have guessed comes from the script=foo key in the configuration file). This is so the same version directory can be reused for both functional and update runs.

It's important to recognize that we have a lot more versions and locales downloading now, not just one English build per env. The mac subdirectory, for example, will now have installers for:

4.0 US and Japanese
4.0.1 US and German
5.0 US and Japanese
5.0.1 US and French
6.0 US and Japanese
6.0.1 US and German
6.0.2 US and Korean

...for a total of 14 builds being tested as update sources on Mac. This is fairly typical since we went to the configuration-oriented system. When things were downloaded more manually, tended to only be three source versions and two locales each for a total of six builds, but with more ease of use came a scale-up.

Step 2)

Again, same basic process, but now it's testrun_update.py, run once per env. As before, in real life this is usually called from a release_update.py front end script, or by my execution daemon.

The parameters are version being tested, name of current env, and a new one: the update channel being tested.

Behavior's the same. Any given env will look in its corresponding subdirectory and iterate through every installer there. It will install it, initiate the update process, and look to be on 7.0 candidate 2 when it's done in order to define success.

Once again, the file/directory structure is what determines execution, and Step 1 creates the directory structure. The configuration file only indirectly defines execution by defining how the structure is created.

As before, Brasstacks has no way to bind the results together, so you get one result per version, per locale. With so many locales, this is a ginormous pain so Henrik implemented an overview view that mostly works. Execution logs all go into the update_logs directory with similar naming as above.