Sisyphus - JavaScript Tests

Running the JavaScript Tests

detect-universe.sh

Utility bash script to generate the particular values for the universe data for the current test machine. Run this script and append the data to the current universe.data file in the JavaScript Tests directory prior to running the tests on a new machine. This will allow the current public-failures.txt and confidential-failures.txt files to be expanded with values appropriate for your machine.

usage: detect-universe.sh -p products -b branches -T buildtypes

Outputs to stdout the universe data for this machine.

variable            description
===============     ============================================================
-p products         required. one or more of firefox, thunderbird, js
-b branches         required. one or more of 1.8.0, 1.8.1, 1.9.0, 1.9.1
-T buildtype        required. one or more of opt debug

if an argument contains more than one value, it must be quoted.

$ detect-universe.sh -p "js firefox" -b "1.9.0 1.9.1" -T "opt debug"

TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=debug, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=debug, TEST_TYPE=shell
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=opt, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=opt, TEST_TYPE=shell
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=debug, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=debug, TEST_TYPE=shell
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=opt, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=opt, TEST_TYPE=shell

runtests.sh

runtests.sh is the main entry point for running the JavaScript Tests using Sisyphus. It is very similar in usage to Sisyphus_-_Usage#builder.sh.

usage: runtests.sh -p products -b branches -e extra\
                   -T  buildtypes -B buildcommands  \
                   [-v] [-S] [-X excludetests] [-I includetests] [-c] [-t] \
                   [-Z n]

variable            description
===============     ============================================================
-p products         space separated list of js, firefox
-b branches         space separated list of branches 1.8.0, 1.8.1, 1.9.0, 1.9.1
-e extra            optional. extra qualifier to pick build tree and mozconfig.
-T buildtypes       space separated list of build types opt debug
-B buildcommands    optional space separated list of build commands
                    clean, checkout, build. If not specified, defaults to
                    'clean checkout build'. 
                    If you wish to skip any build steps, simply specify a value
                    not containing any of the build commands, e.g. 'none'.
-v                  optional. verbose - copies log file output to stdout.
-S                  optional. summary - output tailered for use with
                    Buildbot|Tinderbox
-X excludetests     optional. By default the test will exclude the 
                    tests listed in spidermonkey-n-$branch.tests, 
                    performance-$branch.tests. excludetests is a list of either
                    individual tests, manifest files or sub-directories which 
                    will override the default exclusion list.
-I includetests     optional. By default the test will include the 
                    JavaScript tests appropriate for the branch. includetests is a
                    list of either individual tests, manifest files or 
                    sub-directories which will override the default inclusion 
                    list.
-c                  optional. By default the test will exclude tests 
                    which crash on this branch, test type, build type and 
                    operating system. -c will include tests which crash. 
                    Typically this should only be used in combination with -R. 
                    This has no effect on shell based tests which execute crash
                    tests regardless.
-t                  optional. By default the test will exclude tests 
                    which time out on this branch, test type, build type and 
                    operating system. -t will include tests which timeout.
-Z n                optional. Set gczeal to n. Currently, only valid for 
                    debug builds of Gecko 1.8.1.15, 1.9.0 and later.

if an argument contains more than one value, it must be quoted.

For example, to checkout and build opt and debug builds of the JavaScript Shell and Firefox for the 1.8.1 and 1.9.0 branches, and then run the shell and browser JavaScript Tests you would use:

$TEST_DIR/tests/mozilla.org/js/runtests.sh -p "js firefox" -b "1.8.1 1.9.0" -T "opt debug" -B "checkout build" 

Like builder.sh, runtests.sh will write the names of the log files it is generating to stdout.

log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-20-47-0400,js,1.8.1,,linux,sophie.bclary.com,checkout.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-05-0400,js,1.9.0,,linux,sophie.bclary.com,checkout.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-17-0400,js,1.8.1,opt,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-29-0400,js,1.8.1,debug,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-36-0400,js,1.9.0,opt,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-48-0400,js,1.9.0,debug,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-59-0400,js,1.8.1,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log 
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-13-26-04-0400,js,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log 
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-16-19-28-0400,js,1.9.0,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log 
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-17-00-31-0400,js,1.9.0,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log 
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-59-0400,js,1.8.1,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-59-0400,js,1.8.1,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-13-26-04-0400,js,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-13-26-04-0400,js,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-16-19-28-0400,js,1.9.0,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-16-19-28-0400,js,1.9.0,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-17-00-31-0400,js,1.9.0,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-17-00-31-0400,js,1.9.0,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log

The checkout and build logs should be familiar from builder.sh.

The test logs have a similar format:

2008-07-09-11-21-59-0400,firefox,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log
---- -- -- -- -- ------- ------- ----- ----- ----- ----------------- --------------------
U Y  M  D  H  M  S T O      P      B    B T   O S     M N              T P
T e  o  a  o  i  e i f      r      r    u y   p y     a a              e a
C a  n  y  u  n  c m f      o      a    i p   e s     c m              s t
  r  t     r  u  o e s      d      n    l e   r t     h e              t h
     h        t  n z e      u      c    d     a e     i                
              e  d o t      c      h          t m     n                
              s  s n        t                 i       e                
                   e                          n
                                              g

The possible fixes and regressions log names are constructed from the test logs during a post processing phase where the raw log files are converted into a common format. In addition to the "possible-fixes.log" and the "possible-regressions.log", an "all" version containing all test results and a "failures" version containing only failures are produced.

universe.data, public-failures.txt and confidential-failures.txt

public-failures.txt and confidential-failures.txt contain an "encoded" list of known failures in a format very similar to that of the post process test result logs (minus the TEST_MACHINE and TEST_DATE fields). The files are used to create a working file which is used to detect the possible fixes and regressions during a test run.

These files are "encoded" by using the wild card "any" pattern in a field to represent occurences of all possible values within a set "universe" of values. The file universe.data contains the set of test run configurations which have been used to collect the known failure information. When a test run begins, the Perl script pattern-expander.pl is used to expand the "encoded" files public-failures.txt and confidential-failures.txt into public-failures.txt.expanded and confidential-failures.txt.expanded. The "expanded" known failures files are combined into a single file failures.txt which is used to detect fixes and regressions.

When initially running the JavaScript Tests on a new platform or machine, you should run the detect-universe.sh script to determine the universe values to add to the universe.data file.

If you already have a collection of post-processed log files, you can obtain the necessary universe data from them by running the get-universe.sh script against the "all" versions of your test logs.

For example,

get-universe.sh *all.log > new-universe.data

Once you have updated the universe.data file with values for your test environment, reprocess the log files using the script process-logs.sh in the same directory as the log files to regenerate the possible fixes and failures.

process-logs.sh -l '*_js.log'

If you have more than one test run, for example tests for more than one branch, test type, or build type or results from several different machines, you can easily combine the results into a file for fixes and a file for regressions using the changes.sh script in the same directory as the log files.

changes.sh

This will combine all of the *-possible-fixes.log into a single file possible-fixes.log and all of the *-possible-regressions.log into a single file possible-regressions.log for easy review of the combined results.

Updating public-failures.txt and confidential-failures.txt

TODO

JavaScript Test Environment Variables

Some of the JavaScript tests can take quite a long time to run. In fact, some tests will never complete unless they are killed. In cases where a "too much gc" build, gczeal is used or when valgrind is used, the time for a test to complete may increase dramatically. To control the time alloted for the JavaScript tests to run, the following environment variables are used:

• TEST_JSALL_TIMEOUT=${TEST_JSALL_TIMEOUT:-21600}

The time in seconds for the entire test run to complete before it is terminated. By default, it is 21600 seconds or 8.5 hours.

• TEST_JSEACH_PAGE_TIMEOUT=${TEST_JSEACH_PAGE_TIMEOUT:-480}

The time in seconds for an individual test to load in the browser before the JavaScript test itselfs attempts to time out the test. By default it is 480 seconds or 8 minutes.

• TEST_JSEACH_TIMEOUT=${TEST_JSEACH_TIMEOUT:-485}

The time in seconds for an individual test to load/execute in the browser before it is forcibly terminated. By default it is 485 seconds or 8 minutes 5 seconds.

• TEST_JSSHELL_TIMEOUT=${TEST_JSSHELL_TIMEOUT:-480}

The time in seconds for an individual test to run in the shell.

Back to Sisyphus