Sisyphus - JavaScript Tests: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Running the JavaScript Tests =
This content is obsolete.
 
== detect-universe.sh ==
 
Utility bash script to generate the particular values for the universe data for the current test machine. Run this script and append the data to the current universe.data file in the JavaScript Tests directory prior to running the tests on a new machine. This will allow the [[#universe.data, public-failures.txt and confidential-failures.txt|current public-failures.txt and confidential-failures.txt]] files to be expanded with values appropriate for your machine.
 
<pre><nowiki>
usage: detect-universe.sh -p products -b branches -T buildtypes
 
Outputs to stdout the universe data for this machine.
 
variable            description
===============    ============================================================
-p products        required. one or more of firefox, thunderbird, js
-b branches        required. one or more of 1.8.0, 1.8.1, 1.9.0, 1.9.1
-T buildtype        required. one or more of opt debug
 
if an argument contains more than one value, it must be quoted.
</nowiki></pre>
 
<pre>
$ detect-universe.sh -p "js firefox" -b "1.9.0 1.9.1" -T "opt debug"
 
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=debug, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=debug, TEST_TYPE=shell
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=opt, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.0, TEST_BUILDTYPE=opt, TEST_TYPE=shell
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=debug, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=debug, TEST_TYPE=shell
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=opt, TEST_TYPE=browser
TEST_OS=linux, TEST_KERNEL=2.6.18, TEST_PROCESSORTYPE=intel64, TEST_MEMORY=18, TEST_CPUSPEED=fast, TEST_TIMEZONE=-0400, TEST_BRANCH=1.9.1, TEST_BUILDTYPE=opt, TEST_TYPE=shell
</pre>
 
== runtests.sh ==
 
<code>runtests.sh</code> is the main entry point for running the JavaScript Tests using Sisyphus. It is very similar in usage to [[Sisphus - Usage:#builder.sh|builder.sh]].
 
<pre><nowiki>
usage: runtests.sh -p products -b branches -e extra\
                  -T  buildtypes -B buildcommands  \
                  [-v] [-S] [-X excludetests] [-I includetests] [-c] [-t] \
                  [-Z n]
 
variable            description
===============    ============================================================
-p products        space separated list of js, firefox
-b branches        space separated list of branches 1.8.0, 1.8.1, 1.9.0, 1.9.1
-e extra            optional. extra qualifier to pick build tree and mozconfig.
-T buildtypes      space separated list of build types opt debug
-B buildcommands    optional space separated list of build commands
                    clean, checkout, build. If not specified, defaults to
                    'clean checkout build'.
                    If you wish to skip any build steps, simply specify a value
                    not containing any of the build commands, e.g. 'none'.
-v                  optional. verbose - copies log file output to stdout.
-S                  optional. summary - output tailered for use with
                    Buildbot|Tinderbox
-X excludetests    optional. By default the test will exclude the
                    tests listed in spidermonkey-n-$branch.tests,
                    performance-$branch.tests. excludetests is a list of either
                    individual tests, manifest files or sub-directories which
                    will override the default exclusion list.
-I includetests    optional. By default the test will include the
                    JavaScript tests appropriate for the branch. includetests is a
                    list of either individual tests, manifest files or
                    sub-directories which will override the default inclusion
                    list.
-c                  optional. By default the test will exclude tests
                    which crash on this branch, test type, build type and
                    operating system. -c will include tests which crash.
                    Typically this should only be used in combination with -R.
                    This has no effect on shell based tests which execute crash
                    tests regardless.
-t                  optional. By default the test will exclude tests
                    which time out on this branch, test type, build type and
                    operating system. -t will include tests which timeout.
-Z n                optional. Set gczeal to n. Currently, only valid for
                    debug builds of Gecko 1.8.1.15, 1.9.0 and later.
 
if an argument contains more than one value, it must be quoted.
</nowiki></pre>
 
 
For example, to checkout and build opt and debug builds of the JavaScript Shell and Firefox for the 1.8.1 and 1.9.0 branches, and then run the shell and browser JavaScript Tests you would use:
 
<pre>
$TEST_DIR/tests/mozilla.org/js/runtests.sh -p "js firefox" -b "1.8.1 1.9.0" -T "opt debug" -B "checkout build"
</pre>
 
Like <code>builder.sh</code>, <code>runtests.sh</code> will write the names of the log files it is generating to stdout.
 
<pre>
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-20-47-0400,js,1.8.1,,linux,sophie.bclary.com,checkout.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-05-0400,js,1.9.0,,linux,sophie.bclary.com,checkout.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-17-0400,js,1.8.1,opt,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-29-0400,js,1.8.1,debug,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-36-0400,js,1.9.0,opt,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-48-0400,js,1.9.0,debug,linux,sophie.bclary.com,build.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-59-0400,js,1.8.1,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-13-26-04-0400,js,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-16-19-28-0400,js,1.9.0,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-17-00-31-0400,js,1.9.0,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-59-0400,js,1.8.1,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-11-21-59-0400,js,1.8.1,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-13-26-04-0400,js,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-13-26-04-0400,js,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-16-19-28-0400,js,1.9.0,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-16-19-28-0400,js,1.9.0,opt,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-17-00-31-0400,js,1.9.0,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-fixes.log
log: /work/mozilla/mozilla.com/test.mozilla.com/www/results/2008-07-09-17-00-31-0400,js,1.9.0,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log-results-possible-regressions.log
</pre>
 
The checkout and build logs should be familiar from <code>builder.sh</code>.
 
The test logs have a similar format:
 
<pre>
2008-07-09-11-21-59-0400,firefox,1.8.1,debug,linux,sophie.bclary.com,tests_mozilla.org_js.log
---- -- -- -- -- ------- ------- ----- ----- ----- ----------------- --------------------
U Y  M  D  H  M  S T O      P      B    B T  O S    M N              T P
T e  o  a  o  i  e i f      r      r    u y  p y    a a              e a
C a  n  y  u  n  c m f      o      a    i p  e s    c m              s t
  r  t    r  u  o e s      d      n    l e  r t    h e              t h
    h        t  n z e      u      c    d    a e    i               
              e  d o t      c      h          t m    n               
              s  s n        t                i      e               
                  e                          n
                                              g
</pre>
 
The possible fixes and regressions log names are constructed from the test logs during a post processing phase where the raw log files are converted into a common format. In addition to the "possible-fixes.log" and the "possible-regressions.log", an "all" version containing all test results and a "failures" version containing only failures are produced.
 
= universe.data, public-failures.txt and confidential-failures.txt =
 
<code>public-failures.txt</code> and <code>confidential-failures.txt</code> contain an "encoded" list of known failures in a format very similar to that of the post process test result logs (minus the TEST_MACHINE and TEST_DATE fields). The files are used to create a working file which is used to detect the possible fixes and regressions during a test run.
 
These files are "encoded" by using the wild card "any" pattern in a field to represent occurences of all possible values within a set "universe" of values. The file <code>universe.data</code> contains the set of test run configurations which have been used to collect the known failure information.  When a test run begins, the Perl script <code>pattern-expander.pl</code> is used to expand the "encoded" files <code>public-failures.txt</code> and <code>confidential-failures.txt</code> into <code>public-failures.txt.expanded</code> and <code>confidential-failures.txt.expanded</code>. The "expanded" known failures files are combined into a single file <code>failures.txt</code> which is used to detect fixes and regressions.
 
When initially running the JavaScript Tests on a new platform or machine, you should run the [[#detect-universe.sh|detect-universe.sh]] script to determine the universe values to add to the universe.data file.
 
If you already have a collection of post-processed log files, you can obtain the necessary universe data from them by running the <code>get-universe.sh</code> script against the "all" versions of your test logs.
 
For example,
 
<pre>
get-universe.sh *all.log > new-universe.data
</pre>
 
Once you have updated the <code>universe.data</code> file with values for your test environment, reprocess the log files using the script <code>process-logs.sh</code> in the same directory as the log files to regenerate the possible fixes and failures.
 
<pre>
process-logs.sh -l '*_js.log'
</pre>
 
If you have more than one test run, for example tests for more than one branch, test type, or build type or results from several different machines, you can easily combine the results into a file for fixes and a file for regressions using the <code>changes.sh</code> script in the same directory as the log files.
 
<pre>
changes.sh
</pre>
 
This will combine all of the <code>*-possible-fixes.log</code> into a single file <code>possible-fixes.log</code> and all of the <code>*-possible-regressions.log</code> into a single file <code>possible-regressions.log</code> for easy review of the combined results.
 
= Updating public-failures.txt and confidential-failures.txt =
 
TODO
 
= JavaScript Test Environment Variables =
 
Some of the JavaScript tests can take quite a long time to run. In fact, some tests will never complete unless they are killed. In cases where a "too much gc" build, gczeal is used or when valgrind is used, the time for a test to complete may increase dramatically. To control the time alloted for the JavaScript tests to run, the following environment variables are used:
 
* TEST_JSALL_TIMEOUT=${TEST_JSALL_TIMEOUT:-21600}
 
The time in seconds for the entire test run to complete before it is terminated. By default, it is 21600 seconds or 8.5 hours.
 
* TEST_JSEACH_PAGE_TIMEOUT=${TEST_JSEACH_PAGE_TIMEOUT:-480}
 
The time in seconds for an individual test to load in the browser before the JavaScript test itselfs attempts to time out the test. By default it is 480 seconds or 8 minutes.
 
* TEST_JSEACH_TIMEOUT=${TEST_JSEACH_TIMEOUT:-485}
 
The time in seconds for an individual test to load/execute in the browser before it is forcibly terminated. By default it is 485 seconds or 8 minutes 5 seconds.
 
* TEST_JSSHELL_TIMEOUT=${TEST_JSSHELL_TIMEOUT:-480}
 
The time in seconds for an individual test to run in the shell.
 
Back to [[Sisyphus]]

Latest revision as of 19:08, 13 September 2011

This content is obsolete.

Retrieved from "https://wiki.mozilla.org/index.php?title=Sisyphus_-_JavaScript_Tests&oldid=348516"