Litmus:Web Services

Litmus Web Services Specification

Sending testcases to the client

To be determined at a later date.

Sending results to the server

Litmus will provide a web services interface to allow our automated testing infrastructure to submit test results to Litmus, where they can be tracked and reports of failing tests generated. For the time being, the testcases themselves will be managed as they currently are, outside of Litmus, but test metadata will be tracked in Litmus. In other words, each test will have a corresponding test id (we can do bulk imports of existing automated tests), and the machines running the tests will send the results to Litmus.

To send results, testing machines would identify themselves to the server, describe the system configuration of the product under test (i.e. platform, buildid, locale, etc...), and submit a batch of results, giving the testcase id and the result (pass or fail) for each. In addition, testing machines can send logs to be attached to the results. Logs can be either general configuration information to be linked with the entire batch of results (environment variables, buildconfig)

Submission

Testing clients will open an http connection to 'http://litmus.mozilla.org/process_test.cgi'. Clients must send via HTTP POST an XML document containing test results and metadata for submission. Results are formatted as described in the Test Result Format DTD. The parameter name will be 'data'.

Upon receipt, the server will process the test results. If a fatal error occurs, then no results will be added to the database. If some results are formatted properly, those results will be added even though other results may have errors.

The Test::Litmus perl module automates the process of generating a properly formatted xml result file and sending it to the server. It provides an object-oriented interface to allow test scripts to easily submit results to Litmus.

The following example (perl) shows how to send your results to the server using the module:

use Test::Litmus;
  
  $t = Test::Litmus->new(-machinename => 'mycomputer',
                           -username => 'user', 
                           -password => 'pass');
              
  $t->sysconfig(-product => 'Firefox',
                  -platform => 'Windows', 
                  -opsys => 'Windows XP', 
                  -branch => 'Trunk', 
                  -buildid => '2006061314',
                  -locale => 'en-US');
  
  my $result = Test::Litmus::Result->new(
              -testid => 27,
              -resultstatus => 'pass', # valid results are 'pass'
                                       # or 'fail'
              -exitstatus => 0,
              -duration => 666,
              -timestamp => 20051111150944,
              -comment => 'optional comment here', # optional
              -bugnumber => 300010,                # optional
              -log => [Test::Litmus::Log->new(     # optional
                          -type => 'STDOUT',
                          -data => 'foobar'),
                       Test::Litmus::Log->new(
                           -type => 'Extensions Installed',
                           -data => 'log information here')]
              );
  $t->addResult($result);
  # $t->addResult($someOtherResult);
  # etc...
  
  # add log information that should be linked with 
  # all results (i.e. env variables, config info)
  $t->addLog(Test::Litmus::Log->new(
                  -type => 'STDOUT',
                  -data => 'log data')); 
  
  my $res = $t->submit();
  
  # $res is 0 for non-fatal errors (some results were submitted), and 
  # undef for fatal errors (no results were submitted successfully)
  
  if ($t->errstr()) { die $t->errstr() }

Or, if you have already generated the xml result data manually, you may use the following to submit it to the server.

# $results contains the proper xml result data to send
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $req = HTTP::Request->new(POST => 'http://litmus.mozilla.org/process_test.cgi');
$req->content_type('application/x-www-form-urlencoded');
$req->content('data='.$results);
my $res = $ua->request($req);

if (!$res->is_success) {
    # an error occurred in the submission 
}

my $content = $res->content;

if ($content =~ /^Fatal error/) {
    # a fatal error occurred
} elsif ($content =~ /^Error processing result for test (\d+)/) {
    # result $1 contained an error
}

Response

In the interest of keeping both the client and the server simple, the responses from the server are in plain text. After processing the results, the server will return (following the text/plain content type):

• The string 'ok' if all results were added to the database successfully.

• An error string beginning with the words 'Fatal error' if the result data cannot not be parsed or lacks required information.

• An error string beginning with the words 'Error processing result for test n' where n is the id of the test that resulted in the error. This line will be repeated for each result that contained an error.