Litmus:Web Services

Litmus Web Services Specification

Sending testcases to the client

To be determined at a later date.

Sending results to the server

Litmus will provide a web services interface to allow our automated testing infrastructure to submit test results to Litmus, where they can be tracked and reports of failing tests generated. For the time being, the testcases themselves will be managed as they currently are, outside of Litmus, but test metadata will be tracked in Litmus. In other words, each test will have a corresponding test id (we can do bulk imports of existing automated tests), and the machines running the tests will send the results to Litmus.

To send results, testing machines would identify themselves to the server, describe the system configuration of the product under test (i.e. platform, buildid, locale, etc...), and submit a batch of results, giving the testcase id and the result (pass or fail) for each. In addition, testing machines can send logs to be attached to the results. Logs can be either general configuration information to be linked with the entire batch of results (environment variables, buildconfig), or detailed information about a specific test result only.

The following data would be sent for each communication with Litmus:

• Username
• Password
• Machine name
• Product
• Platform
• Branch
• Build ID
• Locale
• (optional) Log(s) that relate to the entire test run

The following data would be sent for each result sent:

• Test id
• Result status (pass or fail)
• Exit status
• Duration
• Timestamp
• (optional) Comment
• (optional) Bug Number
• (optional) Log(s) that relate to this specific result

The following data would be sent for each log:

• Log type (coop has a long list types that we're using now, and more can be added easily)
• Log data - freeform text

Authentication

Authentication of the client poses a difficult problem. While it would seem that clients could send an encrypted version of their password to the server, Litmus (like Bugzilla) uses a random secret salt for additional security. Because of this, it is impossible for Litmus to know if a password is correct without having the plaintext version of the password the user entered. As such, we can't just send encrypted passwords over the wire.

Some options:

• Send the username and password in cleartext
  • This is no different then what ordinary users do when submitting results through the web
• Associate authentication tokens with accounts used to submit automated testing results that can be sent encrypted over the wire.

I'm not particularly concerned about security here. Consider that it is trivial to spoof data for other mozilla.org services like Bonsai and Tinderbox (Tinderbox will accept results from anyone who knows the correct email address) and that the passwords for these services are generally known. I think that sending the username nad password in cleartext provides an acceptable level of security and simplicity.

Thoughts on how authentication should be accomplished? Zachlipton 11:25, 15 June 2006 (PDT)

Code-level Details

Testing clients will open an http connection to 'http://litmus.mozilla.org/process_test.cgi'. Clients must send via HTTP POST an XML document containing test results and metadata for submission. Results are formatted as described in the Test Result Format DTD. The parameter name will be 'data'.

Upon receipt, the server will process the test results. If a fatal error occurs, then no results will be added to the database. If some results are formatted properly, those results will be added even though other results may have errors.

The Test::Litmus perl module automates the process of generating a properly formatted xml result file and sending it to the server. It provides an object-oriented interface to allow test scripts to easily submit results to Litmus.

The following example (perl) shows how to send your results to the server using the module:

use Test::Litmus;
  
  $t = Test::Litmus->new(-machinename => 'mycomputer',
                           -username => 'user', 
                           -password => 'pass');
              
  $t->sysconfig(-product => 'Firefox',
                  -platform => 'Windows', 
                  -opsys => 'Windows XP', 
                  -branch => 'Trunk', 
                  -buildid => '2006061314',
                  -locale => 'en-US');
  
  my $result = Test::Litmus::Result->new(
              -testid => 27,
              -resultstatus => 'pass', # valid results are 'pass'
                                       # or 'fail'
              -exitstatus => 0,
              -duration => 666,
              -timestamp => 20051111150944,
              -comment => 'optional comment here', # optional
              -bugnumber => 300010,                # optional
              -log => [Test::Litmus::Log->new(     # optional
                          -type => 'STDOUT',
                          -data => 'foobar'),
                       Test::Litmus::Log->new(
                           -type => 'Extensions Installed',
                           -data => 'log information here')]
              );
  $t->addResult($result);
  # $t->addResult($someOtherResult);
  # etc...
  
  # add log information that should be linked with 
  # all results (i.e. env variables, config info)
  $t->addLog(Test::Litmus::Log->new(
                  -type => 'STDOUT',
                  -data => 'log data')); 
  
  my $res = $t->submit();
  
  # $res is 0 for non-fatal errors (some results were submitted), and 
  # undef for fatal errors (no results were submitted successfully)
  
  if ($t->errstr()) { die $t->errstr() }

Or, if you have already generated the xml result data manually, you may use the following to submit it to the server.

# $results contains the proper xml result data to send
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $req = HTTP::Request->new(POST => 'http://litmus.mozilla.org/process_test.cgi');
$req->content_type('application/x-www-form-urlencoded');
$req->content('data='.$results);
my $res = $ua->request($req);

if (!$res->is_success) {
    # an error occurred in the submission 
}

my $content = $res->content;

if ($content =~ /^Fatal error/) {
    # a fatal error occurred
} elsif ($content =~ /^Error processing result for test (\d+)/) {
    # result $1 contained an error
}

Response

In the interest of keeping both the client and the server simple, the responses from the server are in plain text. After processing the results, the server will return (following the text/plain content type):

• The string 'ok' if all results were added to the database successfully.

• An error string beginning with the words 'Fatal error' if the result data cannot not be parsed or lacks required information.

• An error string beginning with the words 'Error processing result for test n' where n is the id of the test that resulted in the error. This line will be repeated for each result that contained an error.