Auto-tools/Projects/Autolog

Goal

The Autolog project seeks to implement a TBPL-like system for viewing test results produced by the a-team's various tools, at least those of which aren't hooked up to TBPL. Such projects potentially include mobile automation, Crossweave, profile manager, etc.

Proposed Implementation

The system will be comprised of both a front-end web UI, and a backend database. The database which will be used is ElasticSearch, the same instance of which we're using for OrangeFactor, since our experience with that seems to indicate that it is fast, reliable, and easy to use.

Communication with the db will be provided by two channels: a REST API, which we can probably build by extending Orange Factor's woo_server.py, and a python library that automation tools can use to post results to the database.

The front-end UI should at least superficially resemble TBPL. Whether or not we will re-use TBPL code is open to question; the current implementation of TBPL is JS-based and is tightly tied to Tinderbox, so it may or may not be a good starting point for a new UI.

asuth of MoMo created an alternative view to tbpl called ArbPL (code). Might be interesting for possible UI ideas and as a source of more code that interfaces with tinderbox.

Data Structure

There are two types of data structures we are concerned with. One is the structure of data that a test suite will have to provide in order to insert test results into ElasticSearch. The second is the structure of data inside ElasticSearch itself.

For the former:

{
  // testrun definition
  'testrun': 'mochitest-other',
  'machine': 'talos-r3-fed64-044',
  'testrun_count': 1,             // supplied by python lib
  'starttime': 1297879654,
  'date': '2011-02-16',           // supplied by python lib
  'logurl': '...',
  'os': 'fedora12',
  'platform': 'linux',
  
  // Base product definition: the primary product under test.  For
  // Crossweave, this is the fx-sync code; for Android it is
  // the mobile-browser code.  In a TBPL-like display, this
  // product's rev would be displayed in the "commit" column.
  'tree': 'fx-sync',
  'branch': '1.7',
  'revision': '553f7b1974a3',
  'buildtype': 'xpi',
  'buildid': '20110210030206',    // optional
  'version': '1.7.pre',           // optional
  'buildurl': '...',              // optional
  
  // Secondary product definitions:  additional products involved
  // in the test.  For Crossweave or Android, this might be 
  // 'mozilla-central', etc. There can be as many secondary
  // products as needed.
  'tree2': 'mozilla-central',
  'branch2': 'default',
  'revision2': '553f7b1974a3',
  'buildtype2': 'opt',
  'buildid2': '20110210030206',   // optional
  'version2': '4.0b13pre',        // optional
  'buildurl2': '...',             // optional
  
  // Testsuite definition.  This is an array (to support cases like
  // mochitest-other); only one member is shown in the example below.
  'testsuites': [
    // for cases other than mochitest-other, this is probably the
    // same as 'testrun' above
    'suitename': 'mochitest-ally1',
    'cmdline': '...',             // optional
    'testfailure_count': 1,       // provided by python lib
    'elapsedtime': 665,           // in seconds
    'passed': 85152,
    'failed': 1,
    'todo': 124,
    // These errors are testsuite errors, that cannot be pinned to a
    // specific test case (e.g., a crash that occurred after all tests
    // had finished).  Each member of 'errors' can contain additional
    // keys, depending on type of test, e.g., 'stacktrace' for crashes,
    // etc.
    'errors': [
      'status': 'PROCESS-CRASH',
      'text': 'application crashed (minidump found)',
    ],
    
    // These are failures that occur during specific test cases.
    'testfailures': [
      'test': 'xpcshell/tests/toolkit/components/places/tests/autocomplete/test_download_embed_bookmarks.js',
      // Like testsuite errors, each member of 'failures' can contain
      // additional metadata depending on failure type.
      'failures': [
        'status': 'TEST-UNEXPECTED-FAIL',
        'text': 'Acceleration enabled on Windows XP or newer - didn't expect 0, but got it'
      ]
    ],
    
    // Do we want/need to record passed/todo tests?
  ]
}

For the structure in ElasticSearch, the data will be separated into three document types (by the python library if that's used; the test suite will have to do this if it's posting to ES via), similar to the way that tinderbox logs are spread across three document types at present:

• a testrun document (corresponding to tinderbox build documents, example here)
• one or more testsuite documents (corresponding to tinderbox testrun documents, example here)
• one or more testfailure documents (corresponding to tinderbox testfailure

documents, example here)

Tasks

Task	Owner	Notes
Investigate TBPL code and determine how much to re-use	Mcote?
Propose common data structure for test results in db	Jgriffin
Clean up woo_server.py and modularize in order to make future additions easier	?
Create python library for test tools to use to post results to db	?