Auto-tools/Projects/Autolog

Goal

The Autolog project seeks to implement a TBPL-like system for viewing test results produced by the a-team's various tools, at least those of which aren't hooked up to TBPL. Such projects potentially include mobile automation, Crossweave, profile manager, etc.

Proposed Implementation

The system will be comprised of both a front-end web UI, and a backend database. The database which will be used is ElasticSearch, the same instance of which we're using for OrangeFactor, since our experience with that seems to indicate that it is fast, reliable, and easy to use.

Communication with the db will be provided by two channels: a REST API, which we can probably build by extending Orange Factor's woo_server.py, and a python library that automation tools can use to post results to the database.

The front-end UI should at least superficially resemble TBPL. Whether or not we will re-use TBPL code is open to question; the current implementation of TBPL is JS-based and is tightly tied to Tinderbox, so it may or may not be a good starting point for a new UI.

asuth of MoMo created an alternative view to tbpl called ArbPL (code). Might be interesting for possible UI ideas and as a source of more code that interfaces with tinderbox.

Data Structure

There are two types of data structures we are concerned with. One is the structure of data that a test suite will have to provide in order to insert test results into ElasticSearch. The second is the structure of data inside ElasticSearch itself.

For the former:

{
  // testgroup definition
  'testgroup': 'mochitest-other',
  'machine': 'talos-r3-fed64-044',
  'testsuite_count': 1,           // supplied by python lib
  'starttime': 1297879654,
  'date': '2011-02-16',           // supplied by python lib
  'logurl': '...',
  'os': 'fedora12',
  'platform': 'linux',
  
  // Base product definition: the primary product under test.  For
  // Crossweave, this is the fx-sync code; for Android it is
  // the mobile-browser code.  In a TBPL-like display, this
  // product's rev would be displayed in the "commit" column.
  'tree': 'fx-sync',
  'branch': '1.7',
  'revision': '553f7b1974a3',
  'buildtype': 'xpi',
  'buildid': '20110210030206',    // optional
  'version': '1.7.pre',           // optional
  'buildurl': '...',              // optional
  
  // Secondary product definitions:  additional products involved
  // in the test.  For Crossweave or Android, this might be 
  // 'mozilla-central', etc. There can be as many secondary
  // products as needed.
  'tree2': 'mozilla-central',
  'branch2': 'default',
  'revision2': '553f7b1974a3',
  'buildtype2': 'opt',
  'buildid2': '20110210030206',   // optional
  'version2': '4.0b13pre',        // optional
  'buildurl2': '...',             // optional
  
  // Testsuite definition.  This is an array (to support cases like
  // mochitest-other); only one member is shown in the example below.
  'testsuites': [
    // for cases other than mochitest-other, this is probably the
    // same as 'testgroup' above
    'suitename': 'mochitest-ally1',
    'cmdline': '...',             // optional
    'testfailure_count': 1,       // provided by python lib
    'elapsedtime': 665,           // in seconds
    'passed': 85152,
    'failed': 1,
    'todo': 124,
    // These errors are testsuite errors, that cannot be pinned to a
    // specific test case (e.g., a crash that occurred after all tests
    // had finished).  Each member of 'errors' can contain additional
    // keys, depending on type of test, e.g., 'stacktrace' for crashes,
    // etc.
    'errors': [
      'status': 'PROCESS-CRASH',
      'text': 'application crashed (minidump found)',
    ],
    
    // These are failures that occur during specific test cases.
    'testfailures': [
      'test': 'xpcshell/tests/toolkit/components/places/tests/autocomplete/test_download_embed_bookmarks.js',
      // Like testsuite errors, each member of 'failures' can contain
      // additional metadata depending on failure type.
      'failures': [
        'status': 'TEST-UNEXPECTED-FAIL',
        'text': 'Acceleration enabled on Windows XP or newer - didn't expect 0, but got it'
      ]
    ],
    
    // Do we want/need to record passed/todo tests?
  ]
}

For the structure in ElasticSearch, the data will be separated into three document types (by the python library if that's used; the test suite will have to do this if it's posting to ES directly via HTTP), similar to the way that tinderbox logs are spread across three document types at present:

• a testgroup document (corresponding to tinderbox build documents, example here)
• one or more testsuite documents (corresponding to tinderbox testrun documents, example here)
• one or more testfailure documents (corresponding to tinderbox testfailure documents, example here)

Q: Why do we separate the data into three document types, why not just use one big document?
A: Because searches in ElasticSearch are must faster and easier with basic data types; searching inside complex nested JSON is slower and the syntax is much more complex.

Q: Can't the python library automatically provide 'os' and 'platform'?
A: It would be nice, wouldn't it? Unfortunately, there are lots of things which can confuse the issue; e.g., if you're using mozilla-build on Windows, it will see your 64-bit version of Windows as win32, regardless of what you're testing. Similarly, we sometimes test 32-bit Mac stuff on macosx64. It seems safest to have the test tools provide this data instead of trying to guess.

Q: Why do we have both testgroup and testsuite?
A: It's entirely to support mochitest-other.  :( In most cases, each testgroup will have 1 testsuite.

Q: Where are the test runs in this structure?
A: We've been using the term 'testrun' to mean different things in different places. In this structure, I imagine 'testrun' to mean the same thing as it does in OrangeFactor: that is, a collection of testgroups that are run against the same primary changeset.

Q: Is this really the best way to include data about multiple products, or code from multiple repos?
A: I'm not sure. I suggested this structure because it's easy to use when searching ES. Other structures are possible. For instance, we could create a 'product' document type, and store all the products there, and then just include indexes to this document in the 'testgroup' document. The downside to this is that getting certain data out of ES would require multiple queries.

Tasks

Task	Owner	Notes
Investigate TBPL code and determine how much to re-use	Mcote?
Propose common data structure for test results in db	Jgriffin
Clean up woo_server.py and modularize in order to make future additions easier	?
Create python library for test tools to use to post results to db	?