Ateam GSoC Projects 2014

Structured Logging for Mochitest

Project Goal

Implementation of Structured Logging for Mochitest

Background

When running testsuites in automation it is necessary to extract the results of the testcases so that they can be summarised and analysed. Traditionally Mozilla has used a text-based format for writing these results in a way that is optimised for human readability. However this human readability has come at the cost of making the results difficult to interpret by tools, so test harnesses often implement their own unique test summarisation using internal data, and tools that have to rely on the external result stream tend to be fragile.

In order to solve these problems we have developed a new approach to storing testrun data; "structured logging". Instead of optimising for human readability we optimise for machine parsability using a JSON-based serialisation format. This means that test harnesses no longer need to include their own summarisation code and code that parses the logs no longer needs to employ imprecise regexp. Instead we have a suite of reporting tools that can work with any harness that produces structured output.

Structured Logging has now been implemented as a library in python and is being used for the new web-platform-tests test type. However we need to port it to our other test types, of which Mochitest is one of the most significant.

About Mochitest

Mochitest is a test type which is used to test a great deal of the browser engine code in Gecko, as well as some of the UI layers. The test execution code is largely written in javascript running in the browser itself, but the browser is launched by a python harness. Both the python and javascript code may both produce messages that need to be included, in order, in the run log.

Detailed Goals

• Become familiar with structured logging

• Implement the structured logging API in javascript

• Implement a backend for the javascript API that will allow it to synchronise messages with python (the expected means of achieving this is by writing the messages to a socket which python can listen on, but other designs are worth discussing).

• Implement any python-side changes required to receive the structured logging messages in the python harness.

• Fix the interaction with existing systems (mozharness, mach) so that results are reported correctly.

• Ensure that mochitests all still give the expected results

• Provide appropriate documentation for the code produced.

Suggestions for Preparation

• Fix some simple Mochitest bugs.

• Understand the structured logging format and API by examining the existing python library.

• See how structured logging is used in production by looking at the web-platform-tests on cedar.

Getting Help

• Code Firefox has documentation for getting a Mozilla build environment set up.

• Ask in #ateam on irc.mozilla.org or /msg me (jgraham). Note that I am in Europe and not always able to respond immediately.