Ateam GSoC Projects 2014
Structured Logging for Mochitest
Implementation of Structured Logging for Mochitest
When running testsuites in automation it is necessary to extract the results of the testcases so that they can be summarised and analysed. Traditionally Mozilla has used a text-based format for writing these results in a way that is optimised for human readability. However this human readability has come at the cost of making the results difficult to interpret by tools, so test harnesses often implement their own unique test summarisation using internal data, and tools that have to rely on the external result stream tend to be fragile.
In order to solve these problems we have developed a new approach to storing testrun data; "structured logging". Instead of optimising for human readability we optimise for machine parsability using a JSON-based serialisation format. This means that test harnesses no longer need to include their own summarisation code and code that parses the logs no longer needs to employ imprecise regexp. Instead we have a suite of reporting tools that can work with any harness that produces structured output.
Structured Logging has now been implemented as a library in python and is being used for the new web-platform-tests test type. However we need to port it to our other test types, of which Mochitest is one of the most significant.
- Become familiar with structured logging
- Implement any python-side changes required to receive the structured logging messages in the python harness.
- Fix the interaction with existing systems (mozharness, mach) so that results are reported correctly.
- Ensure that mochitests all still give the expected results
- Provide appropriate documentation for the code produced.
Suggestions for Preparation
- Fix some simple Mochitest bugs.
- Understand the structured logging format and API by examining the existing python library.
- See how structured logging is used in production by looking at the web-platform-tests on cedar.
- Code Firefox has documentation for getting a Mozilla build environment set up.
- Ask in #ateam on irc.mozilla.org or /msg me (jgraham). Note that I am in Europe and not always able to respond immediately.