Community:SummerOfCode14:Brainstorming: Difference between revisions

| Traditionally at Mozilla, test harnesses have written test results as arbitrary strings to stdout.  Other tools that want to gather data about test runs are forced to parse the entire logfile produced by capturing stdout, and using a set of regex's on each line in order to identify test results, test summaries, and other items of interest.

We'd like to replace this very inefficient approach to test logging with a more robust mechanism...structured logging.  In this approach, a test harness would write to a file structured data in JSON for each test event that needed to be logged, in addition to the current approach of spewing everything to stdout.  Other tools can then easily iterate over this structure when they want to know what happened during the test run.  We've already implemented this approach for our xpcshell harness.

 

Now, we want to apply this to our Mochitest harness.  The harness should utilize structured logging for both test events generated in the JavaScript harness that's running in-browser, and also the Python test runner.  The harness should accept structured logging data from both sources and write them to the same file in chronological order.  Mochitests running in continuous integration should upload their structured log files at the end of the test as a test artifact.