Auto-tools/New Contributor/Quarter of Contribution/WPT Results

From MozillaWiki
Jump to: navigation, search

Title: Web Platform Tests (WPT) Results Explorer
Mentor: James Graham (:jgraham, irc, github)

The Web Platform Tests are a set of cross browser validation tests designed to validate common functionality on the web between browsers. The source code can be found on github. Mozilla runs these tests inside our CI system, as do many vendors.

This project will result in creating a tool which ingests the logs from a test run and displays them in a webUI. Keep in mind there are thousands of tests, so an easy way to filter and sort the data will be critical here. Once this is achieved, we should be able to take two or more different log files and compare them side by side in the tool.

To learn more about this project we recommend:

  • Join IRC! It's the best place to ask questions about the project and get help if you are stuck. #testing on and #ateam on are great channels to be in. Note that many people are most active in European timezones.
  • Checkout web-platform-tests from GitHub. Make sure that you are able to start the server and load tests in your browser; see the README file for instructions on how to do this.
  • If you like, try fixing one of the difficulty:easy bugs listed on the web-platform-tests issue tracker. These are generally not strictly related to this project, but they will help you understand the workings of web-platform-tests.
  • Run the manifest script in your web-platform-tests checkout to generate a test manifest.
  • Checkout wptrunner and try running a test against your local browser. See the README for more instructions, but something like:
wptrunner --binary=/path/to/firefox/binary --tests /path/to/web-platform-tests --metadata /path/to/web-platform-tests --ssl-type none --log-mach - dom/historical.html

Alternatively, if you have a firefox source tree and build, you can run web-platform-tests using mach:

mach web-platform-tests dom/historical.html
  • Repeat your testrun but add a command line option --log-raw output.log. This will produce test output in mozlog JSON format. Check the contents of that file after the test run and compare against the documentation for mozlog. Each line is a JSON object. Hopefully you find, amongst other things, one line with "action": "test_start", several with "action": "test_status" detailing the results of various tests and one with "action": "test_end".
  • Look at the test results for web-platform-tests on Mozilla's treeherder tool. This page shows all the finished web-platform-tests runs for all the current builds in the mozilla-inbound repository. Click on one of the green or orange numbers to select a job. You should see a panel at the bottom of the page. Select the "job details" tab and there should be a line with text "artifact uploaded: wpt_raw.log". Clicking on the log should load a much larger log like the one from your local run. The ultimate goal of the project is to be able to visualise the data from several of these larger log files at the same time.
  • The first part of the project is then producing a web page that can read mozlog format files. For comparison we have some python code that reads the files in the mozlog source.

The list of good first bugs and other specs/subprojects are outlined here.

Things we are looking for in an ideal contributor for this project:

  • Basic knowledge of Python and Javascript
  • Great ideas for displaying and filtering data
  • Willing to learn or apply skills in building a web based tool
  • Willing to learn and work with standardized tests
  • Eagerness to try ideas out, ask questions, accept feedback, and learn new things