From MozillaWiki
Jump to: navigation, search

Personal deliverables for :maja_zf, in addition to ongoing work.

2017 Q1

  • [DONE] Add wdspec web-platform tests and wdclient support for key actions.
  • [ON TRACK] Implement dispatch of mouse actions in Marionette according to WebDriver spec.

2016 Q4

  • [DONE] Add support for debug builds for --artifact flag in try syntax.
  • [DONE] Implement dispatch key actions in Marionette according to WebDriver spec.

2016 Q3

Other tasks: continue mentoring Outreachy participant (Marionette harness unit tests), deploy Marionette Fennec tests on TaskCluster

2016 Q2

2016 Q1

2015 Q4

This work is happening in coordination with major changes to firefox-ui-tests and firefox-media-tests: both test suites are moving in-tree, and firefox-media-tests jobs will move to the same CI as firefox-ui-tests which means that various aspects of the Jenkins configurations will be consolidated.

2015 Q3

  • [DONE] Move a subset of Firefox Media Tests jobs (EME/MSE testing) to buildbot:
    • Short firefox-media-tests jobs should run per commit in buildbot and report results to Treeherder Tier-2.
    • Supporting tasks include: reduce dependency on firefox-ui-tests; refactor existing mozharness script for running firefox-media-tests; move firefox-media-tests from Github to
  • [DONE] Fulfill Treeherder Tier-2 requirements for Firefox Media Tests (EME/MSE testing).
    • Modify job failure output summary to match format expected for Treeherder bug suggestions.
    • Parse job logs into “steps” so they can be browsed more easily in Treeherder’s Log Viewer.
    • Collect firefox-media-tests documentation in a Mozilla wiki page, including key information for tree sheriffs.

Meanwhile, I've also made firefox-media-tests much more reliable by fixing a few intermittent failures related to YouTube ads. In addition, I've made several changes in our CI to make the test environment more reproducible and deterministic.

2015 Q2

Team goal: Extend MSE/EME automation to provide additional test coverage: create new MSE/EME automation and extend the existing automation to provide additional stress and functional coverage.

  • [DONE] Add instrumentation to HTML5 video playback tests and produce reports targeted for MSE/EME. Modify test harness to output debugging information in case of failure: MSE player state (like about:media); screenshots, YouTube debug data, crash stacks.
    • Also improved logging of test progress and improved detection of playback stalls
    • Extra: tests can trigger recording of a memory report

Team goal: Improve the visibility of QA automation: make new and existing QA automation more valuable by migrating it to buildbot or reporting test results to Treeherder.

  • [DONE] Get qualifying subset of Jenkins-based media automation (MSE/EME/WebRTC) reporting to Treeherder. Implement the generation/submissio​n​ of appropriate data structures to treeherder-service. This applies to all Firefox branches: short MSE/EME tests and WebRTC tests should report to Treeherder as Tier 2 jobs.
    • WebRTC Steeplechase tests and firefox-media-tests (MSE/EME) all report to Treeherder staging: Win 7, Win 8, Mac 10.10 on mozilla-aurora, mozilla-central. mozilla-beta is turned off on pf-jenkins until package downloads move to using a Pulse monitor.
    • All MSE firefox-media-tests (YouTube) report to Treeherder prod as Tier-2.
    • We won't report WebRTC tests to Treeherder production until we come to an agreement with sheriffs and Treeherder team about which job symbols to use and how to present the data. Right now, the Treeherder UI is not well-suited to present job results for combinations of branches and platforms.

Supporting tasks:

  • Created a mozharness script for running firefox-media-tests, which helps with virtual env management, collecting resources, debugging harness failures. It was then easy to extend the script to upload logs to S3, submit data to Treeherder, and so on.
  • Refactored firefox-media-tests to distinguish between generic video tests versus site/vendor-specific tests

2015 Q1

Joined mid-quarter, so no deliverables. These are "supporting tasks".