Top of the List

Highest of highest priority

• Bugzilla - Shift focus toward Bugzilla as a platform
• Services Automation - Automation for Services Projects - focused on webRTC
• Treeherder - Deploy Treeherder
• B2g - Complete Gaia Automation and update B2G automation to increase its robustness
• Performance - Create foundation of comprehensive performance measurement overhaul

Automation

P0: WebRTC/Services Automation

• Define and implement the minimally viable "services-based" automation system:
  • [DONE] Allows configuration of different networking environments between end points
  • [DONE] Built in-cloud on AWS
  • [DEFER] Integration with Treeherder
  • [DONE] Allows tests to be defined simply in the main m-c tree
  • [DONE] Runs WebRTC tests
  • [DEFER] Could also use this system to do a sanity check on the social API partner integration once the marionette based tests for social API are written
  • [DONE] Should use an off-the-shelf scheduler - either jenkins or buildbot or something.

P0: B2G General

• [DONE] Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds
• [DONE] Finish getting gaia-unit-tests and gaia-ui-tests running on all trees in TBPL
• [MISSED] (Depends on James Lal's dev team): Deploy gaia-integration tests into TBPL
  • NOTE: We have an npm mirror; need to write mozharness script
• [MISSED] Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming)
  • NOTE: Waiting for rel-eng to schedule 4.2 builds/tests on cedar
• [DONE] Get b2g jsreftests working, so we can turn off no-ion builds/tests (stretch goal)
  • NOTE: We have these running in single-process mode, but reftest in general doesn't work in B2G in OOP mode, which is the reason for running them on no-ion builds, so we can't turn off the no-ion builds yet.
• [DONE] Get tests running using full-stack emulator builds
• [DEFER] Implement heuristics to notify dev managers of "real" failures from B2G device test runs on Jenkins
• [MISSED] (Depends on rel-eng): Help implement a gaia try-server
• [MISSED] Resolve various tap-related issues (including modal dialogs) in Marionette in order to support gaia-ui-tests and gaia-integration-tests

P0: Eideticker

• [DONE] Create responsiveness test for firefox on Android test using Eideticker
• [DONE] Create a set (at least 2) of tests for b2g Eideticker to measure user perceived application startup and scrolling performance, and run on a daily basis

P1: Marionette

• [DONE] Clean up oranges
• [DONE] Better error reporting
• [DROPPED] Add support for nested content processes
  • NOTE: not needed
• [DONE] Continue support for B2G testing / developer ergonomics (webqa, mach support) ... originally scheduled for Q2 but dropped for focus on gaia tests
• [DROPPED] Support social API tests
  • NOTE: The social API team is using the 'old' FirefoxDriver for now

P0: Performance

High level details:

• update talos tests to measure relevant parts of the code while providing detailed reports to the developers
• collect competitive benchmark numbers on a regular schedule
• deliver and refine tools for development teams as their performance needs change

Finer details:

• Networking:
  • [DONE] deliver webpagetest as a tool. This will be 100% offline for non ssl pages, and we will provide a similar setup but 100% online for ssl traffic.
  • [DONE] work with necko team to create an easy to use interface with local builds, try builds, nightly builds running on wpt and reporting in a meaningful manner

• Benchmarking:
  • [DONE] work with the games and js teams to run competitive benchmarks on android, b2g, and desktop platforms
  • [DONE] report numbers at least once a month
  • [DONE] integrate additional benchmarks, browsers, platforms (i.e. IOS) as well as more hands off automation

• Talos:
  • [DONE] add additional alerts for disk flushes, too many disk accesses with talos xperf
  • [DONE] continue modifying existing tests to make the data more useful, improve documentation on what all tests measure
  • [DONE] modify talos ts_dirty tests to have updated profiles and measure useful scenarios
  • [MISSED] have sheriff/developer dashboards for datazilla, statistical analysis of all test types with alerts for detected regressions at <2% noise

• Other:
  • [DONE] continue working with developers to refine microperf
  • [DONE] deliver a prototype for real world benchmark using the live internet to aid the js team
  • [DEFER] provide help to the servo/rust team as they start measuring performance
  • [DONE] investigate ways to involve community into the process of measurement and analysis

P1: MozBase

• [DONE] unittests for all modules
• [MISSED] m-c commit watcher to notify mozbase owners of unexpected commits
• [MISSED] full mochitest and reftest integration with mozprofile/mozprocess/mozrunner (excluding Fennec)
  • NOTE: mozprofile work done; the remainder is just being started
• [DONE] continue to standardize + impove dev practices
• [DONE] structured logging
• [DONE] manifestdestiny API improvements and fixes
• [DONE] add mozprocess documentation

P2: mozmill 2.0 released

• [DONE] Release 2.0 (should be done Q2)
• [DONE] SV leads metro testing using mozmill

Tools

P0: Bugzilla

Optimize bugzilla as a platform that we can build on top of.

• [DONE] Deploy native REST API
• [DONE] Optimize and expand native REST API as needed to support 3rd party UI building efforts
• [DONE] Stand up reviewboard as a semi-independent reviewing service
  • NOTE: Up at https://reviewboard.mozilla.org/, but waiting on proper cert and http -> https redirect from IT.
• [DONE] Tracking flag migration
  • Migration script is ready, but IT outage will not happen until early Q4.

P0: TreeHerder

• [MISSED] Deliver UI parity with existing TBPL
• [DONE] Implement UI plugin architecture
• [DONE] Set up publicly accessible instance
• [DONE] Develop a logging strategy:
  • for the data adapters
  • for the restful api
• Full Project Listing Is Here

P0: Datazilla

• [DEFER] Automate datazilla database rollover and rollover talos data. What do we want to do with legacy data? (Stretch goal)
• [DONE] Refactor datazilla to remove all portions of the old UI, unnecessary controllers, and legacy code not in production
• [DONE] Provide datazilla landing page that displays data for all projects
• [MISSED] Replace graph server for talos on m-c,m-i
  • Everything in place, except alerting which will carry into Q4

Process

P1: Infrastructure Load

• [DONE] parallelize xpcshell test framework
• [DONE] analyze always passing tests and determine a strategy for turning tests off selectively/running intermittently
• [DONE] work with releng to turn on code coverage builds

P1: Sheriff goals

• [DONE] Provide documentation and how to steps to enable a community sheriff to be effective
• [MISSED] Wire documentation into bugsAhoy/whatcanIdoForMozilla
• [DONE] Fix P1/low hanging fruit of the Sheriffing Tools (see full goals list)
• [DONE] Drive treeherder UI functional specification
• Full Goals here

P1: Bugmaster

• [DONE] Create a contribution pipeline for 30 incoming bugmasters and track them through each stage of involvement in order to get 1 to the state where they regularly do triage.
• [DONE] Improve Bugzilla Help pages.
• Full goals here