Auto-tools/Goals/2014Q1

From MozillaWiki
Jump to: navigation, search

Top Level

Highest priority, multi-team dependent goals

Goal Measurement Status Owner Bug Area
Turn on Android 2.3 tests on emulators; turn off tegras All non-perf suites currently running on tegras should be running on emulators in EC2 partially [DONE] ? Got many tests working on emulators, but work will continue into next quarter. Gbrown bug 910092 Android
Get more B2G tests running in TBPL All P1 suites in B2G running in TBPL [DONE] Ahal bug 916356 B2G
Get B2G AWSY tests running per-commit to mozilla-central B2G data should be reported to https://areweslimyet.com/ on a regular basis [BLOCKED] waiting for jschoenick to add B2G results to the web site Rwood bug 899194 B2G
Continue work on Bugzilla as a platform by investigating and implementing performance enhancements Increase memcache use; add instrumentation to identify perf bottlenecks [DONE] Glob Project statuses Bugzilla
Report B2G startup and fps metrics to the Eideticker dashboard consistently App startup times and fps metrics should be reported regularly to the Eideticker dashboard [DONE] Wlach bug TBD Eideticker
Make Marionette test failures easier to debug Implement better logging and error handling so that Marionette test failures are more easily debugged [DONE] Mdas bug TBD Marionette
Deploy Datazilla alerting system Alerts should be raised whenever datazilla ingestion rates fall below a configurable threshold, or when performance regressions occur [DONE] Mcote bug 921218 Performance
Add latency tests to Talos Successful completion by deadline [BLOCKED] due to issues with test machines, low priority with dev team jmaher bug 938644 Performance
Implement a prototype of a flexible test scheduler that can be used by Gaia's Shepherd project, and make Treeherder display the test results which are produced by this system Successful completion by deadline [DROPPED] Ted bug TBD Project X
Deploy a version of Treeherder that can be used for sheriffing Sheriffing should be possible without using TBPL by the end of Q1 [DONE] Jeads Pivotal Tracker Treeherder

Detailed Lists

These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The current backlog is here

Android (gbrown)

  • P1: [DONE] Stand-up Android 2.3 tests on Graydon's emulator environment - to replace Android 2.2/tegra bug 910092
  • P2: [DEFER] Finish Android x86 emu work - get most test jobs working such that sheriffs are okay for trunk with the least amount of effort possible (i.e., turning off failing tests)
  • P1: [bc] [DONE] Investigate the usefulness of autolog, and attempt to quantify the value we're getting from it; use this to make a decision to keep or kill it
Ongoing and supporting projects
  • Continue improving developer ergonomics by adding mach targets
  • Maintain skia GL tests in Autophone with a limited number of devices

B2G (jgriffin)

  • [ahal] Get the following tests running in TBPL on all trunk trees:
    • [DONE] P1: mochitests on debug emulator builds
    • [DONE] P1: OOP reftests on emulator builds
    • [DONE] P1: reftests on B2G desktop builds
    • [MISSED] P1: WebAPI tests on debug emulator builds
    • [MISSED] P2: Gaia-ui-tests on opt emulator builds
    • [DROPPED] P2: Get all suites running against emulator-jb builds on cedar
  • [DONE] P1: Instrument B2G test runners for better logging and error handling
  • [ON TRACK] P1: (ongoing) Work with engineering to get B2G reftests off of Fedora slaves, bug 818968
  • [DONE] P1: [ahal] Make sure crash reporting is working reliably in all B2G test jobs
  • [DONE] P2: [ahal] Convert B2G mochitests to using manifests
  • [rwood] Long-running tests:
    • [DONE] P1: (carry over) Finish standing up memory tests reporting to AWSY and have them running per-commit to mozilla-central, bug 899194
    • [DEFER] P1: Help Taipei team implement crash/error detection routines for MTBF that can be used in other harnesses as well
    • [DEFER] P2: Review the endurance tests:
      • Determine if these are needed or if their purpose can be rolled into MTBF tests and/or AWSY tests
      • If needed, make sure we're reporting useful metrics
  • [DONE] P1: [davehunt] Work with engineering to add per-app memory consumption metrics to datazilla, bug 917717
  • [DONE] P1: [davehunt] Add ability to perform semi-automated bisection of perf regressions in Jenkins, bug 943611
Ongoing and supporting projects
  • Continue to investigate and fix intermittent oranges
  • Continue work to make B2G tests more sheriffable
  • Continue work to make B2G tests more developer-friendly
  • Continue to support the WebQA team's Jenkins device testing

Bugzilla (mcote/glob)

See BMO#Current_Projects for status.

  • [MISSED] P1 (carryover): Finish and deploy bzAPI compatibility layer.
    • This ended up requiring a lot more patching of the core code than anticipated, and getting the bzAPI test suite adapted and running correctly was difficult. First revision of patch up for review.
  • [DONE] P1: Performance instrumentation (new relic or other)
    • [DONE] P1: Identify areas for improvement in API performance.
      • Several bugs filed, but still more investigation to do.
  • [DONE] P1: Deploy production Change Notification System.
  • [DONE] P1: Increased memcache support
  • [DONE] P2: Provide accurate review history.

Eideticker (wlach)

  • [DONE] P1: Get B2G app startup tests running smoothly and consistently on the London hardware, reporting to the Eideticker dashboard
  • [DONE] P1: Get B2G fps tests running smoothly and consistently, as above, with a first iteration that ignores the partial frame problem
  • [DONE] P1: Work with the FxOS_Perf team to figure out how to produce accurate fps measurements and deal with noise created by the partial frame problems
  • [DROPPED] P2: Implement a framework that can be used to run B2G responsiveness tests (first target: virtual keyboard responsiveness)
  • [DROPPED] P2: Get Eideticker reporting to datazilla, and data files stored in S3

General Automation

  • [DONE] P1: [jmaher] chunk mochitest-browser-chrome
  • [DEFER] P1: [jgraham] Get a snapshot of W3C web-platform tests running on TBPL
  • [BLOCKED] P1: [dkl] Reconfigure Pulse so that it can act as a robust backbone for Treeherder and other projects
  • [DONE] P1: [ahal] Move mozbase to m-c
  • [DEFER] P2: [dminor] Finish removing jit-tests and cpp-tests from 'make check' for desktop Firefox

Marionette (mdas)

  • [DONE] P1: [ato] Make Marionette compatible with WebDriver at the JSON Wire Protocol Level
  • [DEFER] P2: [ato] Get a passing subset of the WebDriver spec suite tests running on cedar
  • [DEFER] P1: [ato] Add modal dialog support for desktop Firefox
  • [DEFER] P1: [mdas] Investigate and fix frequent intermittent oranges in Marionette unit test suite
  • [DONE] P1: [mdas] Make Marionette's logging and error reporting more robust
    • Smarter handling of socket errors
    • Better crash handling
    • Conditional logging of all requests/responses and other significant activity
  • [DEFER] P2: [mdas] Create a suite of Marionette endurance tests designed to find memory leaks in Marionette and gaiatest components
Ongoing and supporting projects
  • Triage bugs and fix those affecting Gaia testing (e.g., WebQA)
  • Support Taipei team in their efforts to use Marionette for MTBF and other tests

Performance (jmaher)

  • [DONE] P1: [glob, jeads] Add error bars and ability to toggle between mean and median to datazilla
  • [DONE] P1: [glob] Add b2g performance data ingestion rate notifications to datazilla
  • [DONE] P1: [kyle, mcote] Add b2g perf regression notifications to datazilla
  • [MISSED] P1: [jmaher] Continue work on dashboards (carryover from Q4)
  • [BLOCKED] P1: [jmaher] Add new talos tests for latency and webRTC
  • [DEFER] P2: [jmaher] Bring up automated power testing for desktop Firefox
  • [DEFER] P2: [kyle, jmaher] Add talos perf regression notifications to datazilla
Ongoing and supporting projects
  • Continue supporting game benchmarking effort
  • Continue to work with team leads so we can understand upcoming performance needs and get them in a pipeline instead of dealing with them on an emergency basis
  • Have tools, data, tests fully documented and easy to find
  • Continue maintenance of webpagetest

Project X (ted)

Note: ownership of this project transitioned to James Lal

  • [DROPPED] P1: Work with Releng and the Gaia team to deliver a prototype of a flexible test scheduling system that can be used by Gaia's Shepherd project to schedule tests based on Gaia pull requests
  • [DROPPED] P1: Make it possible to display the results of this test execution in Treeherder

Sheriffs (edmorley)

  • [DONE] P1: Perform initial Treeherder testing side-by-side with TBPL.
  • [DONE] P1: Add tooling to allow sheriffs to make use of the new SlaveAPI. This is best integrated into the existing slave health dashboard owned by releng - which has been done in bug 914805.
  • [DROPPED] [Not possible unless we switch to git.mozilla.org as the canonical repo] P2: Add tooling to allow sheriffs to easily close the Gaia repo

Treeherder (jeads)

  • [DONE] P1: Achieve feature parity with TBPL in terms of sheriffing
  • [MISSED] P2: Deliver a production system that can be used for sheriffing independently of TBPL
  • [DONE] P1: Ensure that Treeherder can ingest and display data from non-buildbot sources, such as Travis or Jenkins