Auto-tools/Goals/2013Q4

Marionette

P0 - (highlight)Get passing WebDriver conformance tests running in TBPL P0 - (nonpublic) Continue to support WebQA and Gaia teams with bug fixes and new features P1 - (highlight)Fix WebDriver compatibility problems; implement missing commands P1 - (nonpublic) Split monolithic Python repo into separate packages P2 - (nonpublic) Improve handling of socket errors

= B2G = (going to need to cut down the size here after the work week). TBD depending on outcome of oslo work week, likely candidates:

• (perf team) power consumption testing - DANGER not yet sure how to get actionable data out of this, cross team measurements
• perf tryserver? - DANGER unspecifed even less than perf.
• (cross-team highlight)(with gaia team) github autolanding
• (continuing with investigation)(with gaia and rel-eng) running tests via github pull requests (or this could be Q1-2014)
• (nonpublic) continued burn down of intermittent oranges
• (highlight)continued work to make B2G tests more sheriffable
• tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
• memory tests
• gaia integration tests on tbpl - still don't exist yet.
• build smoke testing on nexus 4 devices
• expanding hardware rig in mtn view - (stephend team, IT) joint with perf system

Performance (theme is visibility)

• (highlight) deploy performance project dashboard to outliine all projects, health of data, impact of the data, and make it easy to understand the data via docs and views on the data.
• have tools, data, tests fully documented and easy to find (all)
• hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority (jmaher)
• proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher)

Datazilla

• (tied to 1st perf goal) Implement alert system that monitors data ingestion rates for all branches in all projects.
• (tied to 1st perf goal) NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
• Fine tune automated regression detection system as needed
• Implement the performance dashboard described in the performance section
  • Set up eideticker projects and backfill eideticker data (https://bugzilla.mozilla.org/show_bug.cgi?id=909796)
  • Integrate documentation in the homepage UI and in the b2g UI
  • Remove all legacy code from deprecated UI's \o/

NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of.

Treeherder

• Implement TBPL *starring* features
• Implement Orange Factor compatability
• Release production instance on AWS
• (mid-q) Get sheriffs to drive side-by-side usability testing with TBPL
  • Make sure we're on target for full TBPL replacement in Q1 2014.

Bugzilla

• Continue Bugzilla as a Platform (BaaP):
  • [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
  • [dkl] (highlight) bzAPI compatibility layer
  • [dkl] plan for versioning APIs, first draft of interface for API v2
• [glob] (highlight) memcached minimum viable product (also helps BaaP above)
• [mcote] ReviewBoard integration plan based on feedback
• [mcote] somehow get over to git, or at least have a really serious plan for it
• [kyle] deploy new bzETL

Android

• (highlight) Fix printf debugging so it doesn't go to logcat by default
• (highlight) Update x86 emulator to SDK 18 fix agent 4.3
• panda stablity
• bandwidth metrics???
• Eideticker responsiveness (possible carry over, depending on Q3).
• C++ remote tests

WebRTC

• Wire up existing AWS system, see if we can give to releng to run from here on out
• Investigate hardware set up (joint with desktop QA?)

platforms

• mac os x 10.9 - still waitiing to see what happens here.
• official 8.1 windows release
• releng working on EC2 windows...not sure of status

Bugmaster

• Build out community involvement at the summit
• Get tools we've built deployed some place permanent (bug board, tutorial, bugzilla user profile report)
• Continue triaging and imrpove the groundswell.
• Research into new Bugzilla home page for logged-in/logged-out users (separate pages)

Desktop QA Automation

• System expansion to support locale tests
• (highlight) Conversion to mozmill 2.0
• (highlight) metro tests in production
• stetch (work on the community idea with jdm)
• (highlight) SV to lead the automation for QA with our oversight
• web app to generate the release/beta test manfiest

sheriffs

• DEPEND: depdening on treeherder
• testing treeherder
• complete documentation goals/operationalize sheriffing

Research

• implement something to know how often our dashboards are used
  • web page tracking/analytics software
  • irc keyword counters
  • bugzilla activity
• Orange Quarantine Experiment
• turn off passing tests
  • directory based
• forecastingtool
• define a good orangefactor number? 3.14
• manifests - is this goal worthy, or a side project until it finishes

mozbase

• add structured logging to more harnesses
  • need to merge the logs from python and js
  • need a long term vision here

Community

   [henrik] Create a tool to help watch and observe community member's involvement / engagement across projects; to highlight community members in danger of dropping out and/or community members who are ready for bigger challenge

Mozmill

   [henrik] Setup a tryserver similar to mozmill-ci to get assistence in running tests with local patches applied.