Auto-tools/Goals/2013Q1

Well, the world didn't end, so we are going to keep rocking automation into 2013!

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

For Q1, I'd like to take some time to focus on some back-to-basics stuff so we can spend the rest of the year building atop a solid foundation. That said, there is front-line automation that we will continue needing to deliver for performance and the various mobile products. Note that this isn't a real statement of goals yet, it's more a statement of the projects we plan to undertake. Once we have agreement on this, we'll craft it into a set of high level goals.

For some inspiration on this please see our proposed goals and gps's wishlist

High Level Goals

This is my first crack at the high level goals based on the below projects

• Goal: Build to support future capacity for workflow and performance by:
  • Deploying Bugzilla 4.2
  • Publicizing and starting on an effort to revamp TBPL
  • Updating test harnesses to use mozbase and mozharness and creating a migration plan for moving to better test reporting across the board (migration planned for Q2 to coincide with TBPL v2 delivery)
  • Revamp GoFaster measurement tools
  • Create triage plan
  • Make all QA centered automation ready for rapid betas to allow the desktop team to work faster
• Goal: Continue to deliver B2G automation support for in house developers and partners
• Goal: Create new Performance measurement tool with performance team to measure startup using the best of our new tools - mozharness/mozbase/datazilla.
• Goal: Support next generation android efforts by getting Android x86 virtualized environments to run correctness tests
• Goal: Improve test slave capacity by allowing linux tests to be entirely virtualized for tests that do not need hardware support.

Building Foundations

Bugzilla

• [ON TRACK] Deploy Bugzilla 4.2
• [ON TRACK] Deploy tracking flags optimization
• [ON TRACK] Fix persona support
• [ON TRACK] Make BMO work with Zeus's caching mechanism
• [ON TRACK] Create plan for user workflow integration into Bugzilla

TBPL & Sheriffing

• [ON TRACK] Create the "Single Source of Truth" data extraction layer and deploy into buildbot
• [ON TRACK] Create the Requirements and Back End design for TBPL v2
• [ON TRACK] Complete 3 sprints on TBPL v2 code (estimated delivery end of Q2) TBD: Decide what features will be completed
• [ON TRACK] Investigate improving test harness logging and create plan for structured data/log formats - create ability to hang rich meta data (Cycle counters, memory usage etc) in test results. Deliver transition plan from current logging insanity into a modern results reporting situation, optimize for machine readable output that can be transformed for humans

Performance

• [ON TRACK] Analyze & Improve data in Datazilla with an eye to switching off Graphs.m.o sometime in Q1/Q2

Build/Test Capacity

• [ON TRACK] Regenerate the GoFaster dashboards to understand where our time goes in automation
• [ON TRACK] Complete the manifests for mochitests so we can provide future versatility on that harness
• [ON TRACK] Get linux desktop tests running in EC2 VMs (for tests that don't require hardware)
• [ON TRACK] Work with Releng to deliver first steps to make buildbot scheduling more configurable to support "bisect in the cloud"

General Automation

• [ON TRACK] Complete the move to Python 2.7 on all build and test automation harnesses.
• [ON TRACK] with releng) -- Deploy mozharness for all desktop test automation to m-c
• [ON TRACK] Move mochitest atop mozbase
  • [ON TRACK] Finish documenting mozbase modules
  • [ON TRACK] Re-tool mozprocess to have a more sane API

Automation Development

• [ON TRACK] Complete and test pulse integration with Mozmill CI for rapid beta readiness
• [ON TRACK] Continue working on further WebRTC tests as fake devices come online
• [CARRY OVER] Complete Documentation overhaul

Triage

• [ON TRACK] Create and announce early bug lifecycle plan for triage
• [ON TRACK] Identify and drive creation of triage workflows into overall bugzilla user workflow overhaul plan
• [ON TRACK] Do "testing" of triage plan by triaging TBD components using new plan, tweaking as necessary

Front Line Automation

B2G

• [CARRY OVER] Gaia QA tests on panda
• [ON TRACK] Gaia JS integration tests on panda
• [ON TRACK] Speed up emulator tests by reducing setup times, integrating with goFaster
• [ON TRACK] Enable more mochitest and reftests
• [ON TRACK] Work with B2G partners and help support them as they start running our automation

Mobile

• [ON TRACK] Improve MozPool so it can be used to help us manage the influx of mobile platforms - should work seamlessly with android and b2g - and should be able to handle different platforms of android (4.x on x86 (tbd) or 2.2 on ARM (tegra) or 4.x on ARM (panda) etc)
• [ON TRACK] Get Android x86 VMs or Android x86 Emulators running tests solidly enough that buildbot integration can occur
• [ON TRACK] Do preliminary groundwork with partners to verify an android x86 devboard solution - no devboards delivered to automation this quarter).

Bisect in the Cloud v1

• [ON TRACK] Deliver plan and requirements
• [ON TRACK] Build v1 of tool using existing mozregression framework
  • [ON TRACK] Ideally, use more flexible buildbot scheduling, if that falls through, then use hourly builds

Performance

• [ON TRACK] With performance team -- create new system (based on mozbase and datazilla entirely to free our hands from the old talos constraints) for startup measurement on desktop

WebRTC

• [ON TRACK] Create requirements for building a fully configurable webRTC network automation rig
• [ON TRACK] Determine if we can build a webRTC rig "on the cheap"
• [ON TRACK] Prototype the webRTC rig on the cheap (if possible)

Stretch Goals/Proposals

• stretch/intern: Build pseudo-automated Eideticker-with-camera rig for user perceived performance testing of devices without HDMI out.
• Stretch: Deploy mozpool for on device testing in haxxor to flesh out further on-real-device testing abilities for both android and b2g