Auto-tools/Goals/2011Q4

• [ON TRACK] Achieve and maintain a 5% or less intermittent orange rate for Android automation (talos and tests) to achieve parity with desktop intermittent orange rates.
  • [ON TRACK] Land modifications to make it easier for developers to run Android talos (to make it easier to debug failures)
  • [ON TRACK] Create and land specialpowers-core JS module to simplify test utilities and reduce bugs in "alive test", talos, reftest, and mochi* test families

• [ON TRACK] Expand and enhance our test and performance automation to anticipate and address needs for Fennec and B2G.
  • [ON TRACK] Deliver video capture data from an alive test (Ts/Tp) an a "driver" test (Tuser) from eideticker to developers for analysis. Stretch: have system ready for duplication by end of Quarter
  • [ON TRACK] Deploy Talos RSS metrics in Android production
  • [ON TRACK] Land marionette with ability to "runscript" in B2G context and enough action support to run Tuser eideticker test.
  • [ON TRACK] Land enhancements to special powers to entirely remove enablePrivilege from all testing frameworks

• [ON TRACK] Deploy user responsiveness automation into production
  • [ON TRACK] Deploy Talos User Responsiveness test on desktop and android automation
  • [ON TRACK] Deploy Regression User Responsiveness (peptest) on desktop production automation

Supporting Projects

Many of our projects are inter-related, and there are many supporting projects that we want to call out as they are important and cannot be entirely neglected in favor of these goals. This section calls these out.

Bughunter

• Q4 Outcomes
  • [ON TRACK] UI Deployed, UI taken through find/fix cycle
  • [ON TRACK] Conduct demo/training for QA, Dev, and Product Mgmt on tool
  • [ON TRACK] Create plan for soccoro integration

• Community Involvement
  • Targeted involvement with other areas of Mozilla (see training outcome)

• Interconnections
  • Increasing quality and security for the gecko platform by providing good data on crashes.
  • Visualization technologies will be repurposed
  • Future: Android integration, re-using technology for "classify the web" effort

Pandaboards

• Q4 Outcomes
  • [ON TRACK] Create repeatable, deployable build/flash system to get a functioning board running
  • [ON TRACK] Analyze results from unit and performance tests on board, but not in production this quarter

• Community Involvement
  • Working with linaro, having presence in pandaboard community, publicizing the created scripts

• Interconnections
  • Aids with the "expand mobile test and performance automation" Q4 goal.

Mozbase

• Q4 Outcomes
  • [ON TRACK] Work with Release Engineering MozHarness to deploy a MozHarness/MozBase Talos system
  • [ON TRACK] Ensure mozbase utilities are robust enough underpinnings for Peptest
  • [ON TRACK] Have our own automated tests running for Mozbase utilities to ensure their robustness (reporting to autolog)

• Community Involvement
  • Use github to pull in new contributors toward our work
  • Publicize what benefits these changes will bring

• Interconnections
  • Aids with the "make talos easier for developers to run" sub goal in "Make android automation more reliable"
  • Crucial support for PepTest in user responsiveness goal.

W3C Browser Test Protocol Effort

• Q4 Outcomes
  • Attend meeting, ensure actions created from it are helpful to our efforts
  • Situate ourselves for greater involvement in Q1.

• Community Involvement
  • This is a community involvement goal

• Interconnections
  • The JSON protocol that marionette is using will be based upon the protocol developed by this group.

Mozmill E10S

• Q4 Outcomes
  • Adapt the harness to run in E10S firefox and fennec
  • All Mutt tests should pass in E10S mode

• Community Involvement
  • Publicize the "lessons learned" for other addon developers to use

• Interconnections
  • The later phases of Peptest will re-use some of the mozmill UI driver code to provide a simple API for driving the browser to perform user actions.

Bugzilla

• Q4 Outcomes
  • Complete Bugzilla REST API to be native to bugzilla with all parameters of the existing REST API
  • Drive creation of plan for Pulse integration with bugzilla (involves IT coordination, experimentation)
  • Deploy 4.2 (provided upstream is completed).

• Community Involvement
  • Be more public about what these changes (REST, pulse, 4.2 etc) bring to the wider mozilla community.

• Interconnections
  • REST API and Pulse integration directly improve Orange Factor, which aids us in making our test and performance automation more stable.