Auto-tools/Goals/2011Q4

Official Q4 Goals

• [DONE] Achieve and maintain a 5% or less intermittent orange rate for Android automation (talos and tests) to achieve parity with desktop intermittent orange rates.
  • [DONE] Land modifications to make it easier for developers to run Android talos (to make it easier to debug failures)
  • [DROPPED] Create and land specialpowers-core JS module to simplify test utilities and reduce bugs in "alive test", talos, reftest, and mochi* test families

• [DONE] Expand and enhance our test and performance automation to anticipate and address needs for Fennec and B2G.
  • [DONE] Deliver video capture data from an alive test (Ts/Tp) an a "driver" test (Tuser) from eideticker to developers for analysis. Stretch: have system ready for duplication by end of Quarter
  • [DONE] Deploy Talos RSS metrics in Android production
  • [DONE] Land marionette with ability to "runscript" and enough action support to run Tuser eideticker test on both B2G and Fennec.
  • [DROPPED] Land enhancements to special powers to entirely remove enablePrivilege from all testing frameworks

• [DONE] Deploy user responsiveness automation into production
  • [DONE] Deploy Talos User Responsiveness test on desktop and android automation
  • [DONE] Deploy Regression User Responsiveness (peptest) on desktop production automation

Crucial Projects

These projects must be completed to achieve the above goals.

GOAL: Improve Android Stability

• Q4 Outcomes
  • [DONE] Reproduce and fix intermittent android issues
  • [DONE] Add Android Talos to Orange Factor to aid in tracking
  • [DONE] "Make Android Remote Talos" must be landed
  • [DROPPED] Create specialpowers core to use in other tests, may not prove necessary

• Community Involvement
  • All these changes directly impact developers, ensure good publicizing (blog, newsgoup etc) done around these changes

• Depends On
  • Orange Factor
  • Mozbase/Mozharness
  • SpecialPowers API work

GOAL: Expand Android Performance and Unit Tests

Eideticker

• Q4 Outcomes
  • [DONE] Deliver video capture data from an alive test (Ts/Tp) an a "driver" test (Tuser) from eideticker to developers for analysis.
  • [DONE] Have system stable enough to be ready for duplication in MV by end of quarter (stretch)
  • [DONE] Define two tests for completion at end of Quarter
  • [DROPPED] Integrate with marionette JSON protocol to drive device under test
  • [DROPPED] Potentially use JSON protocol and shim (supplied by selenium project) to drive non-gecko browsers on device (stretch/Q1)
    • NOTE: We got marionette working with eideticker, but decided it would be more important to simplify the harness and make it easier to run for developers.
• Community Involvement
  • Cross-team project, ensure and maintain feedback loops with teams Eideticker is being developed for - close customer integration

• Depends on
  • Marionette
  • Mozbase/Talos

Marionette

• Q4 Outcomes
  • [DONE] Create receiver for JSON protocol
  • [DONE] Implement enough action to support the Eideticker tests
  • [MISSED] Land in m-c by end of quarter
    • NOTES: We were gated on landing in m-c by the remote debugger code landing in m-c. We drove it hard, but in the end the remote debugger code was held up by reviews and there will be no way to land in m-c by the end of the quarter. However, we will be landing in the b2g repo by the end of the quarter

• Community Involvement
  • Cross-team project, ensure and maintain feedback loops with teams Eideticker is being developed for - close customer integration.

• Depends on
  • Developer Tools Remote Debugging Code
  • A11y interfaces
  • W3C Browser Test Protocol definition (soft dependency)

SpecialPowers API

• Q4 Outcomes
  • [DROPPED] Remove all enablePrivilege calls in all test frameworks and land patches in m-c
    • NOTES: We decided with the de-prioritization of E10S to drop SpecialPowers work and double down on native fennec work
• Community Involvement
  • Be public and visible, try to recruit JS hackers to aid with project, engage developers to help expand APIs as needed

• Depends on
  • Developer aid (original test writers usually) when issues arise

GOAL: User Responsiveness Automation

Talos User Responsiveness

• Q4 Outcomes
  • [DONE] Deploy into production for desktop and android
  • [DONE] Finalize equation for responsiveness metric
  • [DONE] Determine if test can be deployed as addition to Tp5
  • [DONE] Work with developers and Releng for deployment
    • NOTES: Still tracking down linux failure, mac and windows landed

• Community Involvement
  • Ensure blogs, newsgroups updated with changes and notices surrounding this system

• Depends on
  • Developers creating the metric
  • Release Engineering for deployment
  • Stable android automation

Peptest User Responsiveness

• Q4 Outcomes
  • [DONE] Land test harness in m-c with full integration (make targets etc)
  • [MISSED] Deploy into production for desktop automation
    • NOTES: Aki and Ahal made heroic efforts to get peptest deployed by end of Quarter. We did everything - interoperability bugs filed, debugged, and fixed, buildbot code staged, peptest landed in m-c, etc. Everything is ready to turn on for deployment to try, we just ran out of time before christmas vacations hit us. Expect to deploy to try in first two weeks of Q1.
  • [DONE] Provide simple API for driving UI (post-landing, stretch)

• Community Involvement
  • Ensure community is up to date on the new harness and expectations surrounding it.
  • Ensure log parsers ready to accept the log format and developers educated on how to run it and parse the data from it

• Depends on
  • Mozbase/mozharness
  • Release Engineering for deployment
  • Developers creating the metric for Talos User Responsiveness
  • Mozmill E10s (for driving API)

Supporting Projects

Many of our projects are inter-related, and there are many supporting projects that we want to call out as they are important and cannot be entirely neglected in favor of these goals. This section calls these out.

Bughunter

• Q4 Outcomes
  • [DONE] UI Deployed, UI taken through find/fix cycle
  • [MISSED] Conduct demo/training for QA, Dev, and Product Mgmt on tool
  • [DONE] Create plan for soccoro integration
    • NOTE: We concentrated on the UI and pushed the demo/training into early Q1.

• Community Involvement
  • Targeted involvement with other areas of Mozilla (see training outcome)

• Interconnections
  • Increasing quality and security for the gecko platform by providing good data on crashes.
  • Visualization technologies will be repurposed
  • Future: Android integration, re-using technology for "classify the web" effort

Pandaboards

• Q4 Outcomes
  • [DROPPED] Create repeatable, deployable build/flash system to get a functioning board running
  • [DROPPED] Analyze results from unit and performance tests on board, but not in production this quarter

• Community Involvement
  • Working with linaro, having presence in pandaboard community, publicizing the created scripts

• Interconnections
  • Aids with the "expand mobile test and performance automation" Q4 goal.

Mozbase

• Q4 Outcomes
  • [DONE] Work with Release Engineering MozHarness to deploy a MozHarness/MozBase Talos system
    • NOTE: It's not fully deployed yet due to a talos bug that we hit on the way. Decided it was more important to continue improving mozharness and peptest and talos for native fennec than to drive this the rest of the way to completion.
  • [DONE] Ensure mozbase utilities are robust enough underpinnings for Peptest
  • [DONE] Have our own automated tests running for Mozbase utilities to ensure their robustness (reporting to autolog)

• Community Involvement
  • Use github to pull in new contributors toward our work
  • Publicize what benefits these changes will bring

• Interconnections
  • Aids with the "make talos easier for developers to run" sub goal in "Make android automation more reliable"
  • Crucial support for PepTest in user responsiveness goal.

W3C Browser Test Protocol Effort

• Q4 Outcomes
  • [DONE] Attend meeting, ensure actions created from it are helpful to our efforts
  • [DONE] Situate ourselves for greater involvement in Q1.

• Community Involvement
  • This is a community involvement goal

• Interconnections
  • The JSON protocol that marionette is using will be based upon the protocol developed by this group.

Mozmill E10S

• Q4 Outcomes
  • [DROPPED] Adapt the harness to run in E10S firefox and fennec
  • [DROPPED] All Mutt tests should pass in E10S mode

• Community Involvement
  • Publicize the "lessons learned" for other addon developers to use

• Interconnections
  • The later phases of Peptest will re-use some of the mozmill UI driver code to provide a simple API for driving the browser to perform user actions.

Bugzilla

• Q4 Outcomes
  • [MISSED] Complete Bugzilla REST API to be native to BMO with all parameters of the existing REST API
    • NOTES: Good progress but not complete. Will need to plan a beta run and security review first of the quarter before full roll out. Should be done before the end of Q1.
  • [DONE] Drive creation of plan for Pulse integration with BMO (involves IT coordination, experimentation, re-architecture of current code)
    • NOTES: Made good progress on this, will carry forward into Q1
  • [MISSED] Deploy 4.2 (provided upstream is completed).
    • NOTES: Upstream 4.2 hasn't been released; we have been working to resolve some of he upstream roadblocks and remaining issues, however there are issues that we can't address directly.

• Community Involvement
  • Be more public about what these changes (REST, pulse, 4.2 etc) bring to the wider mozilla community.

• Interconnections
  • REST API and Pulse integration directly improve Orange Factor, which aids us in making our test and performance automation more stable.
  • Investigate updating the Elastic-Search based bug cache directly from BMO, and make available to the wider community (as an alternative to polling BMO directly)

Tegra Pool Automation

• Q4 Outcomes
  • [DONE] Ensure prototype up and running with a few tegras for checkout
  • [DONE] Demo at Mobile Summit for developers as a solution for new/hard to get hardware

• Community Involvement
  • See above demo.

• Interconnections
  • Aids developers to more easily fix android issues by providing them debug-ready tegras upon checking them out from the system.

Back Burner

These projects are on the back burner for the quarter. We'll aid with support and maintenance as needed, but they will not be our primary focus

• Autolog
• War on Orange
• Speed Tests
• DXR automation
• TPS (crossweave)