View source for Auto-tools/Projects/B2G Automation

= Goal 2013 Q2 =

== Milestone 1 - 2 weeks (April 1 - 12) ==
* [jgriffin] {{miss|Get}} Gaia Unit tests running in TBPL (blocked by {{bug|855049}})
* [jgriffin] {{done|Triage}} all existing B2G failures and identify a concrete set of problems that need to be addressed
# test_click errors {{bug|838607}} - dburns looking at it.
# reftest/crashtest crash {{bug|818103}} - assigned to mike habicher.
# {{done|exception}} in window_manager.js {{bug|851666}} - WORKSFORME
# {{done|Timeout}} for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts.  Rob will fix.
# socket.timeout errors caused by actorPool error {{bug|859417}}
# socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in {{bug|829551}}
# chrome tests occasionally can't find elements (timing or page loading error?) {{bug|859423}}
# B2G crashes during WebAPI tests {{bug|859439}} - will be easier to handle once crash detection is working here
* [jgriffin] {{miss|Make}} mach runners for B2G mochitests and reftests
* [ahal] {{done|Enable}} crash detection on mozilla-b2g18 for mochitest/reftest
* [ahal] {{done|Add}} crash detection to Marionette tests
* [ahal] {{miss|Write}} a script to download the latest emulator
* [mdas] {{done|Use}} weakref in marionette-elements ({{bug|736592}})
* [mdas] {{ok|Work}} on getting Gaia UI tests running on b2g desktop builds
* [yiming] {{done|Complete}} touch/mouse event code for action chains
* [yiming] {{miss|Convert}} code from synthetic gestures to action chains
* [rwood] {{done|Develop}} bluetooth Gaia UI test to pair unagi with host machine
* [rwood] {{done|Finish}} integrating long running tests with datazilla

== Milestone 2 - 2 weeks (April 22 - May 3) ==
* [jgriffin] {{done|Get}} Gaia Unit tests running in TBPL (scheduled on cedar)
* [jgriffin] {{done|Make}} mach runners for Marionette
* [jgriffin] {{miss|Add}} mach to B2G so we can build B2G mach targets
* [jgriffin] {{miss|Improve}} Marionette's error reporting with specific, clear error messages and stack traces
* [jgriffin] {{miss|Integrate}} JS client Pt 1:  Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
* [jgriffin] {{miss|Get}} mozilla-central b2g desktop builds working with Marionette
* [ahal] {{done|Write}} a B2G bootstrap script to make downloading and running tests easier
* [ahal] {{miss|Use}} in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
* [mdas] {{miss|Work}} on refactoring touch events and getting the mapping between action chains and mouse events working.
* [mdas] {{miss|Finish}} converting synthetic gestures to action chains
* [rwood] {{done|Get}} gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review) was pulled out of the repo, new location tbd
* [rwood] {{ok|Improve}} gaia-ui endurance test stability
* [rwood] {{done|Automate}} entire process via Jenkins

== Milestone 3 - 2 weeks (May 13 - May 24) ==
* {{ok|Work}} on top B2G test stability problems and either fix or get them to a state where developers can take over
# {{done|test_click}} errors {{bug|838607}} - dburns looking at it.
# {{done|reftest/crashtest}} crash {{bug|818103}} - assigned to mike habicher.
# {{drop|socket.timeout}} errors caused by actorPool error {{bug|859417}} - dropped due to marionette refactor
# {{miss|socket.timeout}} errors that occur at startup in newSession, before a new Gecko is copied - see logs in {{bug|829551}}
* [mdas] {{deferred|Finish}} converting synthetic gestures to action chains - may not be necessary.
* (carry over) [mdas] {{done|On}} on refactoring touch events and getting the mapping between action chains and mouse events working.
* [mdas] {{done|Develop}} manifest of stable gaia-ui-tests on b2g desktop builds
* [mdas] {{ok|Work}} on improving Marionette docs
* [mdas] {{ok|Fix}} tap/touch related bugs that are blocking webQA
* (carry over) [jgriffin] {{done|Get}} mozilla-central b2g desktop builds working with Marionette 
* [jgriffin] {{done|Add}} manifest support to gaia-unit-tests
* [jgriffin] {{done|write}} mozharness script for gaia-ui-tests on b2g desktop builds
* {{drop|Add}} debugger support to B2G testrunners
* (carry over) [ahal] {{miss|Use}} in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
* [wlach] {{miss|Get}} rid of debugger server dependency {{Bug|797529}}

== Milestone 4 - 2 weeks (June 3 - June 14) ==
* (carry over) [wlach] {{done|Get}} rid of debugger server dependency {{Bug|797529}}
* [wlach] {{ok|Integrate}} JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner 
* (ongoing) [mdas] {{miss|Resolve}} issues with tap
* (ongoing) [mdas] {{ok|Update}} marionette docs
* (carry over) [ahal] {{done|Get}} mozilla-central B2G unit tests to use in-tree mozbase instead of puppetagain
* [jgriffin] {{miss|Disable}} gaia-unit-tests that are still failing on cedar
* [jgriffin + releng] {{miss|Get}} gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
* [jgriffin] {{miss|Update}} mozharness script for gaia-ui-tests
* [jgriffin + releng] {{miss|Get}} gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
* [jgriffin] {{miss|Identify}} current B2G top oranges
* [rwood] {{done|Try}}rel-eng m-c unagi-eng on gaia-ui-tests
* [rwood] {{done|Develop}} some heuristics to help determine when gaia-ui-test Jenkins failures are "real" or reproducible enough to warrant sending out an e-mail notification to engineering managers

== Milestone 5 - 2 weeks (June 24 - July 5) ==
* [wlach] {{drop|Add}} JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner 
* (ongoing) [mdas] {{ok|Resolve}} blocking issues with tap
* [mdas] {{defer|Resolve}} blocking scroll issues - blocked on is_displayed fix
* (ongoing) [mdas] {{done|Add}} some mechanisms to make debugging touch related problems easier
* [jgriffin] {{ok|Disable}} gaia-unit-tests that are still failing on cedar
* [jgriffin + releng] {{miss|Get}} gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
* [jgriffin] {{done|Update}} mozharness script for gaia-ui-tests
* [jgriffin + releng] {{done|Get}} gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
* [jgriffin] {{defer|Identify}} current B2G top oranges
* [ahal] {{done|Identify}} and fix issues with full-stack emulator builds
* [rwood] {{done|Fix}} intermittent gaia-ui endurance test failures
* [rwood] {{done|Expand}} gaia-ui endurance test coverage
* [rwood] {{done|Write}} script to email/flag b2g memory spikes via gaia-ui endurance tests

= Goal 2013 Q3 =

== Milestone 6 - 2 weeks (July 8 - July 19) ==
* (carry over) [jgriffin + releng] {{done|Get}} gaia-unit-tests scheduled on all branches
* [jgriffin] {{miss|Implement}} script to compare two sources.xml
* [jgriffin] {{done|Identify}} source of memory leak in LG automated tests {{bug|897684}}
* [mdas, davehunt, webqa] {{done|Get}} gaia-ui-tests on cedar running stable green
* [mdas] {{miss|Implement}} modal dialog support in Marionette
* (carry over) [mdas] {{miss|Fix}} remaining blocking touch issues
* [wlach] {{done|Implement}} get_active_frame command - {{bug|855327}}
* [rwood] {{miss|Get}} games benchmarking tests running on Firefox OS
* [rwood] {{ok|Investigate/fix}} intermittent WebAPI emulator failures
* [rwood] {{ok|Maintain endurance}} tests as needed; expand checks for b2g memory changes over time
* [rwood] {{done|Get}} endurance tests running on Leo on Master
* [rwood] {{defer|Look}} into data requirements for gaia-ui heuristics

== Milestone 7 - 2 weeks (July 29 - August 9) ==
* ongoing - patches in hand - [ahal] Add mach to B2G so we can build B2G mach targets
* scheduled on cedar - [mihnea] Get JSreftests working for B2G
* [jgriffin] {{done|Implement}} script to compare two sources.xml
* {{done|final patch up}} - (carry over) [mdas] Implement modal dialog support in Marionette
* current focus -> (carry over) [mdas] Fix remaining blocking touch issues
* (carry over) [rwood] Get games benchmarking tests running on Firefox OS
* (ongoing) [rwood] {{ok|Investigate/fix}} intermittent WebAPI emulator failures
* (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
* currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
* [rwood] Create app-launch and page-browser tsets for AWSY
* (carry over) [rwood] Look into data requirements for gaia-ui heuristics

== Milestone 8 - 2 weeks (August 19 - August 30) ==
* (carry over) [ahal] Add mach to B2G so we can build B2G mach targets
* {{done|scheduled}} on cedar - [mihnea] Get JSreftests working for B2G
* P1! [jgriffin + all] Update all testing docs to make it easy for developers to get started with tests
* [mdas] Investigate modal dialog test failure
* (carry over) [rwood] Get games benchmarking tests running on Firefox OS
* (ongoing) [rwood] {{ok|Investigate/fix}} intermittent WebAPI emulator failures
* (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
* currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
* [rwood] Create app-launch and page-browser tsets for AWSY

== Sept 19 - Oct 2 ==

* (jgriffin PTO)
* (carry over) {{done|[mdas]}} land the modal dialog fix
* [mdas] fix the import script leak
* [mdas] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
* [ahal] write mozharness script for running mochitests on B2G desktop builds
* [wlach] finish implementation of npm mirror for rel-eng (re-writing in node.js and documenting)
* [rwood] Endurance test fixes/maintenance for inari master/1.3
* [rwood] Further work on AWSY emulator test driver
* [rwood] Continued support for games benchmarking on Firefox OS

== Future milestones ==
* Look into data requirements for gaia-ui heuristics
* {{drop|Gaia}} bisection tool - implemented by jhford
* Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds 
* Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming, depends on 4.2 emu builds in TBPL)
* Get b2g jsreftests working
* Implement device test failure notification heuristics (depends on results being uploaded to TBPL)
* Fix more top oranges
* Get gaia-ui-tests running on all branches (depends on gaia-ui-tests to gaia coupling, gaia try-server)
* {{drop|Add}} Finish integrating JS Marionette client into Python testrunner
* Get Gaia integration tests (JS UI tests) in TBPL (depends on Gaia devs)
* Write tests for new bluetooth support in emulator?