Goal 2013 Q2

B2G Automation

Approach and Milestones

TODO: Add bugzilla bug queries to this by attaching proper whiteboard flags to the relevant bugs that are filed.

[jgriffin] [MISSED] Get Gaia Unit tests running in TBPL (blocked by bug 855049)
[jgriffin] [DONE] Triage all existing B2G failures and identify a concrete set of problems that need to be addressed

test_click errors bug 838607 - dburns looking at it.
reftest/crashtest crash bug 818103 - assigned to mike habicher.
[DONE] exception in window_manager.js bug 851666 - WORKSFORME
[DONE] Timeout for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts. Rob will fix.
socket.timeout errors caused by actorPool error bug 859417
socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
chrome tests occasionally can't find elements (timing or page loading error?) bug 859423
B2G crashes during WebAPI tests bug 859439 - will be easier to handle once crash detection is working here

[jgriffin] [DONE] Get Gaia Unit tests running in TBPL (scheduled on cedar)
[jgriffin] [DONE] Make mach runners for Marionette
[jgriffin] [MISSED] Add mach to B2G so we can build B2G mach targets
[jgriffin] [MISSED] Improve Marionette's error reporting with specific, clear error messages and stack traces
[jgriffin] [MISSED] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
[jgriffin] [MISSED] Get mozilla-central b2g desktop builds working with Marionette
[ahal] [DONE] Write a B2G bootstrap script to make downloading and running tests easier
[ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
[mdas] [MISSED] Work on refactoring touch events and getting the mapping between action chains and mouse events working.
[mdas] [MISSED] Finish converting synthetic gestures to action chains
[rwood] [DONE] Get gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review) was pulled out of the repo, new location tbd
[rwood] [ON TRACK] Improve gaia-ui endurance test stability
[rwood] [DONE] Automate entire process via Jenkins

[ON TRACK] Work on top B2G test stability problems and either fix or get them to a state where developers can take over

[DONE] test_click errors bug 838607 - dburns looking at it.
[DONE] reftest/crashtest crash bug 818103 - assigned to mike habicher.
[DROPPED] socket.timeout errors caused by actorPool error bug 859417 - dropped due to marionette refactor
[MISSED] socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551

[mdas] [DEFERRED] Finish converting synthetic gestures to action chains - may not be necessary.
(carry over) [mdas] [DONE] On on refactoring touch events and getting the mapping between action chains and mouse events working.
[mdas] [DONE] Develop manifest of stable gaia-ui-tests on b2g desktop builds
[mdas] [ON TRACK] Work on improving Marionette docs
[mdas] [ON TRACK] Fix tap/touch related bugs that are blocking webQA
(carry over) [jgriffin] [DONE] Get mozilla-central b2g desktop builds working with Marionette
[jgriffin] [DONE] Add manifest support to gaia-unit-tests
[jgriffin] [DONE] write mozharness script for gaia-ui-tests on b2g desktop builds
[DROPPED] Add debugger support to B2G testrunners
(carry over) [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
[wlach] [MISSED] Get rid of debugger server dependency bug 797529

(carry over) [wlach] [DONE] Get rid of debugger server dependency bug 797529
[wlach] [ON TRACK] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
(ongoing) [mdas] Resolve issues with tap
(ongoing) [mdas] Update marionette docs
(carry over) [ahal] [DONE] Get mozilla-central B2G unit tests to use in-tree mozbase instead of puppetagain
[jgriffin] Disable gaia-unit-tests that are still failing on cedar
[jgriffin + releng] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
[jgriffin] Update mozharness script for gaia-ui-tests
[jgriffin + releng] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
[jgriffin] Identify current B2G top oranges
[rwood] Try rel-eng m-c unagi-eng on gaia-ui-tests
[rwood] [DONE] Develop some heuristics to help determine when gaia-ui-test Jenkins failures are "real" or reproducible enough to warrant sending out an e-mail notification to engineering managers

[wlach] [ON TRACK] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
(ongoing) [mdas] Resolve issues with tap
(ongoing) [mdas] Update marionette docs
[jgriffin] Disable gaia-unit-tests that are still failing on cedar
[jgriffin + releng] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
[jgriffin] Update mozharness script for gaia-ui-tests
[jgriffin + releng] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
[jgriffin] Identify current B2G top oranges
[ahal] Identify and fix issues with full-stack emulator builds