Auto-tools/Projects/B2G Automation

Goal 2013 Q2

B2G Automation

Approach and Milestones

• TODO: Add bugzilla bug queries to this by attaching proper whiteboard flags to the relevant bugs that are filed.

Milestone 1 - 2 weeks (April 1 - 12)

• [jgriffin] [MISSED] Get Gaia Unit tests running in TBPL (blocked by bug 855049)
• [jgriffin] [DONE] Triage all existing B2G failures and identify a concrete set of problems that need to be addressed

1. test_click errors bug 838607 - dburns looking at it.
2. reftest/crashtest crash bug 818103 - assigned to marshall_law.
3. exception in window_manager.js bug 851666 - asked David Scravaglieri for help
4. [DONE] Timeout for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts. Rob will fix.
5. socket.timeout errors caused by actorPool error bug 859417
6. socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
7. chrome tests occasionally can't find elements (timing or page loading error?) bug 859423
8. B2G crashes during WebAPI tests bug 859439 - will be easier to handle once crash detection is working here

• [jgriffin] [MISSED] Make mach runners for B2G mochitests and reftests
• [ahal] [DONE] Enable crash detection on mozilla-b2g18 for mochitest/reftest
• [ahal] [DONE] Add crash detection to Marionette tests
• [ahal] [MISSED] Write a script to download the latest emulator
• [mdas] [DONE] Use weakref in marionette-elements (bug 736592)
• [mdas] [ON TRACK] Work on getting Gaia UI tests running on b2g desktop builds
• [yiming] [DONE] Complete touch/mouse event code for action chains
• [yiming] [MISSED] Convert code from synthetic gestures to action chains
• [rwood] [DONE] Develop bluetooth Gaia UI test to pair unagi with host machine
• [rwood] [DONE] Finish integrating long running tests with datazilla

Milestone 2 - 2 weeks (April 22 - May 3)

• [jgriffin] [CARRY OVER] Get Gaia Unit tests running in TBPL (just unblocked on April 24)
• [jgriffin] [CARRY OVER] Make mach runners for Marionette
• [jgriffin] Add mach to B2G so we can build B2G mach targets
• [jgriffin] Improve Marionette's error reporting with specific, clear error messages and stack traces
• [jgriffin] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
• [ahal] [CARRY OVER] Write a B2G bootstrap script to make downloading and running tests easier
• [ahal] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
• [mdas] [CARRY OVER] Work on refactoring touch events and getting the mapping between action chains and mouse events working.
• [mdas] [CARRY OVER] Finish converting synthetic gestures to action chains
• [mdas] Get mozilla-central b2g desktop builds working with Marionette
• [rwood] Get gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review)
• [rwood] Improve gaia-ui endurance test stability
• [rwood] Automate entire process via Jenkins

Milestone 3 - 2 weeks (tentative start: May 13)

• Finish integrating JS Marionette client into Python testrunner
• Work on top B2G test stability problems and either fix or get them to a state where developers can take over

1. socket.timeout errors caused by actorPool error bug 859417
2. socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551

• Get Gaia UI tests running on B2G desktop builds in TBPL (will need a mozharness script)
• Add debugger support to B2G testrunners

Milestone 4 - 2 weeks (planned)

• Run mochitests on b2g desktop builds on TBPL?
• Resolve more of the top TBPL intermittent oranges

Future milestones

• Get rid of debugger server dependency bug 797529
• Get Gaia integration tests (JS UI tests) in TBPL
• Update marionette docs