Auto-tools/Projects/B2G Automation
From MozillaWiki
< Auto-tools | Projects
This page is mostly obsolete. FirefoxOS Automation is now handled by the FirefoxOS Automation Team. See https://github.com/mozilla-b2g/automation-projects or irc.mozilla.org#fxos-automation
Etherpad
https://etherpad.mozilla.org/b2g-automation-meeting-notes
Goal 2013 Q2
Milestone 1 - 2 weeks (April 1 - 12)
- [jgriffin] [MISSED] Get Gaia Unit tests running in TBPL (blocked by bug 855049)
- [jgriffin] [DONE] Triage all existing B2G failures and identify a concrete set of problems that need to be addressed
- test_click errors bug 838607 - dburns looking at it.
- reftest/crashtest crash bug 818103 - assigned to mike habicher.
- [DONE] exception in window_manager.js bug 851666 - WORKSFORME
- [DONE] Timeout for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts. Rob will fix.
- socket.timeout errors caused by actorPool error bug 859417
- socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
- chrome tests occasionally can't find elements (timing or page loading error?) bug 859423
- B2G crashes during WebAPI tests bug 859439 - will be easier to handle once crash detection is working here
- [jgriffin] [MISSED] Make mach runners for B2G mochitests and reftests
- [ahal] [DONE] Enable crash detection on mozilla-b2g18 for mochitest/reftest
- [ahal] [DONE] Add crash detection to Marionette tests
- [ahal] [MISSED] Write a script to download the latest emulator
- [mdas] [DONE] Use weakref in marionette-elements (bug 736592)
- [mdas] [ON TRACK] Work on getting Gaia UI tests running on b2g desktop builds
- [yiming] [DONE] Complete touch/mouse event code for action chains
- [yiming] [MISSED] Convert code from synthetic gestures to action chains
- [rwood] [DONE] Develop bluetooth Gaia UI test to pair unagi with host machine
- [rwood] [DONE] Finish integrating long running tests with datazilla
Milestone 2 - 2 weeks (April 22 - May 3)
- [jgriffin] [DONE] Get Gaia Unit tests running in TBPL (scheduled on cedar)
- [jgriffin] [DONE] Make mach runners for Marionette
- [jgriffin] [MISSED] Add mach to B2G so we can build B2G mach targets
- [jgriffin] [MISSED] Improve Marionette's error reporting with specific, clear error messages and stack traces
- [jgriffin] [MISSED] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
- [jgriffin] [MISSED] Get mozilla-central b2g desktop builds working with Marionette
- [ahal] [DONE] Write a B2G bootstrap script to make downloading and running tests easier
- [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
- [mdas] [MISSED] Work on refactoring touch events and getting the mapping between action chains and mouse events working.
- [mdas] [MISSED] Finish converting synthetic gestures to action chains
- [rwood] [DONE] Get gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review) was pulled out of the repo, new location tbd
- [rwood] [ON TRACK] Improve gaia-ui endurance test stability
- [rwood] [DONE] Automate entire process via Jenkins
Milestone 3 - 2 weeks (May 13 - May 24)
- [ON TRACK] Work on top B2G test stability problems and either fix or get them to a state where developers can take over
- [DONE] test_click errors bug 838607 - dburns looking at it.
- [DONE] reftest/crashtest crash bug 818103 - assigned to mike habicher.
- [DROPPED] socket.timeout errors caused by actorPool error bug 859417 - dropped due to marionette refactor
- [MISSED] socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
- [mdas] [DEFERRED] Finish converting synthetic gestures to action chains - may not be necessary.
- (carry over) [mdas] [DONE] On on refactoring touch events and getting the mapping between action chains and mouse events working.
- [mdas] [DONE] Develop manifest of stable gaia-ui-tests on b2g desktop builds
- [mdas] [ON TRACK] Work on improving Marionette docs
- [mdas] [ON TRACK] Fix tap/touch related bugs that are blocking webQA
- (carry over) [jgriffin] [DONE] Get mozilla-central b2g desktop builds working with Marionette
- [jgriffin] [DONE] Add manifest support to gaia-unit-tests
- [jgriffin] [DONE] write mozharness script for gaia-ui-tests on b2g desktop builds
- [DROPPED] Add debugger support to B2G testrunners
- (carry over) [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
- [wlach] [MISSED] Get rid of debugger server dependency bug 797529
Milestone 4 - 2 weeks (June 3 - June 14)
- (carry over) [wlach] [DONE] Get rid of debugger server dependency bug 797529
- [wlach] [ON TRACK] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
- (ongoing) [mdas] [MISSED] Resolve issues with tap
- (ongoing) [mdas] [ON TRACK] Update marionette docs
- (carry over) [ahal] [DONE] Get mozilla-central B2G unit tests to use in-tree mozbase instead of puppetagain
- [jgriffin] [MISSED] Disable gaia-unit-tests that are still failing on cedar
- [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
- [jgriffin] [MISSED] Update mozharness script for gaia-ui-tests
- [jgriffin + releng] [MISSED] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
- [jgriffin] [MISSED] Identify current B2G top oranges
- [rwood] [DONE] Tryrel-eng m-c unagi-eng on gaia-ui-tests
- [rwood] [DONE] Develop some heuristics to help determine when gaia-ui-test Jenkins failures are "real" or reproducible enough to warrant sending out an e-mail notification to engineering managers
Milestone 5 - 2 weeks (June 24 - July 5)
- [wlach] [DROPPED] Add JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
- (ongoing) [mdas] [ON TRACK] Resolve blocking issues with tap
- [mdas] [DEFER] Resolve blocking scroll issues - blocked on is_displayed fix
- (ongoing) [mdas] [DONE] Add some mechanisms to make debugging touch related problems easier
- [jgriffin] [ON TRACK] Disable gaia-unit-tests that are still failing on cedar
- [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
- [jgriffin] [DONE] Update mozharness script for gaia-ui-tests
- [jgriffin + releng] [DONE] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
- [jgriffin] [DEFER] Identify current B2G top oranges
- [ahal] [DONE] Identify and fix issues with full-stack emulator builds
- [rwood] [DONE] Fix intermittent gaia-ui endurance test failures
- [rwood] [DONE] Expand gaia-ui endurance test coverage
- [rwood] [DONE] Write script to email/flag b2g memory spikes via gaia-ui endurance tests
Goal 2013 Q3
Milestone 6 - 2 weeks (July 8 - July 19)
- (carry over) [jgriffin + releng] [DONE] Get gaia-unit-tests scheduled on all branches
- [jgriffin] [MISSED] Implement script to compare two sources.xml
- [jgriffin] [DONE] Identify source of memory leak in LG automated tests bug 897684
- [mdas, davehunt, webqa] [DONE] Get gaia-ui-tests on cedar running stable green
- [mdas] [MISSED] Implement modal dialog support in Marionette
- (carry over) [mdas] [MISSED] Fix remaining blocking touch issues
- [wlach] [DONE] Implement get_active_frame command - bug 855327
- [rwood] [MISSED] Get games benchmarking tests running on Firefox OS
- [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
- [rwood] [ON TRACK] Maintain endurance tests as needed; expand checks for b2g memory changes over time
- [rwood] [DONE] Get endurance tests running on Leo on Master
- [rwood] [DEFER] Look into data requirements for gaia-ui heuristics
Milestone 7 - 2 weeks (July 29 - August 9)
- ongoing - patches in hand - [ahal] Add mach to B2G so we can build B2G mach targets
- scheduled on cedar - [mihnea] Get JSreftests working for B2G
- [jgriffin] [DONE] Implement script to compare two sources.xml
- [DONE] final patch up - (carry over) [mdas] Implement modal dialog support in Marionette
- current focus -> (carry over) [mdas] Fix remaining blocking touch issues
- (carry over) [rwood] Get games benchmarking tests running on Firefox OS
- (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
- (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
- currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
- [rwood] Create app-launch and page-browser tsets for AWSY
- (carry over) [rwood] Look into data requirements for gaia-ui heuristics
Milestone 8 - 2 weeks (August 19 - August 30)
- (carry over) [ahal] Add mach to B2G so we can build B2G mach targets
- [DONE] scheduled on cedar - [mihnea] Get JSreftests working for B2G
- P1! [jgriffin + all] Update all testing docs to make it easy for developers to get started with tests
- [mdas] Investigate modal dialog test failure
- (carry over) [rwood] Get games benchmarking tests running on Firefox OS
- (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
- (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
- currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
- [rwood] Create app-launch and page-browser tsets for AWSY
Sept 19 - Oct 2
- (jgriffin PTO)
- (carry over) [DONE] [mdas] land the modal dialog fix
- [mdas] [MISSED] fix the import script leak (landed, but backed out)
- [mdas] [DEFER] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
- [ahal] [DONE] write mozharness script for running mochitests on B2G desktop builds
- [wlach] [DONE] finish implementation of npm mirror for rel-eng (re-writing in node.js and documenting)
- [rwood] [ON TRACK] Endurance test fixes/maintenance for inari master/1.3
- [rwood] [ON TRACK] Further work on AWSY emulator test driver
- [rwood] [ON TRACK] Continued support for games benchmarking on Firefox OS
Goal 2013 Q4
Oct 10 - Oct 24
- [jgriffin] [ON TRACK] investigate problem with B2G unit test timeouts not being caught by mozprocess
- [jgriffin] [DONE] add B2G support to existing Marionette mach command
- [mdas] [DONE] investigate issues with the modal dialog fix
- (carry over) [mdas] [DONE] fix the import script leak (landed, but backed out)
- [mdas] [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
- [ahal] [ON TRACK] get mochitests on b2g desktop builds rolled out everywhere
- [ahal] [ON TRACK] get B2G reftests running OOP
- [wlach] document Eideticker in such a way that it's easy for partners and people new to the project to set it up from scratch
- [wlach] [DONE] instrument better app startup times for B2G bug 888103
- [wlach] [ON TRACK] do some fps comparisons with 120fps camera
- (ongoing) [wlach] attend FxOS_Perf standups and continue to provide feedback wrt to power management testing
- [rwood] Endurance test fixes/maintenance for inari master/1.3
- [rwood] Get AWSY emulator test driver running locally
- [rwood] [ON TRACK] Set up Jenkins cluster on new VMs for AWSY tests
- [rwood] [DONE] Continued support for games benchmarking on Firefox OS
Oct 24 - Nov 7
[jgriffin]
- [DONE] get gaia-integration tests in TBPL (depends on rel-eng)
- not running green, but Gareth Aye owns this
- [MISSED] get green set of debug emulator mochitests running everywhere
- in progress on cedar; see https://tbpl.mozilla.org/?tree=Cedar&jobname=emulator&showall=1; devs working on greening up problems also on https://tbpl.mozilla.org/?tree=Pine
[mdas]
- [ON TRACK] refactor runtests.py so we can use a mixin pattern bug 925398
- (carry over) (time consuming) [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045
- [DONE] Stop making marionette.log when unnecessary bug 918408
[ahal]
- [DONE] get mochitests on b2g desktop builds rolled out everywhere
- [DONE] address B2G crash reporting problems
[wlach]
- Get the B2G Eideticker dashboard running
- running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
- bug filed for this, wlach will look up and post here after meeting
- also need to file some follow-up bugs on test reproducibility (e.g. for contact loading we don't last long enough to display all pictures)
- the contacts issue at least is ours to fix: pictures loading slowly and asynchrously is "by design" according to gaia devs. will file bug with davehunt to not load avatar pictures for eideticker
- (carry over) do some fps comparisons with 120fps camera
- should have some results shortly
[rwood]
- (carry over) Get a full set of AWSY emulator test driver running locally
- done except now blocked by Bug 931198 about_memory fails with > 30 memory files
- Assist the Taipei team with MTBF tests
- Leaving tomorrow eve
- Deliver a set of memory test reports to the AWSY maintainer
- Get AWSY prodution VMs setup (1 master, others as nodes)
- Get gaia-ui endurance tests running on hamachi b2g26/1.2
- Up and running but now need to cherry-pick endurance test changes that arent in b2g26
- Fix/investigate gaia-ui endurance test failures
[gkw]
- Spoke about marifuzz, a project to use marionette for fuzzing, based on orangfuzz but using marionette instead.
- Emailed discussion outcome (drawings from TW office board) to jgriffin, mdas and wlach
Nov 7 - Nov 21
[jgriffin]
- [MISSED] Finish adding OOP support to WebAPI tests bug 926280
- This is ready to land, but I'm going to wait to land it until Malini's mixin refactor lands (bug 925398), since this will bitrot that patch.
- (carry over) [MISSED] get green set of debug emulator mochitests running everywhere
- These are mostly green on cedar, but intermittents still appear. I've filed a bug to schedule these on all trunk branches, but we'll likely have to hide these initially until the intermittents fall to an acceptable level. bug 940161
[ahal]
- get emulator reftests running oop
- should also help getting b2g desktop reftests going oop
- done and running on cedar (if the implementation is ok)
- still some problems (crashes, timeouts, and an error that happens between tests but doens't seem to affect them)
- [MISSED] get reftests running on b2g desktop builds
- investigate emulator startup crashes?
[mdas]
- (carry over) [MISSED] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045 (I was actually working on bug 925398 in order to address the other bugs, milestone updated for next sprint)
[wlach]
- [MISSED] Get the B2G Eideticker dashboard running
- running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
- Still blocked on IT AFAIK
- (carry over) [DONE] do some fps comparisons with 120fps camera
- DONE
- improve test stability (launch time tests)
- Dave is on this
- Stabilizing contacts app: https://bugzilla.mozilla.org/show_bug.cgi?id=937730
- Fixing b2g settings scrolling -DONE https://bugzilla.mozilla.org/show_bug.cgi?id=939191
- Get Eideticker working with Hamachi - DONE (mostly)
- https://bugzilla.mozilla.org/show_bug.cgi?id=936121 (eideticker) - DONE pending review
- https://bugzilla.mozilla.org/show_bug.cgi?id=936120 (orangutan, dependency) - DONE pending review
- NEW: Improve fps metrics (917404 -- old bug, but issue is more pressing now)
- NEW: Figure out why captures with 150 fps camera are so oversaturated -- they aren't with pointgrey's utility... (no bug filed yet but will do tomorrow)
[ato]
- Updating Selenium Proxy (programming sitting on top of Marionette that provides Selenium-compatible HTTP server) with missing endpoints to run Selenium tests against Marionette
- Allows for better overview of our existing compatibility level
- Will file bugs on incompatibilities
[rwood]
- Nov 4-8th was MTBF week in Taipei with Taiwan QA Team; good progress
- (carry over) Get a full set of AWSY emulator test driver running locally
- DONE except now blocked by Bug 931198 about_memory fails with > 30 memory files
- [missed] Deliver a set of memory test reports to the AWSY maintainer
- [missed - in progress] Get AWSY prodution VMs setup (1 master, others as nodes)
- blocked by Bug 931198 however get as much setup as possible outside of that
- [DONE] Get gaia-ui endurance tests running on hamachi b2g26/1.2
- Up and running but now need to cherry-pick endurance test changes that arent in b2g26
- Fix/investigate gaia-ui endurance test failures
Nov 22 - Dec 5
[jgriffin]
- (carry over) Finish adding OOP support to WebAPI tests bug 926280
- (carry over) get green set of debug emulator mochitests running everywhere bug 916359
[ahal]
- (carry over) get emulator reftests running oop
- should also help getting b2g desktop reftests going oop
- done and running on cedar (if the implementation is ok)
- still some problems (crashes, timeouts, and an error that happens between tests but doens't seem to affect them)
- get reftests running on b2g desktop builds
- investigate emulator startup crashes?
[mdas]
- Fix gaia testing blocking issue with getting tabs bug 941749
- (carry over) (time consuming, landing will take more time) refactor marionette/gaiatest to use mixins: bug 925398
- (carry over) write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045
[wlach]
- Get the B2G Eideticker dashboard running
- running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
- NEW: Improve fps metrics (917404 -- old bug, but issue is more pressing now)
- NEW: Figure out why captures with 150 fps camera are so oversaturated -- they aren't with pointgrey's utility... (no bug filed yet but will do tomorrow)
[ato]
- (ongoing) Updating Selenium Proxy (programming sitting on top of Marionette that provides Selenium-compatible HTTP server) with missing endpoints to run Selenium tests against Marionette
- Add screen orientation implementation, using emulator build - bug 804515
[rwood]
- (carry over) Deliver a set of memory test reports to the AWSY maintainer
- (carry over) Get AWSY prodution VMs setup (1 master, others as nodes)
- Get another hamachi and replace the Inari endurance test (on master) device with hamachi
- Fix endurance test failures (update tests according to latest gaia changes)
Future milestones
- Fix frequent Mn orange - bug 930430 (test_submit.py failures)
- add tests for modal dialog bug 906175
- Investigate B2G "timed out after 1000s" errors: bug 906716 - these seem to be different flavors of emulator startup problems. See also a related bug, bug 905324, " Intermittent B2G mochitest "AssertionError: Timed out waiting for port!", which is probably another manifestation of the same problem.
- Stand up new builds and tests for B2G - bug 916356
- Put the Eideticker backend on Datazilla
- Look into data requirements for gaia-ui heuristics
- Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds
- Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming, depends on 4.2 emu builds in TBPL)
- Implement device test failure notification heuristics (depends on results being uploaded to TBPL)
- Fix more top oranges
- Write tests for new bluetooth support in emulator?
- improve server side logging bug 928491
- shear away some of Marionette's MDN documentation and make it more concise and relevant. The device specific instructions may need updating as well.
- [DROPPED] Finish integrating JS Marionette client into Python testrunner
- [DROPPED] Gaia bisection tool - implemented by naoki