Auto-tools/Goals/2014Q2

From MozillaWiki
Jump to: navigation, search

Top Level Goals

A list of high priority goals we're committed to achieving in Q2.

B2G

FxOS Certification Suite [owner: jgriffin]

Support Mozilla's goal of expanding the uptake of FirefoxOS by OEM's by delivering the next version of the FxOS Certification Suite, which will contain the following:

  • [DONE] A guided (semi-auto) WebAPI test suite (to verify WebAPI functionality)
  • [DONE] The web-platform-tests (to cover HTML and DOM verification)
  • [DONE] Static analysis tools (to verify changes to WebAPI IDL's and omni.ja)
  • [DONE] A suite of tests to verify implementation of the permissions model
  • [DONE] Related documentation

Tracking bug: bug 970615, etherpad: https://etherpad.mozilla.org/fxos-cert-suite

MarionetteJS tests [owner: ahal]

Increase the impact and usefulness of the Gaia developer's integration tests, by working with them to enable these tests to run on devices and emulators.

  • [MISSED] Build a common harness that can run both Python and MarionetteJS tests.
  • [MISSED] Enable the harness to run on emulators and in on-device automation.
    • On track for delivery in early Q3

Performance

Talos Regression Dashboard [owner: jmaher]

Improve our ability to make Talos test results more actionable by deploying a Talos Regression Dashboard. This dashboard will consume Talos alerts currently sent to dev.tree-management, identify regressions in the data, consolidate those regressions, and allow them to be "starred". In this quarter, we will:

  • [DONE] Deploy the dashboard to allow talos results to be "sheriffed"
  • [MISSED] Configure it to work additionally with regression alerts from datazilla (e.g., B2G alerts)

Eideticker [owner: wlach]

Get more value out of Eideticker, one of our more successful out-of-tree performance tools, by increasing its ability to scale:

  • [DONE] Make it easier to add new device types in B2G
  • [DONE] Add test runs against the Tarako device
  • [DONE] Add test runs against the Flame devices
  • [DROPPED] Investigate ways to make it more cost efficient to scale for Android tests, by looking into external DeckLink boxes for HDMI capture
  • [DONE] Run tests more frequently

Datazilla [owner: jmaher]

Improve our ability to monitor our products for performance regressions.

  • [DONE] Minimal alerts for Talos regressions.
  • [DONE] Minimal alerts for mobile Eideticker regressions.
    • More complicated than originally thought, and Kyle's time being shared with charts.

Mobile

Android [owner: mcote]

Continue efforts to move our Android 2.3 testing to VM's and remove our reliance on tegras.

  • [DONE] Run reftests on Android 2.3 emulators on trunk.
  • [DONE] Designate Eideticker and/or Autophone as the official perf testing system for Android 2.3.

Tools

Bugzilla [owner: mcote]

Continue work on performance improvements; move off old systems to allow them to be decommissioned.

  • [DONE] Migrate BMO and upstream Bugzilla CI from Tinderbox to travis-ci.
  • [DONE] Continue and deepen analysis of BMO perf data.
  • [DONE] Improve the slowest MySQL queries or include them in memcache.
  • [DONE] Investigate and, if appropriate, implement improvements to page-load times, including CSS concatenation/minification and delayed loading of bug link tooltips.
    • Delayed-loading of bug-link tooltips was too complicated for the gains; going with a different solution (next quarter) that should give comparable benefits (bug 1028795)

See BMO#Current_Projects for status.

Treeherder [owner: jeads]

Increase our ability to leverage existing and upcoming tools by deploying Treeherder, a TBPL successor. Treeherder will eventually allow us to have a single source of truth for test status, including on-device and extra-buildbot tests. In Q2:

  • [DONE] Deploy a production version of Treeherder in coordination with Laura's team
    • Carryover a socket.io using a different virtual host - that will be done in Q3
  • [MISSED] Define and implement a sheriff transition plan to move from TBPL to Treeherder
    • Treeeherder writes failure classifications to tbpl, and treeherder reports data to orange factor were what were needed to do this, but the transition plan was not implemented. Sheriffing transition will happen in early Q3.
  • [DONE] Have Treeherder report data to OrangeFactor
  • [DONE] Start planning on work needed to get performance data into Treeherder; this work will likely be fully delivered in Q3
    • Will go into Q3, but ahead of where we thought we'd be

Community

Good First Bugs [owner: David Burns :automatedtester]

Improve our ability to connect with contributors by making better use of the existing good first bugs list:

  • [DONE] Refresh this list on a monthly basis
  • [DONE] Review good first bugs and make sure they have adequate information for contributors to start with
  • [DONE] Keep the numbers of open good first bugs steady or increasing

Supporting and Ongoing Tasks

A list of ongoing and supporting tasks that the A*Team is actively engaged in. The delivery date for these will vary depending on the resource requirements of our Top Level goals, as well as the amount of support requests that come in during the quarter, and their respective priorities.

B2G

Tests in Continuous Integration

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
997244 -- Refactor emulator.py out of marionette and into mozrunner RESOLVED FIXED Andrew Halberstadt [:ahal] ateam-b2g-task
999719 -- [Tracking bug] Stand up OOP tests on b2g desktop builds side-by-side non-OOP tests RESOLVED INCOMPLETE Jonathan Griffin (:jgriffin) ateam-b2g-task
999086 -- Add gaia linter tests to TBPL RESOLVED FIXED Jonathan Griffin (:jgriffin) ateam-b2g-task
990635 P1 [marionette-js-runner] Create host for connecting to python harness over a socket RESOLVED FIXED Andrew Halberstadt [:ahal] ateam-b2g-task
1040658 P1 Add Gaia build unit tests to TBPL RESOLVED FIXED Jonathan Griffin (:jgriffin) ateam-b2g-task
989126 P1 Add Gaia build tests to TBPL RESOLVED FIXED Jonathan Griffin (:jgriffin) ateam-b2g-task
943326 P2 run-if = os == 'b2g' does not work in b2g-desktop NEW ateam-b2g-task
987406 P2 Add ssltunnel to xre.zip bundle used for b2g tests RESOLVED FIXED Jonathan Griffin (:jgriffin) ateam-b2g-task
994920 P2 Run media mochitests on B2G emulators on faster VMs NEW ateam-b2g-big
916368 P2 Get gaia-ui-tests running on emulators on TBPL RESOLVED WONTFIX Dave Hunt (:davehunt) ateam-b2g-task
969146 P2 Crashes from child processes don't get processed in b2g mochitests RESOLVED FIXED Andrew Halberstadt [:ahal] ateam-b2g-task
1027597 P2 B2G reftest timeouts are outputting only "TimeoutException: TimeoutException: Connection timed out" RESOLVED WONTFIX ateam-b2g-task
987746 P3 Create a mach target for running gaia-ui tests RESOLVED WONTFIX ateam-b2g-task

13 Total; 2 Open (15.38%); 11 Resolved (84.62%); 0 Verified (0%);


On-device tests

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
1264471 P3 Norton Support Phone Number 1-800-445-2790 USA, Norton Antivirus Support Phone Number, Norton Technical Support Phone number RESOLVED INVALID ateam-b2g-device-big
979077 P2 Get MarionetteJS device tests running and reporting to treeherder NEW ateam-b2g-device-big
992405 P2 B2G Spot check (and future automation) system needed to analyze data usage for app interaction RESOLVED WONTFIX ateam-b2g-device-big
990842 P1 Start Running Graphics App On Datazilla to test Graphics Performance RESOLVED WONTFIX Mason Chang [Inactive] [:mchang] ateam-b2g-device-task, perf
991507 P1 [mozversion] BadZipfile raised when run against tarako device RESOLVED FIXED Dave Hunt (:davehunt) ateam-b2g-device-task
994764 P1 [Tracking] Gaia integration tests using python toolchain over a socket RESOLVED WONTFIX ateam-b2g-device-big
995404 P1 Automated testing of Marketplace Catalog RESOLVED FIXED Thomas Elin [:relaas] ateam-b2g-device-task
996102 P1 [tarako] Add Tarako Marketplace to Datazilla's fxOS Dashboard RESOLVED FIXED Dave Hunt (:davehunt) ateam-b2g-device-task, perf
1011567 P1 [meta] Run Performance Tests on a per checkin-basis RESOLVED DUPLICATE ateam-b2g-device-big, perf
1026710 -- Add comprehensive logging to DeviceManagerADB RESOLVED FIXED Armen [:armenzg] ateam-b2g-device-task

10 Total; 1 Open (10%); 9 Resolved (90%); 0 Verified (0%);


Performance

B2G

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
899197 P2 Add script to report endurance test results to AWSY RESOLVED WONTFIX ateam-b2g-perf-task, perf
994552 P1 Set up Tarako device in Eideticker CI RESOLVED FIXED Dave Hunt (:davehunt) ateam-b2g-perf-task, perf
1020215 P1 Set up Flame device in Eideticker CI RESOLVED FIXED Dave Hunt (:davehunt) ateam-b2g-perf-task, perf
995983 -- Add support for Tarako device RESOLVED FIXED Dave Hunt (:davehunt) ateam-b2g-perf-task

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);


Desktop

  • Implement desktop power tests, reporting to datazilla
    • deploy rvitillio's power benchmark
    • define set of websites to track
    • on windows 7, automate browsing to the list of sites and recording power readings
    • compare between aurora and central at least once a week, ideally nightly
    • priority 2 is osx and linux
  • mainthread I/O tracking
    • work with aklotz to get this new feature tracked on an existing tbpl based job
  • possible new benchmarks to be run nightly (not per commit) [p2]
    • work with cpeterson to get additional benchmarks (think Tom's Hardware) which do not overlap with the games benchmarks

Datazilla

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
850476 P2 B2G FPS test reports data in fps not milliseconds RESOLVED WONTFIX ateam-datazilla-task, perf
962378 P1 Datazilla Alerts (meta) RESOLVED WONTFIX Kyle Lahnakoski [:ekyle] ateam-datazilla-big
974860 -- Expand b2g graph to current date/time to visually highlight lack of incoming data RESOLVED WONTFIX ateam-datazilla-task
993202 -- Add ability to order data points by gaia or gecko push dates RESOLVED WONTFIX ateam-datazilla-task

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);


Talos

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
909524 P1 Monitor end-to-end audio quality in automation RESOLVED FIXED Joel Maher ( :jmaher) (UTC-5) ateam-talos-big
936630 -- Add a Talos regression test for session restore times RESOLVED FIXED David Teller [:Yoric] (please use "needinfo") ateam-talos-task, dev-doc-complete
938644 -- Add web latency benchmark to talos RESOLVED WONTFIX ateam-talos-big
967635 -- Add profiling hooks for all the major Talos Test RESOLVED FIXED Viktor Stanchev [:vikstrous] ateam-talos-task
1010473 -- Fix stand-alone TART and tscrollx (Profiler.js broken) RESOLVED FIXED Avi Halachmi (:avih) ateam-talos-task
1016089 -- Allow TART to control the profiler outside of talos RESOLVED WONTFIX Avi Halachmi (:avih) ateam-talos-task
1016675 -- Talos TART - correct profiler pause/resume timing RESOLVED FIXED Avi Halachmi (:avih) ateam-talos-task
1020365 -- Add AnimationBenchmarks to talos RESOLVED WONTFIX ateam-talos-task
1020663 -- Add WebGL to talos RESOLVED FIXED Avi Halachmi (:avih) ateam-talos-task
1020677 -- Talos: We need higher resolution regression alerts RESOLVED FIXED ateam-talos-task
1021842 -- Talos: improve graphserver's formula for "final test result" RESOLVED FIXED Joel Maher ( :jmaher) (UTC-5) ateam-talos-task
1021910 -- Make talos WebGL tests work on b2g/FxOS and Android/Fennec RESOLVED WONTFIX ateam-talos-task
1035794 -- Make talos pageloader addon logs more readable/informative for humans RESOLVED FIXED ateam-talos-task

13 Total; 0 Open (0%); 13 Resolved (100%); 0 Verified (0%);


Mobile

Android

  • Green up more x86 tests and run them on trunk - bug 936226.
  • Get some Android debug tests running on trunk - bug 940068.

Tools

Bugzilla

  • Improve Review Board Bugzilla extension to deepen integration.
  • Extend history APIs to return all changes to bugs, instead of just bugs_activity entries, and allow the REST version to query a set of bugs instead of just one.

Marionette

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
1046512 P3 Marionette breaks in e10s enabled Nightly RESOLVED FIXED Chris Manchester (:chmanchester) ateam-marionette-goal, ateam-marionette-server
1101046 P3 Code Bug RESOLVED INVALID ateam-marionette-task
962645 P2 Action chains only handle one press->release chain RESOLVED FIXED Ting-Yu Lin [:TYLin] (UTC-6) ateam-marionette-goal, ateam-marionette-spec, ateam-marionette-userinput
946130 P1 Flick and other actions not working on System app RESOLVED FIXED ateam-marionette-goal, ateam-marionette-userinput, regression
958036 P1 Unable to automate scrolling in apps with APZ enabled RESOLVED FIXED Malini Das [:mdas] - Away, not checking bugmail ateam-marionette-goal, ateam-marionette-userinput
994176 P1 Create branched versions of marionette-client and consumer packages RESOLVED FIXED Dave Hunt (:davehunt) ateam-marionette-runner, ateam-marionette-task
1107706 P1 Refactor server by introducing a more expressive internal architecture RESOLVED FIXED Andreas Tolfsen ‹:ato› ateam-marionette-big, ateam-marionette-server
1118201 P1 Update gaia imports to check if it has marionette_driver package, if not use marionette still RESOLVED FIXED David Burns :automatedtester ateam-marionette-client, ateam-marionette-goal
1123506 P1 Evaluate scripts in content with lasting side-effects RESOLVED FIXED Andreas Tolfsen ‹:ato› ateam-marionette-goal, ateam-marionette-server, ateam-marionette-spec
1309336 P1 Unable to select and element from dropdown in UI in mozilla firefox 49.0.1 RESOLVED INVALID ateam-marionette-big
849183 -- [B2G] Marionette tap in Mail app is not working RESOLVED WORKSFORME David Burns :automatedtester ateam-marionette-goal, ateam-marionette-userinput
865191 -- 'args' parameter should be allowed to be null in execute* calls RESOLVED WORKSFORME David Burns :automatedtester ateam-marionette-task
867720 -- Make flick a webdriver action chain command RESOLVED WONTFIX ateam-marionette-goal, ateam-marionette-userinput
876411 -- Click works but tap does not on tabs on browser app RESOLVED DUPLICATE ateam-marionette-goal, ateam-marionette-userinput
877163 -- still need to scrollIntoView when using el.tap() in email app RESOLVED WONTFIX ateam-marionette-goal, ateam-marionette-userinput
877604 -- long_press action failing on B2G desktop client when installing an Everything.me app RESOLVED WONTFIX ateam-marionette-goal, ateam-marionette-userinput
877651 -- [B2G] Not scrolling into view in System app RESOLVED DUPLICATE ateam-marionette-goal, ateam-marionette-userinput
878027 -- el.tap() not working on: Contacts - Settings - Done button RESOLVED WONTFIX ateam-marionette-goal, ateam-marionette-userinput
902711 -- Split the monolithic marionette Python package RESOLVED FIXED Jonathan Griffin (:jgriffin) ateam-marionette-task
956739 -- Move marionette tests to structured logging RESOLVED FIXED Chris Manchester (:chmanchester) ateam-marionette-task
979220 -- webelement.tap() has no effect on button from alert screen RESOLVED WORKSFORME ateam-marionette-goal, ateam-marionette-userinput
1043325 -- Support for finding anonymous content RESOLVED DUPLICATE ateam-marionette-big, ateam-marionette-server
1046873 -- Marionette does not shutdown properly if test waits for element RESOLVED INCOMPLETE ateam-marionette-big, ateam-marionette-userinput
1068726 -- Enable testAllPrintableKeys for Marionette NEW ateam-marionette-server, ateam-marionette-task
1068728 -- testNumericShiftKeys needs to be re enabled NEW ateam-marionette-server, ateam-marionette-task
1068733 -- sendkeys() fails in adding a white-space for Keys.SPACE NEW ateam-marionette-server, ateam-marionette-task
1068735 -- testUppercaseAlphaKeys needs to be re enabled NEW ateam-marionette-server, ateam-marionette-task
1085717 -- Investigate issue with Marionette's test_set_window_size.py and availWidth/availHeight on linux RESOLVED INCOMPLETE ateam-marionette-task
1112905 -- [Meta] Create and release executable that allows webdriver compatible clients to speak to marionette NEW ateam-marionette-goal, ateam-marionette-httpd
1112910 -- Release marionette-httpd executable RESOLVED FIXED James Graham [:jgraham] ateam-marionette-goal, ateam-marionette-httpd
1112913 -- Screenshots should return only the view port RESOLVED FIXED Julien Pagès (:parkouss) ateam-marionette-goal, ateam-marionette-server
1113373 -- Marionette Screenshot needs to return viewport as the default RESOLVED DUPLICATE ateam-marionette-goal, ateam-marionette-server
1113554 -- Have capabilities return which type of user inputs it has access to RESOLVED WONTFIX David Burns :automatedtester ateam-marionette-goal, ateam-marionette-server, ateam-marionette-spec
1131860 -- Re-enable TestKeyActions.test_open_in_new_window_shortcut for e10s RESOLVED DUPLICATE ateam-marionette-task
1156070 -- Rename test_errors.py's Exception tests so they don't confuse mozharness and thus Treeherder into thinking there are six failures in every single Mn run RESOLVED FIXED David Burns :automatedtester ateam-marionette-task
1198797 -- Centralise creation of valid UUID in Marionette RESOLVED FIXED Shaif ateam-marionette-server, ateam-marionette-task
1301902 -- Unable to save files automatically while using MarionetteDriver (selenium) RESOLVED INCOMPLETE ateam-marionette-big

37 Total; 5 Open (13.51%); 32 Resolved (86.49%); 0 Verified (0%);


  • Fix up major touch-related bugs [owner: mdas]
  • WebDriver Specification incompatibility bugs [owner: David Burns :AutomatedTester]
  • Selenium Java and Python command executor interfaces [owner: ato]
    • Running Selenium test suite on cedar [owner: ato]
  • Add structured logging support [owner: ahal]
  • Support Loop testing for dmose [owner: AutomatedTester]

Misc

  • Release a new version of mozilla-build on Windows that uses the latest version of msys
  • Help to stabilize the existing Steeplechase tests and expand the suite with new tests and capabilities
  • Create and execute on a plan to connect mozharness and mach [owner: ahal]

Unit tests

Full Query
ID Priority Summary Status Resolution Assigned to Keywords
886570 P2 Mochitest should have ability to output structured messages RESOLVED FIXED Ahmed Kachkach [:akachkach] ateam-unittests-goal
994216 P1 Fix linux mochitest-other on AWS nodes RESOLVED FIXED Myk Melez [:myk] [@mykmelez] ateam-unittests-task
920569 -- Add support for webapprt-test-chrome test jobs & enable them per push on Cedar RESOLVED FIXED Jonathan Griffin (:jgriffin) ateam-unittests-task
945221 -- Add a test framework for running W3C web-platform-tests and compatible testsuites RESOLVED FIXED James Graham [:jgraham] ateam-unittests-big
984930 -- Create mochitest-dt and move all mochitests under browser/devtools into that suite RESOLVED FIXED Joel Maher ( :jmaher) (UTC-5) ateam-unittests-big
987892 -- Clear up oranges for debug mochitest-browser-chrome jobs on mozilla-aurora, mozilla-beta, mozilla-release and mozilla-esr24 RESOLVED FIXED Joel Maher ( :jmaher) (UTC-5) ateam-unittests-task
989583 -- Install all tests (even disabled) from manifests RESOLVED FIXED Armen [:armenzg] ateam-unittests-task
992911 -- add the ability to run mochitests per directory in a loop RESOLVED FIXED Joel Maher ( :jmaher) (UTC-5) ateam-unittests-task
996183 -- write a tool which reports the total number of tests we have and which are skipped/disabled for all configurations RESOLVED FIXED Andrew Halberstadt [:ahal] ateam-unittests-big

9 Total; 0 Open (0%); 9 Resolved (100%); 0 Verified (0%);


Sheriffing

  • More work on version-control-tools repo (eg bzexport, qimportbz, and other repo tools)

Backlog

A list of things we'd like to do, but are not actively working on, due to resource constraints, poor requirements, or outside dependencies.

Marionette

  • Further WebDriver spec compatibility work [owner: ato] - Backlog
    • Goal: Getting Marionette-backed FirefoxDriver deployed at major companies

B2G

  • Set up a separate Jenkins instance for B2G so we can put the Jenkins scripts in version control
  • Make B2G test jobs fail on uncaught JS exceptions [owner: ahal, bug 959493]
  • Add JSReftests to B2G [owner: ahal, bug 873179]
  • Get WebAPI tests runnning on debug emulators [owner: ahal]
  • Get tests running on emulator-kk [owner: ahal]
  • Get tests running on Firefox Mulet, as soon as it's available [owner: ahal]
  • Auto-submit crash reports from automated tests [owner: TBD, bug 962700]

Tools

Unit tests

  • WebDriver support in web-platform-tests
  • Create a tool to identify inter-test dependencies (GSOC project, Q2-Q3) [owner: jmaher]
  • Run valgrind on test slaves [bug 977240, blocked on RelEng making "real" valgrind builds with uploaded build artifacts - bug 977351]

Performance

B2G

  • Work with engineering to identify better ways to use gaia endurance and AWSY tests [owner: rwood]
    • Identify overlap between the two and turn off one if appropriate
  • Work with the FxOS_Perf team to stand up B2G power tests on the Flame [owner: TBD]

Talos

  • svg-opacity cleanup (revisit the test, possible rewrite/revision bump)
  • new scroll test (in addition to tscrollx) - work with vladan/avih