	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

Marionette (mdas)

P0 - (highlight)Get passing WebDriver conformance tests running in TBPL
P0 - (highlight)Fix touch related bugs
P0 - (nonpublic) Continue to support WebQA and Gaia teams with bug fixes and new features
P1 - (highlight)Fix WebDriver compatibility problems; implement missing commands
P1 - (nonpublic) Split monolithic Python repo into separate packages
P2 - (nonpublic) Improve handling of socket errors

B2G (jgriffin)

P0 [wlach supporting] (highlight) (perf team) power consumption testing
P0 [rwood] (highlight) memory tests
P1 [wlach supporting] (cross-team highlight)(with gaia team) github autolanding
P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable
P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system
P2 [jgriffin, ahal] (nonpublic) continued burn down of intermittent oranges
P2 [ahal] continued work to make B2G tests more developer friendly
(defer to Q1-2014) perf tryserver? - DANGER no clear requirements
(defer to Q1-2014)(with gaia and rel-eng) running tests via github pull requests

new builds and tests

P0 [ahal] (highlight) mochitests, reftests, crashtests, jsreftests, gaia-ui-tests on b2g desktop builds: debug/opt linux32/64, debug/opt macosx64
P0 [ahal] reftests/crashtests running out of process
P0 [jgriffin] gaia integration tests on tbpl
P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds: linux64
P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
P2 [jgriffin] (nonpublic) run "sanity test" of emulator on osx

Performance (jmaher)

Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)

(highlight) deploy performance project dashboard to outline all projects, health of data, impact of the data, and make it easy to understand the data via docs and views on the data.
have tools, data, tests fully documented and easy to find (all)
hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority (jmaher)
proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher)

Datazilla (jeads)

The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.

(tied to 1st perf goal) Implement alert system that monitors data ingestion rates for all branches in all projects.
(tied to 1st perf goal) NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
Fine tune automated regression detection system as needed
Implement the performance dashboard described in the performance section
- Set up eideticker projects and backfill eideticker data (https://bugzilla.mozilla.org/show_bug.cgi?id=909796)
- Integrate documentation in the homepage UI and in the b2g UI
- Remove all legacy code from deprecated UI's \o/

NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of.

NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, bug 913726, not sure how these should be prioritized in Q4.

Treeherder (mdoglio/jeads)

Implement TBPL *starring* features
Implement Orange Factor compatability
Release production instance on AWS
(mid-q) Get sheriffs to drive side-by-side usability testing with TBPL
- Make sure we're on target for full TBPL replacement in Q1 2014.

Bugzilla (glob/mcote)

Continue Bugzilla as a Platform (BaaP):
- [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
- [dkl] (highlight) bzAPI compatibility layer
- [dkl] plan for versioning APIs, first draft of interface for API v2
[glob] (highlight) memcached minimum viable product (also helps BaaP above)
[mcote] ReviewBoard integration plan based on feedback
[mcote] somehow get over to git, or at least have a really serious plan for it
[kyle] deploy new bzETL

Android (gbrown)

(highlight) Fix printf debugging so it doesn't go to logcat by default
(highlight) Update x86 emulator system images to SDK 18; fix agent for android 4.3
panda stability
- [dminor] biggest problem is retries
bandwidth metrics???
Eideticker responsiveness (possible carry over, depending on Q3).
C++ remote tests
[dminor] Android 2.3.x on pandas?
[dminor] Update panda images (maybe using AOSP, gbrown had been looking at this)?
[dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?

WebRTC (ted)

Wire up existing AWS system, see if we can give to releng to run from here on out
Investigate hardware set up (joint with desktop QA?)

platforms (ctalbert)

mac os x 10.9 - still waitiing to see what happens here.
official 8.1 windows release
releng working on EC2 windows...not sure of status

Bugmaster (lizzard)

Build out community involvement at the summit
Get tools we've built deployed some place permanent (bug board, tutorial, bugzilla user profile report)
Continue triaging and imrpove the groundswell.
Research into new Bugzilla home page for logged-in/logged-out users (separate pages)

Desktop QA Automation (whimboo)

System expansion to support locale tests
(highlight) Conversion to mozmill 2.0
(highlight) metro tests in production
stretch (work on the community ideas with jdm and lizzard)
(highlight) SV to lead the automation for QA with our oversight
web app to generate the release/beta test manfiest

sheriffs (edmorley)

DEPEND: depdening on treeherder
testing treeherder
complete documentation goals/operationalize sheriffing

Research (ctalbert)

implement something to know how often our dashboards are used
- web page tracking/analytics software
- irc keyword counters
- bugzilla activity
Orange Quarantine Experiment
turn off passing tests
- directory based
- [dminor] are we still interested in looking at code coverage as well?
forecastingtool
define a good orangefactor number? 3.14
manifests - is this goal worthy, or a side project until it finishes
[dminor] investigate using Heka on a small project?
[dminor] initial support for servo automation?

mozbase (ahal)

add structured logging to more harnesses
- need to merge the logs from python and js
- need a long term vision here

Suggestions/Thoughts

[henrik] Create a tool to help watch and observe community member's involvement / engagement across projects; to highlight community members in danger of dropping out and/or community members who are ready for bigger challenge
- [ctalbert] - Could this not leverage/build atop the work that Lizzard is doing with her bugboard leaderboard to help track community involvement from the bugmaster side?
[henrik] Setup a tryserver similar to mozmill-ci to get assistance in running tests with local patches applied.
- [ctalbert] How many people do you think would actually be using such a system? Would it be better to merely create a staging system that can be rolled back rather than having something that is integrated into source repos?

Auto-tools/Goals/2013Q4

Contents