Auto-tools/Goals/2013Q4: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(70 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{{draft}}
= Top Level =
= Marionette (mdas)=
Highest priority, multi-team dependent goals
* P0 - (highlight)Get passing WebDriver conformance tests running in TBPL
* P0 - (highlight)Fix touch related bugs
* P0 - (nonpublic) Continue to support WebQA and Gaia teams with bug fixes and new features
* P1 - (highlight)Fix WebDriver compatibility problems; implement missing commands
* P1 - (nonpublic) Split monolithic Python repo into separate packages
* P2 - (nonpublic) Improve handling of socket errors


= B2G (jgriffin) =
{| class="fullwidth-table sortable"
* P0 [wlach supporting] (highlight) (perf team) power consumption testing
|-
* P0 [rwood] (highlight) memory tests
| style="font-weight: bold; background: #DDD; width: 200px" | Goal
* P1 [wlach supporting] (cross-team highlight)(with gaia team) github autolanding
| style="font-weight: bold; background: #DDD; width: 200px" | Measurement
| style="font-weight: bold; background: #DDD; width: 50px" | Status
| style="font-weight: bold; background: #DDD; width: 50px" | Owner
| style="font-weight: bold; background: #DDD; width: 25px" | Bug
| style="font-weight: bold; background: #DDD; width: 25px" | Area
|-
| Run endurance tests without Marionette issues
| Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues
| {{done| }}
| Mdas
| {{bug|908261}} has been marked a dupe, what will help here is {{bug|931044}} which we could hook up to run for 3 days
| [[Auto-tools/Goals/2013Q4#Marionette_.28mdas.29|Marionette]]
|-
| Deploy new B2G desktop tests, reftest running OOP, and gaia integration to TBPL
| All these suites running on B2G Inbound
| {{done| }}
| Jgriffin
| {{bug|916356}}
| [[Auto-tools/Goals/2013Q4#B2g_.28jgriffin.29|B2G]]
|-
| Support B2G Perf Team with automated power testing and Gaia Productivity team with github autoland
| The power testing goal was deferred due to lack of a suitable device, and the Gaia team didn't end up needing our help with the first iteration of github autoland.
| {{defer| }}
| Wlach
| {{bug|917375}}, [https://github.com/mozilla-b2g-bot Github autoland bot]
| [[Auto-tools/Goals/2013Q4#B2g_.28jgriffin.29|B2G]]
|-
| Deploy automated B2G Eideticker system to run nightly
| Successful completion on deadline
| {{done| }}
| Wlach
| {{bug|916989}}
| [[Auto-tools/Goals/2013Q4#B2g_.28jgriffin.29|B2G]]
|-
| Create and Deploy one dashboard for docs, usage statistics, and tools for all our performance tools
| This goal's priority was downgraded to allow Joel to spend time on higher-priority perf items, including desktop power testing, xperf, and webpagetest.
| {{miss| }}
| Jmaher
| {{bug|919619}}
| [[Auto-tools/Goals/2013Q4#Performance_.28jmaher.29|Performance]]
|-
| Implement and deploy a fully functional alerting system for performance regressions and data ingestion monitoring.
| Jeads time was better spent on Treeherder; we will target this goal for Q12014.
| {{miss| }}
| Jeads
| {{bug|921218}}
| [[Auto-tools/Goals/2013Q4#Datazilla_.28jeads.29|Datazilla]]
|-
| Ensure sheriffs can perform side by side usability testing in TBPL
| Complete starring, orange factor integration, plugin detail pane, (possibly) regex to tag transformation by end of quarter.  This should be done, except for starring and orange factor integration.
| {{done| }}
| Mdoglio
| [https://www.pivotaltracker.com/s/projects/749519 Pivotal Tracker Instance]
| [[Auto-tools/Goals/2013Q4#Treeherder_.28mdoglio.2Fjeads.29|Treeherder]]
|-
| Continue Bugzilla as a Platform transition
| Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter
| {{done| }} (except bzAPI compatibility layer, eta early Q1)
| Glob
| {{bug|885464}}
| [[Auto-tools/Goals/2013Q4#Bugzilla_.28glob.2Fmcote.29|Bugzilla]]
|-
| Force Android reftests to run within 15% of their original, pre skia time
| Ensure panda and x86 reftests run within 15% of their original time
| {{done| }}
| Gbrown
| {{bug|916657}}
| [[Auto-tools/Goals/2013Q4#Android_.28gbrown.29|Android]]
|-
| Ensure steeplechase webRTC automation as hands off as possible
| Deployed mochitests, ability to change machine configuration to system by end of quarter. C++ tests were not created (not our responsibility) so did not get integrated.
| {{done| }}
| Ted
| {{bug|894559}}
| [[Auto-tools/Goals/2013Q4#WebRTC_.28ted.29|WebRTC]]
|-
| Deploy bugmaster community building tools: bug board, tutorial, and bugzilla user profile report
| Tools deployed by end of quarter
| {{miss| }}
| Lizzard
| {{bug|919624}}
| [[Auto-tools/Goals/2013Q4#Bugmaster_.28lizzard.29|Bugmaster]]
|-
| Deploy Mozmill 2.0, Metro infrastructure and tests, and aid SoftVision to take over day to day operations for QA automation with our oversight
| Deployed projects, tried to get the SV team to take over the reigns, and we had marginal success. Had to bail them out a few times.
| {{done| }}
| Whimboo
| {{bug|922200}}
| [[Auto-tools/Goals/2013Q4#Desktop_QA_Automation_.28whimboo.29|QA Automation]]
|-
| Operationalize sheriffing role and complete documentation so that we can pull in community sheriffs
| Have documentation completed, identify one part time community sheriff.
| {{done| }}
| Edmorley
| {{bug|928960}}
| [[Auto-tools/Goals/2013Q4#sheriffs_.28edmorley.29|Sheriffing]]
|}
 
== Detailed Lists ==
These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The [[Auto-tools/Project_Backlog|current backlog is here]]
 
=== Marionette (mdas) ===
* {{done|}} P0 - Run set of endurance tests for 3 days without marionette issues
** space issues, memory usage, import log, and don't regress socket.timeout
 
{| class="wikitable collapsible"
! Ongoing and Supporting Projects
|-
|
* P0 - (crossteam)Get passing WebDriver conformance tests running in TBPL
* P0 - (crossteam)Fix touch related bugs
* P0 - (ongoing) Continue to support WebQA and Gaia teams with bug fixes and new features
 
|}
 
=== B2G (jgriffin) ===
* {{done|}} P0 [ahal] (highlight) mochitests, reftests, gaia-ui-tests on b2g desktop builds:  debug/opt linux32/64, debug/opt macosx64
* {{miss|}} P0 [ahal] crashtests, jsreftests on b2g desktop builds
* {{done|}} P0 [ahal] reftests/crashtests running out of process
* {{done|}} P0 [jgriffin] gaia integration tests on tbpl
* {{defer|}} P0 [wlach supporting] (crossteam) (perf team) power consumption testing
* {{miss|}} P0 [rwood] (crossteam) memory tests
* {{done|}} P0 [wlach,davehunt] Deploy automated Eideticker system to run nightly for Firefox OS
* {{miss|}} P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS
* {{drop|}} P1 [wlach supporting] (crossteam)(with gaia team) github autolanding
* {{done|}} P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system
 
{| class="wikitable collapsible"
! Ongoing and Supporting Projects
|-
|
* P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable
* P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable
* P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system
* P2 [jgriffin, ahal] (internal) continued burn down of intermittent oranges
* P2 [jgriffin, ahal] (nonpublic) continued burn down of intermittent oranges
* P2 [ahal] continued work to make B2G tests more developer friendly
* P2 [ahal] continued work to make B2G tests more developer friendly
* (defer to Q1-2014) perf tryserver? - DANGER no clear requirements
|-
* (defer to Q1-2014)(with gaia and rel-eng) running tests via github pull requests
New Builds and Tests
 
|-
== new builds and tests ==
* {{defer|}} P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
* P0 [ahal] (highlight) mochitests, reftests, crashtests, jsreftests, gaia-ui-tests on b2g desktop builds:  debug/opt linux32/64, debug/opt macosx64
* {{miss|}} P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
* P0 [ahal] reftests/crashtests running out of process
* {{miss|}} P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds:  linux64
* P0 [jgriffin] gaia integration tests on tbpl
* {{miss|}} P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
* P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
* {{defer|}} P2 [jgriffin] run "sanity test" of emulator on osx
* P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
|}
* P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds:  linux64
* P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
* P2 [jgriffin] (nonpublic) run "sanity test" of emulator on osx


= Performance (jmaher) =
=== Performance (jmaher) ===
* Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)
* {{miss|}} Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)


* {{done|}} P0 [jmaher] Complete investigation into Talos suites with desktop performance team
** tsvg, tscroll, ts, tpaint are all refreshed/updated accordingly.
* {{miss|}} P0 [jmaher, ekyle] Complete alerting analysis for datazilla data
* {{done|}} P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority
* {{done|}} P1 [jmaher, bc] Investigate desktop power testing


* (highlight) deploy performance project dashboard to outline all projects, health of data, impact of the data, and make it easy to understand the data via docs and views on the data.
{| class="wikitable collapsible"
! Ongoing and supporting projects
|-
|
* have tools, data, tests fully documented and easy to find (all)
* have tools, data, tests fully documented and easy to find (all)
* hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority (jmaher)
* proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher)
* proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher)
* Ensure JS Team is satisfied with real world benchmarking system, investigate automation
* Build performance lab
* Continue supporting games benchmarking effort
|}


= Datazilla (jeads)=
=== Datazilla (jeads)===
* The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.
* {{miss|}} The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.


*(tied to 1st perf goal) Implement alert system that monitors data ingestion rates for all branches in all projects.
* Implement alert system that monitors data ingestion rates for all branches in all projects.
* (tied to 1st perf goal) NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
** NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
* Fine tune automated regression detection system as needed
* Fine tune automated regression detection system as needed
* Implement the performance dashboard described in the performance section
* Implement the performance dashboard described in the performance section
** Set up eideticker projects and backfill eideticker data (https://bugzilla.mozilla.org/show_bug.cgi?id=909796)
 
{| class="wikitable collapsible"
! Ongoing and supporting projects
|-
|
* Set up eideticker projects and backfill eideticker data (https://bugzilla.mozilla.org/show_bug.cgi?id=909796)
** Integrate documentation in the homepage UI and in the b2g UI
** Integrate documentation in the homepage UI and in the b2g UI
** Remove all legacy code from deprecated UI's \o/
** Remove all legacy code from deprecated UI's \o/
NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of.
NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of.
NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, {{bug|913726}}, not sure how these should be prioritized in Q4.
|}
=== Treeherder (mdoglio/jeads) ===
* {{done|}} Ensure sheriffs can perform side by side usability testing in TBPL


NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, {{bug|913726}}, not sure how these should be prioritized in Q4.
* {{miss|}} Implement TBPL *starring* features
* {{miss|}} Implement  Orange Factor compatability
* {{miss|}} Release production instance on AWS
* {{done|}} Make sure we're on target for full TBPL replacement in Q1 2014.


= Treeherder (mdoglio/jeads) =
=== Bugzilla (glob/mcote)===
* Implement TBPL *starring* features
* {{done|}} Q4 top priority is to continue to focus on "Bugzilla as a Platform".  This is a multifaceted goal covering API design, documentation authoring, and community engagement, and of course development.
* Implement  Orange Factor compatability
* Release production instance on AWS
* (mid-q) Get sheriffs to drive side-by-side usability testing with TBPL
** Make sure we're on target for full TBPL replacement in Q1 2014.


= Bugzilla (glob/mcote)=
* Continue Bugzilla as a Platform (BaaP):
* Continue Bugzilla as a Platform (BaaP):
** [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
** {{done|}} [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
** [dkl] (highlight) bzAPI compatibility layer
** {{miss|}} [dkl] (crossteam) bzAPI compatibility layer
** [dkl] plan for versioning APIs, first draft of interface for API v2
** {{miss|}} [dkl] plan for versioning APIs, first draft of interface for API v2
* [glob] (highlight) memcached minimum viable product (also helps BaaP above)
*** We had some interruptions due to more API work, developer tools, and antispam efforts.
* [mcote] ReviewBoard integration plan based on feedback
* {{done|}} [glob] (crossteam) memcached minimum viable product (also helps BaaP above)
* [mcote] somehow get over to git, or at least have a really serious plan for it
* [kyle] deploy new bzETL


= Android (gbrown) =
{| class="wikitable collapsible"
* (highlight) Fix printf debugging so it doesn't go to logcat by default
! Ongoing and supporting projects
* (highlight) Update x86 emulator system images to SDK 18; fix agent for android 4.3
|-
* panda stability
|
** [dminor] biggest problem is retries
* {{defer|}} [mcote] ReviewBoard integration plan based on feedback
* bandwidth metrics???
** Discussion is still ongoing as to whether Review Board can meet our needs or not.
* Eideticker responsiveness (possible carry over, depending on Q3).
* {{done|}} [mcote] somehow get over to git, or at least have a really serious plan for it
* C++ remote tests
** Plan in place.
* [dminor] Android 2.3.x on pandas?
* {{done|}} [kyle] deploy new bzETL
* [dminor] Update panda images (maybe using AOSP, gbrown had been looking at this)?
|}
* [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?


= WebRTC (ted)=
=== Android (gbrown) ===
* Wire up existing AWS system, see if we can give to releng to run from here on out
* {{done|}} Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86.
* Investigate hardware set up (joint with desktop QA?)


= platforms (ctalbert) =
* {{miss|}} mach commands for running remote tests
* mac os x 10.9 - still waitiing to see what happens here.
** automatically get xpcshell/xre for remote tests
* official 8.1 windows release
** [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?
* releng working on EC2 windows...not sure of status
* {{done|}} Eideticker responsiveness (possible carry over, depending on Q3).


= Bugmaster (lizzard)=
{| class="wikitable collapsilbe"
* Build out community involvement at the summit
! Ongoing and supporting projects
* Get tools we've built deployed some place permanent (bug board, tutorial, bugzilla user profile report)
|-
* Continue triaging and imrpove the groundswell.
|
* Research into new Bugzilla home page for logged-in/logged-out users (separate pages)
* Continue working with releng to increase panda stability
* [bc] Use Autophone to do one-off cross browser bandwidth metrics
* {{done|}} [dminor] Deploy C++ remote tests
|}


= Desktop QA Automation (whimboo)=
=== WebRTC (ted) ===
* System expansion to support locale tests
* {{drop|}} P0: Get Steeplechase into releng staging (joint with releng)
* (highlight) Conversion to mozmill 2.0
* {{miss|}} P0: Deploy C++ tests for SteepleChase
* (highlight) metro tests in production
* {{done|}} P0: Deploy ability to change machine configuration per test
* stretch (work on the community ideas with jdm and lizzard)
* {{defer|}} P1: Investigate hardware testing set up (joint with desktop QA)
* (highlight) SV to lead the automation for QA with our oversight
* web app to generate the release/beta test manfiest


= sheriffs (edmorley)=
=== Bugmaster (lizzard) ===
* DEPEND: depdening on treeherder
* {{miss|Deploy}} bugmaster community building tools (bug board, tutorial, and bugzilla user profile report)
* testing treeherder
* {{done|Use}} summit to build out community involvement
* complete documentation goals/operationalize sheriffing
* {{done|Continue}} triaging and imrpove the groundswell.
* {{miss|Research}} into new Bugzilla home page for logged-in/logged-out users (separate pages)


= Research (ctalbert)=
=== Desktop QA Automation (whimboo) ===
* implement something to know how often our dashboards are used
* {{done|(crossteam) Get}} SoftVision to lead the automation efforts for QA with our oversight
** web page tracking/analytics software
* {{done|(crossteam) Switch}} Mozmill CI from Mozmill 1.5.x to Mozmill 2.0
** irc keyword counters
* {{miss|(crossteam) Get}} Firefox Metro tests executed in production
** bugzilla activity
* Orange Quarantine Experiment
* turn off passing tests
** directory based
** [dminor] are we still interested in looking at code coverage as well?
* forecastingtool
* define a good orangefactor number?  3.14
* manifests - is this goal worthy, or a side project until it finishes
* [dminor] investigate using Heka on a small project?
* [dminor] initial support for servo automation?


= mozbase (ahal)=
{| class="wikitable collapsible"
* add structured logging to more harnesses
! Ongoing and supporting projects
|-
|
* {{miss|}} Extend Mozmill CI to cover most of our localized beta and release candidate builds
* {{done|}} Create web application to generate the beta and release Jenkins job configuations
* (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches
* (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros
|}
 
=== sheriffs (edmorley) ===
Meta bug: {{bug|928960}}
* {{defer|}} [https://bugzilla.mozilla.org/show_bug.cgi?id=928964 Test treeherder throughout the quarter, file bugs]
* {{done|}} [https://bugzilla.mozilla.org/show_bug.cgi?id=928962 Complete documentation goals/operationalize sheriffing]
* {{done|}} [https://bugzilla.mozilla.org/show_bug.cgi?id=928965 Have 1 new community sheriff]
 
=== mozbase (ahal) ===
* {{miss|}} add structured logging to the mochitest harness
** need to merge the logs from python and js
** need to merge the logs from python and js
** need a long term vision here
* {{done|}} determine a long term vision
 
= Suggestions/Thoughts =
* [henrik] Create a tool to help watch and observe community member's involvement / engagement across projects; to highlight community members in danger of dropping out and/or community members who are ready for bigger challenge
** [ctalbert] - Could this not leverage/build atop the work that Lizzard is doing with her bugboard leaderboard to help track community involvement from the bugmaster side?
* [henrik] Setup a tryserver similar to mozmill-ci to get assistance in running tests with local patches applied.
** [ctalbert] How many people do you think would actually be using such a system? Would it be better to merely create a staging system that can be rolled back rather than having something that is integrated into source repos?

Latest revision as of 14:15, 8 January 2014

Top Level

Highest priority, multi-team dependent goals

Goal Measurement Status Owner Bug Area
Run endurance tests without Marionette issues Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues [DONE] Mdas bug 908261 has been marked a dupe, what will help here is bug 931044 which we could hook up to run for 3 days Marionette
Deploy new B2G desktop tests, reftest running OOP, and gaia integration to TBPL All these suites running on B2G Inbound [DONE] Jgriffin bug 916356 B2G
Support B2G Perf Team with automated power testing and Gaia Productivity team with github autoland The power testing goal was deferred due to lack of a suitable device, and the Gaia team didn't end up needing our help with the first iteration of github autoland. [DEFER] Wlach bug 917375, Github autoland bot B2G
Deploy automated B2G Eideticker system to run nightly Successful completion on deadline [DONE] Wlach bug 916989 B2G
Create and Deploy one dashboard for docs, usage statistics, and tools for all our performance tools This goal's priority was downgraded to allow Joel to spend time on higher-priority perf items, including desktop power testing, xperf, and webpagetest. [MISSED] Jmaher bug 919619 Performance
Implement and deploy a fully functional alerting system for performance regressions and data ingestion monitoring. Jeads time was better spent on Treeherder; we will target this goal for Q12014. [MISSED] Jeads bug 921218 Datazilla
Ensure sheriffs can perform side by side usability testing in TBPL Complete starring, orange factor integration, plugin detail pane, (possibly) regex to tag transformation by end of quarter. This should be done, except for starring and orange factor integration. [DONE] Mdoglio Pivotal Tracker Instance Treeherder
Continue Bugzilla as a Platform transition Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter [DONE] (except bzAPI compatibility layer, eta early Q1) Glob bug 885464 Bugzilla
Force Android reftests to run within 15% of their original, pre skia time Ensure panda and x86 reftests run within 15% of their original time [DONE] Gbrown bug 916657 Android
Ensure steeplechase webRTC automation as hands off as possible Deployed mochitests, ability to change machine configuration to system by end of quarter. C++ tests were not created (not our responsibility) so did not get integrated. [DONE] Ted bug 894559 WebRTC
Deploy bugmaster community building tools: bug board, tutorial, and bugzilla user profile report Tools deployed by end of quarter [MISSED] Lizzard bug 919624 Bugmaster
Deploy Mozmill 2.0, Metro infrastructure and tests, and aid SoftVision to take over day to day operations for QA automation with our oversight Deployed projects, tried to get the SV team to take over the reigns, and we had marginal success. Had to bail them out a few times. [DONE] Whimboo bug 922200 QA Automation
Operationalize sheriffing role and complete documentation so that we can pull in community sheriffs Have documentation completed, identify one part time community sheriff. [DONE] Edmorley bug 928960 Sheriffing

Detailed Lists

These contain the top level goals as well as the supporting, ongoing, and additional projects we'll try to complete this quarter. Note that additional projects may be moved to the backlog to cope with changes in priority or new projects. The current backlog is here

Marionette (mdas)

  • [DONE] P0 - Run set of endurance tests for 3 days without marionette issues
    • space issues, memory usage, import log, and don't regress socket.timeout
Ongoing and Supporting Projects
  • P0 - (crossteam)Get passing WebDriver conformance tests running in TBPL
  • P0 - (crossteam)Fix touch related bugs
  • P0 - (ongoing) Continue to support WebQA and Gaia teams with bug fixes and new features

B2G (jgriffin)

  • [DONE] P0 [ahal] (highlight) mochitests, reftests, gaia-ui-tests on b2g desktop builds: debug/opt linux32/64, debug/opt macosx64
  • [MISSED] P0 [ahal] crashtests, jsreftests on b2g desktop builds
  • [DONE] P0 [ahal] reftests/crashtests running out of process
  • [DONE] P0 [jgriffin] gaia integration tests on tbpl
  • [DEFER] P0 [wlach supporting] (crossteam) (perf team) power consumption testing
  • [MISSED] P0 [rwood] (crossteam) memory tests
  • [DONE] P0 [wlach,davehunt] Deploy automated Eideticker system to run nightly for Firefox OS
  • [MISSED] P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS
  • [DROPPED] P1 [wlach supporting] (crossteam)(with gaia team) github autolanding
  • [DONE] P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system
New Builds and Tests
  • [DEFER] P1 [ahal] tests on 4.2 emulators (https://bugzilla.mozilla.org/show_bug.cgi?id=912249)
  • [MISSED] P1 [ahal] jsreftests running out of process and enabled on all branches (and disable android noion)
  • [MISSED] P1 [jgriffin, ahal] mochitests, reftests, crashtests, jsreftests, webapi, and xpcshell tests on debug emulator builds: linux64
  • [MISSED] P1 [jgriffin, davehunt] get gaia-ui-tests going on emulators on TBPL
  • [DEFER] P2 [jgriffin] run "sanity test" of emulator on osx
Ongoing and Supporting Projects
  • P1 [jgriffin, ahal] continued work to make B2G tests more sheriffable
  • P2 [jgriffin, ahal] (internal) continued burn down of intermittent oranges
  • P2 [ahal] continued work to make B2G tests more developer friendly

Performance (jmaher)

  • [MISSED] Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard)
  • [DONE] P0 [jmaher] Complete investigation into Talos suites with desktop performance team
    • tsvg, tscroll, ts, tpaint are all refreshed/updated accordingly.
  • [MISSED] P0 [jmaher, ekyle] Complete alerting analysis for datazilla data
  • [DONE] P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority
  • [DONE] P1 [jmaher, bc] Investigate desktop power testing
Ongoing and supporting projects
  • have tools, data, tests fully documented and easy to find (all)
  • proactively meet with at least 5 teams or team leaders to get work in a pipeline instead of an emergency. (jmaher)
  • Ensure JS Team is satisfied with real world benchmarking system, investigate automation
  • Build performance lab
  • Continue supporting games benchmarking effort

Datazilla (jeads)

  • [MISSED] The Q4 top priority is to implement a fully functional alerting system for performance regressions and data ingestion monitoring.
  • Implement alert system that monitors data ingestion rates for all branches in all projects.
    • NOTE: Data ingestion rates should be reported in a graph, this could be part of the performance dashboard, abnormal values should trigger alert emails.
  • Fine tune automated regression detection system as needed
  • Implement the performance dashboard described in the performance section
Ongoing and supporting projects

NOTE: It's possible that slow page load performance in the new UI will require us to implement a reference data cache. Not sure if this will be an issue yet but something to be aware of. NOTE: There are several tasks building up in the b2g performance ui. They're assembled here, bug 913726, not sure how these should be prioritized in Q4.

Treeherder (mdoglio/jeads)

  • [DONE] Ensure sheriffs can perform side by side usability testing in TBPL
  • [MISSED] Implement TBPL *starring* features
  • [MISSED] Implement Orange Factor compatability
  • [MISSED] Release production instance on AWS
  • [DONE] Make sure we're on target for full TBPL replacement in Q1 2014.

Bugzilla (glob/mcote)

  • [DONE] Q4 top priority is to continue to focus on "Bugzilla as a Platform". This is a multifaceted goal covering API design, documentation authoring, and community engagement, and of course development.
  • Continue Bugzilla as a Platform (BaaP):
    • [DONE] [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval)
    • [MISSED] [dkl] (crossteam) bzAPI compatibility layer
    • [MISSED] [dkl] plan for versioning APIs, first draft of interface for API v2
      • We had some interruptions due to more API work, developer tools, and antispam efforts.
  • [DONE] [glob] (crossteam) memcached minimum viable product (also helps BaaP above)
Ongoing and supporting projects
  • [DEFER] [mcote] ReviewBoard integration plan based on feedback
    • Discussion is still ongoing as to whether Review Board can meet our needs or not.
  • [DONE] [mcote] somehow get over to git, or at least have a really serious plan for it
    • Plan in place.
  • [DONE] [kyle] deploy new bzETL

Android (gbrown)

  • [DONE] Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86.
  • [MISSED] mach commands for running remote tests
    • automatically get xpcshell/xre for remote tests
    • [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)?
  • [DONE] Eideticker responsiveness (possible carry over, depending on Q3).
Ongoing and supporting projects
  • Continue working with releng to increase panda stability
  • [bc] Use Autophone to do one-off cross browser bandwidth metrics
  • [DONE] [dminor] Deploy C++ remote tests

WebRTC (ted)

  • [DROPPED] P0: Get Steeplechase into releng staging (joint with releng)
  • [MISSED] P0: Deploy C++ tests for SteepleChase
  • [DONE] P0: Deploy ability to change machine configuration per test
  • [DEFER] P1: Investigate hardware testing set up (joint with desktop QA)

Bugmaster (lizzard)

  • [MISSED] Deploy bugmaster community building tools (bug board, tutorial, and bugzilla user profile report)
  • [DONE] Use summit to build out community involvement
  • [DONE] Continue triaging and imrpove the groundswell.
  • [MISSED] Research into new Bugzilla home page for logged-in/logged-out users (separate pages)

Desktop QA Automation (whimboo)

  • [DONE] (crossteam) Get SoftVision to lead the automation efforts for QA with our oversight
  • [DONE] (crossteam) Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0
  • [MISSED] (crossteam) Get Firefox Metro tests executed in production
Ongoing and supporting projects
  • [MISSED] Extend Mozmill CI to cover most of our localized beta and release candidate builds
  • [DONE] Create web application to generate the beta and release Jenkins job configuations
  • (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches
  • (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros

sheriffs (edmorley)

Meta bug: bug 928960

mozbase (ahal)

  • [MISSED] add structured logging to the mochitest harness
    • need to merge the logs from python and js
  • [DONE] determine a long term vision
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Goals/2013Q4&oldid=869649"