QA/Goals/2014q3

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

2014 Yearly Goals

• Support platform and products to ensure what we are shipping is high quality.
• Increase scalability by attaining a level of community participation where we have on average 80 active contributors per week, every week working on QA projects
• Every top level project in every area will establish a comprehensive and integrated set of quality metrics and a known, externally (from qa) referenceable level of quality.
• Drive changes to our processes for increased efficiency and impact as measured from both developer satisfaction (with finding issues early and often) and in end-user quality perception and satisfaction.

Supporting Events

Primary Team	Event	Action	When	Grade
Firefox, Services	Firefox 31	TODO: List high profile features	July 21
Firefox, Services	Firefox 32	Loop, etc	Sept 2
Services	OAuth Services Ship	TODO:NEEDS DEFINITION	TODO:
FxOS	1.4 Completion	Deliver 1.4 through IOT testing and get it shipped and closed down	August 1
FxOS	2.0 FC	Ensure testing completed sufficiently to determine when 2.0 hits the FC quality milestones, and raise issues accordingly	July 21
FxOS	2.0 CS	Deliver FxOS support through CS - Aug 27.
FxOS	2.1 FL	Ensure we have good scope, great test coverage, and manage our quality level and risk for a successful FL on 2.1	Sept 9 TODO: Verify
Marketplace	Support New Payment Provider	Test and Launch the second payment provider
Marketplace	Server-less in-app payments	Test and Support launch of the new in-app payments system which doesn't require the developer to have a server set up
Marketplace	Feed	Fully support the launch of the new Discovery mechanism (v1) for Marketplace
Marketplace	Support New Payment Provider
Marketplace	Desktop Support	Fully support the rehaul and support for Marketplace on Desktop

Other Big Elements - might need to be included above:

• Loop - part of FxOS 2.0 and 32, right?
• Sponsored Tiles - 32?
• Lab Buildout for Automation - part of 2.1 effort on Fx OS - date to be live is July 21.

Internal Goals (By Project)

Firefox QA

Revitalize our Quality Engineering strategy by switching to a feature-based test strategy for all Firefox browser products so that there are test plans for each feature and testing is executed within two sprints of the feature's code landing.

High-level Supporting Tasks

• General Tasks
  • [ON TRACK] Complete Root Cause analysis so that the data auto-updates and use the data to guide testing strategy successfully for Firefox 34 (by July 21): kairo
  • [ON TRACK] Ship Games Inititative with Mozilla and Partner approved quality: mschifer
  • [ON TRACK] Ship Loop client with Mozilla and Partner approved quality: ashughes
• Community Tasks
  • [ON TRACK] Create community infrastructure to involve and onboard 20 new contributors: mschifer
  • [ON TRACK] Create relevant contribution pathways, train the team on using them: tracy
  • [ON TRACK] Ensure all feature owners have one and done tasks that direct contributors toward contribution paths: mschifer
• Automation Tasks
  • [ON TRACK] Automate update testing for beta->release->beta update testing strategy: whimboo
  • [ON TRACK] Stabilize restart testing automation - less than 5% intermittent failure rate: whimboo
  • [ON TRACK] Automatically restart machines between test runs and full automatic start of jenkins: whimboo

Firefox OS

Better metrics for quality analysis, provide opportunities for community contribution, and expanding the role of our on-device automation for reporting to Treeherder. TODO: NEEDS DEFINITION AND METRICS

High-level Supporting Tasks

• General Tasks
  • [ON TRACK] Create an auto-updating, public quality dashboard to display smoketest health status: jlorenzo, geo
  • [ON TRACK] Define and Implement a plan for closer coordination with TPE QA: tchung, brian
    • Flame lab build-out for smoketest automation in TPE
    • Define a strategy for geolocated contract testing resources for release work
  • [ON TRACK] Hire open reqs: tchung
  • [ON TRACK] Report a overall acceptance health status a week prior to to FL and FC: nhirata, jsmith, geo
• Community Tasks
  • [ON TRACK] Ensure One and Done tasks created that deliver contributors into actionable, useful steps with and without a device: kglazko
    • Note - needs to be more quantifiable on # of tasks.
  • [ON TRACK] Define, publish, and use contribution pathways for contributors with and without devices: marcia
    • Note - may need to clarify the word "use" as a quantifiable metric
  • [ON TRACK] Hold one in-person quality event, and hold 3 virtual or async ones: marcia
    • Note - focus on in-person piece, but bug bashes can include for virtual
• Automation Tasks
  • [ON TRACK] Define a set of JS integration sanity testsuite that will run on Flame devices as part of the per-checkin Tree Herder strategy: jason, stephen, zac
    • Testsuite should target ~5-8 minutes of execution time
    • Related/co-dependent is (for the actual implementation of these tests) the Q3 A-team goal, "** Get gaia-integration tests running on-device with full support for handling crashes, timeouts, and test manifests (ahal)")
  • [ON TRACK] Integrate graphics regression tests into the mainline code tree: njpark
  • [ON TRACK] Deploy QA Lab Flame buildout: retornam
    • See https://wiki.mozilla.org/QA/Execution/Web_Testing/Lab for more info

Services

Ensure new offerings have 98% uptime and no more than 2 P1 bugs discovered post launch while building a core contributor base in services

High-level Supporting Tasks

• General Tasks
  • [ON TRACK] Complete automated cross-browser tests for OAuth flows: edwong
  • [ON TRACK] Ensure the DIY sync is well-supported TODO:NEEDS METRIC: edwong
• Community Tasks
  • [ON TRACK] Acquire 2 active contributors for services QA (at least): edwong
    • [ON TRACK] Have at least one meeting with the QA 101 contributors: kthiessen
    • [ON TRACK] Perform one trial seminar of QA 201: kthiessen
    • [ON TRACK] Ensure docs are adequate for on-boarding of contributors in QMO: edwong
    • [ON TRACK] Perform two services test days at least: edwong
• Automation Tasks
  • [ON TRACK] Ensure FxA smoketests landed in gaia-ui-test: rpappa
  • [ON TRACK] Ensure Loop call creation regression performance tests are run and tracked over time: edwong

Marketplace & General Web FE

Ensure marketplace's new payments backend, in-app payments, and the marketplace feed land with high quality releases.

High-level Supporting Tasks

• General Tasks
  • Improve developer visibility into automation tasks
• Community Tasks
  • Continue participation in monthly Marketplace days
  • Work on making the Internal hackathon for apps a success
• Automation Tasks
  • Ensure automation runs on 1.4 using Flame devices set for 256Mb of RAM to simulate dolphin
  • Have automation coverage for Payments including in-app payments

Platform

Deploy into continuous integration a regression testing system for WebRTC along three areas of investigation: 1. "Smoke" tests, 2. Connection tests, and 3. Connection quality tests. These will be performed in a variety of environments, including various network topologies with various performance characteristics, connections between various versions of Firefox, and connections on multiple OS platforms. Test plans will be developed, and we will measure coverage of test plans. Results will be posted and available in Treeherder. We will also build backlog for next Platform Tiger Team tasks.

High-level Supporting Tasks

• General Tasks
  • [ON TRACK] Build out a backlog for the next Tiger team target(s): syd
  • [ON TRACK] Fill open req: mschifer
• Community Tasks
  • [ON TRACK] Investigate mentored bugs/one and done etc as a means to begin community involvement and create a plan for community involvement that will be deployed in Q4: syd
• Automation Tasks
  • [ON TRACK] Complete test plan and deployment of tests in a sunny-day environment with basic TURN-based smoke tests, including reporting results to treeherder: geo
  • [ON TRACK] Complete procedure for deployment of parallel sunny-day environments using virtual machines, including running TURN-based tests: mwargers
  • [ON TRACK] Complete test plan and deployment of connection establishment tests: nils
  • [ON TRACK] Complete test plan and deployment of connection quality tests: geo
  • [ON TRACK] Complete architecture plan and deployment of infrastructure for multi-version, multi-OS, and multiple-network topology tests, including smoke, connection establishment, and connection quality tests: syd

Community

Increase conversion of interested and casual contributors to help them become active contributors with strong ties to the QA community.

High-level Supporting Tasks

(Further detail at https://etherpad.mozilla.org/2014Q3Goals-Community)

• One and Done
  • [ON TRACK] Create an ongoing roadmap for One and Done: rbillings
  • [ON TRACK] Deploy ability to import tasks into One and Done from bugzilla: rbillings
• Metrics
  • [ON TRACK] Integrate moztrap data into Baloo: lizzard
  • [ON TRACK] Popularize, encourage, and standardize use of the Ad-hoc contribution logger: lizzard
• Establish Connections
  • [ON TRACK] Strengthen connections in the QA community. marcia
  • [ON TRACK] In collaboration with the community, develop a long-term strategy to redefine the program as value-driven events designed to teach new skills and connect contributors with various projects. We'll organize at least one event utilizing this strategy and gather feedback to drive more meaningful and active events through Q4 and beyond. ashughes/aaronmt
• Redesign of QMO (quality.mozilla.org)
  • [ON TRACK] Finalize the re-design of QMO with mockups and workflows: pragmatic