QA/Goals/2014q1

2014 Yearly Goals

• Support platform and products to ensure what we are shipping is high quality.
• Increase scalability by attaining a level of community participation where we have on average 80 active contributors per week, every week working on QA projects
• Every top level project in every area will establish a comprehensive and integrated set of quality metrics and a known, externally (from qa) referenceable level of quality.
• Drive changes to our processes for increased efficiency and impact as measured from both developer satisfaction (with finding issues early and often) and in end-user quality perception and satisfaction.

Q1 Breakdown

Supporting Events

Events in the quarter that we will be providing quality engineering for.

Release	Event	ETA
Firefox Desktop and Android 27	Ship	Feb 4
Firefox Desktop and Android 28	Ship	Mar 18
Firefox 29 - Australis, Metro, Sync	Cut to Aurora - expect crash landings	Feb 3
Firefox 29 - Australis, Metro, Sync	Cut to Beta - expecting more than usual uplifts during aurora	Mar 17
Stabilize 1.3 FFxOS	Expected to stabilize	Jan 31
Tarako Readiness (1.3 FFxOS)	Expected to stabilize	Jan 31
Move to 18 week FFxOS Cycle wth 1.4	Expecting rocky start, focus on polish and UI	Through end of quarter
Firefox Accounts (FxA) Sync Ship	Servers ready and tested by 29 on beta	Mar 17

Internal Q1 Goals

Internal team goals. This quarter it's all about building the ground work for increasing our scalability and measurability. Both of these will continue to guide our work throughout the rest of the year.

Goal	Measurement	Status	Primary Owner	Area
Build community infrastructure to support us the rest of the year while putting low-hanging fruit into place	Achieve weekly active contributor count in 10-30 range	[ON TRACK]	Marcia	Scalability::Community
Run Metro test automation reliably by Feb 12	Tests running in automation with a less than 5% failure rate of any kind by the time the 29 release is on aurora	[ON TRACK]	Whimboo	Scalability::Automation
Find and repair on-device automation to eliminate manual smoketest requirement on Buri	Have smoketests completely automated from initiating to reporting with enough confidence we can only investigate failures rather than duplicating smoketest work	[ON TRACK]	Stephend	Scalability::Automation
Ensure new Firefox Accounts Sync Server APIs are robust enough to support our user base	Have ongoing automation in place to test all APIs and be able scale load to 2x user base by Feb 21 by simulating user load over time	[ON TRACK]	Edwong	Scalability::Automation
Define base line quality metrics for FFxOS	Create first set of quality metrics focused on 1.3 stabilization effort, but with an eye to apply them broadly	[ON TRACK]	Geo	Metrics::FFxOS
Define base line quality metrics for Desktop Firefox	Create set of metrics applicable to all channels, but start by focusing on 29	[ON TRACK]	Mschifer	Metrics::Desktop
Define base line quality metrics for Fennec	Create set of metrics applicable to all channels, but start with 29	[ON TRACK]	Kbrosnan	Metrics::Fennec
Define base line quality metrics for Marketplace and Payments		[ON TRACK]	Krupa	Metrics::Marketplace_and_payments
Define base line quality metrics for Web QA		[ON TRACK]	Stephend	Metrics::WebQA
Define base line metrics for Services	Create a set useful broadly, but first focus on quality metrics for FxA sync service	[ON TRACK]	Edwong	Metrics::Services
Create Training/Webcast for what our existing automation can tell us	Create small training demos in order to help increase technical expertise of the team and potential contributors	[ON TRACK]	Ctalbert	[https://wiki.mozilla.org/QA/Goals/2014q1#People Scalability::People

Detailed Breakdown

This is where the supporting elements of our top level projects can be found with their owners and any further information that might be required for tracking

Scalability

Community

• [ON TRACK] Get all teams working on establishing conversion targets and methods for responding to emails from the contribute lists (stephend)
• [ON TRACK] Come up with some kind of interim tracking solution we can use now for community members (karl)
• [ON TRACK] Define an event strategy (marcia)
• And so on...please correct the above if necessary :-)

Automation

• Metro Automation
  • [ON TRACK] Ensure crashes in Jenkins infrastructure fixed (Whimboo)
  • [ON TRACK] Complete and land metro code (AndreeaMatei)

• Gaia Automation
  • [ON TRACK] Identify what prevents current automation from replacing manual smoketests (stephend)
  • [ON TRACK] Fix that ^ (zac)

• Services Automation
  • [ON TRACK] Land API tests for FxA to verify APIs funcitonal (??)
  • [ON TRACK] Create/Build upon load testing system to expand capacity (??)

People

• [ON TRACK] Make webcasts for training so that it is clear how to read our existing tests in automation and understand what it is they tell us (ctalbert)
• [ON TRACK] Investigate JavaScript training for anyone interested in working on expanding code skills (ctalbert/mschifer)

Metrics

Use this area to record specific high level projects that need to be tracked in order to be successful at various metrics projects. There may not be any need to have these projects for your area, which is fine.

• [ON TRACK] Come up with basic set of cross-project metrics using bugzilla, moztrap etc (ctalbert?)

Desktop

• [ON TRACK] Create root cause analysis project and begin low-hanging fruit analysis (KaiRo/Liz)

Fennec

• ??? - Not sure if there is anything specific here beyond what's above

FfxOS

• [ON TRACK] Find/fixed rate for Gaia UI Tests, particularly the smoketest automation? (stephend/geo)
  • 1) how will we measure? 2) who will be responsible for the report? 3) how/when/where will it be delivered?

Marketplace and payments

• ??? - Not sure if there is anything specific here beyond what's above

Web QA

• ??? - Not sure if there is anything specific here beyond what's above

Services

• Auth/Sync Server uptime of 95%
• Code coverage on Auth and Content servers greater than 87%
• Authentication time doesn't regress from baseline
• Less than 3 confirmed crash bugs when in FxA flow. (pending dev meets schedule)