From MozillaWiki
< QA‎ | Goals
Jump to: navigation, search

2014 Yearly Goals

  • Support platform and products to ensure what we are shipping is high quality.
  • Increase scalability by attaining a level of community participation where we have on average 80 active contributors per week, every week working on QA projects
  • Every top level project in every area will establish a comprehensive and integrated set of quality metrics and a known, externally (from qa) referenceable level of quality.
  • Drive changes to our processes for increased efficiency and impact as measured from both developer satisfaction (with finding issues early and often) and in end-user quality perception and satisfaction.

Q1 Breakdown

Supporting Events

Events in the quarter that we will be providing quality engineering for.

Release Event ETA
Firefox Desktop and Android 27 Ship Feb 4
Firefox Desktop and Android 28 Ship Mar 18
Firefox 29 - Australis, Metro, Sync Cut to Aurora - expect crash landings Feb 3
Firefox 29 - Australis, Metro, Sync Cut to Beta - expecting more than usual uplifts during aurora Mar 17
Stabilize 1.3 FFxOS Expected to stabilize Jan 31
Tarako Readiness (1.3 FFxOS) Expected to stabilize Jan 31
Move to 18 week FFxOS Cycle wth 1.4 Expecting rocky start, focus on polish and UI Through end of quarter
Firefox Accounts (FxA) Sync Ship Servers ready and tested by 29 on beta Mar 17

Internal Q1 Goals

Internal team goals. This quarter it's all about building the ground work for increasing our scalability and measurability. Both of these will continue to guide our work throughout the rest of the year.

Goal Measurement Status Primary Owner Area
Build community infrastructure to support us the rest of the year while putting low-hanging fruit into place Achieve weekly active contributor count in 10-30 range [MISSED] Marcia Scalability::Community
Run Metro test automation reliably by Feb 12 Tests running in automation with a less than 5% failure rate of any kind by the time the 29 release is on aurora [DROPPED] Whimboo Scalability::Automation
Have smoketests completely automated from initiating to reporting with enough confidence we can only investigate failures rather than duplicating smoketest work Identify current automation-to-manual smoketest coverage, and report on the opportunities to "close the gap" [DONE] Stephend Scalability::Automation
Ensure new Firefox Accounts Sync Server APIs are robust enough to support our user base Have ongoing automation in place to test all APIs and be able scale load to 2x user base by Feb 21 by simulating user load over time [DONE] Edwong Scalability::Automation
Define baseline quality metrics for FFxOS Create first set of quality metrics focused on 1.3 stabilization effort, but with an eye to apply them broadly [MISSED] Geo Metrics::FFxOS
Define baseline quality metrics for Desktop Firefox Create set of metrics applicable to all channels, but start by focusing on 29 [MISSED] Mschifer Metrics::Desktop
Define baseline quality metrics for Fennec Create set of metrics applicable to all channels, but start with 29 [DROPPED] Kbrosnan Metrics::Fennec
Define baseline quality metrics for Marketplace and Payments Track and group the kind of bugs being found in Marketplace and Payments and understand Bug patterns emerging from it [DONE] Krupa Metrics::Marketplace_and_payments
Define baseline quality metrics for Web QA Determine if it's possible to get reliable trendline metrics for find/fixed rates for both manual + automation, and a correlation atop, if possible, for the two [MISSED] Stephend Metrics::WebQA
Define baseline metrics for Services Create a set useful broadly, but first focus on quality metrics for FxA sync service [DONE] Edwong Metrics::Services
Create Training/Webcast for what our existing automation can tell us Create small training demos in order to help increase technical expertise of the team and potential contributors [MISSED] Ctalbert [ Scalability::People

Detailed Breakdown

This is where the supporting elements of our top level projects can be found with their owners and any further information that might be required for tracking



  • Tools: [MISSED] Build a consistent method to track contributors (lizzard)
  • Tools: [DONE] Seed content into the One and Done tool and deliver meaningful tasks to active and emerging contributors. (karl)
  • Contribution Paths: [DONE] Streamline inquiries coming from by channeling contributors into areas that better match their interests and experiences. (rbillings)
  • Contribution Paths: [DONE] Get all teams working on establishing conversion targets and methods for responding to emails from the contribute lists (stephend)
    • [DONE] Ship One and Done
    • [DONE] Ensure we measure Persona logins (account creations)
    • [DONE] Ensure we measure tasks taken
    • [DONE] Ensure we measure tasks finished
  • Events: [MISSED] Define an event strategy and reinvigorate test days (marcia)
  • Recognition: [DONE] Get the General QA Participation Badge in a position to be awarded to contributors (parul)


  • Metro Automation
    • [DONE] Ensure crashes in Jenkins infrastructure fixed (Whimboo)
    • [DROPPED] Complete and land metro code (AndreeaMatei)
  • Services Automation
    • [DONE] Land API tests for FxA to verify APIs functional (pdehaan/jrgm)
    • [DONE] Create/Build upon load testing system to expand capacity (jbo)


  • [DONE] Make webcasts for training so that it is clear how to read our existing tests in automation and understand what it is they tell us (ctalbert)
  • [DONE] Investigate JavaScript training for anyone interested in working on expanding code skills (ctalbert/mschifer)


Use this area to record specific high level projects that need to be tracked in order to be successful at various metrics projects. There may not be any need to have these projects for your area, which is fine.

  • [DONE] Come up with basic set of cross-project metrics using bugzilla, moztrap etc (ctalbert?)


  • [DONE] Create root cause analysis project and begin low-hanging fruit analysis (KaiRo/Liz)


  • No specific goals here that aren't already mentioned to release work


  • Proposed Product Metrics
    • Number requirements/stories defined vs. tested
      • Acceptance: Defined vs. tested is equal by release (all defined stories landed or moved out of scope, all landed stories tested)
      • Measures knowledge of release quality for new functionality
      • Query from User Story metas in Bugzilla if possible, otherwise gathering from EPMs/teams.
    • Add/Remove/Total for release blockers
      • Acceptance: Total is 0, either via being explicitly moved out of scope or fixed.
      • Expectations around add/remove softer--fit expected curve. Possibly not a baseline, but something to track.
      • Similar to find/fix/total, but specifically what moves in/out of release+ blocking status.
      • Defers blocking status to triage team, but ensures that triaged issues are all addressed
      • Query from Bugzilla
    • Smoketest regressions
      • Acceptance: 0 by release, all fixed, none moved out of scope
      • Adds "not moved out of scope" to regressions agreed upon as smoketest criteria
      • Query from Bugzilla
    • Performance acceptance
      • Acceptance: be within established baselines
      • Performance baselines are established via product
      • Will be observable in Datazilla
    • Stability acceptance
      • Acceptance: be within established baselines
      • Unsure what baselines are already established (open issue?)
      • Will be observable in Crash Stats(?)
  • Other Metrics Discussed
    • # regressions
    • # bugs total
    •  % bugs qualified (bugs r+'ed / total bugs found * 100)
    •  % tests executed (test coverage)
    • # bugs found in last regression cycle
    • # bugs found in IOT (special case of bugs found in field)
    • All worth tracking, but difficult to baseline to understand acceptance since the ship decision is made mostly on blocking flags
  • Progress on Metrics
    • [DONE] Proposal
    • [DONE] Review with team lead stakeholders
    • [DONE] Review with release management
    • [MISSED] Gather existing Bugzilla queries for blocker/keyword tracking
    • [MISSED] Define process for user story measurements (risk due to external dependencies)
    • [MISSED] Get timeline for performance measurements (risk due to external dependencies)
    • [MISSED] Get timeline for stability measurements (risk due to external dependencies)
  • Stretch: vet release management dashboard software for use by QA and deploy a PoC instance
    • [DONE] Demo and project info from release management
    • [MISSED] Deploy into a test instance
    • [MISSED] Translate Bugzilla queries into elasticsearch queries for PoC
    • [MISSED] Create sample quality dashboard
  • Proposed UI Automation Metrics
    • [MISSED] Find/fixed rate for Gaia UI Tests, particularly the smoketest automation? (stephend)
    • 1) how will we measure? 2) who will be responsible for the report? 3) how/when/where will it be delivered?

Marketplace and payments

  • [MISSED] Using Bugzilla, track and group the major types of bugs being filed for Marketplace and Payments
    • [DONE] for Marketplace
    • [MISSED] for payments
  • [DONE] Have an automated smoketest run for stage which is run before every push

Web QA

  • Determine if it's feasible to produce reliable metrics on the following:
    • [MISSED] Trendline of find/fixed rate for manually-filed bugs
    • For automation, the same -- a trendline of find/fixed rate
      • See if we can correlate the two?
    • Big miscommunication on what Tableau can provide "out of the box": was told it can do expose everything Bugzilla has (which is true), but it wasn't built in time and didn't come with what we had asked for
      • This'll have to be a carry-over: we'll either need a new tool, or to ensure that the [fromAutomation] whiteboard status can be measured and correlated to the manual tests (at least for the WebDriver test suites)


  • Auth/Sync Server uptime of 95%
  • Code coverage on Auth and Content servers greater than 87%
  • Authentication time doesn't regress from baseline
  • Less than 2 crash bugs when in FxA flow. (pending dev meets schedule)