2014 Yearly Goals

• Support platform and products to ensure what we are shipping is high quality.
• Increase scalability by attaining a level of community participation where we have on average 80 active contributors per week, every week working on QA projects
• Every top level project in every area will establish a comprehensive and integrated set of quality metrics and a known, externally (from qa) referenceable level of quality.
• Drive changes to our processes for increased efficiency and impact as measured from both developer satisfaction (with finding issues early and often) and in end-user quality perception and satisfaction.

Q1 Breakdown

Supporting Events

Events in the quarter that we will be providing quality engineering for.

Internal Q1 Goals

Internal team goals. This quarter it's all about building the ground work for increasing our scalability and measurability. Both of these will continue to guide our work throughout the rest of the year.

Detailed Breakdown

This is where the supporting elements of our top level projects can be found with their owners and any further information that might be required for tracking

Scalability

Community

• Tools: [MISSED] Build a consistent method to track contributors (lizzard)
• Tools: [DONE] Seed content into the One and Done tool and deliver meaningful tasks to active and emerging contributors. (karl)
• Contribution Paths: [DONE] Streamline inquiries coming from Mozilla.org/contribute by channeling contributors into areas that better match their interests and experiences. (rbillings)
• Contribution Paths: [DONE] Get all teams working on establishing conversion targets and methods for responding to emails from the contribute lists (stephend)
  • [DONE] Ship One and Done
  • [DONE] Ensure we measure Persona logins (account creations)
  • [DONE] Ensure we measure tasks taken
  • [DONE] Ensure we measure tasks finished
• Events: [MISSED] Define an event strategy and reinvigorate test days (marcia)
• Recognition: [DONE] Get the General QA Participation Badge in a position to be awarded to contributors (parul)

Automation

• Metro Automation
  • [DONE] Ensure crashes in Jenkins infrastructure fixed (Whimboo)
  • [DROPPED] Complete and land metro code (AndreeaMatei)

• Gaia Automation
  • [DONE] Identify what prevents current automation from replacing manual smoketests (stephend)
    • https://docs.google.com/spreadsheet/ccc?key=0Ah8f07ewuXU2dEJDREN0RVQ2NC1pQnBDN0lpbEh1OWc&usp=drive_web#gid=0
  • [DONE] (ongoing) Fix that ^ (zac/a-team)

• Services Automation
  • [DONE] Land API tests for FxA to verify APIs functional (pdehaan/jrgm)
  • [DONE] Create/Build upon load testing system to expand capacity (jbo)

People

• [DONE] Make webcasts for training so that it is clear how to read our existing tests in automation and understand what it is they tell us (ctalbert)
• [DONE] Investigate JavaScript training for anyone interested in working on expanding code skills (ctalbert/mschifer)

Metrics

Use this area to record specific high level projects that need to be tracked in order to be successful at various metrics projects. There may not be any need to have these projects for your area, which is fine.

• [DONE] Come up with basic set of cross-project metrics using bugzilla, moztrap etc (ctalbert?)

Desktop

• [DONE] Create root cause analysis project and begin low-hanging fruit analysis (KaiRo/Liz)

Fennec

• No specific goals here that aren't already mentioned to release work

FxOS

• Proposed Product Metrics
  • Number requirements/stories defined vs. tested
    • Acceptance: Defined vs. tested is equal by release (all defined stories landed or moved out of scope, all landed stories tested)
    • Measures knowledge of release quality for new functionality
    • Query from User Story metas in Bugzilla if possible, otherwise gathering from EPMs/teams.
  • Add/Remove/Total for release blockers
    • Acceptance: Total is 0, either via being explicitly moved out of scope or fixed.
    • Expectations around add/remove softer--fit expected curve. Possibly not a baseline, but something to track.
    • Similar to find/fix/total, but specifically what moves in/out of release+ blocking status.
    • Defers blocking status to triage team, but ensures that triaged issues are all addressed
    • Query from Bugzilla
  • Smoketest regressions
    • Acceptance: 0 by release, all fixed, none moved out of scope
    • Adds "not moved out of scope" to regressions agreed upon as smoketest criteria
    • Query from Bugzilla
  • Performance acceptance
    • Acceptance: be within established baselines
    • Performance baselines are established via product
    • Will be observable in Datazilla
  • Stability acceptance
    • Acceptance: be within established baselines
    • Unsure what baselines are already established (open issue?)
    • Will be observable in Crash Stats(?)

• Other Metrics Discussed
  • # regressions
  • # bugs total
  •  % bugs qualified (bugs r+'ed / total bugs found * 100)
  •  % tests executed (test coverage)
  • # bugs found in last regression cycle
  • # bugs found in IOT (special case of bugs found in field)
  • All worth tracking, but difficult to baseline to understand acceptance since the ship decision is made mostly on blocking flags

• Progress on Metrics
  • [DONE] Proposal
  • [DONE] Review with team lead stakeholders
  • [DONE] Review with release management
  • [MISSED] Gather existing Bugzilla queries for blocker/keyword tracking
  • [MISSED] Define process for user story measurements (risk due to external dependencies)
  • [MISSED] Get timeline for performance measurements (risk due to external dependencies)
  • [MISSED] Get timeline for stability measurements (risk due to external dependencies)

• Stretch: vet release management dashboard software for use by QA and deploy a PoC instance
  • [DONE] Demo and project info from release management
  • [MISSED] Deploy into a test instance
  • [MISSED] Translate Bugzilla queries into elasticsearch queries for PoC
  • [MISSED] Create sample quality dashboard

• Proposed UI Automation Metrics
  • [MISSED] Find/fixed rate for Gaia UI Tests, particularly the smoketest automation? (stephend)
  • 1) how will we measure? 2) who will be responsible for the report? 3) how/when/where will it be delivered?

Marketplace and payments

• [MISSED] Using Bugzilla, track and group the major types of bugs being filed for Marketplace and Payments
  • [DONE] for Marketplace
  • [MISSED] for payments
• [DONE] Have an automated smoketest run for stage which is run before every push

Web QA

• Determine if it's feasible to produce reliable metrics on the following:
  • [MISSED] Trendline of find/fixed rate for manually-filed bugs
  • For automation, the same -- a trendline of find/fixed rate
    • See if we can correlate the two?
  • Big miscommunication on what Tableau can provide "out of the box": was told it can do expose everything Bugzilla has (which is true), but it wasn't built in time and didn't come with what we had asked for
    • This'll have to be a carry-over: we'll either need a new tool, or to ensure that the [fromAutomation] whiteboard status can be measured and correlated to the manual tests (at least for the WebDriver test suites)

Services

• Auth/Sync Server uptime of 95%
• Code coverage on Auth and Content servers greater than 87%
• Authentication time doesn't regress from baseline
• Less than 2 crash bugs when in FxA flow. (pending dev meets schedule)