Metrics and dashboards

From MozillaWiki
Jump to: navigation, search

Executive Dashboards

Mozilla gathers a variety of information on the useage and health of our products. This data is critical to evaluating the health of our engineering/product ecosystem and user base. The project intent is to create ways to both query and visualize this data.

V1 is the current live dashboard (april 2015). V2 exists to add data sources and a more fully featured query set. V3 is when this will be handed over to IT for ongoing work.

This is part of the metrics and dashboards effort - see quality dashboard here []

Project Introduction

  • Team Mailing list:dev-metrics-pipeline@mozilla.com
  • Team IRC Channel: #datapipeline
  • Roadmap

Dependencies:

FHR V4

Data sources and key metrics (see doc for summary of key metrics being measured and the known data source gaps).

  • Firefox desktop metrics
  • Firefox mobile metrics
  • Tile Metrics
  • Account metrics
  • Data warehouse
  • Churn Metrics
  • Hello Metrics

Links to Current info

V1 Current live as of April 2015

V2

V3 Hand off to IT

Roles and Responsibilities

  • Dev Manager: Katie Parlante
  • Data Scientist: Rebecca Weiss
  • TPM: Thomas Huelbert

Meetings & Communications

Meeting Day of week Pacific Time Eastern Time Central European Time Vidyo Room Notes
Weekly meeting Friday 10:00AM - 10:30AM 1:00PM - 1:30PM 5:00PM - 5:30PM MTV2 - 221 Doctor Who etherpad
"Retrospective" TBD [1]
  • Links to mailing list or all communication channels

Development

Schedule

Roadmap

WIP - user stories : user stories

  • Q2 2015: Pipeline officially supports business use cases
    • FHR v4 feeds executive dashboard
    • Complete set of use cases WIP (link to bugzilla tree)
    • Complete set of monitoring and reporting outputs tbd: dashboards, data warehouse, monitoring, self-service access to data
    • FHR+telemetry hits full release 2015-05-19, handle full production load
  • Q3 2015: Fill out monitoring and reporting capabilities; add sources and use cases

Upcoming Priorities and Project Health

As we plan what's coming next, these are areas being discussed. This is not a commitment to the next projects - just our scratch area, but it is in order of relative priority. (after the work we've pulled for the sprints).

  • finishing Theme 1 (large change)
  • Theme 2: any time constraints/dependencies
  • Theme 3: quick why

Things we want to do - but parking lot until higher priorities are done:

  • Process Improvement: Theme
  • Other platform Investigation
  • Tech-debt: any time constraints/dependencies
  • Theme 4
  • Quality: description of areas
  • Theme 5

Risks

  • Risk 1 : See this doc for known data gaps see data gaps section
  • Risk 2 : iOS is not currently in scope but will be released on June 2 (planned). There seems to be a risk of not including this product is the intent is to provide an over arching look at the ecosystem.

Definitions of start and done

Start (its possible not all these items need to be defined or agreed on by all members of the team, but the lack of these items should indicate red flags)

  • PRD's have been created: in progress
  • User stories created: in progress
  • Above requirements are understood (including dependencies such as: access to staging, new hardware, skill set development etc): TBD
  • Dev, QE and PM resources are mapped to intended goals: TBD
  • schedules create and agreed on: TBD

Done:

  • TBD

Current Status

example from Jenn's

Project Health

IOS,Desktop Yellow Risk.png

  • (get central location for graphics from Erin)

Include clear, executive level summary that will be included at the [mana page overall view level:

  • for company goal x, we are in _____ state because ______. Please consider ______(propose fix or adjustment of goal).
  • for release goal for ______ , we are in _____ state because ______. Please consider ______(propose fix or adjustment of goal).

Current (queries)

Past

  • Pipeline of work that has been completed and moving through Aurora and Beta to release.
  • Ideally these aren't just queries, but human readable so folks know what is coming in x release.

Product Backlog

Key Bugzilla Queries

Link to backlog (scrubbed weekly)

Bugzilla Agile fields

Several Fields have been added this year into bugzilla for Agile tracking that you may want to leverage:

  • Iteration: right now this has the 3 previous and future 2 week sprints for Firefox release schedule - shows iteration (40.1) and uplift date.
  • Points: Fibonacci series 1,2,3,5,8,13]
  • Rank: see Rank use description under Triage Guidelines
    • Rank will not show under your Product::Component unless you request it by filing a bug with "bugzilla.mozilla.org :: Administration". You need to give a list of acceptable editors (usually the folks that can triage).

Product Backlog

  • There needs to be a way to mark the bugs you have triaged versus not. Firefox uses the flag for "firefox-backlog".
  • If you are not strictly desktop, there's also an option to add you team to the blocking-flag "backlog".
    • ex: the webRTC team uses webRTC+ for bugs they are accepting and "parking lot" for ones they are not. The queries are based on product/component - so multiple teams can use "parkinglot" and just request a new tag <team name>+ tag by filing a bug under "bugzilla.mozilla.org :: Administration"
  • ex: Bugzilla Ranked "Firefox-backlog+" list
    • Add the "Rank" Column to your results and sort on Rank to view order of work.
      • The option to "Change columns" is at bottom of search results
    • Search is based on "Flags" - "contains the string" - firefox-backlog+
  • Un-triaged bugs
    • Search is based on not having the Firefox-backlog+ or Firefox-backlog-

Triage Guidelines

The Product Backlog is continually maintained by the Hello Management team to ensure new priorities are available for each Sprint Planning meeting.

  • Priorities follow this Standard:
    • Priority 1 - Blocker, must-fix before shipping.
    • Priority 2 - Major impact, considering severity × probability. Not a blocker for shipping.
    • Priority 3 - Average Bug. definitely a problem, but doesn't stop someone from using the product.
    • Priority 4 - Minor or polish bugs that are real issues (especially in aggregate) and annoying.
    • Priority 5 - Low-impact. something we'd fix, but mostly only bothers the discerning user. Little impact on usability.
  • RANK: As priority buckets start to have a large amount of bugs in them, the Rank field can be used to call attention to higher or lower rank and provide a way to sort easily in bugzilla. To have some rhyme/reason to the order - Rank should relate to Priority. The "Ranking" number does not need to be unique. Unless there is a reason to for a bug to be considered before (or after) others in the Priority bucket - default to mid-range value.
    • P1 Rank options=1-19, default 15
    • P2 Rank options=20-29, default 25
    • P3 Rank options=30-39, default 35
    • P4 Rank options=40-49, default 45
    • P5 Rank options=50-59, default 55
    • any that we don't think we can get to in the next 6 months should go in "backlog-" area

  • The Firefox-backlog flag is used to track bugs that are approved for the Backlog "+"
  • QE-Verify is a flag that developers should be setting on bugs they are working on. This is used for QE to filter which bugs they check
    • "+" means that QE should look at the bug and it can be verified with human eyes
    • "-" means QE should not look at
      • Typically goes with in-testsuite set to "+", to show testing via another method.
  • "Points" should be set when known (if nothing set - assumed a "1" or very small). Most relevant if taking a bigger bug so we know when bugs are large bits of work.
  • "Iteration" should be updated when a bug is being worked on during a particular Sprint.

Filing a bug

    • Open a bug under Product:"Loop" || Component: "General" or "Client"
    • Hello team reviews for inclusion into a release backlog every 2 weeks, and will mark "firefox-blocking" accordingly
  • If there is a bug that should be considered for taking ASAP - you can mark "firefox-backlog"+
    • Before it can be given a Rank it should:
      • be in an actionable state
      • for defects, the problem is ready for Engineering or UX: diagnosis, measurement, design, or fixing
      • for feature requests or enhancements, it means that there's a clear problem statement or suggestion
      • has a difficulty/user-impact ratio low enough that we can reasonably expect to spend time fixing the bug within the next 6 months

Templates

Go to Templates page for sample etherpads, invite details, etc. for the following types of meetings.

  • stand-ups (daily, weekly, or other)
  • retrospective invites with questions/structure of the meeting in etherpad
  • cross-team/company meeting invites/etherpads to keep efficient and only when needed
  • postmortems

Archive

as iteration information or links get old or if this is replacing an older page and we don't want to lose path to the information...link to an archive page from here.

Reference materials

Sample bugzilla Queries

Please add your queries and descriptions of what they are doing here to help other create without having to learn bugzilla!

  • Queries to pull info for an iteration

Tools

  • Aha
  • Trello

https://waffle.io/mozilla/payments

Reading

Retrieved from "https://wiki.mozilla.org/index.php?title=Metrics_and_dashboards&oldid=1074248"