Identity/Persona-Analytics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 40: Line 40:
|Two Metrics That Matter  
|Two Metrics That Matter  
|Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard.  
|Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard.  
|John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane (browserid data source)
|John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane Tomlinson (browserid data source)
|Oct?
|Oct?
|-
|-
Line 46: Line 46:
|Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text.
|Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text.
|Ryan Feeley, John Gruen, Dan Callahan
|Ryan Feeley, John Gruen, Dan Callahan
|
|-
|-
|
|Dialog A/B Testing
|
|Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog.
|
|Shane Tomlinson (browserid infrastructure), Katie Parlante (results display)
|
|Oct: dialog infrastructure, Nov: results display
|}
|}


Revision as of 18:54, 10 September 2013

Overview

The Persona team makes use of metrics to help make good decisions and stronger products.

Public Metrics

Private Metrics

We have two dashboards that support the #signin team (LDAP required):

Privacy

Like all of us at Mozilla, we take user privacy seriously. We only collect the data that we use for decision making, and we take care to remove personally identifying information from that data before it reaches these dashboards. While we aim to be open in our decision making, our metrics dashboards that make use of production data are currently private.

Stakeholders

  • UX: A/B testing of UX differences, where are we losing users, etc.
  • Developers/QA: What's breaking? Data to discover problems, understand what is going wrong, etc.
  • Product Management: What are the success modes/product opportunities? Are we converting users/RPs through stages of engagement?

Note: "Is the service up and functioning properly?" is not a question answered by these dashboards -- that is handled by monitoring, which is a separate function.

Meetings and Status

The focus for Persona Analytics is outlined in order of priority below:

Task Description Crew Target
Two Metrics That Matter Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard. John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane Tomlinson (browserid data source) Oct?
RP A/B Testing Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text. Ryan Feeley, John Gruen, Dan Callahan
Dialog A/B Testing Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog. Shane Tomlinson (browserid infrastructure), Katie Parlante (results display) Oct: dialog infrastructure, Nov: results display

KPI Dashboard

User Stories

Top Priorities:

  • As a UX designer, I want to know if new users are making it through the dialog and successfully signing in/up. [ new user views: funnel, etc. ]
  • As a Product Manager, I want to know how well we're doing at acquiring new users and RPs [cohort/funnel]
  • As a UX designer, I want to make a change to the UX and know if it improves the user's experience. [ A/B testing ]

Also on the horizon:

  • As a developer, I want to know the rate users run into error screens.
  • As a developer, I want to know if users experience performance problems with the service.
  • As a product manager, developer or UX designer, I want the dashboard to be reliable and updated regularly. [move to full production setup, etc.]

Code & Issue tracking

Documentation

Identity Dashboard

Open bugs:

People/Project contacts

  • Katie Parlante is driving this project, and maintaining the KPI dashboard
  • Shane Tomlinson is the main point of contact for kpi work in the browserid codebase
  • Ryan Feeley and Hannah Quay-de la Vallee are currently the primary "clients" of the KPI dashboard -- the goal is to support UX
  • The metrics team maintains the "Identity" dashboard
Retrieved from "https://wiki.mozilla.org/index.php?title=Identity/Persona-Analytics&oldid=705890"