Identity/Persona-Analytics: Difference between revisions
< Identity
Jump to navigation
Jump to search
| Line 51: | Line 51: | ||
|Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog. | |Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog. | ||
|Shane Tomlinson (browserid infrastructure), Katie Parlante (results display) | |Shane Tomlinson (browserid infrastructure), Katie Parlante (results display) | ||
|Oct | |Oct? dialog infrastructure, Nov? results display | ||
|- | |- | ||
|Move KPI infrastructure to Ops team | |Move KPI infrastructure to Ops team | ||
Revision as of 19:19, 10 September 2013
Overview
The Persona team makes use of metrics to help make good decisions and stronger products.
Public Metrics
- http://www.arewepopularyet.com/ (Based on github search results)
- Third party metrics
Private Metrics
We have two dashboards that support the #signin team (LDAP required):
- KPI dashboard (Maintained by Identity team)
- Identity dashboard (Maintained by Metrics team)
Privacy
Like all of us at Mozilla, we take user privacy seriously. We only collect the data that we use for decision making, and we take care to remove personally identifying information from that data before it reaches these dashboards. While we aim to be open in our decision making, our metrics dashboards that make use of production data are currently private.
Stakeholders
- UX: A/B testing of UX differences, where are we losing users, etc.
- Developers/QA: What's breaking? Data to discover problems, understand what is going wrong, etc.
- Product Management: What are the success modes/product opportunities? Are we converting users/RPs through stages of engagement?
Note: "Is the service up and functioning properly?" is not a question answered by these dashboards -- that is handled by monitoring, which is a separate function.
Meetings and Status
- overall status updates
- UX/User adoption weekly meeting notes
- https://id.etherpad.mozilla.org/metrics-2013-08-19 Meeting with Annie about BI/DW
- A/B testing
- metrics futures discussion 6/17/13
The focus for Persona Analytics is outlined in order of priority below:
| Task | Description | Crew | Target |
|---|---|---|---|
| Two Metrics That Matter - KPI Dashboard | Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard. | John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane Tomlinson (browserid data source) | Oct? |
| RP A/B Testing | Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text. | Ryan Feeley, John Gruen, Dan Callahan | |
| Dialog A/B Testing | Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog. | Shane Tomlinson (browserid infrastructure), Katie Parlante (results display) | Oct? dialog infrastructure, Nov? results display |
| Move KPI infrastructure to Ops team | Host KPI servers in VPC, instead of on ephemeral instances maintained by dev | Gene Wood, Katie Parlante | Dec? |
| Self service access to data -- KPI | Write script to pull kpiggybank data into elasticsearch+cabana | Unscheduled -- prototype might be good freaky project | |
| Self service access to data -- pentaho data | Use heka as transport, route to an elastic search backend, use cabana (or something similar) as frontend | Unscheduled | |
| Self service access to data -- identity log data | Similar to other services, use heka as transport, route to an elastic search backend, use cabana as frontend | Unscheduled | |
| Dashboard phase II | Further work on user adoption related graphs | Unscheduled | |
| Dashboard phase III | Architecture focus, use elastic search as a datasource | Unscheduled | |
| Dashboard phase IV | New graphs enabled by new data sources | Unscheduled |
KPI Dashboard
- KPI Dashboard tasks tracked in github: issue list
- Browserid, all github issues flagged as KPI: https://github.com/mozilla/browserid/issues?labels=KPI&state=open
- identity-ops github issues:
- make RP data available: https://github.com/mozilla/identity-ops/issues/102
- setup kpiggybank for *-metrics.json data: https://github.com/mozilla/identity-ops/issues/117
- move kpiggybanks to production vpc: https://github.com/mozilla/identity-ops/issues/92
- Privacy Reviews:
- review kickoff for RP/email verification features: https://bugzilla.mozilla.org/show_bug.cgi?id=903581
User Stories
Top Priorities:
- As a UX designer, I want to know if new users are making it through the dialog and successfully signing in/up. [ new user views: funnel, etc. ]
- As a Product Manager, I want to know how well we're doing at acquiring new users and RPs [cohort/funnel]
- As a UX designer, I want to make a change to the UX and know if it improves the user's experience. [ A/B testing ]
Also on the horizon:
- As a developer, I want to know the rate users run into error screens.
- As a developer, I want to know if users experience performance problems with the service.
- As a product manager, developer or UX designer, I want the dashboard to be reliable and updated regularly. [move to full production setup, etc.]
Code & Issue tracking
Documentation
- https://wiki.mozilla.org/Identity/BrowserID/KPI_Dashboard (a bit out of date right now, but the overall picture is right)
Identity Dashboard
Open bugs:
People/Project contacts
- Katie Parlante is driving this project, and maintaining the KPI dashboard
- Shane Tomlinson is the main point of contact for kpi work in the browserid codebase
- Ryan Feeley and Hannah Quay-de la Vallee are currently the primary "clients" of the KPI dashboard -- the goal is to support UX
- The metrics team maintains the "Identity" dashboard