Identity/Persona-Analytics
< Identity
Overview
The Persona team makes use of metrics to help make good decisions and stronger products.
Internal (Private) Metrics
We have two dashboards that support the #signin team (LDAP required):
- KPI dashboard (Maintained by Identity team)
- Identity dashboard (Maintained by Metrics team)
Privacy
Like all of us at Mozilla, we take user privacy seriously. We only collect the data that we use for decision making, and we take care to remove personally identifying information from that data before it reaches these dashboards. While we aim to be open in our decision making, our metrics dashboards that make use of production data are currently private.
Public Metrics
- Exploratory project -- based on github search results: http://www.arewepopularyet.com/
- Persona related metrics collected by other sites: Third party metrics
Stakeholders
- Project leadership: Are we growing? Are we succeeding at our goals?
- UX: Do users proceed successfully through the dialog? Does UX A vs UX B improve our bounce rate?
- Developers/QA: What's breaking? Are the dialog response times acceptable? How frequent is a particular bug/issue?
- Product Management: What are the success modes/product opportunities? Are we converting users/RPs through stages of engagement?
Note: "Is the service up and functioning properly?" is not a question answered by these dashboards -- that is handled by monitoring, which is a separate function.
Meetings and Status
- Weekly meeting notes
- https://id.etherpad.mozilla.org/metrics-2013-08-19 Meeting with Annie about BI/DW
- A/B testing
- metrics futures discussion 6/17/13
- old status updates
The focus for Persona Analytics is outlined in order of priority below:
| Task | Description | Crew | Target |
|---|---|---|---|
| Two Metrics That Matter - KPI Dashboard | Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard. | John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane Tomlinson (browserid data source) | Oct? |
| RP A/B Testing | Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text. | Ryan Feeley, John Gruen, Dan Callahan, Katie Parlante | |
| Dialog A/B Testing | Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog. | Shane Tomlinson (browserid infrastructure), Katie Parlante (results display) | dialog infrastructure: Oct? results display: Nov? |
| Move KPI infrastructure to Ops team | Host KPI servers in VPC, instead of on ephemeral instances maintained by dev | Gene Wood, Katie Parlante | Dec? |
| Self service access to data -- KPI | Write script to pull kpiggybank data into elasticsearch + out-of-box front end (like kibana) | Unscheduled -- prototype might be good freaky project | |
| Self service access to data -- pentaho data | Use heka as transport, route to an elastic search backend, use kibana as frontend | Unscheduled | |
| Self service access to data -- identity log data | Similar to other services, use heka as transport, route to an elastic search backend, use kibana as frontend | Unscheduled | |
| Email verification in Dashboard | Use email verification *metrics data to cross check the KPI data | Unscheduled | |
| System architecture cleanup | Replace https (transport) and couchdb (data store) with hekka and elasticsearch across the board | Unscheduled | |
| Replace pentaho dashboards | Add vanity dashboards, etc. so that we can shut down kettle scripts, once we no longer need the pentaho dashboards | Unscheduled |
KPI Dashboard
- KPI Dashboard tasks tracked in github: issue list
- Browserid, all github issues flagged as KPI: https://github.com/mozilla/browserid/issues?labels=KPI&state=open
- identity-ops github issues:
- make RP data available: https://github.com/mozilla/identity-ops/issues/102
- move kpiggybanks to production vpc: https://github.com/mozilla/identity-ops/issues/92
- Privacy Reviews:
- review kickoff for RP/email verification features: https://bugzilla.mozilla.org/show_bug.cgi?id=909980
User Stories
Top Priorities:
- As a UX designer, I want to know if new users are making it through the dialog and successfully signing in/up. [ new user views: funnel, etc. ]
- As a Product Manager, I want to know how well we're doing at acquiring new users and RPs [cohort/funnel]
- As a UX designer, I want to make a change to the UX and know if it improves the user's experience. [ A/B testing ]
Also on the horizon:
- As a developer, I want to know the rate users run into error screens.
- As a developer, I want to know if users experience performance problems with the service.
- As a product manager, developer or UX designer, I want the dashboard to be reliable and updated regularly. [move to full production setup, etc.]
Code & Issue tracking
Documentation
- https://wiki.mozilla.org/Identity/BrowserID/KPI_Dashboard (a bit out of date right now, but the overall picture is right)
Identity Dashboard
Open bugs:
People/Project contacts
- Katie Parlante is driving this project, and maintaining the KPI dashboard
- Shane Tomlinson is the main point of contact for kpi work in the browserid codebase
- Ryan Feeley and Hannah Quay-de la Vallee are currently the primary "clients" of the KPI dashboard -- the goal is to support UX
- The metrics team maintains the "Identity" dashboard