Identity/Persona-Analytics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(42 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Overview ==
{{LastUpdated}}
The Persona team makes use of metrics to help make good decisions and stronger products.


== Public Metrics ==
== Internal (Private) Metrics ==
 
* http://www.arewepopularyet.com/ (Based on github search results)
* [[/ThirdPartyMetrics | Third party metrics]]
 
== Private Metrics ==


We have two dashboards that support the #signin team (LDAP required):
We have two dashboards that support the #signin team (LDAP required):
Line 13: Line 7:
* [https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics2&path=identity&file=identity.wcdf Identity dashboard] (Maintained by Metrics team)
* [https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics2&path=identity&file=identity.wcdf Identity dashboard] (Maintained by Metrics team)


==== Privacy ====
== Public Metrics ==
Like all of us at Mozilla, we take user privacy seriously. We only collect the data that we use for decision making, and we take care to remove personally identifying information from that data before it reaches these dashboards. While we aim to be open in our decision making, our metrics dashboards that make use of production data are currently private.
* Persona related metrics collected by other sites: [[/ThirdPartyMetrics | Third party metrics]]


== Stakeholders ==
== Stakeholders ==
* UX: A/B testing of UX differences, where are we losing users, etc.
* Project Management: Are we growing? Are we succeeding at our goals? Do we have RPs in the pipeline? Are we converting them?
* Developers/QA: What's breaking? Data to discover problems, understand what is going wrong, etc.
* UX: Do users proceed successfully through the dialog? Does UX A vs UX B improve our bounce rate?
* Product Management: What are the success modes/product opportunities? Are we converting users/RPs through stages of engagement?  
* Developers/QA: What's breaking? Are the dialog response times acceptable? How frequent is a particular bug/issue?


Note: "Is the service up and functioning properly?" is not a question answered by these dashboards -- that is handled by monitoring, which is a separate function.
Note: "Is the service up and functioning properly?" is not a question answered by these dashboards -- that is handled by monitoring, which is a separate function.


== Meetings and Status ==
== Meetings and Status ==
* [[/StatusUpdates|overall status updates]]
* [https://id.etherpad.mozilla.org/kpi-meeting-notes Weekly meeting notes]
* [https://id.etherpad.mozilla.org/kpi-meeting-notes UX/User adoption weekly meeting notes]
* https://id.etherpad.mozilla.org/metrics-2013-08-19 Meeting with Annie about BI/DW
* https://id.etherpad.mozilla.org/metrics-2013-08-19 Meeting with Annie about BI/DW
* [https://groups.google.com/forum/#!topic/mozilla.dev.identity/ydF0QHzJ_tU A/B testing]
* [https://groups.google.com/forum/#!topic/mozilla.dev.identity/ydF0QHzJ_tU A/B testing]
* [https://id.etherpad.mozilla.org/kpi-privacy-review-2013 metrics futures discussion 6/17/13]
* [https://id.etherpad.mozilla.org/kpi-privacy-review-2013 metrics futures discussion 6/17/13]
* [[/StatusUpdates|old status updates]]


The focus for Persona Analytics is outlined in order of priority below:
== Projects ==
A list of all Persona Metrics projects we'd like to do someday:


{| class="fullwidth-table"
{| class="fullwidth-table"
Line 38: Line 34:
!Target
!Target
|-
|-
|Two Metrics That Matter
|Two Metrics That Matter - KPI Dashboard
|
|Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard.
|
|John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane Tomlinson (browserid data source)
|Postponed
|-
|RP A/B Testing
|Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text.
|Ryan Feeley, John Gruen, Dan Callahan, Katie Parlante
|Postponed
|-
|Dialog A/B Testing
|Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog.
|Shane Tomlinson (browserid infrastructure), Katie Parlante (results display)
|Unscheduled
|-
|Move KPI infrastructure to Ops team
|Host KPI servers in VPC, instead of on ephemeral instances maintained by dev.
|Gene Wood, Katie Parlante
|Postponed
|-
|Self service access to data
|Use RabbitMQ or Heka as transport, route to an elastic search backend, use kibana (or similar) as frontend
|
|
|Unscheduled -- prototype might be good freaky project
|-
|-
|System architecture cleanup
|Replace https (transport) with Heka/RabbitMQ and couchdb (data store) with elasticsearch .
|
|
|
|Unscheduled
|
|-
|-
|Dashboard Improvements
|Performance visualizations, email verification stats, etc.
|
|
|
|Unscheduled
|
|
|}
|}


Line 59: Line 76:
* identity-ops github issues:
* identity-ops github issues:
** make RP data available: https://github.com/mozilla/identity-ops/issues/102  
** make RP data available: https://github.com/mozilla/identity-ops/issues/102  
** setup kpiggybank for *-metrics.json data: https://github.com/mozilla/identity-ops/issues/117
** move kpiggybanks to production vpc: https://github.com/mozilla/identity-ops/issues/92
** move kpiggybanks to production vpc: https://github.com/mozilla/identity-ops/issues/92
* Privacy Reviews:
* Privacy Reviews:
** review kickoff for RP/email verification features: https://bugzilla.mozilla.org/show_bug.cgi?id=903581
** review kickoff for RP/email verification features: https://bugzilla.mozilla.org/show_bug.cgi?id=909980
 
* UX Redesign (TMTM): https://github.com/johngruen/kpi_des
=== User Stories ===
 
Top Priorities:
* As a UX designer, I want to know if new users are making it through the dialog and successfully signing in/up. [ new user views: funnel, etc. ]
* As a Product Manager, I want to know how well we're doing at acquiring new users and RPs [cohort/funnel]
* As a UX designer, I want to make a change to the UX and know if it improves the user's experience. [ A/B testing ]
 
Also on the horizon:
* As a developer, I want to know the rate users run into error screens.
* As a developer, I want to know if users experience performance problems with the service.
* As a product manager, developer or UX designer, I want the dashboard to be reliable and updated regularly. [move to full production setup, etc.]


=== Code & Issue tracking ===
=== Code & Issue tracking ===
Line 81: Line 86:


=== Documentation ===
=== Documentation ===
* https://www.lucidchart.com/documents/view/42df-90fc-5240a53c-8ea1-5c2f0a0089aa (Diagram)
* https://wiki.mozilla.org/Identity/BrowserID/KPI_Dashboard (a bit out of date right now, but the overall picture is right)
* https://wiki.mozilla.org/Identity/BrowserID/KPI_Dashboard (a bit out of date right now, but the overall picture is right)


== Identity Dashboard ==
== Identity Dashboard ==
Open bugs:
Open bugs:
*[https://bugzilla.mozilla.org/show_bug.cgi?id=885467 Recognize FFOS]
*[https://bugzilla.mozilla.org/show_bug.cgi?id=935709 Identity Dashboard missing data starting 10/29]
*[https://bugzilla.mozilla.org/show_bug.cgi?id=885423 Screen out testing RPs ]
*[https://bugzilla.mozilla.org/show_bug.cgi?id=885423 Screen out testing RPs ]
== People/Project contacts ==
* Katie Parlante is driving this project, and maintaining the KPI dashboard
* Shane Tomlinson is the main point of contact for kpi work in the browserid codebase
* Ryan Feeley and Hannah Quay-de la Vallee are currently the primary "clients" of the KPI dashboard -- the goal is to support UX
* The metrics team maintains the "Identity" dashboard

Latest revision as of 20:32, 22 November 2013

Last updated: 2013/11/22

Internal (Private) Metrics

We have two dashboards that support the #signin team (LDAP required):

Public Metrics

Stakeholders

  • Project Management: Are we growing? Are we succeeding at our goals? Do we have RPs in the pipeline? Are we converting them?
  • UX: Do users proceed successfully through the dialog? Does UX A vs UX B improve our bounce rate?
  • Developers/QA: What's breaking? Are the dialog response times acceptable? How frequent is a particular bug/issue?

Note: "Is the service up and functioning properly?" is not a question answered by these dashboards -- that is handled by monitoring, which is a separate function.

Meetings and Status

Projects

A list of all Persona Metrics projects we'd like to do someday:

Task Description Crew Target
Two Metrics That Matter - KPI Dashboard Two metrics that matter: one for RP adoption and one for user adoption. These metrics should be accessible in a usable (intelligibly badass) dashboard. John Gruen (UX and dashboard coding), Ryan Feeley (UX), Katie Parlante (dashboard coding, lead), Shane Tomlinson (browserid data source) Postponed
RP A/B Testing Co-ordinating A/B testing hosted by RPs. Tests placement of initial Persona buttons/links/introductory text. Ryan Feeley, John Gruen, Dan Callahan, Katie Parlante Postponed
Dialog A/B Testing Test different options on the first screen, optimizing to minimize bounce rate. Create our own infrastructure to run tests, as we don't want to use 3rd party options in our dialog. Shane Tomlinson (browserid infrastructure), Katie Parlante (results display) Unscheduled
Move KPI infrastructure to Ops team Host KPI servers in VPC, instead of on ephemeral instances maintained by dev. Gene Wood, Katie Parlante Postponed
Self service access to data Use RabbitMQ or Heka as transport, route to an elastic search backend, use kibana (or similar) as frontend Unscheduled -- prototype might be good freaky project
System architecture cleanup Replace https (transport) with Heka/RabbitMQ and couchdb (data store) with elasticsearch . Unscheduled
Dashboard Improvements Performance visualizations, email verification stats, etc. Unscheduled

KPI Dashboard

Code & Issue tracking

Documentation

Identity Dashboard

Open bugs:

Retrieved from "https://wiki.mozilla.org/index.php?title=Identity/Persona-Analytics&oldid=768645"