Identity/Persona-Analytics/StatusUpdates

Status Updates

Detailed meeting notes here: https://id.etherpad.mozilla.org/kpi-meeting-notes

Sept 10

General

• Prioritized the work, splitting everything out into projects. Philosophy for prioritization: Usable dashboard/metrics that assist day to day decision making first, self service access to data next, infrastructure improvements when availability becomes important.
• Shane actively working on KPI for a few months.

==== KPI Dashboard

Aug 20, 2013

KPI Dashboard

• New visualization: General progress through dialog, general bounce rate
• John Gruen to help with UX/Design/Front-end coding
• Next: 4 star bugs (in particular new-idp-users-per-day, bounce rate for new users): https://github.com/mozilla/kpi-dashboard/issues?labels=%E2%98%85%E2%98%85%E2%98%85%E2%98%85&page=1&state=open

A/B Testing

• Hannah analyzing the results of A/B test run with help of codingbat.com
• Interest in more A/B tests with the help of RPs
• Not using Optimizely for dialog testing, implement ourselves
• Next: Adding dialog A/B testing to next week's meeting agenda.
• Next: Meet up with communications team, Be more structured about RP A/B testing efforts, Optimizely as a resource?

RP Adoption

• Austin has a proposal for the "metric that matters"
• Jared's working on making /sign_in and /verify metrics data available to dashboard
• Will soon be blocking on privacy review: https://bugzilla.mozilla.org/show_bug.cgi?id=903581

General investigations

• Katie working with Gene to audit the RP data: https://github.com/mozilla/identity-ops/issues/119
• Katie looked into whether or not performance has been degrading over time (answer appears to be no, but more users are using IdPs, which have worse performance than fallback).
• Jared's freaky project
• Next: Turn initial performance investigations into graphs: https://github.com/mozilla/kpi-dashboard/issues/74, https://github.com/mozilla/kpi-dashboard/issues/73

Ops work

• Next: Need to get an RPM for kpiggybank and kpi dashboard: https://github.com/mozilla/identity-ops/issues/92

July 30, 2013

• KPI Dashboard (User adoption, UX)
  • New data landed from browserid, new user visualizations have all been updated
  • FirefoxOS and desktop/tablet/mobile segmentations landed
  • New visualization: new user per day/week (only fallback users)
  • Scoped out new queries for bounce rate (and overall progress through the dialog, including error rates)
  • Next steps:
    • Finish bounce rate visualization
    • Incorporate "primary" users better into visualizations (and/or new visualizations)
    • Full work list here: https://github.com/mozilla/kpi-dashboard/issues?state=open
  • This project is my highest priority right now

• A/B testing
  • Proposal: use Optimizely. Would need to pass security and privacy review.
  • Perhaps do A/B testing on buttons with help of "friendly" RP -- sumo seems like an ideal candidate
  • Next steps:
    • getting set up with Optimizely account today
    • make sure Optimizely passes the sniff test before kicking off privacy/security reviews
    • Prototype?
    • Next steps for making SUMO collaboration happen?

• RP adoption
  • proposed two visualizations: new RP per week and RP churn
  • Next steps:
    • dependency on https://github.com/mozilla/identity-ops/issues/102 (make metrics data available to kpi dashboard)
    • above needs privacy review

• General problem solving support for sign-in team (where kpi data might help)
  • Next steps: custom queries on demand

• Other cross functional team issues:
  • Metrics team has a new dashboard in the works, waiting on us to give feedback
  • Need to sync up with developer engagement team about Persona metric used
  • Identity-ops starting work on https://github.com/mozilla/identity-ops/issues/92 (kpiggybank on production vpc)

• Weekly meeting Tuesdays at 1:30 to discuss UX/User adoption metrics (KPI dashboard)