Identity/Metrics

Last updated: 2014/02/11

Firefox Accounts

Stakeholders

• Product Managers, including PMs for services that rely on FxA: what does FxA adoption look like?
• UX: are users successful, where do they get stuck?
• QA, Dev, Ops: Is the service responsive? Where are errors happening?
• OpsSec, AppSec: Fraud detection

Nutshell Summary

Quick summary of what's planned (in priority order).

Accounts created per day

• Also logins per day, and other events that correspond to fxa-auth-server endpoints
• segmented by device, service, locale
• derived from fxa-auth-server logs: log line at each endpoint contains the raw information, processed by heka, lands in an elasticsearch index.
• device is derived from user-agent
• locale is sent on request header
• service is passed in on some endpoints

Active daily users

• derived from fxa-auth-server logs: # calls to cert/sign endpoint, bucketed by userid.
• alternative could be derived from verifier logs, again bucketed by userid
• heka can do processing to count unique userid's/day, so that userid is not stored in elastic search

UX flow analysis

• count number of people who make it to each stage of a given UX flow (per day)
• derived from fxa-auth-server logs initially, can add in events from fxa-content-server logs, etc. eventually
• not 100% clear to me yet whether we can identify the "flow" in the server, or if we need help by information passed in from the client
• ideally, we could get timing information as well (ms since the "start" of the flow), but this requires help from the client
• other segmentations come along for the ride: service, device, locale

Additional segmentations

• UX would like screen size segmentations, would require info from the client/device

Additional stages for UX flows

• Would be great to track # people who have opportunity to create account but then skip it (in FTU flows)
• Would be great to tie email validation stage into FTU flow analysis (happens at different point in time)

Sync specific metrics

• # people who make some sort of customization was mentioned, haven't thought this through
• FHS report work happening separately:

User Stories

• PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories
• QA/SE related user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories

Timeline

• Unsorted
  • First pass PM/UX metrics logged: https://github.com/mozilla/fxa-auth-server/issues/372
  • ADU metric logged
  • ADU working on dev stack
• Feb 7
  • All FxA prod data in Heka/ES/somewhere kparlante and trink can access
  • PM/UX use cases logged in bugzilla : Bugzilla
  • QA/SE use cases logged in bugzilla
  • Roadmap/timeline documented
  • kparlante to document endpoints for trink, so he can configure heka for response time use cases
  • ADU metric defined
• Feb 14
  • Response time dashboard working end to end on dev stack (with production data)
• Feb 21
  • Accounts created/day use case working on dev stack
• Feb 28

Resources

• Minimum Viable Metrics : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).
• Shared Services : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
• Svcops Timeline Planning
• Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
• https://wiki.mozilla.org/ADI

Older Notes

• https://id.etherpad.mozilla.org/fxacct-metrics
• https://id.etherpad.mozilla.org/fxacct-mvm
• https://id.etherpad.mozilla.org/fxacct-mvm-details
• https://id.etherpad.mozilla.org/fxacct-metrics-fraud-detection

Persona

• Persona Metrics