Identity/Metrics

Last updated: 2014/03/21

Firefox Accounts

Dashboards

• Links to all dashboards here: https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Accounts+Logging+and+Monitoring+Services
• Docs helpful for building kibana dashboards:
  • Query syntax used by kibana dashboards: http://www.lucenetutorial.com/lucene-query-syntax.html

User Stories

Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29).

Infrastructure

1. general query access to nginx, server log files (via kibana)
2. facility for writing custom alerts, monitors (via heka/lua)

Metrics/Growth (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385)

1. One Metric That Matters! Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105
2. Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973
   • Higher priority segmentations: https://bugzilla.mozilla.org/show_bug.cgi?id=968990
   • Lower priority segmentatons: https://bugzilla.mozilla.org/show_bug.cgi?id=969003
   • We'll get other auth server endpoints as well, in particular account/login and cert/sign
3. Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983
4. UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority)

Monitoring/Health (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065)

1. Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130
2. Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851
3. Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867
4. hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329

Fraud/Security

1. observe/alert mass account creation
2. observe/alert large # of brute force attempts (also observing any that succeed)

Dev/Debug

1. monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409

Repositories

• Heka/elasticsearch/kibana configuration: https://github.com/mozilla-services/puppet-config
• fxa-auth-server: https://github.com/mozilla/fxa-auth-server/issues?labels=metrics

Timeline

• Svcops Timeline Planning
• Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring

Future Work/Plans

• Telemetry from devices
• Pirate Metrics (AARRR), ala: https://wiki.mozilla.org/MDN/Development/Metrics
• A/B testing

Resources

• all metrics story bugs in bugzilla
• Nutshell Summary written for a Feb 2014 meeting
• PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories
• QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories
• OpSec user stories: https://mana.mozilla.org/wiki/display/SECURITY/Firefox+Accounts+Security+Design+Review#FirefoxAccountsSecurityDesignReview-SecurityConcepts
• Shared Services : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
• FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600)
• https://wiki.mozilla.org/ADI
• Minimum Viable Metrics : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).

Older Notes

• https://id.etherpad.mozilla.org/fxacct-metrics
• https://id.etherpad.mozilla.org/fxacct-mvm
• https://id.etherpad.mozilla.org/fxacct-mvm-details
• https://id.etherpad.mozilla.org/fxacct-metrics-fraud-detection

Persona

• Persona Metrics