Identity/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(92 intermediate revisions by 2 users not shown)
Line 3: Line 3:
= Firefox Accounts =
= Firefox Accounts =


== Stakeholders ==
== Dashboards ==
* Product Managers, including PMs for services that rely on FxA: what does FxA adoption look like?
* UX: are users successful, where do they get stuck?
* QA, Dev, Ops: Is the service responsive? Where are errors happening?
* OpsSec, AppSec: Fraud detection


== Use Cases ==
* Links to all dashboards here: https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Accounts+Logging+and+Monitoring+Services
* Docs helpful for building kibana dashboards:
** Query syntax used by kibana dashboards: http://www.lucenetutorial.com/lucene-query-syntax.html


* PM/UX related use cases: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories
== User Stories ==
* QA/SE related use cases: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories
 
Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29).
 
'''Infrastructure'''
# general query access to nginx, server log files (via kibana)
# facility for writing custom alerts, monitors (via heka/lua)
 
'''Metrics/Growth''' (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385)
# '''One Metric That Matters!''' Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105
# Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973
#* Higher priority segmentations: https://bugzilla.mozilla.org/show_bug.cgi?id=968990
#* Lower priority segmentatons: https://bugzilla.mozilla.org/show_bug.cgi?id=969003
#* We'll get other auth server endpoints as well, in particular account/login and cert/sign
# Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983
# UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority)
 
'''Monitoring/Health''' (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065)
# Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130
# Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851
# Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867
# hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329
 
'''Fraud/Security'''
# observe/alert mass account creation
# observe/alert large # of brute force attempts (also observing any that succeed)
 
'''Dev/Debug'''
# monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409
 
== Repositories ==
 
* Heka/elasticsearch/kibana configuration: https://github.com/mozilla-services/puppet-config
* fxa-auth-server: https://github.com/mozilla/fxa-auth-server/issues?labels=metrics


== Timeline ==
== Timeline ==


* Unsorted
* [https://mana.mozilla.org/wiki/display/SVCOPS/SvcOps+FxA+and+Sync+Timeline+Planning Svcops Timeline Planning]
** First pass PM/UX metrics logged: https://github.com/mozilla/fxa-auth-server/issues/372
* Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
** ADU metric logged
 
** ADU working on dev stack
== Future Work/Plans ==
* Feb 7
* Telemetry from devices
** All FxA prod data in Heka/ES/somewhere kparlante and trink can access
* Pirate Metrics (AARRR), ala: https://wiki.mozilla.org/MDN/Development/Metrics
** <strike>PM/UX use cases logged in bugzilla</strike> : [https://bugzilla.mozilla.org/buglist.cgi?list_id=9407994&short_desc=[Metrics%20Story]&resolution=---&resolution=DUPLICATE&classification=Client%20Software&query_format=advanced&short_desc_type=allwordssubstr&component=Server%3A%20Firefox%20Accounts&product=Mozilla%20Services Stories in Bugzilla]
* A/B testing
** QA/SE use cases logged in bugzilla
** Roadmap/timeline documented
** <strike>kparlante to document endpoints for trink, so he can configure heka for response time use cases</strike>
** <strike>ADU metric defined</strike>
* Feb 14
** Response time dashboard working end to end on dev stack (with production data)
* Feb 21
** Accounts created/day use case working on dev stack
* Feb 28


== Resources ==
== Resources ==


* [https://wiki.mozilla.org/Identity/Firefox-Accounts/Minimum_Viable_Metrics Minimum Viable Metrics] : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=9407994&short_desc=Metrics%20Story&resolution=---&resolution=DUPLICATE&classification=Client%20Software&query_format=advanced&short_desc_type=allwordssubstr&component=Server%3A%20Firefox%20Accounts&product=Mozilla%20Services all metrics story bugs in bugzilla]
* [[/FxA_Metrics_Nutshell_Summary|Nutshell Summary]] written for a Feb 2014 meeting
* PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories
* QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories
* OpSec user stories: https://mana.mozilla.org/wiki/display/SECURITY/Firefox+Accounts+Security+Design+Review#FirefoxAccountsSecurityDesignReview-SecurityConcepts
* [https://mana.mozilla.org/wiki/display/SVCOPS/Shared+Services Shared Services] : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
* [https://mana.mozilla.org/wiki/display/SVCOPS/Shared+Services Shared Services] : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
* [https://mana.mozilla.org/wiki/display/SVCOPS/SvcOps+FxA+and+Sync+Timeline+Planning Svcops Timeline Planning]
* FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600)
* Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
* https://wiki.mozilla.org/ADI
* https://wiki.mozilla.org/ADI
* [https://wiki.mozilla.org/Identity/Firefox-Accounts/Minimum_Viable_Metrics Minimum Viable Metrics] : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).


== Older Notes ==
== Older Notes ==

Latest revision as of 19:18, 21 March 2014

Last updated: 2014/03/21

Firefox Accounts

Dashboards

User Stories

Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29).

Infrastructure

  1. general query access to nginx, server log files (via kibana)
  2. facility for writing custom alerts, monitors (via heka/lua)

Metrics/Growth (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385)

  1. One Metric That Matters! Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105
  2. Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973
  3. Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983
  4. UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority)

Monitoring/Health (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065)

  1. Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130
  2. Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851
  3. Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867
  4. hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329

Fraud/Security

  1. observe/alert mass account creation
  2. observe/alert large # of brute force attempts (also observing any that succeed)

Dev/Debug

  1. monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409

Repositories

Timeline

Future Work/Plans

Resources

Older Notes

Sync.next

Persona

Retrieved from "https://wiki.mozilla.org/index.php?title=Identity/Metrics&oldid=954895"