Identity/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(40 intermediate revisions by the same user not shown)
Line 3: Line 3:
= Firefox Accounts =
= Firefox Accounts =


== Stakeholders ==
== Dashboards ==
* Product Managers, including PMs for services that rely on FxA: what does FxA adoption look like?
* UX: are users successful, where do they get stuck?


== Nutshell Summary ==
* Links to all dashboards here: https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Accounts+Logging+and+Monitoring+Services
 
* Docs helpful for building kibana dashboards:
[[/FxA_Metrics_Nutshell_Summary|Nutshell Summary]]
** Query syntax used by kibana dashboards: http://www.lucenetutorial.com/lucene-query-syntax.html


== User Stories ==
== User Stories ==


Priorities & open issues ([https://bugzilla.mozilla.org/buglist.cgi?list_id=9407994&short_desc=Metrics%20Story&resolution=---&resolution=DUPLICATE&classification=Client%20Software&query_format=advanced&short_desc_type=allwordssubstr&component=Server%3A%20Firefox%20Accounts&product=Mozilla%20Services All Metrics Story Bugs In Bugzilla])
Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29).


'''Infrastructure'''
'''Infrastructure'''
* Blocking on development environment: logging stack with production data (heka, elasticsearch, kibana)
# general query access to nginx, server log files (via kibana)
 
# facility for writing custom alerts, monitors (via heka/lua)
'''0. One Metric That Matters: Total number of Firefox Accounts'''
* https://bugzilla.mozilla.org/show_bug.cgi?id=972105


'''1. Accounts created per day'''
'''Metrics/Growth''' (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385)
* https://bugzilla.mozilla.org/show_bug.cgi?id=968973
# '''One Metric That Matters!''' Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105
* '''Higher Priority Segmentations''': https://bugzilla.mozilla.org/show_bug.cgi?id=968990
# Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973
* '''Lower Priority Segmentatons''': https://bugzilla.mozilla.org/show_bug.cgi?id=969003
#* Higher priority segmentations: https://bugzilla.mozilla.org/show_bug.cgi?id=968990
* How is device name derived? What devices do we explicitly want to track/monitor? -trink
#* Lower priority segmentatons: https://bugzilla.mozilla.org/show_bug.cgi?id=969003
** "agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
#* We'll get other auth server endpoints as well, in particular account/login and cert/sign
** "agent":"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
# Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983
** "agent":"curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5"
# UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority)
* What locales do we explicitly want to track/monitor? -trink
* Is there a fixed list of services? Again, if not which ones do we explicitly want to track/monitor -trink
* Above q's answered in above bugs -kparlante


'''2. Active daily users'''
'''Monitoring/Health''' (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065)
* https://bugzilla.mozilla.org/show_bug.cgi?id=968983
# Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130
* heka can do processing to count unique userid's/day, so that userid is not stored in elastic search
# Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851
** note: The sandbox is intentionally resource constrained, it is highly unlikely it will have enough memory to track every active user in a day as usage increases. -trink.
# Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867
# hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329


'''UX flow analysis'''
'''Fraud/Security'''
* https://bugzilla.mozilla.org/show_bug.cgi?id=969007
# observe/alert mass account creation
'''Sync specific metrics'''
# observe/alert large # of brute force attempts (also observing any that succeed)
* FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600)


*
'''Dev/Debug'''
# monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409


== Dependencies ==
== Repositories ==


* First pass PM/UX metrics logged: https://github.com/mozilla/fxa-auth-server/issues/372
* Heka/elasticsearch/kibana configuration: https://github.com/mozilla-services/puppet-config
* All FxA prod data in Heka/ES/somewhere kparlante and trink can access
* fxa-auth-server: https://github.com/mozilla/fxa-auth-server/issues?labels=metrics
* fxa-auth-server API changes to collect pass through information: https://github.com/mozilla/fxa-auth-server/issues/351


== Timeline ==
== Timeline ==


* [https://mana.mozilla.org/wiki/display/SVCOPS/SvcOps+FxA+and+Sync+Timeline+Planning Svcops Timeline Planning]
* [https://mana.mozilla.org/wiki/display/SVCOPS/SvcOps+FxA+and+Sync+Timeline+Planning Svcops Timeline Planning]
* Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
== Future Work/Plans ==
* Telemetry from devices
* Pirate Metrics (AARRR), ala: https://wiki.mozilla.org/MDN/Development/Metrics
* A/B testing


== Resources ==
== Resources ==


* Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
* [https://bugzilla.mozilla.org/buglist.cgi?list_id=9407994&short_desc=Metrics%20Story&resolution=---&resolution=DUPLICATE&classification=Client%20Software&query_format=advanced&short_desc_type=allwordssubstr&component=Server%3A%20Firefox%20Accounts&product=Mozilla%20Services all metrics story bugs in bugzilla]
* [[/FxA_Metrics_Nutshell_Summary|Nutshell Summary]] written for a Feb 2014 meeting
* PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories  
* PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories  
* QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories
* QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories
* OpSec user stories: https://mana.mozilla.org/wiki/display/SECURITY/Firefox+Accounts+Security+Design+Review#FirefoxAccountsSecurityDesignReview-SecurityConcepts
* [https://mana.mozilla.org/wiki/display/SVCOPS/Shared+Services Shared Services] : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
* [https://mana.mozilla.org/wiki/display/SVCOPS/Shared+Services Shared Services] : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
* FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600)
* https://wiki.mozilla.org/ADI
* https://wiki.mozilla.org/ADI
* [https://wiki.mozilla.org/Identity/Firefox-Accounts/Minimum_Viable_Metrics Minimum Viable Metrics] : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).
* [https://wiki.mozilla.org/Identity/Firefox-Accounts/Minimum_Viable_Metrics Minimum Viable Metrics] : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).

Latest revision as of 19:18, 21 March 2014

Last updated: 2014/03/21

Firefox Accounts

Dashboards

User Stories

Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29).

Infrastructure

  1. general query access to nginx, server log files (via kibana)
  2. facility for writing custom alerts, monitors (via heka/lua)

Metrics/Growth (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385)

  1. One Metric That Matters! Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105
  2. Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973
  3. Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983
  4. UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority)

Monitoring/Health (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065)

  1. Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130
  2. Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851
  3. Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867
  4. hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329

Fraud/Security

  1. observe/alert mass account creation
  2. observe/alert large # of brute force attempts (also observing any that succeed)

Dev/Debug

  1. monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409

Repositories

Timeline

Future Work/Plans

Resources

Older Notes

Sync.next

Persona

Retrieved from "https://wiki.mozilla.org/index.php?title=Identity/Metrics&oldid=954895"