Identity/Metrics: Difference between revisions
< Identity
Jump to navigation
Jump to search
| (27 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
= Firefox Accounts = | = Firefox Accounts = | ||
== | == Dashboards == | ||
* | |||
* | * Links to all dashboards here: https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Accounts+Logging+and+Monitoring+Services | ||
* Docs helpful for building kibana dashboards: | |||
** Query syntax used by kibana dashboards: http://www.lucenetutorial.com/lucene-query-syntax.html | |||
== User Stories == | == User Stories == | ||
Metrics/Monitoring requirements/priorities driven by user stories | Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29). | ||
'''Infrastructure''' | '''Infrastructure''' | ||
# general query access to nginx, server log files (via kibana) | |||
# facility for writing custom alerts, monitors (via heka/lua) | |||
'''Metrics/Growth''' | '''Metrics/Growth''' (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385) | ||
# '''One Metric That Matters | # '''One Metric That Matters!''' Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105 | ||
# | # Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973 | ||
#* | #* Higher priority segmentations: https://bugzilla.mozilla.org/show_bug.cgi?id=968990 | ||
#* | #* Lower priority segmentatons: https://bugzilla.mozilla.org/show_bug.cgi?id=969003 | ||
#* We'll get other auth server endpoints as well, in particular account/login and cert/sign | #* We'll get other auth server endpoints as well, in particular account/login and cert/sign | ||
# | # Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983 | ||
# | # UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority) | ||
'''Monitoring/Health''' | '''Monitoring/Health''' (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065) | ||
# | # Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130 | ||
# Timing information for create/login | # Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851 | ||
# Email queue and latency | # Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867 | ||
# hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329 | |||
'''Fraud/Security''' | '''Fraud/Security''' | ||
# observe mass account creation | # observe/alert mass account creation | ||
# observe large # of brute force attempts | # observe/alert large # of brute force attempts (also observing any that succeed) | ||
'''Dev/Debug''' | '''Dev/Debug''' | ||
# monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409 | |||
# monitor V8 heap stats | |||
== | == Repositories == | ||
* | * Heka/elasticsearch/kibana configuration: https://github.com/mozilla-services/puppet-config | ||
* fxa-auth-server: https://github.com/mozilla/fxa-auth-server/issues?labels=metrics | |||
* fxa-auth-server | |||
== Timeline == | == Timeline == | ||
* [https://mana.mozilla.org/wiki/display/SVCOPS/SvcOps+FxA+and+Sync+Timeline+Planning Svcops Timeline Planning] | * [https://mana.mozilla.org/wiki/display/SVCOPS/SvcOps+FxA+and+Sync+Timeline+Planning Svcops Timeline Planning] | ||
* Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring | |||
== Future Work/Plans == | |||
* Telemetry from devices | |||
* Pirate Metrics (AARRR), ala: https://wiki.mozilla.org/MDN/Development/Metrics | |||
* A/B testing | |||
== Resources == | == Resources == | ||
* | * [https://bugzilla.mozilla.org/buglist.cgi?list_id=9407994&short_desc=Metrics%20Story&resolution=---&resolution=DUPLICATE&classification=Client%20Software&query_format=advanced&short_desc_type=allwordssubstr&component=Server%3A%20Firefox%20Accounts&product=Mozilla%20Services all metrics story bugs in bugzilla] | ||
* [[/FxA_Metrics_Nutshell_Summary|Nutshell Summary]] written for a Feb 2014 meeting | |||
* PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories | * PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories | ||
* QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories | * QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories | ||
* OpSec user stories: https://mana.mozilla.org/wiki/display/SECURITY/Firefox+Accounts+Security+Design+Review#FirefoxAccountsSecurityDesignReview-SecurityConcepts | |||
* [https://mana.mozilla.org/wiki/display/SVCOPS/Shared+Services Shared Services] : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one. | * [https://mana.mozilla.org/wiki/display/SVCOPS/Shared+Services Shared Services] : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one. | ||
* FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600) | * FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600) | ||
Latest revision as of 19:18, 21 March 2014
Last updated: 2014/03/21
Firefox Accounts
Dashboards
- Links to all dashboards here: https://mana.mozilla.org/wiki/display/SVCOPS/Firefox+Accounts+Logging+and+Monitoring+Services
- Docs helpful for building kibana dashboards:
- Query syntax used by kibana dashboards: http://www.lucenetutorial.com/lucene-query-syntax.html
User Stories
Metrics/Monitoring requirements/priorities as driven by user stories. Below is the work planned for go live (Fx29).
Infrastructure
- general query access to nginx, server log files (via kibana)
- facility for writing custom alerts, monitors (via heka/lua)
Metrics/Growth (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=977385)
- One Metric That Matters! Total number of Firefox Accounts : https://bugzilla.mozilla.org/show_bug.cgi?id=972105
- Accounts created per day : https://bugzilla.mozilla.org/show_bug.cgi?id=968973
- Higher priority segmentations: https://bugzilla.mozilla.org/show_bug.cgi?id=968990
- Lower priority segmentatons: https://bugzilla.mozilla.org/show_bug.cgi?id=969003
- We'll get other auth server endpoints as well, in particular account/login and cert/sign
- Active daily users : https://bugzilla.mozilla.org/show_bug.cgi?id=968983
- UX flow analysis : https://bugzilla.mozilla.org/show_bug.cgi?id=969007 (lower priority)
Monitoring/Health (Tracking Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=972065)
- Users failing to create/sign in to service : https://bugzilla.mozilla.org/show_bug.cgi?id=974130
- Timing information for create/login : https://bugzilla.mozilla.org/show_bug.cgi?id=976851
- Email queue and latency : https://bugzilla.mozilla.org/show_bug.cgi?id=976867
- hitting max failed password attempts : https://bugzilla.mozilla.org/show_bug.cgi?id=977329
Fraud/Security
- observe/alert mass account creation
- observe/alert large # of brute force attempts (also observing any that succeed)
Dev/Debug
- monitor V8 heap stats: https://bugzilla.mozilla.org/show_bug.cgi?id=977409
Repositories
- Heka/elasticsearch/kibana configuration: https://github.com/mozilla-services/puppet-config
- fxa-auth-server: https://github.com/mozilla/fxa-auth-server/issues?labels=metrics
Timeline
- Svcops Timeline Planning
- Notes from meetings with svcops: https://id.etherpad.mozilla.org/fxa-metrics-monitoring
Future Work/Plans
- Telemetry from devices
- Pirate Metrics (AARRR), ala: https://wiki.mozilla.org/MDN/Development/Metrics
- A/B testing
Resources
- all metrics story bugs in bugzilla
- Nutshell Summary written for a Feb 2014 meeting
- PM/UX related user stories: https://id.etherpad.mozilla.org/fxacct-metrics-user-stories
- QA/SE Monitoring user stories: https://id.etherpad.mozilla.org/fxa-monitoring-user-stories
- OpSec user stories: https://mana.mozilla.org/wiki/display/SECURITY/Firefox+Accounts+Security+Design+Review#FirefoxAccountsSecurityDesignReview-SecurityConcepts
- Shared Services : Describes shared services for heka/elasticsearch/kibana deployments. FxA will likely use its own deployment, based on this one.
- FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600)
- https://wiki.mozilla.org/ADI
- Minimum Viable Metrics : The original design for minimum set of metrics required before going live with real accounts. Original plan was WMF & Marketplace on FxOs, Sync actually went first (Desktop & Android).
Older Notes
- https://id.etherpad.mozilla.org/fxacct-metrics
- https://id.etherpad.mozilla.org/fxacct-mvm
- https://id.etherpad.mozilla.org/fxacct-mvm-details
- https://id.etherpad.mozilla.org/fxacct-metrics-fraud-detection