View source for Identity/Metrics/FxA Metrics Nutshell Summary

Quick summary of what's planned (in priority order). Written for a meeting in Feb 2014.

'''Infrastructure'''
* fxa-auth-server (and fxa-content-server) log events locally
* heka processes those logs and sends data to log aggregator
* more heka processing at the log aggregator, which creates elasticsearch indexes and mappings in elasticsearch db (as well as anonymization and other aggregation)
* kibana dashboard(s) access elasticsearch db (all metrics described below can be expressed as counts/time)
* log aggregator, elastic search & kibana are either on shared services, or an FxA specific mirror of that setup
* a development environment will exist that has mirror of production data (to hack on heka filters, elasticsearch mappings, kibana dashboards etc.)

'''0. One Metric That Matters: Total number of Firefox Accounts'''
* Web page that Mark can show to anyone -- this is the top line goal for FxA
* At minimum, could be a single number
* Display a graph over time

'''1. Accounts created per day'''
* Also logins per day, and other events that correspond to fxa-auth-server endpoints
* segmented by device (Android, Desktop, FxOS, browser), service (Marketplace, Sync, WMF, Settings), locale
* derived from fxa-auth-server logs: log line at each endpoint contains the raw information, processed by heka, data eventually lands in an elasticsearch index.
* device is derived from user-agent (How is device name derived? What devices do we explicitly want to track/monitor? -trink)
** "agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
** "agent":"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
** "agent":"curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5"
* locale is sent on request header (What locales do we explicitly want to track/monitor? -trink)
* service is passed in on some endpoints (Is there a fixed list of services? Again, if not which ones do we explicitly want to track/monitor -trink)

'''2. Active daily users'''
* derived from fxa-auth-server logs: # calls to cert/sign endpoint, bucketed by userid. 
* alternative could be derived from verifier logs, again bucketed by userid
* heka can do processing to count unique userid's/day, so that userid is not stored in elastic search
** note: The sandbox is intentionally resource constrained, it is highly unlikely it will have enough memory to track every active user in a day as usage increases. -trink.

'''3. UX flow analysis'''
* count number of people who make it to each stage of a given UX flow (per day)
* note that it is not important to connect events in a flow for an individual user: associate each event with a flow type (create account, login, reset password) and assume that the flow happens in the context of one day
* derived from fxa-auth-server logs initially, can add in events from fxa-content-server logs, etc. eventually (as long as the "flow type" matches)
* not 100% clear to me yet whether we can identify the "flow" in the server, or if we need help by information passed in from the client
* segmentations come along for the ride: service, device, locale

'''Additional segmentations'''
* UX would like screen size segmentations, would require info from the client/device

'''UX Flow Timing Information'''
* It would be great to get timing information for each stage in a UX flow (ms since the "start" of the flow), but this requires help from the client.
* Getting great visualizations from this data might be a challenge in kibana (without some custom hacking)

'''Additional stages for UX flows, additional UX flows'''
* Would be great to track # people who have opportunity to create account but then skip it (in FTU flows)
* Would be great to tie email validation stage into FTU flow analysis (happens at different point in time -- possibly even on a different day)
* Create and login flows are prioritized. Once things are up and running it should be easy to add password reset, etc. 

'''Sync specific metrics'''
* # people who make some sort of customization was mentioned (not currently prioritized)
* FHR work happening separately (Desktop: https://bugzilla.mozilla.org/show_bug.cgi?id=958561, Android: https://bugzilla.mozilla.org/show_bug.cgi?id=891600)