Mobile/Metrics: Difference between revisions

← Older edit

Mobile/Metrics (view source)

Revision as of 15:54, 13 May 2016

2,708 bytes added , 13 May 2016

→‎Dashboards

Margaret

Confirmed users

1,111

edits

@@ Line 1: / Line 1: @@
-Turns out we track a lot of data around Firefox for Mobile.  Here are links to those things:
+This page explains the different tools available to analyze data about how people use Firefox for Android and iOS. Most of these tools currently only support Android, but eventually there should be more support for iOS.
-* '''[[Mobile/Metrics/Tracking | Tracking]]'''
+Note: some metrics are only available to employees and may require sign-in.
-* [https://wiki.mozilla.org/Mobile/Triage Triage lists]
+== Usage Metrics ==
-* [http://gbrownmozilla.wordpress.com/ Gbrown's monthly analysis blog]
+=== Dashboards ===
-* [http://input.mozilla.org/en-US/?product=mobile&version=-- Input.mozilla Mobile feed]
+*[https://gecko.readthedocs.org/en/latest/mobile/android/fennec/adjust.html Adjust]: Third-party SDK initially added to track install referrers, but currently our most trusted source of usage numbers. Also used to track retention.
-* [https://moztrap.mozilla.org/results/runs/? Moztrap Test Runs] (Filter on "Mobile Firefox (Product)")
+*[https://dataviz.mozilla.org/workbooks/FennecADI ADI dashboard]: Gives us more granular daily use numbers per release version. Very old way of collecting usage numbers, based off blocklist pings.
+*[https://itunesconnect.apple.com/itc/static/login?view=1&path=%2FWebObjects%2FiTunesConnect.woa%2Fra%2Fng%2F%3F iTunesConnect]: Apple's metrics tools for iOS apps.  Provided detailed information for installs, daily users, retention, etc. Some measures are "Opt-In" only.
+*[https://sql.telemetry.mozilla.org/dashboard/mobile-users-engagement Users & Engagement]: Based off core ping.
-* [https://crash-stats.mozilla.com/products/FennecAndroid Mobile Crash Reports]
+=== Telemetry ===
-* [http://eideticker.wrla.ch/# Eideticker Dashboard]
-* [http://eideticker.wrla.ch/#/samsung-gn/startup-abouthome-cold/timetostableframe Eideticker: cold startup to about:home]
-* [http://mrcote.info/phonedash/#/org.mozilla.fennec/throbberstart/local-twitter/2012-12-28/2013-01-04 Raw Fennec startup results]
-* [https://metrics.mozilla.com/data/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2Ftelemetry&file=LandingPage.wcdf Telemetry Dashboards] (Look for Fennec probes)
-* [http://people.mozilla.com/~mfinkle/dashboard/ Talos Performance Dashboard - Central]
-* [http://people.mozilla.com/~mfinkle/dashboard/inbound.html Talos Performance Dashboard - Inbound]
-* [https://areweslimyet.com/mobile Are We Slim Yet (mobile)] memory usage dashboard
-* [https://metrics.mozilla.com/pentaho/content/pentaho-cdf/RenderHTML?solution=metrics&path=blocklist_fennec/blocklist_fennec0&dashboard=template.html&template=new-metrics&title=Fennec%20Usage Fennec Usage] (LDAP req)
+Our telemetry systems are currently in flux, but we are working towards a unified telemetry data pipeline to measure sessions and events. We have two levels of telemetry, opt-out and opt-in, which are controlled by data sharing preferences in our settings UI. Opt-in telemetry is enabled by default up to beta, but is really opt-in on release.
-* [https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=/blocklist_fennec/blocklist_fennec10&file=fennecGrowth.wcdf Fennec Growth] (LDAP req)
-* [https://metrics.mozilla.com/pentaho/content/pentaho-cdf/RenderHTML?solution=metrics&path=downloads/downloads_1&dashboard=template.html&template=new-metrics&title=Downloads&product=fennec Fennec Downloads] (LDAP req)
+*[https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/index.html Telemetry docs]: Some of this is used on both desktop and Android, but some of it is specific to desktop only. Use caution when reading these docs in a mobile context.
-* [https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics&path=/blocklist_fennec/fennec_beta_uptake&file=MinorReleaseUptake.wcdf Fennec Uptake] (LDAP req) (seems broken or at least insanely slow)
+*[https://gecko.readthedocs.org/en/latest/mobile/android/fennec/uitelemetry.html UI telemetry]: Mobile-specific UI telemetry implementation. You can query this data with re:dash, or look at the [https://people.mozilla.org/~mfinkle/uitelemetry/ UI telemetry dashboard]. Learn more about creating probes in this [[Mobile/Metrics/UITelemetry_probes|guide]].
+*[https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/core-ping.html Core ping]: New opt-out telemetry ping that we're working on to replace FHR.
+==== Pipeline ====
+Getting data into the system requires multiple steps:
+* Client submits pings.
+** Pings are generated and an attempt to send is made.
+** Due to network connectivity, server issues or any other real-world issue, the ping could fail to be sent.
+** Failed pings are archived locally and resend attempts are made at a later time.
+* Server receives pings and does a small amount of validation and routing.
+* Pings are ready for streaming style analyses. (~1 minute after receiving)
+* Pings are saved in durable storage. (~15 minutes after receiving)
+* Ping data is available to Spark analyses. (~15 minutes after receiving)
+* Ping data is run through ETL scripts and imported into Presto/Re:dash. Ready for snapshot style analyses. (~24 hours after receiving)
+=== Exploring Telemetry Data ===
+*[https://telemetry.mozilla.org/ Histogram dashboard]: Histogram probes are used mostly for engineering metrics. They are not as useful for product decisions, and we may stop sending histogram telemetry data on mobile.
+*[https://analysis.telemetry.mozilla.org/ Self-serve telemetry analysis]: IPython-based notebooks, great for doing more hardcore, specific telemetry analysis. Also used to create [https://en.wikipedia.org/wiki/Extract,_transform,_load ETL] scripts which load data into other systems.
+*[https://people.mozilla.org/~mfinkle/uitelemetry/ UI telemetry dashboard]: Simple pivot-table summary of UI Telemetry. Re:dash can replicate the table, so we'll probably be moving away from this dashboard.
+*[https://sql.telemetry.mozilla.org/ Re:dash via SQL]: Handy tool for performing analysis against telemetry data. Right now this includes opt-in UI telemetry data and core ping data. Learn more about the data structure and how to query in this [[Mobile/Metrics/Redash|guide]].
+=== Deprecated ===
+*[https://metrics.services.mozilla.com/fennec-dashboard/ Executive metrics]: Mostly not useful since we removed FHR, but a re-booted version is planned.
+== Stability and Performance ==
+=== Dashboards ===
+*[https://crash-stats.mozilla.com/home/products/FennecAndroid Crash stats]: Where we monitor crashes.
+*[https://treeherder.mozilla.org/perf.html#/graphs Perfherder]: Charts to monitor memory use, build times, and APK size.
+*[http://phonedash.mozilla.org Autophone]: Startup performance tests.
+=== See also ===
+*[https://gbrownmozilla.wordpress.com/ gbrown's testing & perf analysis blog]
+== Qualitative Data ==
+*[https://input.mozilla.org/en-US/?product=mobile&version=-- Mobile user feedback (input.mozilla.org)]