Connected Devices/Projects/Metrics: Difference between revisions

Jump to navigation Jump to search
→‎Project Overview: updating Metrics info due to the new approach followed (using Google Analytics)
(Added the link for Instructions to integrate.)
(→‎Project Overview: updating Metrics info due to the new approach followed (using Google Analytics))
Line 5: Line 5:
In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).
In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).


== Project Overview ==
== Background ==
Our goal is to build a framework that will allow each train to collect the specific data needed, but still use the [http://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/index.html unified Telemetry Pipeline] to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.
At the beginning of this project our goal was to build a framework that will allow each train to collect the specific data needed, but still use the [http://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/index.html unified Telemetry Pipeline] to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.


Having the Telemetry API as a common bearer is not enough as we need to ensure the information sent is consistent. For doing so, we will rely on the [https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/common-ping.html Telemetry "Ping" mechanism] that is already used in many Mozilla products.
After some months working to ensure the information sent was consistent, relying on the [https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/common-ping.html Telemetry "Ping" mechanism] already used in many Mozilla products and defining one common [https://docs.google.com/a/mozilla.com/document/d/1nMdnuYWGSXRzLramzzAWpR78ZRA5645bpWAWKIyTvp0/edit?usp=sharing|Connected Connected Devices Ping Format proposal] that could be implemented by each train, we changed our approach.


In order to ensure information is consistent, can be easily processed and aggregated, we have defined one common [https://docs.google.com/a/mozilla.com/document/d/1nMdnuYWGSXRzLramzzAWpR78ZRA5645bpWAWKIyTvp0/edit?usp=sharing|Connected Connected Devices Ping Format proposal] that could be implemented by each train at the gate 1+ phase.
Due to the benefits of using Telemetry are clear for Gecko-based projects, but this might not apply to other our CD trains.
* The CD projects are intended to be initially small projects and Google Analytics and other 3rd party providers offer kind of dashboards really easy to be used.
* Some other projects in Mozilla have already started to use this kind of solutions
 
== Framework Decision ==
After studying [https://docs.google.com/spreadsheets/d/1zMgrunttlH0qSr-2qpiIAPb6DPo837V68AGYmJYCPME/edit#gid=0 several third party solutions] the team decided to go for Google Analytics.
* It fits our needs both in terms of metric collection and visualization
* GA has been already vetted in Mozilla and we are already paying for a premium service
* Implementing GA will be faster and less effort than adapting the Mozilla telemetry pipeline for use with CD projects
We have been working in three major activities:
* Implementing a library for Rust and JS that simplifies the adoption of Google Analytics by the trains
* Documenting the parameters that can be used for collection of events
* Evaluating reporting/visualization tools to ensure that metric data can be easily analyzed by the trains


Connected Devices will be aiming to use a common ping format across all of Connected Devices.  In summary, this currently consists of three ping formats: 
The target of this is ensuring all the trains send information in a consistent way and dashboards/reporting are consistent cross-train.
* FTU Ping - Typically sent after the device is started for the first time.  This helps to determine how many Active devices there are.
* Core Ping - Contains histograms that developers have instrumented in their code using the Metrics Library.  Sent in a recurring interval such as once per day or once every two weeks depending on requirements.  Whether or not this is sent is dependent on privacy choice of the user.
* Crash Ping - Sent when there is a crash.  Also dependent on privacy choice of user.


These pings will all be submitted to the Telemetry data pipeline at incoming.telemetry.mozilla.org.Having a common ping format still leaves freedom to CD trains to send specific payloads within the pings as every product will have different aspects to be measured.
=== Project Status and CD metrics integration ===
The CD Metrics Project uses Google Analytics capabilities as a collection point for the various trains/projects. We are using the Google Analytics Measurement Protocol that is more appropriate for IoT projects than Classic Google Analytics which is more e-Commerce related.


=== Next Steps ===
CD Metrics is just a framework but the usage of the data is up to trains. Once data is collected, there are several visualization options available for them. Have a look at them together with the steps to follow to integrate CD Metrics in your train in our [[Connected_Devices/Projects/Metrics/Integration|'''CD Metrics Integration''']] page
Team Objectives for Q2:
* Create a Rust-based Metrics library ready for integration with Connected Devices trains. API will be documented and tested and will enable capture of three types of app metrics: (1) Crashes, (2) FTU, (3) Core usage/histogram data.
* Evangelize the Metrics project with CD trains and solicit technical feedback.
* Have a clear plan on how to query/dashboard CD metrics data through the central Mozilla telemetry pipeline, for implementation & delivery in Q3.
* Assist with integration of the library into one of the existing CD trains or publish a sample project to illustrate usage of the library


== Program Status ==
== Program Status ==
Confirmed users
1,225

edits

Navigation menu