Connected Devices/Projects/Metrics
With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.
For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.
In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).
Rationale
Nearly all the Connected Devices train are likely, at some point of time, to require collection of data. Data gathered from the projects is extremely useful to understand how people use our products but also to check how changes implemented in the Products affect how people use them.
In summary The idea of the CD Metrics Framework is providing all the CD trains an easy way to:
- Collect metrics in the products
- Get Access to the metrics information
- Visualize the metrics information
The Framework
After firstly assessing Mozilla Telemetry Pipeline, and later several third party solutions the team decided to go for Google Analytics (GA) as the Framework for Collecting Metrics. The key reasons are:
- Technical Fit: Google Analytics meets our needs both in terms of metric collections, accessing to raw data and visualizing it.
- Adoption: The integration of GA with CD trains seemed to be the faster-one (e.g. in one day we managed to get some results in one of the trains: SmartHome).
- Legally/Commercially: GA has been already vetted in Mozilla and we are already paying for a premium service
It’s important to clarify that the framework is using the Google Analytics Measurement Protocol that is more appropriate for IoT projects than Classic Google Analytics which is more e-Commerce related. The metrics are stored by GA in a Google BigQuery instance.
The Data can be later on retrieved via the Google Analytics Console that is very convenient to check if the data is being received and getting simple charts about the trains (e.g. how many users are currently “live”?).
In addition to the Google Analytics Console, the trains could also use Google Analytics Data Studio.
Furthermore, Periscope can be used to formulate SQL queries against the BigQuery Database. Periscope is a third party toolmthat can be used to get more sophisticated reports. Additional tools exist to create reports, that could be used by the CD trains, if they deemed they are necessary.
Additional Tools and Activities
Apart from identifying what is the best toolchain and framework for being used in CD trains, the CD Metrics Team has been working in different activities to ease adoption, ensure consistency and help with the visualisation activities.
- Ease adoption: By Implementing a library for Rust and JS that simplifies the adoption of Google Analytics by the trains. (Note: The Rust library could be also used in other programming languages)
- Ensure consistency: By documenting the parameters that can be used for collection of events
- Ease visualisation: By evaluating Reporting/visualization tools to ensure that metric data can be easily analyzed by the trains and providing some example charts.
In other words, we have done different tasks to ensure all the trains can onboard easily and send information in a consistent way that can be used to get easily dashboards/reports that are also consistent cross-train.
Project Status and onboarding on the metrics framework
CD Metrics is a framework to ease data collection and some tools to ease data manipulation and visualization. The target of the metrics team is not creating reports on behalf of the trains but giving them access to the data and support to getting started with the framework tools. I.e. data ownership relies on the trains themselves: they are the ones that have full access to the data so they can view/query/report as they see fit.
A first version of the framework is already available, so any CD train could start the on boarding now. For doing so, we have developed a set of guidelines to help with the on boarding process to ingrate CD Metrics in your train: CD Metrics Integration
Why Not Telemetry?
The initial approach we explored was leveraging the unified Telemetry Pipeline to send data to the metrics infrastructure and process it. The idea was reusing an architecture already existing and widely used at Mozilla. In that way could rely on the capability the platform has for validating, sanitizing and even automatically analyzing device data.
After some months working to ensure the information sent was consistent, relying on the Telemetry "Ping" mechanism already used in many Mozilla products and defining one common Connected Devices Ping Format proposal that could be implemented by each train, we changed our approach.
The benefits of using Telemetry are clear for Gecko-based projects, but this might not apply to the CD trains:
- The CD projects are intended to be initially small projects and Google Analytics and other 3rd party providers offer frameworks with lower learning and adoption curves. Using Telemetry could be an overkill to them.
- Gecko and Telemetry are very coupled and some of the Telemetry advantages require Gecko. Indeed, some other projects in Mozilla that are not using Gecko have already started to use alternative solutions for Metrics.
Program Status
| Milestone | Date | Status | Comments |
|---|---|---|---|
| Architecture Design to be used with Telemetry pipeline | 4/08/2016 | Done but obsolete | |
| Connected Devices Metrics Library to simplify the adoption of Google Analytics by the trains. | 5/13/2016 | Done | CD Metrics library in Rust that includes also a Javascript wrapper that facilitates metrics gathering for web based applications and a C interface that can be used from C and Java applications. |
| London All Hands Presentation | 6/12/2016 | On target |
Status Key
| Color | Status | Key |
|---|---|---|
| On Target | The project or deliverable is expected to meet its due date. | |
| Challenged | The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track. | |
| At Risk or Late | The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late. | |
| Done | The project or deliverable has been completed. | |
| On Hold or Not Started | The project or deliverable has either not been started or has been placed on hold. |
Archived Completed Deliverables
| Milestone | Date | Status | Status Notes | |
|---|---|---|---|---|
| Connected Devices Ping Format for Telemetry pipeline approach | 3/04/2016 | Done but obsolete | CD Common Ping format | |
| Connected Devices Crash Ping Format drafted for Telemetry pipeline approach | 3/11/2016 | Done but obsolete | Connected Connected Devices Ping Format proposal | |
| High level Histograms Architecture Design for Telemetry pipeline approach | 4/08/2016 | Done but obsolete | Histograms Architecture Design Diagram | |
| Connected Devices Dashboard and Analysis Requirements for Telemetry pipeline approach | 4/08/2016 | Done but obsolete | Connected Devices Dashboard and Analysis Requirements | |
| Metrics Tools Analysis | 4/29/2016 | Done | Metrics Tools comparative Investigation of other third party solutions (instead of using Telemetry pipeline) and also visualization tools | |
| Connected Devices Metrics Presentation for London All Hands | 6/16/2016 | On target |
Program Timeline
This timeline is simply a placeholder. Timeline to be inserted once milestones and schedule are known.
Project Management
We use Trello board for project management. All ongoing tasks are listed there.
Sprint Backlog
- Sprint 4 (May 2nd - May 13th) Current Sprint
- Sprint 3 (April 11st - April 21st)
- Sprint 2 (March 28th - April 8th)
- Sprint 1 (March 15th - March 25th)
Development process
Repositories
- Metrics controller: https://github.com/tamarahills/metrics_controller
IRC
You can find us on irc.mozilla.org in the #cd-metrics channel.
Methodology
The project is in early state but we have planned to use some agile practices:
- Weekly meetings in CD_Metrics Vidyo room on Wednesday at 5:30 PM UTC (9:30 AM, PST), notes in Google doc
- Sprint Planning meeting every other Monday in CD_Metrics Vidyo room at 5:30 PM UTC (9:30 AM, PST)
- Daily standup meetings everyday except the days with Sprint Planning or Weekly meeting in CD_Metrics Vidyo room at 5:30 PM UTC (9:30 AM, PST). You can include your daily report in this etherpad in case you cannot attend the meeting
Calendar
The CD Metrics team has a public calendar with every team meeting Calendar
Instructions for Adding to your Calendar
- Open the calendar.
- Click on the "+ Google Calendar" button in the very bottom right of your screen.
You can also use the XML and ICS methods, but these are not recommended.
Note: The "Find a Time" feature will not work for other people if you import this calendar. As a consequence, others will not see that you are unavailable when attending a FlyWeb meeting. We suggest either accepting this, or adding the meetings to your main calendar as well.
Adding Metrics to your Project/Train
The following link has instructions and details for adding metrics to your Project/train
References
Team
- Dylan Oliver - Engineering Manager
- Tamara Hills - Engineering
- Russ Nicoletti - Engineering
- Michael Bryant - QA/Engineering
- Maria Oteo - EPM