Connected Devices/Projects/Metrics

With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.

For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.

In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).

Background

At the beginning of this project our goal was to build a framework that will allow each train to collect the specific data needed, but still use the unified Telemetry Pipeline to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.

After some months working to ensure the information sent was consistent, relying on the Telemetry "Ping" mechanism already used in many Mozilla products and defining one common Connected Devices Ping Format proposal that could be implemented by each train, we changed our approach.

Due to the benefits of using Telemetry are clear for Gecko-based projects, but this might not apply to other our CD trains.

• The CD projects are intended to be initially small projects and Google Analytics and other 3rd party providers offer kind of dashboards really easy to be used.
• Some other projects in Mozilla have already started to use this kind of solutions

Framework Decision

After studying several third party solutions the team decided to go for Google Analytics.

• It fits our needs both in terms of metric collection and visualization
• GA has been already vetted in Mozilla and we are already paying for a premium service
• Implementing GA will be faster and less effort than adapting the Mozilla telemetry pipeline for use with CD projects

We have been working in three major activities:

• Implementing a library for Rust and JS that simplifies the adoption of Google Analytics by the trains
• Documenting the parameters that can be used for collection of events
• Evaluating reporting/visualization tools to ensure that metric data can be easily analyzed by the trains

The target of this is ensuring all the trains send information in a consistent way and dashboards/reporting are consistent cross-train.

Project Status and CD metrics integration

The CD Metrics Project uses Google Analytics capabilities as a collection point for the various trains/projects. We are using the Google Analytics Measurement Protocol that is more appropriate for IoT projects than Classic Google Analytics which is more e-Commerce related.

CD Metrics is just a framework but the usage of the data is up to trains. Once data is collected, there are several visualization options available for them. Have a look at them together with the steps to follow to integrate CD Metrics in your train in our CD Metrics Integration page

Program Status

Milestone	Date	Status	Comments
Connected Devices Metrics Library Architecture Design	4/08/2016	ON TARGET
Presentation for Connected Devices Product Management	End of April	ON TARGET	Product Management meetup has been delayed, no final date yet but the earlier date would end of April.
CD Metrics Milestones		NOT STARTED	Metrics milestones will be added as milestones once they're identified
London All Hands Conference Presentation	6/13/2016	NOT STARTED

Status Key

Color	Status	Key
	On Target	The project or deliverable is expected to meet its due date.
	Challenged	The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
	At Risk or Late	The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
	Done	The project or deliverable has been completed.
	On Hold or Not Started	The project or deliverable has either not been started or has been placed on hold.

Archived Completed Deliverables

Milestone	Date	Status	Status Notes
Connected Devices Ping Format	3/04/2016	Done	CD Common Ping format
Connected Devices Crash Ping Format drafted	3/11/2016	Done	Connected Connected Devices Ping Format proposal
High level Histograms Architecture Design	4/08/2016	Done	Histograms Architecture Design Diagram
Connected Devices Dashboard and Analysis Requirements	4/08/2016	Done	Connected Devices Dashboard and Analysis Requirements
Connected Devices Metrics Presentation for Product Management team	No final date yet	On target	Connected Devices Metrics Presentation (working on it)

Program Timeline

This timeline is simply a placeholder. Timeline to be inserted once milestones and schedule are known.

Project Management

We use Trello board for project management. All ongoing tasks are listed there.

Sprint Backlog

• Sprint 4 (May 2nd - May 13th) Current Sprint
• Sprint 3 (April 11st - April 21st)
• Sprint 2 (March 28th - April 8th)
• Sprint 1 (March 15th - March 25th)

Development process

Repositories

• Metrics controller: https://github.com/tamarahills/metrics_controller

IRC

You can find us on irc.mozilla.org in the #cd-metrics channel.

Methodology

The project is in early state but we have planned to use some agile practices:

• Weekly meetings in CD_Metrics Vidyo room on Wednesday at 5:30 PM UTC (9:30 AM, PST), notes in Google doc
• Sprint Planning meeting every other Monday in CD_Metrics Vidyo room at 5:30 PM UTC (9:30 AM, PST)
• Daily standup meetings everyday except the days with Sprint Planning or Weekly meeting in CD_Metrics Vidyo room at 5:30 PM UTC (9:30 AM, PST). You can include your daily report in this etherpad in case you cannot attend the meeting

Calendar

The CD Metrics team has a public calendar with every team meeting Calendar

Instructions for Adding to your Calendar

1. Open the calendar.
2. Click on the "+ Google Calendar" button in the very bottom right of your screen.

You can also use the XML and ICS methods, but these are not recommended.

Note: The "Find a Time" feature will not work for other people if you import this calendar. As a consequence, others will not see that you are unavailable when attending a FlyWeb meeting. We suggest either accepting this, or adding the meetings to your main calendar as well.

Adding Metrics to your Project/Train

The following link has instructions and details for adding metrics to your Project/train

• Integration

References

• Telemetry Pipeline
• Common Ping format

Team

• Dylan Oliver - Engineering Manager
• Tamara Hills - Engineering
• Russ Nicoletti - Engineering
• Michael Bryant - QA/Engineering
• Maria Oteo - EPM