Connected Devices/Projects/Metrics

With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.

For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.

In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).

Project Overview

Our goal is to build a framework that will allow each train to collect the specific data needed, but still use the unified Telemetry Pipeline to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.

Having the Telemetry API as a common bearer is not enough as we need to ensure the information sent is consistent. For doing so, we will rely on the Telemetry "Ping" mechanism that is already used in many Mozilla products.

In order to ensure information is consistent, can be easily processed and aggregated, we have defined one common Connected Devices Ping Format proposal that could be implemented by each train at the gate 1+ phase.

Connected Devices will be aiming to use a common ping format across all of Connected Devices. In summary, this currently consists of three ping formats:

• FTU Ping - Typically sent after the device is started for the first time. This helps to determine how many Active devices there are.
• Core Ping - Contains histograms that developers have instrumented in their code using the Metrics Library. Sent in a recurring interval such as once per day or once every two weeks depending on requirements. Whether or not this is sent is dependent on privacy choice of the user.
• Crash Ping - Sent when there is a crash. Also dependent on privacy choice of user.

These pings will all be submitted to the Telemetry data pipeline at incoming.telemetry.mozilla.org.Having a common ping format still leaves freedom to CD trains to send specific payloads within the pings as every product will have different aspects to be measured.

Next Steps

Team Objectives for Q2:

• Create a Rust-based Metrics library ready for integration with Connected Devices trains. API will be documented and tested and will enable capture of three types of app metrics: (1) Crashes, (2) FTU, (3) Core usage/histogram data.
• Evangelize the Metrics project with CD trains and solicit technical feedback.
• Have a clear plan on how to query/dashboard CD metrics data through the central Mozilla telemetry pipeline, for implementation & delivery in Q3.
• Assist with integration of the library into one of the existing CD trains or publish a sample project to illustrate usage of the library

Program Status

Milestone	Date	Status	Comments
Connected Devices Metrics Library Architecture Design	4/08/2016	ON TARGET
Presentation for Connected Devices Product Management	End of April	ON TARGET	Product Management meetup has been delayed, no final date yet but the earlier date would end of April.
CD Metrics Milestones		NOT STARTED	Metrics milestones will be added as milestones once they're identified
London All Hands Conference Presentation	6/13/2016	NOT STARTED

Status Key

Color	Status	Key
	On Target	The project or deliverable is expected to meet its due date.
	Challenged	The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
	At Risk or Late	The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
	Done	The project or deliverable has been completed.
	On Hold or Not Started	The project or deliverable has either not been started or has been placed on hold.

Archived Completed Deliverables

Milestone	Date	Status	Status Notes
Connected Devices Ping Format	3/04/2016	Done	CD Common Ping format
Connected Devices Crash Ping Format drafted	3/11/2016	Done	Connected Connected Devices Ping Format proposal
High level Histograms Architecture Design	4/08/2016	Done	Histograms Architecture Design Diagram
Connected Devices Dashboard and Analysis Requirements	4/08/2016	Done	Connected Devices Dashboard and Analysis Requirements
Connected Devices Metrics Presentation for Product Management team	No final date yet	On target	Connected Devices Metrics Presentation (working on it)

Program Timeline

This timeline is simply a placeholder. Timeline to be inserted once milestones and schedule are known.

Project Management

We use Trello board for project management. All ongoing tasks are listed there.

Sprint Backlog

• Sprint 4 (May 2nd - May 13th) Current Sprint
• Sprint 3 (April 11st - April 21st)
• Sprint 2 (March 28th - April 8th)
• Sprint 1 (March 15th - March 25th)

Development process

Repositories

• Metrics controller: https://github.com/tamarahills/metrics_controller

IRC

You can find us on irc.mozilla.org in the #cd-metrics channel.

Methodology

The project is in early state but we have planned to use some agile practices:

• Weekly meetings in CD_Metrics Vidyo room on Wednesday at 5:30 PM UTC (9:30 AM, PST), notes in Google doc
• Sprint Planning meeting every other Monday in CD_Metrics Vidyo room at 5:30 PM UTC (9:30 AM, PST)
• Daily standup meetings everyday except the days with Sprint Planning or Weekly meeting in CD_Metrics Vidyo room at 5:30 PM UTC (9:30 AM, PST). You can include your daily report in this etherpad in case you cannot attend the meeting

Calendar

The CD Metrics team has a public calendar with every team meeting Calendar

Instructions for Adding to your Calendar

1. Open the calendar.
2. Click on the "+ Google Calendar" button in the very bottom right of your screen.

You can also use the XML and ICS methods, but these are not recommended.

Note: The "Find a Time" feature will not work for other people if you import this calendar. As a consequence, others will not see that you are unavailable when attending a FlyWeb meeting. We suggest either accepting this, or adding the meetings to your main calendar as well.

Adding Metrics to your Project/Train

The following link has instructions and details for adding metrics to your Project/train

• Integration

References

• Telemetry Pipeline
• Common Ping format

Team

• Dylan Oliver - Engineering Manager
• Tamara Hills - Engineering
• Russ Nicoletti - Engineering
• Michael Bryant - QA/Engineering
• Maria Oteo - EPM