QA/Telemetry

Telemetry Experiments

Feature	Status	Lead Engineer	QA Lead	Status
Telemetry Experiments	Incremental Improvements	Benjamin Smedberg	Kamil Jozwiak	OK

Project Goal

Allow Mozilla to deploy experiments to a statistically-relevant population of users and measure the results.

Project Summary

With telemetry experiments enabled, Firefox will periodically retrieve a list of experiments from a secure server if the user meets the specified condition(s). Possible conditions include:

• Product
• Version
• OS
• Channel
• Build ID
• Language
• Sample ratio

Each experiment will include the following information:

• experiment ID
• specific conditions for running the experiment
• experiment start date & end date
• experiment duration
• XPI URL & hash

Test Plan

• QA Testing Template can be found in the following etherpad: Here
• Telemetry Experiment test plans can be found: Here

Experiments

You can view current and past telemetry experiments Here

• Note: Some experiments that have already been expired may not be grayed out once expired when viewing the experiment list Here. The experiment is technically already disabled and once another experiment is pushed, the list will than be updated and the expired experiment will be grayed out.

Experiment	Start Date	End Date	Max Runtime	Channel	Locale(s)	Sample Rate	Test Plan	Current Status
Reorder about:newtab tiles	01-Jan-2014	15-May-2014	5.0 days	Nightly	N/A	25%	Here	Completed
Search Experiment	01-Jul-2014	30-Sep-2014	28.0 days	BETA	en-US	50%	Here	Completed
Automatic Translation	27-Jun-2014	01-Aug-2014	36.0 days	Aurora	de	100%	Here	Completed
OOPP container unload timeout tester	01-Jul-2014	02-Sep-2014	42.0 days	BETA	N/A	5%	Here	Completed
Automatic Translation	16-Jul-2014	25-Nov-2014	126.0 days	BETA	pl	100%	Here	Completed
Automatic Translation	16-Jul-2014	25-Nov-2014	126.0 days	BETA	tr	100%	Here	Completed
Automatic Translation	16-Jul-2014	25-Nov-2014	126.0 days	BETA	vi	100%	Here	Completed
Invisible test for experiment branch correctness	14-Aug-2014	10-Sep-2014	7.0 days	Nightly	N/A	50%	Here	Completed
Automatic Translation	16-Jul-2014	25-Nov-2014	126.0 days	BETA	es-AR	100%	Here	Completed
Automatic Translation	16-Jul-2014	25-Nov-2014	126.0 days	BETA	es-ES	100%	Here	Completed
New Tab Data	10-Dec-2014	10-Jan-2015	7.0 days	BETA	N/A	25%	Here	Completed
Flash Protected-Mode Testing	15-Dec-2014	15-Jan-2015	32.0 days	BETA (Win Only)	N/A	10%	Here	Completed & Disabled
Plugin Video Usage Metrics	12-Jan-2015	23-Feb-2015	60.0 days	BETA	N/A	10%	Here	Completed
Search Experiment	28-Jan-2015	07-Apr-2015	30.0 days	release	tr	70%	Here	Completed
Search Experiment	25-Mar-2015	07-May-2015	30.0 days	release	zh-TW	50%	Here	Completed
Multi-process Firefox A/B test	20-Sep-2015	01-Nov-2015	10.0 days	aurora	N/A	100%	Here	Completed
Search result pages in history	10-Nov-2015	24-Nov-2015	1.0 days	beta	N/A	10%	Here	Completed
Multi-process Firefox A/B test	14-Dec-2015	04-Jan-2016	10.0 days	beta	N/A	15%	Here	In Progress
Unified URLBar Experiment	12-21-2015	12-30-2015	10.0 days	beta	N/A	10%	Here	In Progress

Telemetry Experiments/FHR Documentation

• Setup & Installing Telemetry Staging Server
• Video example of installing Telemetry Experiments
• Setting about:config Preferences
• Browser Console Commands
• Setting up Telemetry Logging
• Changing Experiments Browser Channel
• Installing Multiple Experiments
• Removing experiments.json
• Experiments Manifests (developers)
• Editing Extensions (XPI Files)
• Interfacing with FHR

QA Contact Information

If you have any questions or issues regarding telemetry, you can contact me via the following methods:

Internet Relay Chat (IRC): [Quickest Response]

• Nick: kjozwiak
• Channels: #fhr #qa
• IRC Setup: https://wiki.mozilla.org/IRC

Email: [Slowest Response]

• kjozwiak [at] mozilla.com
• kamiljoz [at] gmail.com