Telemetry Experiments is a feature that allows Firefox to automatically download and run specially-designed restartless addons based on certain conditions.
Experiments are available to anyone who has Telemetry enabled, which is the default on pre-release versions of Firefox. This feature is currently only available in desktop versions of Firefox.
This page provides a high-level overview of the process to build and deploy an experiment.
Product approval is required to run an experiment. User-facing experiments should be approved by a Firefox product manager (Jeff Griffiths). For experiments which have no user-visible effects, this approval is trivial.
For experiments that intend to collect new data, a data collection review is required. Before building such an experiment, contact bsmedberg or a data-collection peer to talk about experiment design and how the results of the experiment will be measured and reported. Depending on the nature of the experiment and its data-collection system, he may require additional privacy or security review.
Experiment designer should have clear questions that are being answered and should have a written plan for analyzing data to provide answers. If you need help to come up with a plan, please talk to the analyst or data engineer that works with your team. If you don't know who that is, please talk to Rebecca Weiss or Ryan VanderMeulen.
Once collected, you'll analyze the data using Map/Reduce tooling: Refer to the wiki page Custom Telemetry Analysis with Spark.
Experiments are a special type of restartless extension. They can do anything a regular restartless extension can do. Detailed instructions on the mechanics of developing and testing the experiment can be found at Developing a Telemetry Experiment.
Note that Telemetry Experiments need to be signed to run in FF 42+, just like all other add-ons. There are more details on the signing process on the page linked above.
Most experiments live for a fixed period and are uninstalled by the experiment system at the end date or after the specified elapsed time.
A few experiments are intended to collect data once and then uninstall themself. These experiments should uninstall themselves when done by calling
To deploy an experiment, write a patch to http://hg.mozilla.org/webtools/telemetry-experiment-server/ for experiment deployment, as described at Developing a Telemetry Experiment, and bsmedberg or a peer can review. Pushes there auto-deploy to the staging server at https://telemetry-experiment-dev.allizom.org/.
All experiments must receive a QA sign-off before being deployed. For experiments that involve a nontrivial amount of new code or changes, the QA team is available to verify the experience on the staging server, including data collection. Teams with dedicated QA resources can also validate their experiments on their own if they so choose. To request QA sign-off, refer to the QA Guide For Telemetry Experiments - Benjamin Smedberg is the QA lead for experiments.
Final deployment of an experiment must be approved by Lawrence Mandel or a release-management peer. Send an email to email@example.com and cc firstname.lastname@example.org with the following information:
- A brief summary of the experiment, the questions that will be answered, and how the experiment will answer the questions
- bug(s) where the code was developed and reviewed
- details about experiment deployment: what channel(s) will the experiment be deployed to, how many users will be participating, and how long will the experiment run? This data can be copied from the staging server.
Here is an example of an approval email:
To: email@example.com Cc: firstname.lastname@example.org Subject: Intent to ship: Telemetry experiment for verification of telemetry experiments bug This is a request for approval to ship the following Telemetry experiment: Telemetry experiment for verification of telemetry experiments bug. Summary In bug 1052714, I plan to ship a telemetry experiment to 50% of nightly channel users to verify that a fix for a race condition setting telemetry experiment branches is actually fixed. This experiment will have no user effect, and only exists to test the system. The results will be collected and analyzed from FHR payloads. Related Bugs: bug 1052714 Start Date 14-Aug-2014 End Date 10-Sep-2014 Maximum runtime 7.0 days Update Channels nightly Build ID 20140820030202- Instance population All nightly instances Sample Rate 50%
After final approval, file a bug to deploy to production at https://telemetry-experiment.cdn.mozilla.net/ . Use bug 1057464 or bug 1131864 as a model for deployment.
Use http://bsmedberg.github.io/telemetry-experiments-dashboard/ to monitor experiment adoption.