Telemetry/Experiments

Telemetry Experiments is a feature that allows Firefox to automatically download and run specially-designed restartless addons based on certain conditions.

Experiments are available to anyone who has Telemetry enabled, which is the default on pre-release versions of Firefox. This feature is currently only available in desktop versions of Firefox.

Experiments can be targeted at various populations by specifying conditions in an experiment manifest. https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/experiments/manifest.html has details on the possible conditions, and there it is also possible to write complex conditions in JavaScript which can filter on any data in the telemetry or Firefox Health Report payloads.

Product Approval

Product approval is required to run an experiment. User-facing experiments should be approved by a Firefox product manager (Chad Weiner). For experiments which have no user-visible effects, this approval is trivial.

Data Collection

Before building an experiment, contact bsmedberg to talk about experiment design and how the results of the experiment will be measured and reported.

Engineering

Experiments must be developed in a public source repository and must be reviewed just as if they were part of Firefox on mozilla-central. Depending on the nature of the experiment and its data-collection system, additional privacy or security review may be required.

Experiments are restartless .xpi and can do anything a normal Firefox extension can do. In install.rdf, em:type==128 must be used.

Staging

To deploy an experiment, write a patch to http://hg.mozilla.org/webtools/telemetry-experiment-server/ for experiment deployment, and bsmedberg or a peer can review. Pushes there auto-deploy to the staging server at https://telemetry-experiment-dev.allizom.org/. QA should verify the experience on the staging server, including data collection.

• QA Guide For Telemetry Experiments - Kamil Jozwiak is the QA lead for experiments

Final Deployment

Final deployment of an experiment must be approved by release-drivers. Send an email to release-drivers@mozilla.org with the following information:

• A brief summary of the experiment, the questions that will be answered, and how the experiment will answer the questions
• bug(s) where the code was developed and reviewed
• details about experiment deployment: what channel(s) will the experiment be deployed to, how many users will be participating, and how long will the experiment run? This data can be copied from the staging server.

After final approval, file a bug to deploy to production at https://telemetry-experiment.cdn.mozilla.net/ . Use bug 1057464 as a model for deployment.

Monitoring

Use http://bsmedberg.github.io/telemetry-experiments-dashboard/ to monitor experiment adoption.