Account confirmers, Anti-spam team, Bureaucrats, canmove, Confirmed users, Module owners and peers, smwadministrator, smwcurator, Administrators, MozillaWiki team, Widget editors
732
edits
QA/Telemetry/Developing a Telemetry Experiment: Difference between revisions
QA/Telemetry/Developing a Telemetry Experiment (view source)
Revision as of 13:24, 29 October 2019
, 29 October 2019outdated page
(more info) |
(outdated page) |
||
| (18 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
{{outdated}} | |||
= Telemetry Experiments = | = Telemetry Experiments = | ||
This page describes the mechanics of how to build, test, and deploy a Telemetry Experiment. For a more high-level overview of Telemetry Experiments, see [[Telemetry/Experiments]]. | |||
== Getting started == | == Getting started == | ||
| Line 13: | Line 17: | ||
== Manifest file == | == Manifest file == | ||
* | * unlike other restartless extensions, experiments must have a top-level manifest.json file, at experiments/your-experiment/manifest.json. | ||
* this file contains the configuration details for your experiment - you can use manifests for other experiments as examples | |||
* the "sample" value, for example, tells the percentage of users (matching the other criteria) that will receive your experiment | * the "sample" value, for example, tells the percentage of users (matching the other criteria) that will receive your experiment | ||
* Note that the name field is visible to end users, so it should be accurate and not confusing. | |||
== install.rdf file == | == install.rdf file == | ||
| Line 28: | Line 34: | ||
== Building the server == | == Building the server == | ||
* To test, you need to build the telemetry server (which is a simple static folder). To build it, you | * To test, you need to build the telemetry server (which is a simple static folder). To build it, you need to setup a virtualenv (see the README.rst file in the root) and then run in the root of the repo: | ||
** <code>python build.py ../build http://localhost:8080</code> | ** <code>python build.py ../build http://localhost:8080</code> | ||
* you can change ../build to any folder where you want to output it, '''but note:''': this folder will be erased by the build.py script | * you can change ../build to any folder where you want to output it, '''but note:''': this folder will be erased by the build.py script | ||
| Line 49: | Line 55: | ||
** '''devtools.chrome.enabled = true''', in order for the Browser Console to show up | ** '''devtools.chrome.enabled = true''', in order for the Browser Console to show up | ||
** '''toolkit.telemetry.enabled = true''', telemetry must be enabled | ** '''toolkit.telemetry.enabled = true''', telemetry must be enabled | ||
** '''experiments.manifest.uri = http://localhost:8080/firefox-manifest.json''', full path to your generated firefox-manifest.json | ** '''experiments.manifest.uri = "http://localhost:8080/firefox-manifest.json"''', full path to your generated firefox-manifest.json | ||
** '''experiments.force-sample-value = "0.1"''', to make sure your testing always falls into the sample group | |||
** '''experiments.logging.enabled = true''', for logging | ** '''experiments.logging.enabled = true''', for logging | ||
** '''experiments.logging.level = 0''', more logging | ** '''experiments.logging.level = 0''', more logging | ||
** '''extensions.logging.enabled = true''', even more logging from the Add-ons Manager | ** '''extensions.logging.enabled = true''', even more logging from the Add-ons Manager | ||
** '''xpinstall.signatures.required = false''', to allow installing unsigned add-ons | |||
Once you enable '''devtools.chrome.enabled''' from about:config, you can paste the rest into the Browser console and be happy: | |||
<code><pre> | <code><pre> | ||
Services.prefs.setBoolPref(" | Services.prefs.setBoolPref("xpinstall.signatures.required", false); | ||
Services.prefs.setBoolPref("toolkit.telemetry.enabled", true); | Services.prefs.setBoolPref("toolkit.telemetry.enabled", true); | ||
Services.prefs.setBoolPref("experiments.logging.enabled", true); | Services.prefs.setBoolPref("experiments.logging.enabled", true); | ||
Services.prefs.setBoolPref("extensions.logging.enabled", true); | Services.prefs.setBoolPref("extensions.logging.enabled", true); | ||
Services.prefs.setIntPref("experiments.logging.level", 0); | Services.prefs.setIntPref("experiments.logging.level", 0); | ||
Services.prefs.setCharPref("experiments.force-sample-value", "0.1"); | |||
Services.prefs.setCharPref("experiments.manifest.uri", "http://localhost:8080/firefox-manifest.json"); | Services.prefs.setCharPref("experiments.manifest.uri", "http://localhost:8080/firefox-manifest.json"); | ||
</pre></code> | </pre></code> | ||
| Line 93: | Line 102: | ||
== Testing on Beta == | == Testing on Beta == | ||
If | If your experiment is going to run on beta or release, and you want to test on those channels, you'll '''need''' to have the XPI signed (see below), or rebuild beta to remove the requirement that experiments be signed, by [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm?rev=002d7897c93c#267 removing this line]. It's probably better to use one of the following two workflows: | ||
* test on a local beta build modified to allow unsigned add-ons, then get code review, then sign the XPI, and then deploy to staging | |||
* get code review, sign the XPI, test the signed XPI on a stock beta build, and then deploy to staging | |||
These workflows put the signing step after code review, because the reviewer might request changes to the add-on which will require you to re-sign the updated version. Neither of these workflows are mandatory, though - do what makes sense. | |||
== Code Review == | |||
Before deploying to staging, you should get your experiment reviewed. To request reviews, file a bug against the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Health%20Report&component=Client%3A%20Desktop Firefox Health Report product, Desktop Client component]. | |||
== Signing the XPI == | |||
For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. You can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems. | |||
== Deploying to staging == | |||
Once everything looks good, everything is working perfectly, you've got r+, and the experiment signed, you should push it to the repo (this requires [https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/ commit access level 3]). | |||
Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment. | |||
At this point, you can also request QA testing of your experiment if needed. Refer to the [[Telemetry/Experiments#QA_Testing|QA Testing]] section of the main experiment documentation for details. | |||
== Deploying to production == | |||
Before deploying, you need to send an e-mail to release-drivers with an intent to ship notice, and get approval from release mgmt. See the [[Telemetry/Experiments#Final_Deployment|Final Deployment]] section of the main experiment documentation for the email template. | |||
To deploy, you need to update the ''release_tag'' in the repo to point to your newest changeset: | |||
* <code> hg tag -f --rev tip release_tag -m "Bug AAA - Update release_tag to deploy my experiment. r=someone"</code> | |||
Then, file a bug ([https://bugzilla.mozilla.org/show_bug.cgi?id=1247822 example]) for DevOps to deploy it to the [https://telemetry-experiment.cdn.mozilla.net/ production server]. | |||