QA/Telemetry/Developing a Telemetry Experiment - Revision history

Sylvestre: outdated page

2019-10-29T13:24:36Z

outdated page

← Older revision		Revision as of 13:24, 29 October 2019
Line 1:		Line 1:
			{{outdated}}

	= Telemetry Experiments =		= Telemetry Experiments =

Ashughes: /* Testing on Beta */

2017-03-22T20:59:22Z

Testing on Beta

← Older revision		Revision as of 20:59, 22 March 2017
Line 100:		Line 100:
	== Testing on Beta ==		== Testing on Beta ==

	If your experiment is going to run on beta or release, and you want to test on those channels, you'll '''need''' to have the XPI signed (see below), or rebuild beta to remove the requirement that experiments be signed, by [~~http~~://~~mxr~~.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm?rev=002d7897c93c#~~240~~ removing this line]. It's probably better to use one of the following two workflows:		If your experiment is going to run on beta or release, and you want to test on those channels, you'll '''need''' to have the XPI signed (see below), or rebuild beta to remove the requirement that experiments be signed, by [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm?rev=002d7897c93c#267 removing this line]. It's probably better to use one of the following two workflows:

	* test on a local beta build modified to allow unsigned add-ons, then get code review, then sign the XPI, and then deploy to staging		* test on a local beta build modified to allow unsigned add-ons, then get code review, then sign the XPI, and then deploy to staging

Ashughes: /* Signing the XPI */

2016-09-20T17:07:24Z

Signing the XPI

← Older revision		Revision as of 17:07, 20 September 2016
Line 113:		Line 113:
	== Signing the XPI ==		== Signing the XPI ==

	For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. Once {{bug\|1271440}} is fixed, you can use the '''jpm''' tool to sign the experiment, using the instructions at [[EngineeringProductivity/HowTo/SignExtensions\|How to Sign Extensions]]. There is additional documentation on the signing API at [http://addons-server.readthedocs.io/en/latest/topics/api/signing.html] - the '''jpm''' tool uses this infrastructure with a shared automation signing key that your LDAP gives you access to.		For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. You can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems.

	~~Until that bug is fixed, though, you~~ can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems.

	== Deploying to staging ==		== Deploying to staging ==

Ashughes: /* Deploying to staging */

2016-09-20T17:06:06Z

Deploying to staging

← Older revision		Revision as of 17:06, 20 September 2016
Line 119:		Line 119:
	== Deploying to staging ==		== Deploying to staging ==

	Once everything looks good, everything is working perfectly, you've got r+, and the experiment signed, you should push it to the repo.		Once everything looks good, everything is working perfectly, you've got r+, and the experiment signed, you should push it to the repo (this requires [https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/ commit access level 3]).

	Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.		Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.

Kats: Link back to the top-level page where appropriate

2016-05-10T14:49:50Z

Link back to the top-level page where appropriate

← Older revision		Revision as of 14:49, 10 May 2016
Line 1:		Line 1:
	= Telemetry Experiments =		= Telemetry Experiments =

			This page describes the mechanics of how to build, test, and deploy a Telemetry Experiment. For a more high-level overview of Telemetry Experiments, see [[Telemetry/Experiments]].

	== Getting started ==		== Getting started ==
Line 120:		Line 122:

	Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.		Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.

			At this point, you can also request QA testing of your experiment if needed. Refer to the [[Telemetry/Experiments#QA_Testing\|QA Testing]] section of the main experiment documentation for details.

	== Deploying to production ==		== Deploying to production ==

	* Before deploying, you need to send an e-mail to release-drivers with an intent to ship notice, and get approval from release mgmt.		Before deploying, you need to send an e-mail to release-drivers with an intent to ship notice, and get approval from release mgmt. See the [[Telemetry/Experiments#Final_Deployment\|Final Deployment]] section of the main experiment documentation for the email template.
	* To deploy, you need to update the ''release_tag'' in the repo to point to your newest changeset:
	** <code> hg tag -f --rev tip release_tag -m "Bug AAA - Update release_tag to deploy my experiment. r=someone"</code>		To deploy, you need to update the ''release_tag'' in the repo to point to your newest changeset:
	** then file a bug ([https://bugzilla.mozilla.org/show_bug.cgi?id=1247822 example]) for DevOps to deploy it to the [https://telemetry-experiment.cdn.mozilla.net/ production server].		* <code> hg tag -f --rev tip release_tag -m "Bug AAA - Update release_tag to deploy my experiment. r=someone"</code>
			Then, file a bug ([https://bugzilla.mozilla.org/show_bug.cgi?id=1247822 example]) for DevOps to deploy it to the [https://telemetry-experiment.cdn.mozilla.net/ production server].

Kats: /* Signing the XPI */ Link to signing API docs as well

2016-05-10T14:38:20Z

Signing the XPI: Link to signing API docs as well

← Older revision		Revision as of 14:38, 10 May 2016
Line 111:		Line 111:
	== Signing the XPI ==		== Signing the XPI ==

	For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. Once {{bug\|1271440}} is fixed, you can use the '''jpm''' tool to sign the experiment, using the instructions at [[EngineeringProductivity/HowTo/SignExtensions\|How to Sign Extensions]].		For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. Once {{bug\|1271440}} is fixed, you can use the '''jpm''' tool to sign the experiment, using the instructions at [[EngineeringProductivity/HowTo/SignExtensions\|How to Sign Extensions]]. There is additional documentation on the signing API at [http://addons-server.readthedocs.io/en/latest/topics/api/signing.html] - the '''jpm''' tool uses this infrastructure with a shared automation signing key that your LDAP gives you access to.

	Until that bug is fixed, you can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems.		Until that bug is fixed, though, you can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems.

	== Deploying to staging ==		== Deploying to staging ==

Kats: Split out separate sections on beta testing, code review, signing

2016-05-10T14:36:00Z

Split out separate sections on beta testing, code review, signing

← Older revision		Revision as of 14:36, 10 May 2016
Line 98:		Line 98:
	== Testing on Beta ==		== Testing on Beta ==

	If ~~you're testing~~ on beta or release, you'll '''need''' to have the XPI signed, or rebuild beta to remove the requirement that experiments be signed, by [http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm?rev=002d7897c93c#240 removing this line].		If your experiment is going to run on beta or release, and you want to test on those channels, you'll '''need''' to have the XPI signed (see below), or rebuild beta to remove the requirement that experiments be signed, by [http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm?rev=002d7897c93c#240 removing this line]. It's probably better to use one of the following two workflows:

	== ~~Deploying to staging~~ ==		* test on a local beta build modified to allow unsigned add-ons, then get code review, then sign the XPI, and then deploy to staging
			* get code review, sign the XPI, test the signed XPI on a stock beta build, and then deploy to staging

			These workflows put the signing step after code review, because the reviewer might request changes to the add-on which will require you to re-sign the updated version. Neither of these workflows are mandatory, though - do what makes sense.

			== Code Review ==

	Before deploying to staging, you should get your experiment reviewed. To request reviews, file a bug against the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Health%20Report&component=Client%3A%20Desktop Firefox Health Report product, Desktop Client component].		Before deploying to staging, you should get your experiment reviewed. To request reviews, file a bug against the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Health%20Report&component=Client%3A%20Desktop Firefox Health Report product, Desktop Client component].

	Once everything looks good, everything is working perfectly, ~~and~~ you've got r+, you should push it to the repo.		== Signing the XPI ==

			For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. Once {{bug\|1271440}} is fixed, you can use the '''jpm''' tool to sign the experiment, using the instructions at [[EngineeringProductivity/HowTo/SignExtensions\|How to Sign Extensions]].

			Until that bug is fixed, you can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems.

			== Deploying to staging ==

			Once everything looks good, everything is working perfectly, you've got r+, and the experiment signed, you should push it to the repo.

	Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.		Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.

Kats: /* Deploying to staging */ Move review notes from https://wiki.mozilla.org/Telemetry/Experiments#Repository_and_Reviews over to this page

2016-05-10T14:22:59Z

Deploying to staging: Move review notes from https://wiki.mozilla.org/Telemetry/Experiments#Repository_and_Reviews over to this page

← Older revision		Revision as of 14:22, 10 May 2016
Line 101:		Line 101:

	== Deploying to staging ==		== Deploying to staging ==

			Before deploying to staging, you should get your experiment reviewed. To request reviews, file a bug against the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Health%20Report&component=Client%3A%20Desktop Firefox Health Report product, Desktop Client component].

	Once everything looks good, everything is working perfectly, and you've got r+, you should push it to the repo.		Once everything looks good, everything is working perfectly, and you've got r+, you should push it to the repo.

Kats: /* Manifest file */ Move interesting content at https://wiki.mozilla.org/Telemetry/Experiments#manifest.json over into this page

2016-05-10T14:20:05Z

Manifest file: Move interesting content at https://wiki.mozilla.org/Telemetry/Experiments#manifest.json over into this page

← Older revision		Revision as of 14:20, 10 May 2016
Line 13:		Line 13:
	== Manifest file ==		== Manifest file ==

	* ~~the~~ file ~~under~~ experiments/your-experiment/manifest.json contains the configuration details for your experiment		* unlike other restartless extensions, experiments must have a top-level manifest.json file, at experiments/your-experiment/manifest.json.
			* this file contains the configuration details for your experiment - you can use manifests for other experiments as examples
	* the "sample" value, for example, tells the percentage of users (matching the other criteria) that will receive your experiment		* the "sample" value, for example, tells the percentage of users (matching the other criteria) that will receive your experiment
			* Note that the name field is visible to end users, so it should be accurate and not confusing.

	== install.rdf file ==		== install.rdf file ==

Cpeterson: /* Deploying to production */ format list

2016-03-18T00:44:58Z

Deploying to production: format list

← Older revision		Revision as of 00:44, 18 March 2016
Line 105:		Line 105:

	== Deploying to production ==		== Deploying to production ==
	~~Before deploying, you need to send an e-mail to release-drivers with an intent to ship notice, and get approval from release mgmt.~~

	To deploy, you need to update the ''release_tag'' in the repo to point to your newest changeset~~, and file a bug ([https~~:~~//bugzilla.mozilla.org/show_bug.cgi?id=1247822 example]) for DevOps to deploy it to the [https://telemetry-experiment.cdn.mozilla.net/ production server].~~		* Before deploying, you need to send an e-mail to release-drivers with an intent to ship notice, and get approval from release mgmt.
			* To deploy, you need to update the ''release_tag'' in the repo to point to your newest changeset:
	** <code> hg tag -f --rev tip release_tag -m "Bug AAA - Update release_tag to deploy my experiment. r=someone"</code>		** <code> hg tag -f --rev tip release_tag -m "Bug AAA - Update release_tag to deploy my experiment. r=someone"</code>
			** then file a bug ([https://bugzilla.mozilla.org/show_bug.cgi?id=1247822 example]) for DevOps to deploy it to the [https://telemetry-experiment.cdn.mozilla.net/ production server].