1,007
edits
Labs/Test Pilot/ExperimentAPI: Difference between revisions
→Experiments
| Line 93: | Line 93: | ||
= Experiments = | = Experiments = | ||
== Example Experiment == | |||
There is an example experiment file in the hg checkout at testcases/example/example_study.js. You can also see the code for it on the web right now by going to [https://testpilot.mozillalabs.com/testcases/example/example_study.js https://testpilot.mozillalabs.com/testcases/example/example_study.js]. It is heavily commented and intended to be as self-explanatory as possible. The best way to get started writing an experiment is probably to copy that file and start changing stuff. | |||
== Experiment API definition == | |||
Experiments are much more involved than surveys, and must export four objects, named ''exactly'' as follows: | Experiments are much more involved than surveys, and must export four objects, named ''exactly'' as follows: | ||
| Line 103: | Line 109: | ||
These are explained one by one in the following sections. | These are explained one by one in the following sections. | ||
== experimentInfo == | === experimentInfo === | ||
This object contains metadata about the experiment: its name and id, its version number, when it is to be run and for how long, whether it recurs, whether it requires an additional layer of opt-in, and the URLs for further information about it. It should contain the following fields: | This object contains metadata about the experiment: its name and id, its version number, when it is to be run and for how long, whether it recurs, whether it requires an additional layer of opt-in, and the URLs for further information about it. It should contain the following fields: | ||
| Line 112: | Line 118: | ||
* testId: Can be either a number or a string, but must be unique - an experiment must not have the same id as another experiment or survey. These IDs will end up used in preference names, so it must not contain spaces. Required. | * testId: Can be either a number or a string, but must be unique - an experiment must not have the same id as another experiment or survey. These IDs will end up used in preference names, so it must not contain spaces. Required. | ||
* testInfoUrl: the URL of a page where the user can go to get more information about the experiment. | * testInfoUrl: the URL of a page where the user can go to get more information about the experiment. | ||
* optInRequired: Set this to true if the test should not start automatically on the start date, but should instead only start if the user explicitly tells it to start. This should be the case for tests that change the Firefox UI, such as A-B tests. Note that the UI to support optInRequired has not yet been implemented in the client as of version 1.0. | |||
* optInRequired: Set this to true if the test should not start automatically on the start date, but should instead only start if the user explicitly tells it to start. This should be the case for tests that change the Firefox UI, such as A-B tests. Note that the UI to support optInRequired has not yet been implemented in the client as of version 0 | * recursAutomatically: Set this to true if the test should recur automatically after it ends. | ||
* recursAutomatically: Set this to true if the test should recur automatically after it ends | |||
* recurrenceInterval: If recursAutomatically is true, then this is the number of days (measured from the startDate) after which the study will begin again. For obvious reasons, this must be longer than duration. Has no effect if recursAutomatically is false. | * recurrenceInterval: If recursAutomatically is true, then this is the number of days (measured from the startDate) after which the study will begin again. For obvious reasons, this must be longer than duration. Has no effect if recursAutomatically is false. | ||
* versionNumber: An integer for telling apart different versions of the same experiment. Whenever a significant change is made to the experiment code, the versionNumber should be incremented. The version number gets attached to the user's data upload when they submit results for the experiment; it can then be detected on the server side to tell apart submissions from different versions of the experiment. Required. | * versionNumber: An integer for telling apart different versions of the same experiment. Whenever a significant change is made to the experiment code, the versionNumber should be incremented. The version number gets attached to the user's data upload when they submit results for the experiment; it can then be detected on the server side to tell apart submissions from different versions of the experiment. Required. | ||
* minTPVersion: A version string (e.g. "1.0a1") corresponding to a version of the Test Pilot extension. Versions of the extension older than minTPVersion will not attempt to download or run this study. It is optional; if not specified, all versions of the extension will download and run the study. | |||
=== Example === | ==== experimentInfo Example ==== | ||
The experimentInfo object for the Tab open/close study is as follows: | The experimentInfo object for the Tab open/close study is as follows: | ||
| Line 136: | Line 142: | ||
}; | }; | ||
== dataStoreInfo == | === dataStoreInfo === | ||
dataStoreInfo provides parameters for the initialization of a client-side SQLite database table which will be used to store the events that your experiment records. It's basically a schema description, that is fed to an object-relational mapper built into the extension. | dataStoreInfo provides parameters for the initialization of a client-side SQLite database table which will be used to store the events that your experiment records. It's basically a schema description, that is fed to an object-relational mapper built into the extension. | ||
Your | When the Test Pilot extension runs your experiment, it will generate a Store object based on the specification you gave it in dataStoreInfo. Your observers (see below) will be able to call a method to record objects into this store. The info you provide in dataStoreInfo determines how the properties of this event object get mapped to the columns in the database. | ||
dataStoreInfo must have the following properties (all required): | dataStoreInfo must have the following properties (all required): | ||
| Line 153: | Line 159: | ||
** default (not yet implemented): What value to use if the eventObject passed in does not have a property matching this column. | ** default (not yet implemented): What value to use if the eventObject passed in does not have a property matching this column. | ||
=== Example of | ==== Example of dataStoreInfo ==== | ||
exports.dataStoreInfo = { | exports.dataStoreInfo = { | ||
| Line 173: | Line 179: | ||
]}; | ]}; | ||
=== Example of storing event objects === | ==== Example of storing event objects ==== | ||
In | In observer code: | ||
this. | this.record({ | ||
event_code: TabsExperimentConstants.DRAG_EVENT, | event_code: TabsExperimentConstants.DRAG_EVENT, | ||
timestamp: Date.now(), | timestamp: Date.now(), | ||
| Line 188: | Line 194: | ||
This anonymous event object passed to storeEvent has properties matching the properties of the columns specified in dataStoreInfo; it provides all the expected properties except for tab_site_hash, which will therefore default to 0. | This anonymous event object passed to storeEvent has properties matching the properties of the columns specified in dataStoreInfo; it provides all the expected properties except for tab_site_hash, which will therefore default to 0. | ||
== webContent == | === webContent === | ||
(TODO out of date) | |||
The webContent object is used to fill in the content of the experiment status page, which lives on the client side as part of the extension. It's at the url chrome://testpilot/content/status.html . webContent must contain the following properties: | The webContent object is used to fill in the content of the experiment status page, which lives on the client side as part of the extension. It's at the url chrome://testpilot/content/status.html . webContent must contain the following properties: | ||
| Line 207: | Line 215: | ||
* graphUtils: a simple graphing library object containing methods drawTimeSeriesGraph() and drawPieChart(). TODO: Document how to use these methods. | * graphUtils: a simple graphing library object containing methods drawTimeSeriesGraph() and drawPieChart(). TODO: Document how to use these methods. | ||
== Observer == | === Observer === | ||
(TODO WAAAY out of date) | |||
The Observer of your experiment has the very important responsibility of actually doing the data collection, and passing the collected data along to the store you defined in dataStoreInfo. | The Observer of your experiment has the very important responsibility of actually doing the data collection, and passing the collected data along to the store you defined in dataStoreInfo. | ||
| Line 223: | Line 233: | ||
Besides the constructor, your Observer must provide an uninstall() method, which is called (with no arguments) when the observer is no longer needed. Any cleanup that you need to do (such as unregistering listeners, etc.) must be done in the uninstall() method. | Besides the constructor, your Observer must provide an uninstall() method, which is called (with no arguments) when the observer is no longer needed. Any cleanup that you need to do (such as unregistering listeners, etc.) must be done in the uninstall() method. | ||
=== Example === | ==== Example Observer ==== | ||
Here's the skeleton of an Observer: | Here's the skeleton of an Observer: | ||