|
|
| Line 1: |
Line 1: |
| This is a tutorial explaining how to write a simple Test Pilot experiment, step by step.
| | {{warning|The original Mozilla Labs Test Pilot project has been retired. Click the 'View History' link on this page to read the old content.}} |
|
| |
|
| I'm assuming you start with the '''Example Study''', which is a very simple experiment skeleton intended to be mostly self-documenting. You can turn it into a study by replacing the comments with useful code.
| | As of January 2016 a similar initiative, formerly named Idea Town, has been [http://micropipes.com/blog/2016/01/27/meet-the-new-test-pilot/ named Test Pilot]. |
|
| |
|
| = Where to get the example study =
| | '''[[Test_Pilot|Learn more about the new Test Pilot!]]''' |
| | |
| The example study is distributed as part of the development experiment set. To get it, first switch your Firefox profile over to using the development experiment index, '''index-dev.json''':
| |
| | |
| # Open '''about:config'''
| |
| # Find the pref extensions.testpilot.indexFileName
| |
| # Change its value to index-dev.json
| |
| # Delete the old cached index file which you can find in (your profile directory)/TestPilotExperimentFiles/index.json
| |
| | |
| Restart Firefox and open [chrome://testpilot/content/debug.html], the test pilot debug page.
| |
| | |
| At the bottom of the page (after the large text area) is a drop-down menu listing code module names: select "example-study.js" and click "Load Ye Code".
| |
| | |
| The example study code appears in the large text area. You can modify it here but I recommend copying it to an external code editor and modifying it there.
| |
| | |
| You can also get the code from Hg. It's part of the test pilot experiment Hg repository, which is separate from the extension code Hg repo. The experiment repo is at [http://hg.mozilla.org/labs/testpilotweb].
| |
| | |
| and the example experiment code is at
| |
| | |
| testpilotweb/testcases/example/example-study.js
| |
| | |
| But you might as well get on index-dev.json and get used to using the debug page, since we'll be using it in the next step to test our modifications.
| |
| | |
| = How to modify the example study =
| |
| | |
| The example study provides a skeleton for your study. The important functions are filled in with explanatory comments instead of code. Basically, just replace the comments with the right code.
| |
| | |
| The tricky part of writing a study is generally figuring out how to get the data you need - what Firefox events you need to observe for, etc. Designing the right schema is also important; we'll cover that later.
| |
| | |
| When you've made changes to the code, go back to the debug page at chrome://testpilot/content/debug.html and then select "example-study.js" from the drop-down menu at the bottom of the page. Select all the text in the large text box, delete it, and paste in your modified code. Then click "Save And Run Ye Code".
| |
| | |
| (Make sure you remember to select "example-study.js" from the drop-down menu before clicking "save and run", otherwise you may overwrite another file that you didn't mean to change.)
| |
| | |
| There is no need to restart Firefox. Test Pilot will re-initialize all studies, so your changes will immediately take effect.
| |
| | |
| == Changing the Schema ==
| |
| | |
| If you change the database schema for your study (i.e. your '''exports.dataStoreInfo''') to add or remove columns or change column types, then you will need to drop the old table from the local SQLite database in Test Pilot, otherwise your study will stop working due to having a table that doesn't match the study's desired schema.
| |
| | |
| To drop the table, select the study name from the drop-down menu at the '''top''' of the debug page, and click the button "NUKE". This does a DROP TABLE. You can then click "Save And Run Ye Code" to apply your changes. Finally click "Reset Task" to start the task over again; a new database table with the correct schema will be automatically created.
| |
| | |
| == Resetting a finished or canceled study ==
| |
| | |
| It sometimes happens during the development process that your study finishes running, or that you have to cancel it or force it to finish in order to test something.
| |
| | |
| To continue development, you'll need to reset the study so it can start running again. Go to the debug page and pick the study name from the drop-down menu at the '''top''' of the page. Then click "Reset Task".
| |
| | |
| If you have Test Pilot set up so that studies don't start until after the user is notified (this is the default setting in the version bundled with Firefox 4 beta) you can click "Notify Me" to bring up the notification immediately. Then just close the notification or click the link, and the study will begin.
| |
| | |
| == Debugging ==
| |
| | |
| If your modified study doesn't show up in the drop-down menu at the top of the debug page, and doesn't show up in the usual "All Your Studies" window interface, that means it ran into a problem during startup, such as a syntax error that prevented the study file from being parsed correctly. An error message will usually show up in the error log file:
| |
| | |
| (your profile directory)/TestPilotErrorLog.log
| |
| | |
| So check that if you're not sure what's preventing your study from running.
| |
| | |
| I usually debug my studies by putting dump() statements liberally throughout the code, and then starting up Firefox from a command line so I can see the output of the dump statements in the terminal window. (Make sure you have '''browser.dom.window.dump.enabled''' pref-ed on in order to use this method). You can also use '''console.debug()''' or '''console.trace()''' statements, which will send output to the TestPilotErrorLog.log file.
| |
| | |
| = Steps to a Real Study =
| |
| | |
| Let's start with a very simple study that observes clicks on the back button.
| |
| | |
| == Defining the Global Observer Class ==
| |
| | |
| Our study needs to export an object named '''handlers'''. The base classes in '''study_base_classes.js''' define a class '''GenericGlobalObserver'''. All we need to do is extend this class, instantiate one instance, and export that instance. Find the code for ExampleStudyGlobalObserver and replace it with this:
| |
| | |
| function BackButtonGlobalObserver() {
| |
| // It's very important that our constructor function calls the base
| |
| // class constructor function:
| |
| BackButtonGlobalObserver.baseConstructor.call(this,
| |
| ExampleStudyWindowObserver);
| |
| }
| |
| // use the provided helper method 'extend()' to handle setting up the
| |
| // whole prototype chain for us correctly:
| |
| BaseClasses.extend(BackButtonGlobalObserver,
| |
| BaseClasses.GenericGlobalObserver);
| |
| | |
| | |
| exports.handlers = new BackButtonGlobalObserver();
| |
| | |
| | |
| == Defining the Per-Window Observer Class ==
| |
| | |
| Each Firefox window has its own back button, so we need to put a listener on each one. That means we need to use the per-window observer. We'll use the base class '''BaseClasses.GenericWindowObserver''', defined in the file '''study_base_classes.js''', and extend it.
| |
| | |
| Your study should have this line at the top to import the base classes:
| |
| | |
| BaseClasses = require("study_base_classes.js");
| |
| | |
| Make a subclass of GenericWindowObserver (If you're modifying '''example_study.js''', find the class ExampleStudyWindowObserver and replace it with the following code)
| |
| | |
| function BackButtonWindowObserver(window, globalInstance) {
| |
| // Call base class constructor (Important!)
| |
| BackButtonWindowObserver.baseConstructor.call(this, window, globalInstance);
| |
| }
| |
| // set up BackButtonWindowObserver as a subclass of GenericWindowObserver:
| |
| BaseClasses.extend(BackButtonWindowObserver,
| |
| BaseClasses.GenericWindowObserver);
| |
| BackButtonWindowObserver.prototype.install = function() {
| |
| }
| |
| | |
| We now need to let the global observer class know about our per-window observer class. We do this by passing the per-window observer class as an argument to the global observer's base constructor. The global observer will then automatically instantiate one instance of the per-window observer for each window that is opened, and will handle cleaning up this observer when the window is closed. So our global observer class should now look like this:
| |
| | |
| function BackButtonGlobalObserver() {
| |
| BackButtonGlobalObserver.baseConstructor.call(this,
| |
| BackButtonWindowObserver);
| |
| }
| |
| BaseClasses.extend(BackButtonGlobalObserver,
| |
| BaseClasses.GenericGlobalObserver);
| |
| | |
| == Registering a Listener on the Back Button ==
| |
| | |
| The '''install()''' method of our BackButtonWindowObserver will be called whenever a new window opens. Get the document from the newly opened window, get the back button element from the document, and then attatch a listener to it, like so:
| |
| | |
| BackButtonWindowObserver.prototype.install = function() {
| |
| // this.window already refers to the newly opened window thanks to
| |
| // magic in the base class.
| |
| let backButton = this.window.document.getElementById("back-button");
| |
| if (backButton) {
| |
| this._listen(backButton, "mouseup", function(evt) {
| |
| console.info("You clicked the back button.");
| |
| }, false);
| |
| }
| |
| }
| |
| | |
| The '''_listen()''' method provided by the base class takes the following arguments:
| |
| | |
| *element : the XUL element to listen for events on
| |
| *event name : the name of the event to listen for
| |
| *callback function: the function to be called when the event happens
| |
| *catchCap: boolean, whether or not to use catch capture (if in doubt, use false)
| |
| | |
| _listen() does some magic behind the scenes so that the listening method will automatically be uninstalled when the window closes or the study shuts down.
| |
| | |
| Thus far our callback function is doing nothing but dumping a message to the console. Next we'll see how to record an event to the data store.
| |
| | |
| == Recording an Event to the Data Store ==
| |
| | |
| Recording events is generally done through your GlobalObserver instance. If you're using the base class and you initialized it correctly, you can use the superclass record() method. This method's signature is:
| |
| | |
| record: function(event, callback)
| |
| | |
| Recording to the database is an asynchronous action, so you can optionally provide a callback function which will be called when the recording is done. If you don't need to do anything after the recording is done (you usually won't) then you can leave out the callback argument. If you provide it, your callback function will get a single argument which is true on successful recording and false on failure.
| |
| | |
| The record() method automatically respects the state of Firefox Private Browsing Mode, and will record nothing (passing '''false''' to your callback) when private browsing mode is turned on.
| |
| | |
| Since you're exporting your GlobalObserver instance already, you'll be able to refer to this instance anywhere in your study code module as '''exports.handlers'''. So your call to the record method might look like:
| |
| | |
| exports.handlers.record({});
| |
| | |
| The '''event''' argument to record() must be an object with property names matching the names of the columns in your database. Remember that these columns are defined according to your module's '''exports.dataStoreInfo''' object. So let's define a data store with two columns, one for the name of the button clicked and the other for the timestamp of when it was clicked:
| |
| | |
| exports.dataStoreInfo = {
| |
| fileName: "testpilot_backbutton_results.sqlite",
| |
| tableName: "testpilot_backbutton_study",
| |
| columns: [
| |
| {property: "button_name", type: BaseClasses.TYPE_STRING,
| |
| displayName: "Button Clicked"},
| |
| {property: "timestamp", type: BaseClasses.TYPE_DOUBLE,
| |
| displayName: "Time of Click"}
| |
| ]
| |
| };
| |
| | |
| I've found it to be a good practice to always include a timestamp with any UI event that is recorded, since it lets us reconstruct the user's interactions with more detail when looking at the results later. Timestamps should always be TYPE_DOUBLE as they will get truncated if you use TYPE_INT_32. You can get the current timestamp through Javascript's built-in Date:
| |
| | |
| Date.now()
| |
| | |
| We now need to pass in objects that have a '''button_name''' property and a '''timestamp''' property. Our call to the record method is:
| |
| | |
| exports.handlers.record({button_name: "", timestamp: ""});
| |
| | |
| Let's put that into its proper context in that listener that we registered on the back button in the previous step:
| |
| | |
| BackButtonWindowObserver.prototype.install = function() {
| |
| let backButton = this.window.document.getElementById("back-button");
| |
| if (backButton) {
| |
| this._listen(backButton, "mouseup", function(evt) {
| |
| console.info("You clicked the back button.");
| |
| exports.handlers.record(
| |
| {button_name: "Back Button",
| |
| timestamp: Date.now()});
| |
| }, false);
| |
| }
| |
| }
| |
| | |
| That's it! We're now recording data. Try it out!
| |
| | |
| == Recording Data on Study Startup ==
| |
| | |
| It's often very useful to record some data, and/or do other startup tasks, when the study starts up. For some studies, e.g. ones that can collect all their data immediately rather than waiting to observe UI events, study startup may be the only time we want to collect data at all.
| |
| | |
| For this, we simply override the '''onExperimentStartup()''' method of the global observer class:
| |
| | |
| BackButtonGlobalObserver.prototype.onExperimentStartup = function(store) {
| |
| // store is a reference to the live database table connection
| |
| // you MUST call the base class onExperimentStartup and give it the store
| |
| // reference:
| |
| BackButtonGlobalObserver.superClass.onExperimentStartup.call(this, store);
| |
| // Put your other code here!
| |
| console.info("Back button study is starting up!");
| |
| };
| |
| | |
| I strongly recommend recording a study version number whenever the study starts up. This way, if you have need to modify your study later, you can update the study version number, and then when we're analyzing the data we can tell apart data that came from the earlier version and data that came from the later version. (You could also make a column for the study version and put in the study version with every event recorded, but that will unnecessarily increase the size of the final data upload...)
| |
| | |
| const EVENT_CODE_STUDY_STARTUP = 0;
| |
| exports.experimentInfo = {
| |
| versionNumber: 1,
| |
| // etc etc.
| |
| };
| |
| | |
| BackButtonGlobalObserver.prototype.onExperimentStartup = function(store) {
| |
| BackButtonGlobalObserver.superClass.onExperimentStartup.call(this, store);
| |
| record({event_code: EVENT_CODE_STUDY_STARTUP,
| |
| value: exports.experimentInfo.versionNumber,
| |
| timestamp: Date.now()});
| |
| console.info("Back button study is starting up!");
| |
| };
| |
| | |
| | |
| Coming up with an appropriate schema for this is left as an exercise for the reader.
| |
| | |
| == Setting Values for Experiment Metadata ==
| |
| | |
| TODO
| |
| | |
| == Displaying Cool Graphs to the User ==
| |
| | |
| TODO
| |
| | |
| == Cleaning Up After Our Experiment is Done ==
| |
| | |
| TODO
| |