|
|
| Line 1: |
Line 1: |
| ''
| | {{warning|The original Mozilla Labs Test Pilot project has been retired. Click the 'View History' link on this page to read the old content.}} |
| Information about getting the Test Pilot source code, installing the
| |
| development version, where the important code lives, seeing debugging
| |
| output, useful tricks for debugging and testing changes, etc.''
| |
|
| |
|
| | As of January 2016 a similar initiative, formerly named Idea Town, has been [http://micropipes.com/blog/2016/01/27/meet-the-new-test-pilot/ named Test Pilot]. |
|
| |
|
| ==To install Test Pilot extension on Firefox==
| | '''[[Test_Pilot|Learn more about the new Test Pilot!]]''' |
| | |
| The most recent release of the extension is at
| |
| https://addons.mozilla.org/services/install.php?addon_id=testpilot
| |
| | |
| However, we really want to be trying to reproduce and fix bugs in the
| |
| development trunk, not the latest released version. To install the
| |
| version from the development trunk:
| |
| | |
| # Make sure you have Python and Mercurial installed.
| |
| # hg clone http://hg.mozilla.org/labs/testpilot/
| |
| # cd testpilot
| |
| # python manage.py install --profile <profile> (where <profile> is the name of the Firefox profile to install the extension to; you may want to create a new profile for this purpose first. You may omit --profile <profile> if you wish to install to the 'default' profile)
| |
| | |
| ==The Code==
| |
| Most of the important stuff is in testpilot/extension/modules:
| |
| | |
| testpilot/extension/modules/tasks.js
| |
| | |
| It defines the base class and subclasses for "Tasks". Tasks can be surveys or experiments. Most of the Test Pilot extension code is concerned with managing Tasks: tracking their state, providing a UI for the user to look at them all, etc.
| |
| | |
| testpilot/extension/modules/setup.js
| |
| It is the core module, contains the logic for managing the list of tasks, also handles UI elements such as the task bar menu and popup notifications.
| |
| | |
| | |
| | |
| testpilot/extension/modules/remote-experiment-loader.js
| |
| downloads JS securable modules from testpilot.mozillalabs.com and loads them.
| |
| (experiment logic and survey questions are implemented in these
| |
| securable modules; after being loaded by remote-experiment-loader, they
| |
| are passed back to setup.js, which wraps them in new Task instances and
| |
| manages them.) Remote-experiment-loader uses the "Cuddlefish" library,
| |
| an implementation of the JS Securable Module standard, so for full
| |
| understanding it will help to get familiar with the documentation here:
| |
| https://wiki.mozilla.org/Labs/Jetpack/JEP/25
| |
| | |
| | |
| testpilot/extension/modules/experiment_data_store.js
| |
| Persistence layer; provides interface for each experiment to
| |
| create/read/update/delete its own table in the built-in Firefox SQLite
| |
| database.
| |
| | |
| | |
| testpilot/extension/content/browser.js
| |
| Startup code, some menu item handlers
| |
| | |
| | |
| testpilot/extension/components/TestPilot.js
| |
| startup code
| |
| | |
| | |
| testpilot/extension/content/all-studies-window.xul and all-studies-window.js
| |
| The main Test Pilot UI window showing all the running experiments and their statuses - the one that comes up when you pick "All your studies" from the menu - is implemented in these files.
| |
| | |
| testpilot/extension/content/status.html
| |
| Page that gets displayed (through chrome URL) to user to show current status of an individual experiment and allow user to interact with that experiment by looking at charts, clicking Submit, or opting out. Opens when you click a "More Info" link in the main Test Pilot window.
| |
| | |
| | |
| testpilot/extension/content/experiment-page.js
| |
| Javascript code that populates status.html with relevant information. The entry point is the function loadExperimentPage(). The code that handles clicks on the Upload button is also here.
| |
| | |
| ==Configuring the Test Pilot directory index==
| |
| Test Pilot uses a special directory file to tell it what tests to
| |
| download and run. The default directory file is at
| |
| https://testpilot.mozillalabs.com/testcases/index.json
| |
| | |
| There is a different directory file just for developers and testers
| |
| called index-dev.json. This is so we can tell the extension to run
| |
| experiment code that we are in the process of testing, before we release
| |
| that extension code to the wider audience. So, you'll want to sign up
| |
| for index-dev.json. To do this:
| |
| | |
| # Open about:config
| |
| # search for "testpilot"
| |
| # Find the pref "extensions.testpilot.indexFileName"
| |
| # Change the value to "index-dev.json".
| |
| | |
| == Debugging output and error log ==
| |
| | |
| The Test Pilot extension emits a ton of logging and debugging output, which goes to a file called TestPilotErrorLog.log in your profile directory.
| |
| | |
| | |
| == Manually changing the state of an experiment ==
| |
| | |
| This is very useful for testing, because otherwise you would have to
| |
| wait for a week to pass before a test was ready to submit!
| |
| The status of each task (experiments or surveys) is stored in
| |
| preferences. You can manually edit these through about:config.
| |
| | |
| There are two relevant preferences for each one:
| |
| extensions.testpilot.startDate.(id) and
| |
| extensions.testpilot.taskstatus.(id). So for example, the ID of the
| |
| menu item usage study is 4, so its starting date is stored in
| |
| extensions.testpilot.startdate.4, and its status is in
| |
| extensions.testpilot.taskstatus.4.
| |
| | |
| The taskstatus is one of the following codes (defined in modules/tasks.js):
| |
| | |
| 0: New
| |
| | |
| 1: Pending
| |
| | |
| 2: Starting
| |
| | |
| 3: In progress
| |
| | |
| 4: Finished
| |
| | |
| 5: Cancelled
| |
| | |
| 6: Submitted
| |
| | |
| 7: Results
| |
| | |
| 8: Archived
| |
| | |
| If a task is currently in progress and you want to fast-forward to the
| |
| end (e.g. to test submission of a study), just change its taskstatus
| |
| preference to 4 (finished).
| |
| | |
| If a task has already ended and you want to reset it so that you can run
| |
| it again from the beginning, you can just right-click (or ctrl-click) on
| |
| its taskstatus preference and its startDate preference and "Reset" both
| |
| of them.
| |
| | |
| ID codes for all experiments and surveys that have been run so far:
| |
| | |
| 1 = Tab open/close study
| |
| | |
| 2 = Week-in-the-life study
| |
| | |
| 3 = Accounts and Passwords study
| |
| | |
| 4 = Menu item study
| |
| | |
| 5 = Tab switch/grouping study
| |
| | |
| 1000 = "Ghost study" (this study doesn't do anything; I'm just using it
| |
| for purposes of testing and debugging some server-side stuff.)
| |
| | |
| Surveys have strings instead of numerical ID codes. (There was a reason
| |
| for this difference once, but the reason is no longer valid, so it's
| |
| just for backwards compatibility.) The two surveys that have been
| |
| defined are:
| |
| "account_password_survey" and "basic_panel_survey".
| |
| | |
| ==The Debug Page==
| |
| | |
| When Test Pilot is running, enter chrome://testpilot/content/debug.html
| |
| into the location bar to get to the debug page.
| |
| | |
| There are no links to this page nor any way to load it from the UI,
| |
| because users are never supposed to see it (I made it just for my own
| |
| use), but there are tons of useful things you can do here to debug Test
| |
| Pilot problems. I use it constantly when developing Test Pilot. You
| |
| may find it helpful too.
| |
| | |
| You can enter the ID number for an experiment (e.g. for the Menu Item
| |
| Usage study, ID number = 4) into the field on top that says "Experiment
| |
| number". Then:
| |
| | |
| * Click "Show Raw Data" to see current contents of the database table for this experiment.
| |
| * Click "Wipe my data" to delete everything from that database table.
| |
| * Click "NUKE" (or rather, please don't click it!) to completely drop the database table for the selected experiment.
| |
| * Click "Popup" to test the popup notification (showing status of selected study).
| |
| * Click "Reload all experiments" to do that without having to restart Firefox
| |
| * Click "Run Unit Tests"... oh yeah, I used to have unit tests |:-P
| |
| They have not been maintained lately. I need to get on that. This
| |
| button will not work.
| |
| * Click "Remind me" to test popup notification (showing reminder of pending tasks)
| |
| * Enter a numerical status field in the next field and click "Set" to set the status of the experiment to that code (I usually do this through about:config instead.)
| |
| * Click "Show Metadata" to see the metadata that will be attatched to the header of the uploaded data, including firefox version, locale, operating system, installed extensions, etc. to verify correctness.
| |
| * Choose a study filename from the drop-down menu and click "Load Ye Code" to load up the JS code implementing an experiment. You can then edit the code right in the box (or do what I do, edit it in Emacs and then copy-paste it into this box) and click "Save and Run ye code" to test out changes to the experiment code. This is a shortcut I made so that I could rapidly test changes without having to upload new code to the server and restart Test Pilot every time I wanted to change something.
| |
| | |
| If there are other features that I could add to the debug page to make testing/QA easier, please let me know; it's generally pretty trivial for me to add stuff here (I just don't bother polishing the interface at all, which is why it's so ugly).
| |
| | |
| | |
| If you have more questions not answered here, please contact
| |
| jono@mozilla.com. I will give your questions highest priority and
| |
| answer them as soon as possible.
| |