Labs/Test Pilot/Test Plan

Mobile Test Plan

1. Install the add-on from http://people.mozilla.com/~jdicarlo/tpm.xpi onto Fennec on an Android phone.
2. Upon first-run, you should see a Welcome page. Make sure the page looks good and fits on the mobile screen, and that the links work.
   1. The next time you start up, the welcome page should not reappear. It's only supposed to appear once. If you want to force it to appear again in order to test it again, go to about:config and reset the pref extensions.testpilot.lastversion.
3. Go to the add-ons manager and make sure Test Pilot appears there.
4. Click Test Pilot, then click Options. You should see the following options:
   1. Run User Studies (default to Yes)
   2. Automatically Submit (default to No)
   3. See All Studies (button)
5. Click See All Studies. You should go to a page that says "All Test Pilot Studies" which lists studies called "Mobile Example" and "Mobile Heatmap"
6. Click on the title "Mobile Example".
7. This should take you to a page that says "Thank you, Test Pilot!" and describes the Mobile Example study in progress.
   1. You should be able to click the hide/show buttons for different subsections of the page (usage data, privacy info, contact info) to fold and unfold them
   2. In the usage data section there should be a bar chart here with two bars on it, which should be clearly labeled as "back button" and "forward button" and should be accurate counts of the number of times you hit each button since the study started. (Please test the accuracy by using the forward and back buttons in Fennec a few times and making sure the count is correct for both.)
   3. Make sure the links in the contact info section work.
   4. The "Mobile Example" title link should take you to a web page on the Test Pilot site describing the study, which should open in a new tab. (This is currently an invalid URL, so a new tab will open but not take you anywhere useful.)
   5. The "Quit this study" link should take you to a page where you can optionally enter a reason, and then click "Quit the Study" which should take you to a page letting you know that the study is canceled.
8. If the study is canceled, it should appear at the bottom of the "All Test Pilot Studies" page, with an icon to clearly indicate it's canceled. And it should no longer be collecting data, obviously.
9. Assuming you don't cancel the study, the study should prompt you to submit when it finishes its run. You can test this by forcing the study to end, like so:
   1. Load chrome://testpilot/content/debug.html
   2. Make sure "mobile example" is selcted in the dropdown menu in the upper right
   3. Where it says "set it to", pick "4 (Finished)".
10. This should produce an Android notification, with a puzzle-piece icon, that says that the mobile experiment study is finished running. Check that it's legible. When you click on the text in the notification, it should take you to the Mobile Example Study detail page, which should now display a message that the study has finished collecting data. There should be a green button saying "Submit".
    1. (If you go to the Mobile Example Study detail page by the method described in step 6, then it should look the same way, with a green "Submit" button.)
11. If you click the submit button, then you should a few moments later (assuming you have an internet connection so the upload succeeds) the page should change to tell you about the successful upload.
12. After submitting or canceling the study, here's how to reset it for more testing:
    1. Load chrome://testpilot/content/debug.html
    2. Make sure "mobile example" is selcted in the dropdown menu in the upper left
    3. Click "Reset Task"
    4. Click "Reload All Experiments".
    5. The Mobile Example study should now be back in the running state -- you can check this by looking at the Mobile Example Study detail page and verifying that the text says it's in progress. From here, you can test cancellation and/or submission again.
13. If you turn the "automatically submit" pref on, using the test pilot options panel on the add-ons manager, and then force the study to finish running using the instructions from step 8, then you should not see a notification. Instead, the study should just silently submit. You can test that this worked by loading the Mobile Example Study detail page and checking that it has a message saying "thank you for submitting your data".
14. If you turn the "run user studies" pref off, then the study should not run, notify, submit, or collect data at all regardless of what you do.
    1. (Having this pref may be redundant, since turning it off is the same as disabling the add-on. The pref may be removed in a future version.)
15. To test the new metadata collection: Go to chrome://testpilot/content/debug.html then choose a study from the drop-down menu in the upper-left. (If this drop-down menu is empty, wait a few seconds and then reload the page.) Click "show metadata"; you'll see a bunch of JSON data barfed to the screen. Scroll down past the accessibility preferences and look for the system info, especially "sysinfo-arch", "sysinfo-name", and "sysinfo-version". These should show the correct values for the processor type of your phone, the OS name, and the OS version respectively.