Confirmed users
402
edits
Foundation/Metrics/Optimizely Process: Difference between revisions
Foundation/Metrics/Optimizely Process (view source)
Revision as of 20:07, 19 February 2014
, 19 February 2014Add link to SUMO doc
Adamlofting (talk | contribs) |
Adamlofting (talk | contribs) (Add link to SUMO doc) |
||
| (13 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
This is our process for running front-end tests on Mozilla Foundation websites and tools. The process is new so please feedback on it, and we can improve these guides, and the process. | This is our process for running front-end tests on Mozilla Foundation websites and tools. The process is new so please feedback on it, and we can improve these guides, and the process. | ||
This is a detailed working document on how to run tests. You may also be interested in this higher-level introduction document: [https://support.mozilla.org/en-US/kb/ab-testing-webmaker A/B Testing on Webmaker]. | |||
== Golden rules for A/B testing == | == Golden rules for A/B testing == | ||
| Line 8: | Line 10: | ||
* '''It's a team sport.''' Testing is as good as the number of people who see the results | * '''It's a team sport.''' Testing is as good as the number of people who see the results | ||
== | == How to get something tested == | ||
# If you have quick ideas, dump them here: [https://etherpad.mozilla.org/webmaker-ideas-for-things-to-test Webmaker test ideas Etherpad] | |||
# If you have something specific you definitely want 'actioned', scroll down and follow the guide on how to file the bug and fill out the report template | |||
== What tests are we running? == | == What tests are we running? == | ||
| Line 24: | Line 27: | ||
== Our process to run a test == | == Our process to run a test == | ||
# Open a Bugzilla ticket to assign the test and track it's progress | # Open a Bugzilla ticket under the component the test is being run in to assign the test and track it's progress | ||
# Add [optimizely] to the ticket's whiteboard | |||
# Estimate time to complete the test and record this in the ticket: [http://visualwebsiteoptimizer.com/ab-split-test-duration/ Test duration calculator] | # Estimate time to complete the test and record this in the ticket: [http://visualwebsiteoptimizer.com/ab-split-test-duration/ Test duration calculator] | ||
# Make a copy of this Google doc template and fill out the content: [https://docs.google.com/document/d/1Un0M4Yj749E3NoSr78ZCO0kTR1ZjpmuAsy9MH-0yCWU/edit?usp=sharing Mofo A/B testing report template] | # Make a copy of this Google doc template and fill out the content: [https://docs.google.com/document/d/1Un0M4Yj749E3NoSr78ZCO0kTR1ZjpmuAsy9MH-0yCWU/edit?usp=sharing Mofo A/B testing report template] | ||
| Line 35: | Line 39: | ||
# Put the test live in Optimizely | # Put the test live in Optimizely | ||
# Update the status to live on the [https://wiki.mozilla.org/Webmaker/Testing_Hub Webmaker Webmaker Testing Hub] | # Update the status to live on the [https://wiki.mozilla.org/Webmaker/Testing_Hub Webmaker Webmaker Testing Hub] | ||
# People affected by testing should | # People affected by testing should watch the wiki page for updates | ||
# | # Announce new test to: | ||
# Let the test run until results are significant | #* webmaker on IRC | ||
# Some tests don't produce significant results | #* webmaker@lists.mozilla.org | ||
# Let the test run until results are statistically significant (Optimizely will do the calculations for you) | |||
# Add the results and screenshots into your test write-up | |||
== Concluding a test == | |||
# Some tests don't produce significant results. If this happens, don't be afraid to close the test. There is always something else we can be testing that might have more impact. | |||
# Add the results and Optimizely screenshots into your test write-up | |||
# Move your test into the 'Completed Tests' section in the [https://wiki.mozilla.org/Webmaker/Testing_Hub Webmaker testing hub] | |||
# Announce the write-up to: | |||
#* webmaker on IRC | |||
#* webmaker@lists.mozilla.org | |||
# Close the ticket | |||
# Share this on the next weekly team call and get peer-review on your conclusions | # Share this on the next weekly team call and get peer-review on your conclusions | ||
== Weekly updates == | == Weekly updates == | ||
In the weekly cross-team calls, include a short segment on testing: | In the weekly cross-team calls, include a short segment on testing: | ||
# Announce any tests closed in that week and link to the write up for peer-review | # Announce any tests closed in that week and link to the write up for peer-review | ||
# Remind people to [https://etherpad.mozilla.org/webmaker-ideas-for-things-to-test log new test ideas] | # Remind people to [https://etherpad.mozilla.org/webmaker-ideas-for-things-to-test log new test ideas] | ||
# Ask if any planned tests needed priority | |||
== Potential Complications caused by A/B testing == | |||
When we run A/B tests on the interface of our tools, this can cause confusion to the people using them and teaching with them. Every test we setup requires finding a balance between potential confusion caused and potential improvements made. | |||
All change causes complications (especially in documentation) but the most important thing we can do here is to make our testing process highly visible. If our community know we are running tests, and they know why (i.e. to make the tools better) than they can use this story as a part of their teaching rather than having it cause problems for them. | |||
Some parts of this testing process can be time consuming to implement, but let's be careful not to skip of the communication part of the process. | |||