MozillaWiki

QA/Telemetry: Difference between revisions

< QA
(update QA contacts)
 
(51 intermediate revisions by 5 users not shown)
Line 12: Line 12:
| style="background:#9D9; text-align:center;" | Incremental Improvements
| style="background:#9D9; text-align:center;" | Incremental Improvements
| style="text-align:center;" | [https://mozillians.org/en-US/u/bsmedberg/ Benjamin Smedberg]
| style="text-align:center;" | [https://mozillians.org/en-US/u/bsmedberg/ Benjamin Smedberg]
| style="text-align:center;" | [https://mozillians.org/en-US/?next=/en-US/u/kjozwiak/ Kamil Jozwiak]
| style="text-align:center;" | [https://mozillians.org/en-US/u/bsmedberg/ Benjamin Smedberg]
| style="background:#9D9; text-align:center;" | OK
| style="background:#9D9; text-align:center;" | OK
|}
|}
Line 42: Line 42:
== Test Plan ==
== Test Plan ==


* The Telemetry Experiment test plan can be found: [https://wiki.mozilla.org/QA/Telemetry/TelemetryTestCases Here]
* QA Testing Template can be found in the following etherpad: [https://public.etherpad-mozilla.org/p/TelemetryExperimentQATemplate Here]
* Telemetry Experiment test plans can be found: [https://wiki.mozilla.org/QA/Telemetry/TelemetryTestCases Here]


== Experiments ==
== Experiments ==


You can view current and past telemetry experiments [https://telemetry-experiment.cdn.mozilla.net/ Here]
You can view current and past telemetry experiments [https://telemetry-experiment.cdn.mozilla.net/ Here]
* '''Note:''' Some experiments that have already been expired may not be grayed out once expired when viewing the experiment list [https://telemetry-experiment.cdn.mozilla.net/ Here]. The experiment is technically already disabled and once another experiment is pushed, the list will than be updated and the expired experiment will be grayed out.


{| class="wikitable"
{| class="wikitable"
Line 64: Line 67:
| style="text-align:center;" | 15-May-2014
| style="text-align:center;" | 15-May-2014
| style="text-align:center;" | 5.0 days
| style="text-align:center;" | 5.0 days
| style="text-align:center;" | Nightly
| style="text-align:center;" | nightly
| style="text-align:center;" | N/A
| style="text-align:center;" | N/A
| style="text-align:center;" | 25%
| style="text-align:center;" | 25%
Line 74: Line 77:
| style="text-align:center;" | 30-Sep-2014
| style="text-align:center;" | 30-Sep-2014
| style="text-align:center;" | 28.0 days
| style="text-align:center;" | 28.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | en-US
| style="text-align:center;" | en-US
| style="text-align:center;" | 50%
| style="text-align:center;" | 50%
Line 84: Line 87:
| style="text-align:center;" | 01-Aug-2014
| style="text-align:center;" | 01-Aug-2014
| style="text-align:center;" | 36.0 days
| style="text-align:center;" | 36.0 days
| style="text-align:center;" | Aurora
| style="text-align:center;" | aurora
| style="text-align:center;" | de
| style="text-align:center;" | de
| style="text-align:center;" | 100%
| style="text-align:center;" | 100%
Line 94: Line 97:
| style="text-align:center;" | 02-Sep-2014
| style="text-align:center;" | 02-Sep-2014
| style="text-align:center;" | 42.0 days
| style="text-align:center;" | 42.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | N/A
| style="text-align:center;" | 5%
| style="text-align:center;" | 5%
Line 104: Line 107:
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | pl
| style="text-align:center;" | pl
| style="text-align:center;" | 100%
| style="text-align:center;" | 100%
Line 114: Line 117:
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | tr
| style="text-align:center;" | tr
| style="text-align:center;" | 100%
| style="text-align:center;" | 100%
Line 124: Line 127:
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | vi
| style="text-align:center;" | vi
| style="text-align:center;" | 100%
| style="text-align:center;" | 100%
Line 134: Line 137:
| style="text-align:center;" | 10-Sep-2014
| style="text-align:center;" | 10-Sep-2014
| style="text-align:center;" | 7.0 days
| style="text-align:center;" | 7.0 days
| style="text-align:center;" | Nightly
| style="text-align:center;" | nightly
| style="text-align:center;" | N/A
| style="text-align:center;" | N/A
| style="text-align:center;" | 50%
| style="text-align:center;" | 50%
Line 144: Line 147:
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | es-AR
| style="text-align:center;" | es-AR
| style="text-align:center;" | 100%
| style="text-align:center;" | 100%
Line 154: Line 157:
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 25-Nov-2014
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | 126.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | es-ES
| style="text-align:center;" | es-ES
| style="text-align:center;" | 100%
| style="text-align:center;" | 100%
Line 164: Line 167:
| style="text-align:center;" | 10-Jan-2015
| style="text-align:center;" | 10-Jan-2015
| style="text-align:center;" | 7.0 days
| style="text-align:center;" | 7.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | N/A
| style="text-align:center;" | 25%
| style="text-align:center;" | 25%
Line 174: Line 177:
| style="text-align:center;" | 15-Jan-2015
| style="text-align:center;" | 15-Jan-2015
| style="text-align:center;" | 32.0 days
| style="text-align:center;" | 32.0 days
| style="text-align:center;" | BETA (Win Only)
| style="text-align:center;" | beta (Win Only)
| style="text-align:center;" | N/A
| style="text-align:center;" | N/A
| style="text-align:center;" | 10%
| style="text-align:center;" | 10%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Flash_Protected-Mode_Testing| Here]]
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Flash_Protected-Mode_Testing| Here]]
| style="background:#9D9; text-align:center;" | '''Completed & Disabled'''
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
|-
| Plugin Video Usage Metrics
| Plugin Video Usage Metrics
Line 184: Line 187:
| style="text-align:center;" | 23-Feb-2015
| style="text-align:center;" | 23-Feb-2015
| style="text-align:center;" | 60.0 days
| style="text-align:center;" | 60.0 days
| style="text-align:center;" | BETA
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | N/A
| style="text-align:center;" | 10%
| style="text-align:center;" | 10%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Plugin_Video_Usage_Metrics| Here]]
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Plugin_Video_Usage_Metrics| Here]]
| style="background:#00FFFF; text-align:center;" | '''In Progress'''
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
|-
| Search Experiment
| Search Experiment
Line 198: Line 201:
| style="text-align:center;" | 70%
| style="text-align:center;" | 70%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Search_Experiment_.5Btr.5D| Here]]
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Search_Experiment_.5Btr.5D| Here]]
| style="background:#00FFFF; text-align:center;" | '''In Progress'''
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Search Experiment
| style="text-align:center;" | 25-Mar-2015
| style="text-align:center;" | 07-May-2015
| style="text-align:center;" | 30.0 days
| style="text-align:center;" | release
| style="text-align:center;" | zh-TW
| style="text-align:center;" | 50%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Search_Experiment_.5Bzh-TW.5D| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Multi-process Firefox A/B test
| style="text-align:center;" | 20-Sep-2015
| style="text-align:center;" | 01-Nov-2015
| style="text-align:center;" | 10.0 days
| style="text-align:center;" | aurora
| style="text-align:center;" | N/A
| style="text-align:center;" | 100%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Multi-process_Firefox_A.2FB_test| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Search result pages in history
| style="text-align:center;" | 10-Nov-2015
| style="text-align:center;" | 24-Nov-2015
| style="text-align:center;" | 1.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 10%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Search_result_pages_in_history| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Multi-process Firefox A/B test
| style="text-align:center;" | 14-Dec-2015
| style="text-align:center;" | 04-Jan-2016
| style="text-align:center;" | 10.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 15%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#e10s_A.2FB_on_BETA| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Unified URLBar Experiment
| style="text-align:center;" | 11-Jan-2016
| style="text-align:center;" | 24-Feb-2016
| style="text-align:center;" | 14.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 10%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Unified_URLBar_Experiment| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Multi-process Firefox A/B test Beta 45
| style="text-align:center;" | 25-Jan-2016
| style="text-align:center;" | 11-Feb-2016
| style="text-align:center;" | 15.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 50%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Multi-process_Firefox_A.2FB_Experiment_Beta| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Multi-process Firefox A/B test Beta 45 2nd Experiment
| style="text-align:center;" | 12-Feb-2016
| style="text-align:center;" | 25-Feb-2016
| style="text-align:center;" | 15.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 50%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Multi-process_Firefox_A.2FB_Experiment_Beta 2nd Experiment| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|
|-
| E10s Checkerboard Experiment aka DisplayPort Size Tuning
| style="text-align:center;" | 25-Feb-2016
| style="text-align:center;" | 19-Mar-2016
| style="text-align:center;" | 7.0 days
| style="text-align:center;" | nightly
| style="text-align:center;" | N/A
| style="text-align:center;" | 50%
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#E10s Checkerboard Experiment aka DisplayPort Size Tuning| Here]]
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Multi-process Firefox A/B test 3 (Disabled)
| style="text-align:center;" | 07-Mar-2016
| style="text-align:center;" | 24-Mar-2016
| style="text-align:center;" | 15.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 50%
| style="text-align:center;" | N/A
| style="background:#9D9; text-align:center;" | '''Completed'''
|-
| Plugin Block Experiment
| style="text-align:center;" | 02-May-2016
| style="text-align:center;" | 23-May-2016
| style="text-align:center;" | 21.0 days
| style="text-align:center;" | beta
| style="text-align:center;" | N/A
| style="text-align:center;" | 10%
| style="text-align:center;" | [https://public.etherpad-mozilla.org/p/bug-1248813-plugin-block-experiment-test-plan Here]
| style="background:#00bfff; text-align:center;" | '''Deployed'''
|}
|}


== Telemetry Experiments/FHR Documentation ==
== Telemetry Experiments/FHR Documentation ==
* [[QA/Telemetry/Developing_a_Telemetry_Experiment| Summary of all steps of development process]]
* [[QA/Telemetry/Installing_Staging_Server|Setup & Installing Telemetry Staging Server]]
* [[QA/Telemetry/Installing_Staging_Server|Setup & Installing Telemetry Staging Server]]
* [http://codefirefox.com/video/install-telmetry-experiment Video example of installing Telemetry Experiments]
* [http://codefirefox.com/video/install-telmetry-experiment Video example of installing Telemetry Experiments]
Line 210: Line 315:
* [[QA/Telemetry/InstallSameExperiment|Installing Multiple Experiments]]
* [[QA/Telemetry/InstallSameExperiment|Installing Multiple Experiments]]
* [[QA/Telemetry/RemovingExperimentsJSON|Removing experiments.json]]
* [[QA/Telemetry/RemovingExperimentsJSON|Removing experiments.json]]
* [https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/experiments/manifest.html Experiments Manifests (developers)]
* [http://gecko.readthedocs.org/en/latest/browser/experiments/experiments/manifest.html Experiments Manifests (developers)]
* [[QA/Telemetry/EditingXPIFiles|Editing Extensions (XPI Files)]]
* [[QA/Telemetry/EditingXPIFiles|Editing Extensions (XPI Files)]]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Firefox_Health_Report Interfacing with FHR]
* [https://wiki.mozilla.org/QA/Desktop_Firefox/Firefox_Health_Report Interfacing with FHR]
* [[QA/Telemetry/State%26ErrorCodes|Definitions of States & Error codes]]


== QA Contact Information ==
== QA Contact Information ==
Line 220: Line 326:
'''Internet Relay Chat (IRC):''' [Quickest Response]
'''Internet Relay Chat (IRC):''' [Quickest Response]


* Nick: kjozwiak
* Nick: StefanG, bsmedberg, kjozwiak
* Channels: #fhr #qa
* Channels: #qa
* IRC Setup: https://wiki.mozilla.org/IRC
* IRC Setup: https://wiki.mozilla.org/IRC
'''Email:''' [Slowest Response]
* kjozwiak [at] mozilla.com
* kamiljoz [at] gmail.com

Latest revision as of 18:35, 22 February 2017

Telemetry Experiments

Feature Status Lead Engineer QA Lead Status
Telemetry Experiments Incremental Improvements Benjamin Smedberg Benjamin Smedberg OK

Project Goal

Allow Mozilla to deploy experiments to a statistically-relevant population of users and measure the results.

Project Summary

With telemetry experiments enabled, Firefox will periodically retrieve a list of experiments from a secure server if the user meets the specified condition(s). Possible conditions include:

  • Product
  • Version
  • OS
  • Channel
  • Build ID
  • Language
  • Sample ratio

Each experiment will include the following information:

  • experiment ID
  • specific conditions for running the experiment
  • experiment start date & end date
  • experiment duration
  • XPI URL & hash

Test Plan

  • QA Testing Template can be found in the following etherpad: Here
  • Telemetry Experiment test plans can be found: Here

Experiments

You can view current and past telemetry experiments Here

  • Note: Some experiments that have already been expired may not be grayed out once expired when viewing the experiment list Here. The experiment is technically already disabled and once another experiment is pushed, the list will than be updated and the expired experiment will be grayed out.
Experiment Start Date End Date Max Runtime Channel Locale(s) Sample Rate Test Plan Current Status
Reorder about:newtab tiles 01-Jan-2014 15-May-2014 5.0 days nightly N/A 25% Here Completed
Search Experiment 01-Jul-2014 30-Sep-2014 28.0 days beta en-US 50% Here Completed
Automatic Translation 27-Jun-2014 01-Aug-2014 36.0 days aurora de 100% Here Completed
OOPP container unload timeout tester 01-Jul-2014 02-Sep-2014 42.0 days beta N/A 5% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta pl 100% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta tr 100% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta vi 100% Here Completed
Invisible test for experiment branch correctness 14-Aug-2014 10-Sep-2014 7.0 days nightly N/A 50% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta es-AR 100% Here Completed
Automatic Translation 16-Jul-2014 25-Nov-2014 126.0 days beta es-ES 100% Here Completed
New Tab Data 10-Dec-2014 10-Jan-2015 7.0 days beta N/A 25% Here Completed
Flash Protected-Mode Testing 15-Dec-2014 15-Jan-2015 32.0 days beta (Win Only) N/A 10% Here Completed
Plugin Video Usage Metrics 12-Jan-2015 23-Feb-2015 60.0 days beta N/A 10% Here Completed
Search Experiment 28-Jan-2015 07-Apr-2015 30.0 days release tr 70% Here Completed
Search Experiment 25-Mar-2015 07-May-2015 30.0 days release zh-TW 50% Here Completed
Multi-process Firefox A/B test 20-Sep-2015 01-Nov-2015 10.0 days aurora N/A 100% Here Completed
Search result pages in history 10-Nov-2015 24-Nov-2015 1.0 days beta N/A 10% Here Completed
Multi-process Firefox A/B test 14-Dec-2015 04-Jan-2016 10.0 days beta N/A 15% Here Completed
Unified URLBar Experiment 11-Jan-2016 24-Feb-2016 14.0 days beta N/A 10% Here Completed
Multi-process Firefox A/B test Beta 45 25-Jan-2016 11-Feb-2016 15.0 days beta N/A 50% Here Completed
Multi-process Firefox A/B test Beta 45 2nd Experiment 12-Feb-2016 25-Feb-2016 15.0 days beta N/A 50% Here Completed
E10s Checkerboard Experiment aka DisplayPort Size Tuning 25-Feb-2016 19-Mar-2016 7.0 days nightly N/A 50% Here Completed
Multi-process Firefox A/B test 3 (Disabled) 07-Mar-2016 24-Mar-2016 15.0 days beta N/A 50% N/A Completed
Plugin Block Experiment 02-May-2016 23-May-2016 21.0 days beta N/A 10% Here Deployed

Telemetry Experiments/FHR Documentation

QA Contact Information

If you have any questions or issues regarding telemetry, you can contact me via the following methods:

Internet Relay Chat (IRC): [Quickest Response]

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Telemetry&oldid=1163570"