QA/Telemetry: Difference between revisions
< QA
Jump to navigation
Jump to search
(update QA contacts) |
|||
| (18 intermediate revisions by 5 users not shown) | |||
| Line 12: | Line 12: | ||
| style="background:#9D9; text-align:center;" | Incremental Improvements | | style="background:#9D9; text-align:center;" | Incremental Improvements | ||
| style="text-align:center;" | [https://mozillians.org/en-US/u/bsmedberg/ Benjamin Smedberg] | | style="text-align:center;" | [https://mozillians.org/en-US/u/bsmedberg/ Benjamin Smedberg] | ||
| style="text-align:center;" | [https://mozillians.org | | style="text-align:center;" | [https://mozillians.org/en-US/u/bsmedberg/ Benjamin Smedberg] | ||
| style="background:#9D9; text-align:center;" | OK | | style="background:#9D9; text-align:center;" | OK | ||
|} | |} | ||
| Line 67: | Line 67: | ||
| style="text-align:center;" | 15-May-2014 | | style="text-align:center;" | 15-May-2014 | ||
| style="text-align:center;" | 5.0 days | | style="text-align:center;" | 5.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | nightly | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="text-align:center;" | 25% | | style="text-align:center;" | 25% | ||
| Line 77: | Line 77: | ||
| style="text-align:center;" | 30-Sep-2014 | | style="text-align:center;" | 30-Sep-2014 | ||
| style="text-align:center;" | 28.0 days | | style="text-align:center;" | 28.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | en-US | | style="text-align:center;" | en-US | ||
| style="text-align:center;" | 50% | | style="text-align:center;" | 50% | ||
| Line 87: | Line 87: | ||
| style="text-align:center;" | 01-Aug-2014 | | style="text-align:center;" | 01-Aug-2014 | ||
| style="text-align:center;" | 36.0 days | | style="text-align:center;" | 36.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | aurora | ||
| style="text-align:center;" | de | | style="text-align:center;" | de | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| Line 97: | Line 97: | ||
| style="text-align:center;" | 02-Sep-2014 | | style="text-align:center;" | 02-Sep-2014 | ||
| style="text-align:center;" | 42.0 days | | style="text-align:center;" | 42.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="text-align:center;" | 5% | | style="text-align:center;" | 5% | ||
| Line 107: | Line 107: | ||
| style="text-align:center;" | 25-Nov-2014 | | style="text-align:center;" | 25-Nov-2014 | ||
| style="text-align:center;" | 126.0 days | | style="text-align:center;" | 126.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | pl | | style="text-align:center;" | pl | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| Line 117: | Line 117: | ||
| style="text-align:center;" | 25-Nov-2014 | | style="text-align:center;" | 25-Nov-2014 | ||
| style="text-align:center;" | 126.0 days | | style="text-align:center;" | 126.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | tr | | style="text-align:center;" | tr | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| Line 127: | Line 127: | ||
| style="text-align:center;" | 25-Nov-2014 | | style="text-align:center;" | 25-Nov-2014 | ||
| style="text-align:center;" | 126.0 days | | style="text-align:center;" | 126.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | vi | | style="text-align:center;" | vi | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| Line 137: | Line 137: | ||
| style="text-align:center;" | 10-Sep-2014 | | style="text-align:center;" | 10-Sep-2014 | ||
| style="text-align:center;" | 7.0 days | | style="text-align:center;" | 7.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | nightly | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="text-align:center;" | 50% | | style="text-align:center;" | 50% | ||
| Line 147: | Line 147: | ||
| style="text-align:center;" | 25-Nov-2014 | | style="text-align:center;" | 25-Nov-2014 | ||
| style="text-align:center;" | 126.0 days | | style="text-align:center;" | 126.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | es-AR | | style="text-align:center;" | es-AR | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| Line 157: | Line 157: | ||
| style="text-align:center;" | 25-Nov-2014 | | style="text-align:center;" | 25-Nov-2014 | ||
| style="text-align:center;" | 126.0 days | | style="text-align:center;" | 126.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | es-ES | | style="text-align:center;" | es-ES | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| Line 167: | Line 167: | ||
| style="text-align:center;" | 10-Jan-2015 | | style="text-align:center;" | 10-Jan-2015 | ||
| style="text-align:center;" | 7.0 days | | style="text-align:center;" | 7.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="text-align:center;" | 25% | | style="text-align:center;" | 25% | ||
| Line 177: | Line 177: | ||
| style="text-align:center;" | 15-Jan-2015 | | style="text-align:center;" | 15-Jan-2015 | ||
| style="text-align:center;" | 32.0 days | | style="text-align:center;" | 32.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta (Win Only) | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="text-align:center;" | 10% | | style="text-align:center;" | 10% | ||
| Line 187: | Line 187: | ||
| style="text-align:center;" | 23-Feb-2015 | | style="text-align:center;" | 23-Feb-2015 | ||
| style="text-align:center;" | 60.0 days | | style="text-align:center;" | 60.0 days | ||
| style="text-align:center;" | | | style="text-align:center;" | beta | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="text-align:center;" | 10% | | style="text-align:center;" | 10% | ||
| Line 251: | Line 251: | ||
| style="text-align:center;" | 10% | | style="text-align:center;" | 10% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Unified_URLBar_Experiment| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Unified_URLBar_Experiment| Here]] | ||
| style="background:# | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |||
| Multi-process Firefox A/B test Beta 45 | |||
| style="text-align:center;" | 25-Jan-2016 | |||
| style="text-align:center;" | 11-Feb-2016 | |||
| style="text-align:center;" | 15.0 days | |||
| style="text-align:center;" | beta | |||
| style="text-align:center;" | N/A | |||
| style="text-align:center;" | 50% | |||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Multi-process_Firefox_A.2FB_Experiment_Beta| Here]] | |||
| style="background:#9D9; text-align:center;" | '''Completed''' | |||
|- | |||
| Multi-process Firefox A/B test Beta 45 2nd Experiment | |||
| style="text-align:center;" | 12-Feb-2016 | |||
| style="text-align:center;" | 25-Feb-2016 | |||
| style="text-align:center;" | 15.0 days | |||
| style="text-align:center;" | beta | |||
| style="text-align:center;" | N/A | |||
| style="text-align:center;" | 50% | |||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Multi-process_Firefox_A.2FB_Experiment_Beta 2nd Experiment| Here]] | |||
| style="background:#9D9; text-align:center;" | '''Completed''' | |||
| | |||
|- | |||
| E10s Checkerboard Experiment aka DisplayPort Size Tuning | |||
| style="text-align:center;" | 25-Feb-2016 | |||
| style="text-align:center;" | 19-Mar-2016 | |||
| style="text-align:center;" | 7.0 days | |||
| style="text-align:center;" | nightly | |||
| style="text-align:center;" | N/A | |||
| style="text-align:center;" | 50% | |||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#E10s Checkerboard Experiment aka DisplayPort Size Tuning| Here]] | |||
| style="background:#9D9; text-align:center;" | '''Completed''' | |||
|- | |||
| Multi-process Firefox A/B test 3 (Disabled) | |||
| style="text-align:center;" | 07-Mar-2016 | |||
| style="text-align:center;" | 24-Mar-2016 | |||
| style="text-align:center;" | 15.0 days | |||
| style="text-align:center;" | beta | |||
| style="text-align:center;" | N/A | |||
| style="text-align:center;" | 50% | |||
| style="text-align:center;" | N/A | |||
| style="background:#9D9; text-align:center;" | '''Completed''' | |||
|- | |||
| Plugin Block Experiment | |||
| style="text-align:center;" | 02-May-2016 | |||
| style="text-align:center;" | 23-May-2016 | |||
| style="text-align:center;" | 21.0 days | |||
| style="text-align:center;" | beta | |||
| style="text-align:center;" | N/A | |||
| style="text-align:center;" | 10% | |||
| style="text-align:center;" | [https://public.etherpad-mozilla.org/p/bug-1248813-plugin-block-experiment-test-plan Here] | |||
| style="background:#00bfff; text-align:center;" | '''Deployed''' | |||
|} | |} | ||
== Telemetry Experiments/FHR Documentation == | == Telemetry Experiments/FHR Documentation == | ||
* [[QA/Telemetry/Developing_a_Telemetry_Experiment| Summary of all steps of development process]] | |||
* [[QA/Telemetry/Installing_Staging_Server|Setup & Installing Telemetry Staging Server]] | * [[QA/Telemetry/Installing_Staging_Server|Setup & Installing Telemetry Staging Server]] | ||
* [http://codefirefox.com/video/install-telmetry-experiment Video example of installing Telemetry Experiments] | * [http://codefirefox.com/video/install-telmetry-experiment Video example of installing Telemetry Experiments] | ||
| Line 274: | Line 326: | ||
'''Internet Relay Chat (IRC):''' [Quickest Response] | '''Internet Relay Chat (IRC):''' [Quickest Response] | ||
* Nick: kjozwiak | * Nick: StefanG, bsmedberg, kjozwiak | ||
* Channels: | * Channels: #qa | ||
* IRC Setup: https://wiki.mozilla.org/IRC | * IRC Setup: https://wiki.mozilla.org/IRC | ||
Latest revision as of 18:35, 22 February 2017
Telemetry Experiments
| Feature | Status | Lead Engineer | QA Lead | Status |
| Telemetry Experiments | Incremental Improvements | Benjamin Smedberg | Benjamin Smedberg | OK |
Project Goal
Allow Mozilla to deploy experiments to a statistically-relevant population of users and measure the results.
Project Summary
With telemetry experiments enabled, Firefox will periodically retrieve a list of experiments from a secure server if the user meets the specified condition(s). Possible conditions include:
- Product
- Version
- OS
- Channel
- Build ID
- Language
- Sample ratio
Each experiment will include the following information:
- experiment ID
- specific conditions for running the experiment
- experiment start date & end date
- experiment duration
- XPI URL & hash
Test Plan
- QA Testing Template can be found in the following etherpad: Here
- Telemetry Experiment test plans can be found: Here
Experiments
You can view current and past telemetry experiments Here
- Note: Some experiments that have already been expired may not be grayed out once expired when viewing the experiment list Here. The experiment is technically already disabled and once another experiment is pushed, the list will than be updated and the expired experiment will be grayed out.
| Experiment | Start Date | End Date | Max Runtime | Channel | Locale(s) | Sample Rate | Test Plan | Current Status | |
| Reorder about:newtab tiles | 01-Jan-2014 | 15-May-2014 | 5.0 days | nightly | N/A | 25% | Here | Completed | |
| Search Experiment | 01-Jul-2014 | 30-Sep-2014 | 28.0 days | beta | en-US | 50% | Here | Completed | |
| Automatic Translation | 27-Jun-2014 | 01-Aug-2014 | 36.0 days | aurora | de | 100% | Here | Completed | |
| OOPP container unload timeout tester | 01-Jul-2014 | 02-Sep-2014 | 42.0 days | beta | N/A | 5% | Here | Completed | |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | beta | pl | 100% | Here | Completed | |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | beta | tr | 100% | Here | Completed | |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | beta | vi | 100% | Here | Completed | |
| Invisible test for experiment branch correctness | 14-Aug-2014 | 10-Sep-2014 | 7.0 days | nightly | N/A | 50% | Here | Completed | |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | beta | es-AR | 100% | Here | Completed | |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | beta | es-ES | 100% | Here | Completed | |
| New Tab Data | 10-Dec-2014 | 10-Jan-2015 | 7.0 days | beta | N/A | 25% | Here | Completed | |
| Flash Protected-Mode Testing | 15-Dec-2014 | 15-Jan-2015 | 32.0 days | beta (Win Only) | N/A | 10% | Here | Completed | |
| Plugin Video Usage Metrics | 12-Jan-2015 | 23-Feb-2015 | 60.0 days | beta | N/A | 10% | Here | Completed | |
| Search Experiment | 28-Jan-2015 | 07-Apr-2015 | 30.0 days | release | tr | 70% | Here | Completed | |
| Search Experiment | 25-Mar-2015 | 07-May-2015 | 30.0 days | release | zh-TW | 50% | Here | Completed | |
| Multi-process Firefox A/B test | 20-Sep-2015 | 01-Nov-2015 | 10.0 days | aurora | N/A | 100% | Here | Completed | |
| Search result pages in history | 10-Nov-2015 | 24-Nov-2015 | 1.0 days | beta | N/A | 10% | Here | Completed | |
| Multi-process Firefox A/B test | 14-Dec-2015 | 04-Jan-2016 | 10.0 days | beta | N/A | 15% | Here | Completed | |
| Unified URLBar Experiment | 11-Jan-2016 | 24-Feb-2016 | 14.0 days | beta | N/A | 10% | Here | Completed | |
| Multi-process Firefox A/B test Beta 45 | 25-Jan-2016 | 11-Feb-2016 | 15.0 days | beta | N/A | 50% | Here | Completed | |
| Multi-process Firefox A/B test Beta 45 2nd Experiment | 12-Feb-2016 | 25-Feb-2016 | 15.0 days | beta | N/A | 50% | Here | Completed | |
| E10s Checkerboard Experiment aka DisplayPort Size Tuning | 25-Feb-2016 | 19-Mar-2016 | 7.0 days | nightly | N/A | 50% | Here | Completed | |
| Multi-process Firefox A/B test 3 (Disabled) | 07-Mar-2016 | 24-Mar-2016 | 15.0 days | beta | N/A | 50% | N/A | Completed | |
| Plugin Block Experiment | 02-May-2016 | 23-May-2016 | 21.0 days | beta | N/A | 10% | Here | Deployed |
Telemetry Experiments/FHR Documentation
- Summary of all steps of development process
- Setup & Installing Telemetry Staging Server
- Video example of installing Telemetry Experiments
- Setting about:config Preferences
- Browser Console Commands
- Setting up Telemetry Logging
- Changing Experiments Browser Channel
- Installing Multiple Experiments
- Removing experiments.json
- Experiments Manifests (developers)
- Editing Extensions (XPI Files)
- Interfacing with FHR
- Definitions of States & Error codes
QA Contact Information
If you have any questions or issues regarding telemetry, you can contact me via the following methods:
Internet Relay Chat (IRC): [Quickest Response]
- Nick: StefanG, bsmedberg, kjozwiak
- Channels: #qa
- IRC Setup: https://wiki.mozilla.org/IRC