QA/Telemetry: Difference between revisions
< QA
Jump to navigation
Jump to search
| Line 68: | Line 68: | ||
| style="text-align:center;" | 25% | | style="text-align:center;" | 25% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Reorder_about:newtab_tiles| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Reorder_about:newtab_tiles| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Search Experiment | | Search Experiment | ||
| Line 78: | Line 78: | ||
| style="text-align:center;" | 50% | | style="text-align:center;" | 50% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Search_Experiment| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Search_Experiment| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Automatic Translation | | Automatic Translation | ||
| Line 88: | Line 88: | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bde.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bde.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| OOPP container unload timeout tester | | OOPP container unload timeout tester | ||
| Line 98: | Line 98: | ||
| style="text-align:center;" | 5% | | style="text-align:center;" | 5% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#OOPP_container_unload_timeout_tester| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#OOPP_container_unload_timeout_tester| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Automatic Translation | | Automatic Translation | ||
| Line 108: | Line 108: | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Automatic Translation | | Automatic Translation | ||
| Line 118: | Line 118: | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Automatic Translation | | Automatic Translation | ||
| Line 128: | Line 128: | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Invisible test for experiment branch correctness | | Invisible test for experiment branch correctness | ||
| Line 138: | Line 138: | ||
| style="text-align:center;" | 50% | | style="text-align:center;" | 50% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bpl.2C_tr.2C_vi.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Automatic Translation | | Automatic Translation | ||
| Line 148: | Line 148: | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bes-AR.2C_es-ES.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bes-AR.2C_es-ES.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Automatic Translation | | Automatic Translation | ||
| Line 158: | Line 158: | ||
| style="text-align:center;" | 100% | | style="text-align:center;" | 100% | ||
| style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bes-AR.2C_es-ES.5D| Here]] | | style="text-align:center;" | [[QA/Telemetry/TelemetryTestCases#Automatic_Translation_Experiment_.5Bes-AR.2C_es-ES.5D| Here]] | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| New Tab Data | | New Tab Data | ||
| Line 168: | Line 168: | ||
| style="text-align:center;" | 25% | | style="text-align:center;" | 25% | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="background:#9D9; text-align:center;" | Completed | | style="background:#9D9; text-align:center;" | '''Completed''' | ||
|- | |- | ||
| Flash Protected-Mode Testing | | Flash Protected-Mode Testing | ||
| Line 178: | Line 178: | ||
| style="text-align:center;" | 10% | | style="text-align:center;" | 10% | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="background:#9D9; text-align:center;" | Completed & Disabled | | style="background:#9D9; text-align:center;" | '''Completed & Disabled''' | ||
|- | |- | ||
| Plugin Video Usage Metrics | | Plugin Video Usage Metrics | ||
| Line 188: | Line 188: | ||
| style="text-align:center;" | 10% | | style="text-align:center;" | 10% | ||
| style="text-align:center;" | N/A | | style="text-align:center;" | N/A | ||
| style="background:#00FFFF; text-align:center;" | In Progress | | style="background:#00FFFF; text-align:center;" | '''In Progress''' | ||
|} | |} | ||
Revision as of 15:12, 2 February 2015
Telemetry Experiments
| Feature | Status | Lead Engineer | QA Lead | Status |
| Telemetry Experiments | Incremental Improvements | Benjamin Smedberg | Kamil Jozwiak | OK |
Project Goal
Allow Mozilla to deploy experiments to a statistically-relevant population of users and measure the results.
Project Summary
With telemetry experiments enabled, Firefox will periodically retrieve a list of experiments from a secure server if the user meets the specified condition(s). Possible conditions include:
- Product
- Version
- OS
- Channel
- Build ID
- Language
- Sample ratio
Each experiment will include the following information:
- experiment ID
- specific conditions for running the experiment
- experiment start date & end date
- experiment duration
- XPI URL & hash
Test Plan
- The Telemetry Experiment test plan can be found: Here
Experiments
You can view current and past telemetry experiments Here
| Experiment | Start Date | End Date | Max Runtime | Channel | Locale(s) | Sample Rate | Test Plan | Current Status |
| Reorder about:newtab tiles | 01-Jan-2014 | 15-May-2014 | 5.0 days | Nightly | N/A | 25% | Here | Completed |
| Search Experiment | 01-Jul-2014 | 30-Sep-2014 | 28.0 days | BETA | en-US | 50% | Here | Completed |
| Automatic Translation | 27-Jun-2014 | 01-Aug-2014 | 36.0 days | Aurora | de | 100% | Here | Completed |
| OOPP container unload timeout tester | 01-Jul-2014 | 02-Sep-2014 | 42.0 days | BETA | N/A | 5% | Here | Completed |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | BETA | pl | 100% | Here | Completed |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | BETA | tr | 100% | Here | Completed |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | BETA | vi | 100% | Here | Completed |
| Invisible test for experiment branch correctness | 14-Aug-2014 | 10-Sep-2014 | 7.0 days | Nightly | N/A | 50% | Here | Completed |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | BETA | es-AR | 100% | Here | Completed |
| Automatic Translation | 16-Jul-2014 | 25-Nov-2014 | 126.0 days | BETA | es-ES | 100% | Here | Completed |
| New Tab Data | 10-Dec-2014 | 10-Jan-2015 | 7.0 days | BETA | N/A | 25% | N/A | Completed |
| Flash Protected-Mode Testing | 15-Dec-2014 | 15-Jan-2015 | 32.0 days | BETA (Win Only) | N/A | 10% | N/A | Completed & Disabled |
| Plugin Video Usage Metrics | 12-Jan-2015 | 23-Feb-2015 | 60.0 days | BETA | N/A | 10% | N/A | In Progress |
Telemetry Experiments/FHR Documentation
- Setup & Installing Telemetry Staging Server
- Setting about:config Preferences
- Browser Console Commands
- Setting up Telemetry Logging
- Changing Experiments Browser Channel
- Installing Multiple Experiments
- Removing experiments.json
- Experiments Manifests (developers)
- Editing Extensions (XPI Files)
- Interfacing with FHR
QA Contact Information
If you have any questions or issues regarding telemetry, you can contact me via the following methods:
Internet Relay Chat (IRC): [Quickest Response]
- Nick: kjozwiak
- Channels: #fhr #qa
- IRC Setup: https://wiki.mozilla.org/IRC
Email: [Slowest Response]
- kjozwiak [at] mozilla.com
- kamiljoz [at] gmail.com