QA/Telemetry/TelemetryTestCases

Telemetry Experiments

Test Cases

• Select Telemetry under the Products section on MozTrap

Experiments

Reorder about:newtab tiles

asd

Automatic Translation Experiment

• Metabug: 1029746 - [meta] Search experiment
• Testing was done in: 1022411 - Build Telemetry Experiment for German translation trial

Full Query

ID	Summary	Priority	Status
1017806	Telemetry Experiments: builtin support for multiple branches of an experiment	--	VERIFIED
1029189	[UX] Design experience for new default search test	--	RESOLVED
1029776	[Search experiment] Create experiment add-on and define user group (cell) on installation	--	RESOLVED
1029792	[Search experiment] Implement notification UI	--	RESOLVED
1029799	[Search experiment] Define experiment parameters	--	VERIFIED
1029807	[Search experiment] Host update manifest	--	RESOLVED
1029810	[Search experiment] Add the search experiment to the experiments server	--	VERIFIED
1029818	[Search experiment] QA experiment add-on and sign-off for deployment	--	VERIFIED
1030864	[Search experiment] Notify user about the end of the experiment	--	VERIFIED
1031320	[Search experiment] Display notification UI for the correct groups	--	RESOLVED
1033125	Deploy telemetry-experiment-server for search-beta experiment	--	RESOLVED
1036719	[Search experiment] Search experiment should only target en-US	--	VERIFIED

12 Total; 0 Open (0%); 6 Resolved (50%); 6 Verified (50%);

Automatic Translation Experiment

• Metabug: 973271 - Automatic Translation
• Testing was done in: 1022411 - Build Telemetry Experiment for German translation trial

Full Query

ID	Summary	Priority	Status
1021921	Story Breakdown - [Translation] Launch pre-trial of instant translation	--	VERIFIED
1022410	Ensure all translation-related strings in Fx32 Aurora are localized in German	--	VERIFIED
1022411	Build Telemetry Experiment for German translation trial	--	VERIFIED
1022413	Update list of languages supported by automatic translation	--	RESOLVED
1022856	implement translation provider attribution	--	VERIFIED
1032265	Deploy telemetry-experiment-server for translation-aurora experiment	--	VERIFIED

6 Total; 0 Open (0%); 1 Resolved (16.67%); 5 Verified (83.33%);

Bugs

V1 [Completed]

Current V1 Status:

Completed & Experiment Published

• Metabug: 973990

*Last Updated: June 19th, 2014*

Full Query

ID	Summary	Priority	Status
973992	Telemetry experiments: addon manager UX	P2	RESOLVED
973997	Telemetry experiments: decide on manifest url scheme and format	--	RESOLVED
973998	Telemetry experiments: initial server	--	VERIFIED
973999	Telemetry experiments: fetch and cache the experiment list	--	RESOLVED
974009	Telemetry experiments: test experiment conditions and enable experiments	--	RESOLVED
974024	Telemetry experiments: add monitoring data to FHR payload	--	RESOLVED
974028	Firefox Telemetry Experiments (rev 1)	--	RESOLVED
974151	Production and staging telemetry-experiments.mozilla.org	--	RESOLVED
975000	Telemetry experiments: ensure that experiment addons are compatible, don't update automatically	--	RESOLVED
977708	Annotate crash reports with enabled experiments	--	RESOLVED
979474	Telemetry experiments: Pin the experiment manifest fetch to a particular CA	P2	RESOLVED
981842	Telemetry experiments: include current-experiment details in about:support	--	RESOLVED
982256	Telemetry experiments: test extension	--	RESOLVED
983231	Telemetry experiments: test manifest condition evaluation	--	RESOLVED
983360	Telemetry experiments: add monitoring data to Telemetry payload	--	RESOLVED
984014	Telemetry experiments: test update mechanism	--	RESOLVED
984387	Telemetry experiments: test coverage for disabling and removal of experiments via manifest	--	RESOLVED
984879	Experiment manager shutdown	--	RESOLVED
985084	Experiment add-ons should be disabled at startup	--	VERIFIED
985089	Telemetry experiments: allow QA to force experiment randomization via pref	--	RESOLVED
985670	Telemetry experiments: invalid timer scheduling when addon & experiment id don't match	--	RESOLVED
985682	Experiments aren't loaded correctly from the cache	--	RESOLVED
986040	Telemetry experiments: Assure that no experiment addon is running yet before starting an experiment.	P3	RESOLVED
986093	Icon for Telemetry Experiments add-ons	--	RESOLVED
986530	Deploy a test experiment to the nightly population	--	VERIFIED
986539	Experiments.disableExperiment() should just disable the current experiment	--	RESOLVED
986677	Include time left in experiment in addon-manager	P2	VERIFIED
987207	Telemetry experiments: Assure that cache save & restore works properly	--	RESOLVED
987225	Telemetry experiments: Combine pending tasks into a serial task	--	RESOLVED
988710	Telemetry experiments: local staging server not working with nightly	--	VERIFIED
988873	Telemetry experiments: enable the experiments feature	--	VERIFIED
989137	Telemetry Experiments: remaining AddonManager integration cleanups	--	VERIFIED
990111	Addon Provider for previously-active experiments	--	VERIFIED
992208	Telemetry Experiments not evaluating automatically: packaging error	--	RESOLVED
992232	telemetry-experiment-server: change deployment repo and script	--	VERIFIED
992258	Trigger a refresh of the experiments view in the addon manager when experiments change	--	VERIFIED
993084	14% regression in ts paint on linux64 seen on fx-team	--	VERIFIED
994686	Permaorange nightly-only failure in browser_experiments.js | Experiment tab hidden by default. - false == true	--	VERIFIED
994727	Telemetry experiments: Test and fix how disabling the feature works.	--	VERIFIED
995027	Intermittent Experiments.jsm Shutdown "ABORT: file resource://gre/modules/AsyncShutdown.jsm, line 431" [@ mozalloc_abort(char const*)]	--	VERIFIED
995032	Telemetry Experiments test will fail when run on different channel	--	RESOLVED
997188	testsuite connects to telemetry-experiment.cdn.mozilla.net	--	VERIFIED
1000114	Extend test coverage for experiments in addon manager UI	--	VERIFIED
1001031	Create ongoing telemetry report on experiment uptake/adoption	--	VERIFIED
1001529	Telemetry experiments: "Experiments" container not initially visible when there's a disabled experiment	--	RESOLVED
1004506	Telemetry experiments: NaN days remaining when having multiple experiments under about:addons	--	VERIFIED
1004665	Experiments: exception thrown if activating experiment with crash reporting disabled	--	RESOLVED
1005270	Trigger experiments initialization on opening the addon manager UI	--	VERIFIED
1007663	Avoid showing empty experiments list	--	VERIFIED
1010397	Experiment manager may undo pending uninstalls	P4	RESOLVED
1017744	Telemetry experiments: remove buttons don't initially disable/delete the experiment	--	RESOLVED
1023450	Experiments are running during mochitests	--	RESOLVED

52 Total; 0 Open (0%); 32 Resolved (61.54%); 20 Verified (38.46%);

V2 [In Progress]

Current V2 Status:

Currently in Progress

• Metabug: N/A (No Metabug available, incremental improvements are being made via separate bugs)

Full Query

ID	Summary	Priority	Status
1010397	Experiment manager may undo pending uninstalls	P4	RESOLVED
1012466	exception on shutdown: "Exception calling provider shutdown", NS_ERROR_FAILURE from nsIObserverService.removeObserver from PreviousExperimentProvider.shutdown	--	RESOLVED
1014194	Base implementation: updater addon hotfix	--	VERIFIED
1017806	Telemetry Experiments: builtin support for multiple branches of an experiment	--	VERIFIED
1021002	Nightly test experiment logging doesn't show failures due to incompatible extensions: because the jsfilter isn't working!	--	VERIFIED
1022411	Build Telemetry Experiment for German translation trial	--	VERIFIED
1026850	Experiment system might get active experiment in a confused state if telemetry was disabled	P4	RESOLVED
1026853	Experiment is displayed as "pending removal" in detailed view	--	VERIFIED
1029799	[Search experiment] Define experiment parameters	--	VERIFIED
1029810	[Search experiment] Add the search experiment to the experiments server	--	VERIFIED
1029818	[Search experiment] QA experiment add-on and sign-off for deployment	--	VERIFIED
1030864	[Search experiment] Notify user about the end of the experiment	--	VERIFIED
1032265	Deploy telemetry-experiment-server for translation-aurora experiment	--	VERIFIED

13 Total; 0 Open (0%); 3 Resolved (23.08%); 10 Verified (76.92%);