QA/Telemetry/TelemetryTestCases: Difference between revisions

From MozillaWiki
< QA‎ | Telemetry
Jump to navigation Jump to search
Line 8: Line 8:


* '''Metabug''': {{Bugzilla|1029746}} -  [meta] Search experiment
* '''Metabug''': {{Bugzilla|1029746}} -  [meta] Search experiment
* '''Story Breakdown''': {{Bugzilla|1021921}} - Story Breakdown - [Translation] Launch pre-trial of instant translation
* '''Testing was done in''': {{Bugzilla|1022411}} - Build Telemetry Experiment for German translation trial
* '''Testing was done in''': {{Bugzilla|1022411}} - Build Telemetry Experiment for German translation trial


<bugzilla>
<bugzilla>
{
{
"id": "1022410, 1022411, 1032265, 1022413, 1022856"
"id": "1017806, 1029189, 1029776, 1029792, 1029799, 1029807, 1029810, 1029818, 1030864, 1031320, 1033125, 1036719"
}
}
</bugzilla>
</bugzilla>

Revision as of 23:33, 21 July 2014

Telemetry Experiments

Test Cases

  • Select Telemetry under the Products section on MozTrap

Experiments

Reorder about:newtab tiles

asd

Automatic Translation

  • Metabug: 1029746 - [meta] Search experiment
  • Testing was done in: 1022411 - Build Telemetry Experiment for German translation trial
Full Query
ID Summary Priority Status
1017806 Telemetry Experiments: builtin support for multiple branches of an experiment -- VERIFIED
1029189 [UX] Design experience for new default search test -- RESOLVED
1029776 [Search experiment] Create experiment add-on and define user group (cell) on installation -- RESOLVED
1029792 [Search experiment] Implement notification UI -- RESOLVED
1029799 [Search experiment] Define experiment parameters -- VERIFIED
1029807 [Search experiment] Host update manifest -- RESOLVED
1029810 [Search experiment] Add the search experiment to the experiments server -- VERIFIED
1029818 [Search experiment] QA experiment add-on and sign-off for deployment -- VERIFIED
1030864 [Search experiment] Notify user about the end of the experiment -- VERIFIED
1031320 [Search experiment] Display notification UI for the correct groups -- RESOLVED
1033125 Deploy telemetry-experiment-server for search-beta experiment -- RESOLVED
1036719 [Search experiment] Search experiment should only target en-US -- VERIFIED

12 Total; 0 Open (0%); 6 Resolved (50%); 6 Verified (50%);


Automatic Translation

  • Metabug: 973271 - Automatic Translation
  • Testing was done in: 1022411 - Build Telemetry Experiment for German translation trial
Full Query
ID Summary Priority Status
1021921 Story Breakdown - [Translation] Launch pre-trial of instant translation -- VERIFIED
1022410 Ensure all translation-related strings in Fx32 Aurora are localized in German -- VERIFIED
1022411 Build Telemetry Experiment for German translation trial -- VERIFIED
1022413 Update list of languages supported by automatic translation -- RESOLVED
1022856 implement translation provider attribution -- VERIFIED
1032265 Deploy telemetry-experiment-server for translation-aurora experiment -- VERIFIED

6 Total; 0 Open (0%); 1 Resolved (16.67%); 5 Verified (83.33%);


Bugs

V1 [Completed]

Current V1 Status: Completed & Experiment Published

*Last Updated: June 19th, 2014*

Full Query
ID Summary Priority Status
973992 Telemetry experiments: addon manager UX P2 RESOLVED
973997 Telemetry experiments: decide on manifest url scheme and format -- RESOLVED
973998 Telemetry experiments: initial server -- VERIFIED
973999 Telemetry experiments: fetch and cache the experiment list -- RESOLVED
974009 Telemetry experiments: test experiment conditions and enable experiments -- RESOLVED
974024 Telemetry experiments: add monitoring data to FHR payload -- RESOLVED
974028 Firefox Telemetry Experiments (rev 1) -- RESOLVED
974151 Production and staging telemetry-experiments.mozilla.org -- RESOLVED
975000 Telemetry experiments: ensure that experiment addons are compatible, don't update automatically -- RESOLVED
977708 Annotate crash reports with enabled experiments -- RESOLVED
979474 Telemetry experiments: Pin the experiment manifest fetch to a particular CA P2 RESOLVED
981842 Telemetry experiments: include current-experiment details in about:support -- RESOLVED
982256 Telemetry experiments: test extension -- RESOLVED
983231 Telemetry experiments: test manifest condition evaluation -- RESOLVED
983360 Telemetry experiments: add monitoring data to Telemetry payload -- RESOLVED
984014 Telemetry experiments: test update mechanism -- RESOLVED
984387 Telemetry experiments: test coverage for disabling and removal of experiments via manifest -- RESOLVED
984879 Experiment manager shutdown -- RESOLVED
985084 Experiment add-ons should be disabled at startup -- VERIFIED
985089 Telemetry experiments: allow QA to force experiment randomization via pref -- RESOLVED
985670 Telemetry experiments: invalid timer scheduling when addon & experiment id don't match -- RESOLVED
985682 Experiments aren't loaded correctly from the cache -- RESOLVED
986040 Telemetry experiments: Assure that no experiment addon is running yet before starting an experiment. P3 RESOLVED
986093 Icon for Telemetry Experiments add-ons -- RESOLVED
986530 Deploy a test experiment to the nightly population -- VERIFIED
986539 Experiments.disableExperiment() should just disable the current experiment -- RESOLVED
986677 Include time left in experiment in addon-manager P2 VERIFIED
987207 Telemetry experiments: Assure that cache save & restore works properly -- RESOLVED
987225 Telemetry experiments: Combine pending tasks into a serial task -- RESOLVED
988710 Telemetry experiments: local staging server not working with nightly -- VERIFIED
988873 Telemetry experiments: enable the experiments feature -- VERIFIED
989137 Telemetry Experiments: remaining AddonManager integration cleanups -- VERIFIED
990111 Addon Provider for previously-active experiments -- VERIFIED
992208 Telemetry Experiments not evaluating automatically: packaging error -- RESOLVED
992232 telemetry-experiment-server: change deployment repo and script -- VERIFIED
992258 Trigger a refresh of the experiments view in the addon manager when experiments change -- VERIFIED
993084 14% regression in ts paint on linux64 seen on fx-team -- VERIFIED
994686 Permaorange nightly-only failure in browser_experiments.js | Experiment tab hidden by default. - false == true -- VERIFIED
994727 Telemetry experiments: Test and fix how disabling the feature works. -- VERIFIED
995027 Intermittent Experiments.jsm Shutdown "ABORT: file resource://gre/modules/AsyncShutdown.jsm, line 431" [@ mozalloc_abort(char const*)] -- VERIFIED
995032 Telemetry Experiments test will fail when run on different channel -- RESOLVED
997188 testsuite connects to telemetry-experiment.cdn.mozilla.net -- VERIFIED
1000114 Extend test coverage for experiments in addon manager UI -- VERIFIED
1001031 Create ongoing telemetry report on experiment uptake/adoption -- VERIFIED
1001529 Telemetry experiments: "Experiments" container not initially visible when there's a disabled experiment -- RESOLVED
1004506 Telemetry experiments: NaN days remaining when having multiple experiments under about:addons -- VERIFIED
1004665 Experiments: exception thrown if activating experiment with crash reporting disabled -- RESOLVED
1005270 Trigger experiments initialization on opening the addon manager UI -- VERIFIED
1007663 Avoid showing empty experiments list -- VERIFIED
1010397 Experiment manager may undo pending uninstalls P4 RESOLVED
1017744 Telemetry experiments: remove buttons don't initially disable/delete the experiment -- RESOLVED
1023450 Experiments are running during mochitests -- RESOLVED

52 Total; 0 Open (0%); 32 Resolved (61.54%); 20 Verified (38.46%);


V2 [In Progress]

Current V2 Status: Currently in Progress
  • Metabug: N/A (No Metabug available, incremental improvements are being made via separate bugs)
Full Query
ID Summary Priority Status
1010397 Experiment manager may undo pending uninstalls P4 RESOLVED
1012466 exception on shutdown: "Exception calling provider shutdown", NS_ERROR_FAILURE from nsIObserverService.removeObserver from PreviousExperimentProvider.shutdown -- RESOLVED
1014194 Base implementation: updater addon hotfix -- VERIFIED
1017806 Telemetry Experiments: builtin support for multiple branches of an experiment -- VERIFIED
1021002 Nightly test experiment logging doesn't show failures due to incompatible extensions: because the jsfilter isn't working! -- VERIFIED
1022411 Build Telemetry Experiment for German translation trial -- VERIFIED
1026850 Experiment system might get active experiment in a confused state if telemetry was disabled P4 RESOLVED
1026853 Experiment is displayed as "pending removal" in detailed view -- VERIFIED
1029799 [Search experiment] Define experiment parameters -- VERIFIED
1029810 [Search experiment] Add the search experiment to the experiments server -- VERIFIED
1029818 [Search experiment] QA experiment add-on and sign-off for deployment -- VERIFIED
1030864 [Search experiment] Notify user about the end of the experiment -- VERIFIED
1032265 Deploy telemetry-experiment-server for translation-aurora experiment -- VERIFIED

13 Total; 0 Open (0%); 3 Resolved (23.08%); 10 Verified (76.92%);

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Telemetry/TelemetryTestCases&oldid=998783"