Confirmed users
487
edits
QA/Mobile/TelemetryTestPlan: Difference between revisions
Jump to navigation
Jump to search
restructure doc
Rpappalardo (talk | contribs) m (scenarios) |
Rpappalardo (talk | contribs) (restructure doc) |
||
| Line 80: | Line 80: | ||
#* Drill down on the JSON data and ''verify the correct data has been submitted.'' | #* Drill down on the JSON data and ''verify the correct data has been submitted.'' | ||
= | = Test Execution = | ||
== Summary == | |||
== | |||
This will be a "client-to-server (end-2-end) verification" test using the production telemetry service across the real network (not a mock service on localhost). Execution requires manually triggering a telemetry ping to be sent to the telemetry service by emulating a specific set of user activities in the application. | This will be a "client-to-server (end-2-end) verification" test using the production telemetry service across the real network (not a mock service on localhost). Execution requires manually triggering a telemetry ping to be sent to the telemetry service by emulating a specific set of user activities in the application. | ||
| Line 100: | Line 95: | ||
** Enabling debug mode either returns a unique identifier, or the ID you want to use (see tagPings parameter below), enabling immediate query of test pings | ** Enabling debug mode either returns a unique identifier, or the ID you want to use (see tagPings parameter below), enabling immediate query of test pings | ||
== Setup == | |||
=== Download Application === | |||
# Download the Fenix (or other) [https://tools.taskcluster.net/index/project.mobile.fenix.signed-nightly.nightly/latest release apk file] | # Download the Fenix (or other) [https://tools.taskcluster.net/index/project.mobile.fenix.signed-nightly.nightly/latest release apk file] | ||
#* choose: [https://index.taskcluster.net/v1/task/project.mobile.fenix.signed-nightly.nightly.latest/artifacts/public/target.arm.apk public/target.x86.apk] or [https://index.taskcluster.net/v1/task/project.mobile.fenix.signed-nightly.nightly.latest/artifacts/public/target.x86.apk public/target.aarch64.apk] | #* choose: [https://index.taskcluster.net/v1/task/project.mobile.fenix.signed-nightly.nightly.latest/artifacts/public/target.arm.apk public/target.x86.apk] or [https://index.taskcluster.net/v1/task/project.mobile.fenix.signed-nightly.nightly.latest/artifacts/public/target.x86.apk public/target.aarch64.apk] | ||
| Line 110: | Line 105: | ||
* Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed) | * Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed) | ||
=== Install Application === | |||
# Connect an Android (physical) device via USB | # Connect an Android (physical) device via USB | ||
# Open terminal | # Open terminal | ||
| Line 118: | Line 113: | ||
# NOTE: Do not open the application, Glean will do this for us | # NOTE: Do not open the application, Glean will do this for us | ||
=== Check Network === | |||
# ping Glean Debug Dashboard: [https://debug-ping-preview.firebaseapp.com] | # ping Glean Debug Dashboard: [https://debug-ping-preview.firebaseapp.com] | ||
#* <code> $ ping https://debug-ping-preview.firebaseapp.com</code> | #* <code> $ ping https://debug-ping-preview.firebaseapp.com</code> | ||
== Data Collection == | |||
Next, we will do three things (see "Examples" below): | Next, we will do three things (see "Examples" below): | ||
# Tag our telemetry pings with a custom identifier | # Tag our telemetry pings with a custom identifier | ||
# Trigger ping generation on client | # Trigger ping generation on client | ||
== Data Submission == | |||
Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait! | Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait! | ||
| Line 136: | Line 131: | ||
# Move app to the background (this should immediately trigger send) | # Move app to the background (this should immediately trigger send) | ||
== Data Integrity == | |||
Now let's verify that data has arrived to telemetry service intact. | Now let's verify that data has arrived to telemetry service intact. | ||
see: [# Verify JSON data per [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/pings.md pings doc] | see: [# Verify JSON data per [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/pings.md pings doc] | ||
=== Open Ping === | |||
# Open Glean Debug Dashboard: [https://debug-ping-preview.firebaseapp.com] | # Open Glean Debug Dashboard: [https://debug-ping-preview.firebaseapp.com] | ||
# Verify real-time receipt of tagged ping in dashboard | # Verify real-time receipt of tagged ping in dashboard | ||
| Line 149: | Line 144: | ||
# Drill down to JSON data | # Drill down to JSON data | ||
=== Verify Common Info === | |||
* Verify data in all fields per [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/pings.md#ping-sections common fields doc] | * Verify data in all fields per [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/pings.md#ping-sections common fields doc] | ||
* NOTE: if common info is good on baseline ping, SHOULD BE good on others! (All use same path of execution) | * NOTE: if common info is good on baseline ping, SHOULD BE good on others! (All use same path of execution) | ||
=== Verify Baseline Info === | |||
=== Verify Events Info === | |||
=== Verify Metrics Info === | |||
=== Verify Custom Info === | |||
= | = Examples = | ||
For running the commands with any Firefox Android app, adjust the application ID part in the command: | For running the commands with any Firefox Android app, adjust the application ID part in the command: | ||
== Fenix == | |||
* <code>adb shell -n org.mozilla.fenix.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1</code> | * <code>adb shell -n org.mozilla.fenix.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1</code> | ||
==Android Components (sample apps)== | |||
* <code>adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1</code> | * <code>adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1</code> | ||
==Reference Browser== | |||
* <code>adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1</code> | * <code>adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1</code> | ||
* NOTE | * NOTE | ||
| Line 174: | Line 169: | ||
** suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data: | ** suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data: | ||
= | = Glean CLI = | ||
''logPings'' | ''logPings'' | ||
| Line 190: | Line 185: | ||
#* <code>adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1</code>, then spot <code>rpapa-test1</code> in the frontend to identify your pings. | #* <code>adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1</code>, then spot <code>rpapa-test1</code> in the frontend to identify your pings. | ||
=NOTES= | |||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1522428 Bugzilla: Enable tagging glean pings for the "debug view"] | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1522428 Bugzilla: Enable tagging glean pings for the "debug view"] | ||
= Reference = | = Reference = | ||