Confirmed users
487
edits
QA/Mobile/TelemetryTestPlan: Difference between revisions
adding link for glean on iOS
Rpappalardo (talk | contribs) (metrics) |
Rpappalardo (talk | contribs) (adding link for glean on iOS) |
||
| (22 intermediate revisions by the same user not shown) | |||
| Line 19: | Line 19: | ||
* TBD - pending Glean implementation | * TBD - pending Glean implementation | ||
== | == Testing Frequency == | ||
==== Ad-Hoc Testing ==== | |||
'''First Pass''' | |||
As telemetry is now being tested for the first time on mobile, we'll do a more thorough "first pass" to verify all data points one time for each ping scenario. | |||
This will be a one-time full verification (which will only be repeated on request) | |||
Currently: baseline, metrics, events | |||
TBD: activation, later, sync | |||
'''New Ping Types''' | |||
End-to-end verification of all data points will only be performed for new ping data in new scenarios. | |||
'''Per Management / dev request''' | |||
Upon request from management, fenix or telemetry dev engineers. | |||
==== Scheduled Testing ==== | |||
'''Sprint''' | |||
Spot checking = verify each ping scenario is successfully transmitting data | |||
NOTE: | |||
not verifying every single data point, just eye-balling that data is being transmitted and arrives at telemetry service. | |||
''' Releases ''' | |||
Spot checking = verify each ping scenario is successfully transmitting data | |||
NOTE: | |||
not verifying every single point, just eye-balling that data is being transmitted and arrives at telemetry service. | |||
= Test Strategy = | = Test Strategy = | ||
| Line 49: | Line 85: | ||
== Ping Activities == | == Ping Activities == | ||
=== Description === | === Description === | ||
For each ping scenario, we will conduct a single end-to-end test. Each test can be broken down into | For each ping scenario, we will conduct a single end-to-end test. Each test can be broken down into two high-level steps: | ||
# [[#Data Collection|Data Collection]] | # [[#Data Collection & Submission|Data Collection & Submission]] | ||
#* Ping data is collected on the client (and stored locally on disk). Some data is recorded automatically (i.e. [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/baseline.md baseline] and [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/metrics.md metrics] pings). Other data is collected when certain specific actions are performed by the user. The latter are referred to as "events" (i.e. [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/events.md events] pings). | #* Ping data is collected on the client (and stored locally on disk). Some data is recorded automatically (i.e. [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/baseline.md baseline] and [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/metrics.md metrics] pings). Other data is collected when certain specific actions are performed by the user. The latter are referred to as "events" (i.e. [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/events.md events] pings). | ||
#* Our test coverage must ensure we verify that [[#Data Collection|data collection | #* Our test coverage must ensure we verify that [[#Data Collection & Submission|data collection and submission]] has taken place locally for each of the ping scenarios. We can monitor this local activity with the logcat tool (built into Android Studio). | ||
#* Using the [https://github.com/mozilla-mobile/android-components/tree/master/components/service/glean#debugging-products-using-glean Glean CLI tool], we'll tag and force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc. | #* Using the [https://github.com/mozilla-mobile/android-components/tree/master/components/service/glean#debugging-products-using-glean Glean CLI tool], we'll tag and force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc. | ||
# [[#Data Integrity|Data Integrity Verification]] | # [[#Data Integrity|Data Integrity Verification]] | ||
| Line 71: | Line 106: | ||
# [[#Test Setup|Test Setup]] | # [[#Test Setup|Test Setup]] | ||
# [[#Data Collection|Data Collection | # [[#Data Collection & Submission|Data Collection & Submission]] | ||
# [[#Data Integrity|Data Integrity Verification]] | # [[#Data Integrity|Data Integrity Verification]] | ||
== Test Demo == | |||
This [https://youtu.be/F9OSct32rl0 demo video] illustrates the basic steps for test setup and execution. This is not intended to illustrate a real test case! Additionally, you will want to run all tests on a physical (not an x86 device) and monitor all local activity using logcat on Android Studio. | |||
== Test Setup == | == Test Setup == | ||
=== Download Application === | === Download Application === | ||
# Close and uninstall any previous instances of app-under-test | |||
# Download the Fenix (or other) [https://tools.taskcluster.net/index/project.mobile.fenix.signed-nightly.nightly/latest release apk file] | # Download the Fenix (or other) [https://tools.taskcluster.net/index/project.mobile.fenix.signed-nightly.nightly/latest release apk file] | ||
#* [https://index.taskcluster.net/v1/task/project.mobile.fenix.signed-nightly.nightly.latest/artifacts/public/target.arm.apk public/target.arm.apk] | #* [https://index.taskcluster.net/v1/task/project.mobile.fenix.signed-nightly.nightly.latest/artifacts/public/target.arm.apk public/target.arm.apk] | ||
| Line 98: | Line 138: | ||
<code> $ ping https://debug-ping-preview.firebaseapp.com</code> | <code> $ ping https://debug-ping-preview.firebaseapp.com</code> | ||
== Data Collection == | == Data Collection & Submission == | ||
=== Summary === | === Summary === | ||
Next, we will need to | Next, we will need collect and submit data to the telemetry server: | ||
# Tag our telemetry pings with a custom identifier | # Tag our telemetry pings with a custom identifier | ||
# Trigger ping collection (generation) on the client. Since this is done differently for each of the ping scenarios, we'll list the steps separately below | # Trigger ping collection (generation) on the client. Since this is done differently for each of the ping scenarios, we'll list the steps separately below | ||
# Submit data to telemetry server | |||
#* Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait! | |||
=== Baseline Ping === | === Baseline Ping === | ||
'''Launch Glean''' | '''Launch Glean''' | ||
See test steps in [https://testrail.stage.mozaws.net/index.php?/suites/view/4669&group_by=cases:section_id&group_order=asc TestRail - Fenix - telemetry] | |||
=== Metrics Ping === | === Metrics Ping === | ||
See test steps in [https://testrail.stage.mozaws.net/index.php?/suites/view/4669&group_by=cases:section_id&group_order=asc TestRail - Fenix - telemetry] | |||
'''NOTE''' | '''NOTE''' | ||
* If metrics ping doesn't get submitted, go to Android Settings and change local time to 3: | * If metrics ping doesn't get submitted, go to Android Settings and change local time to 3:58 then launch Glean command | ||
=== Events Ping === | === Events Ping === | ||
See test steps in [https://testrail.stage.mozaws.net/index.php?/suites/view/4669&group_by=cases:section_id&group_order=asc TestRail - Fenix - telemetry] | |||
As of the writing of this test plan, there are currently 37 events defined in the fenix [https://github.com/mozilla-mobile/fenix/blob/master/app/events.yaml events.yaml] (Look for "type: event"). | |||
''' data collection''' | |||
* Pick 4-5 event types and perform those user actions in sequence (NOTE: we will later want to verify that they are recorded on the telemetry server in the correct sequence). | |||
* Reason (per the [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/events.md events ping docs]): "The events ping's purpose is to transport all of the event metric information" | |||
== Data Integrity == | == Data Integrity == | ||
| Line 173: | Line 198: | ||
* [[#Verify_Common_Info|common fields info]] | * [[#Verify_Common_Info|common fields info]] | ||
Verify events | Verify that specific events performed in the [[#Data Collection & Submission|Data Collection & Submission]] section have arrived in Dashboard in the proper sequence. | ||
NOTE: | NOTE: | ||
| Line 181: | Line 206: | ||
Verify the following fields: | Verify the following fields: | ||
* [[#Verify_Common_Info|common fields info]] | * [[#Verify_Common_Info|common fields info]] | ||
* | * Review al fields in [https://github.com/mozilla-mobile/fenix/blob/master/app/metrics.yaml metrics.yaml] ''not of type: event'' | ||
Review fields in [https://github.com/mozilla-mobile/fenix/blob/master/app/metrics.yaml metrics.yaml] | |||
=== Verify Custom Info === | === Verify Custom Info === | ||
TBD | |||
= Test Tools = | = Test Tools = | ||
| Line 214: | Line 235: | ||
''logPings'' | ''logPings'' | ||
# To dump Glean logs to logcat | # To dump Glean logs to logcat | ||
#* <code>adb shell -n ...GleanDebugActivity --ez logPings true</code> | #* <code>adb shell am start -n ...GleanDebugActivity --ez logPings true</code> | ||
''sendPing'' | ''sendPing'' | ||
# To force sending a ping immediately use: | # To force sending a ping immediately use: | ||
#* <code>adb shell -n ...GleanDebugActivity --es sendPing metrics </code> | #* <code>adb shell am start -n ...GleanDebugActivity --es sendPing metrics </code> | ||
#* (or instead of metrics other ping names) | #* (or instead of metrics other ping names) | ||
# Look at Glean frontend to verify data is landing | # Look at Glean frontend to verify data is landing | ||
| Line 224: | Line 245: | ||
''tagPings'' | ''tagPings'' | ||
# To tag a ping with a user label for easy query in [https://debug-ping-preview.firebaseapp.com/ Glean Debug dashboard] | # To tag a ping with a user label for easy query in [https://debug-ping-preview.firebaseapp.com/ Glean Debug dashboard] | ||
#* <code>adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1</code>, then spot <code>rpapa-test1</code> in the frontend to identify your pings. | #* <code>adb shell am start -n ...GleanDebugActivity --es tagPings rpapa-test1</code>, then spot <code>rpapa-test1</code> in the frontend to identify your pings. | ||
== Examples == | == Examples == | ||
| Line 231: | Line 252: | ||
'''Fenix''' | '''Fenix''' | ||
* <code>adb shell -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | * <code>adb shell am start -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | ||
'''Android Components (sample apps)''' | '''Android Components (sample apps)''' | ||
* <code>adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | * <code>adb shell am start -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | ||
'''Reference Browser''' | '''Reference Browser''' | ||
* <code>adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | * <code>adb shell am start -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | ||
** NOTES | ** NOTES | ||
*** even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the ''baseline'' ping. | *** even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the ''baseline'' ping. | ||
| Line 256: | Line 277: | ||
== NOTES== | == NOTES== | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1522428 Bugzilla: Enable tagging glean pings for the "debug view"] | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1522428 Bugzilla: Enable tagging glean pings for the "debug view"] | ||
* glean on iOS platform [https://mozilla.github.io/glean/book/user/debugging/ios.html] | |||