Confirmed users
487
edits
QA/Mobile/TelemetryTestPlan: Difference between revisions
adding link for glean on iOS
Rpappalardo (talk | contribs) (metrics info) |
Rpappalardo (talk | contribs) (adding link for glean on iOS) |
||
| (8 intermediate revisions by the same user not shown) | |||
| Line 19: | Line 19: | ||
* TBD - pending Glean implementation | * TBD - pending Glean implementation | ||
== Testing Frequency == | |||
==== Ad-Hoc Testing ==== | |||
'''First Pass''' | |||
As telemetry is now being tested for the first time on mobile, we'll do a more thorough "first pass" to verify all data points one time for each ping scenario. | |||
This will be a one-time full verification (which will only be repeated on request) | |||
Currently: baseline, metrics, events | |||
TBD: activation, later, sync | |||
'''New Ping Types''' | |||
End-to-end verification of all data points will only be performed for new ping data in new scenarios. | |||
'''Per Management / dev request''' | |||
Upon request from management, fenix or telemetry dev engineers. | |||
==== Scheduled Testing ==== | |||
'''Sprint''' | |||
Spot checking = verify each ping scenario is successfully transmitting data | |||
NOTE: | |||
not verifying every single data point, just eye-balling that data is being transmitted and arrives at telemetry service. | |||
''' Releases ''' | |||
Spot checking = verify each ping scenario is successfully transmitting data | |||
NOTE: | |||
not verifying every single point, just eye-balling that data is being transmitted and arrives at telemetry service. | |||
= Test Strategy = | = Test Strategy = | ||
| Line 113: | Line 150: | ||
'''Launch Glean''' | '''Launch Glean''' | ||
See test steps in [https://testrail.stage.mozaws.net/index.php?/suites/view/4669&group_by=cases:section_id&group_order=asc TestRail - Fenix - telemetry] | |||
=== Metrics Ping === | === Metrics Ping === | ||
See test steps in [https://testrail.stage.mozaws.net/index.php?/suites/view/4669&group_by=cases:section_id&group_order=asc TestRail - Fenix - telemetry] | |||
'''NOTE''' | '''NOTE''' | ||
| Line 144: | Line 160: | ||
=== Events Ping === | === Events Ping === | ||
See test steps in [https://testrail.stage.mozaws.net/index.php?/suites/view/4669&group_by=cases:section_id&group_order=asc TestRail - Fenix - telemetry] | |||
As of the writing of this test plan, there are currently 37 events defined in the fenix [https://github.com/mozilla-mobile/fenix/blob/master/app/events.yaml events.yaml] (Look for "type: event"). | |||
''' | ''' data collection''' | ||
* Pick 4-5 event types and perform those user actions in sequence (NOTE: we will later want to verify that they are recorded on the telemetry server in the correct sequence). | * Pick 4-5 event types and perform those user actions in sequence (NOTE: we will later want to verify that they are recorded on the telemetry server in the correct sequence). | ||
* Reason (per the [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/events.md events ping docs]): "The events ping's purpose is to transport all of the event metric information" | * Reason (per the [https://github.com/mozilla-mobile/android-components/blob/master/components/service/glean/docs/pings/events.md events ping docs]): "The events ping's purpose is to transport all of the event metric information" | ||
== Data Integrity == | == Data Integrity == | ||
| Line 225: | Line 235: | ||
''logPings'' | ''logPings'' | ||
# To dump Glean logs to logcat | # To dump Glean logs to logcat | ||
#* <code>adb shell -n ...GleanDebugActivity --ez logPings true</code> | #* <code>adb shell am start -n ...GleanDebugActivity --ez logPings true</code> | ||
''sendPing'' | ''sendPing'' | ||
# To force sending a ping immediately use: | # To force sending a ping immediately use: | ||
#* <code>adb shell -n ...GleanDebugActivity --es sendPing metrics </code> | #* <code>adb shell am start -n ...GleanDebugActivity --es sendPing metrics </code> | ||
#* (or instead of metrics other ping names) | #* (or instead of metrics other ping names) | ||
# Look at Glean frontend to verify data is landing | # Look at Glean frontend to verify data is landing | ||
| Line 235: | Line 245: | ||
''tagPings'' | ''tagPings'' | ||
# To tag a ping with a user label for easy query in [https://debug-ping-preview.firebaseapp.com/ Glean Debug dashboard] | # To tag a ping with a user label for easy query in [https://debug-ping-preview.firebaseapp.com/ Glean Debug dashboard] | ||
#* <code>adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1</code>, then spot <code>rpapa-test1</code> in the frontend to identify your pings. | #* <code>adb shell am start -n ...GleanDebugActivity --es tagPings rpapa-test1</code>, then spot <code>rpapa-test1</code> in the frontend to identify your pings. | ||
== Examples == | == Examples == | ||
| Line 242: | Line 252: | ||
'''Fenix''' | '''Fenix''' | ||
* <code>adb shell -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | * <code>adb shell am start -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | ||
'''Android Components (sample apps)''' | '''Android Components (sample apps)''' | ||
* <code>adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | * <code>adb shell am start -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | ||
'''Reference Browser''' | '''Reference Browser''' | ||
* <code>adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | * <code>adb shell am start -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1</code> | ||
** NOTES | ** NOTES | ||
*** even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the ''baseline'' ping. | *** even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the ''baseline'' ping. | ||
| Line 267: | Line 277: | ||
== NOTES== | == NOTES== | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1522428 Bugzilla: Enable tagging glean pings for the "debug view"] | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1522428 Bugzilla: Enable tagging glean pings for the "debug view"] | ||
* glean on iOS platform [https://mozilla.github.io/glean/book/user/debugging/ios.html] | |||