QA/Mobile/TelemetryTestPlan
Contents
Summary
Product Coverage
This is the test plan for verification of telemetry service clients for the following Mozilla Android products:
Fenix
Frequency:
- thorough first pass and on releases, spot check w/ sprints (to be verified w/ kbrosnan)
Issues:
Reference Browser
Frequency:
- upon request from dev, management
android-components - sample apps
Frequency:
- upon request from dev, management
Firefox for FireTV
Frequency:
- TBD - pending Glean implementation
Testing Frequency
Ad-Hoc Testing
First Pass
As telemetry is now being tested for the first time on mobile, we'll do a more thorough "first pass" to verify all data points one time for each ping scenario. This will be a one-time full verification (which will only be repeated on request) Currently: baseline, metrics, events TBD: activation, later, sync
New Ping Types
End-to-end verification of all data points will only be performed for new ping data in new scenarios.
Per Management / dev request
Upon request from management, fenix or telemetry dev engineers.
Scheduled Testing
Sprint
Spot checking = verify each ping scenario is successfully transmitting data
NOTE:
not verifying every single data point, just eye-balling that data is being transmitted and arrives at telemetry service.
Releases
Spot checking = verify each ping scenario is successfully transmitting data
NOTE:
not verifying every single point, just eye-balling that data is being transmitted and arrives at telemetry service.
Test Strategy
Summary
The Mobile QA team will perform two test types of telemetry verification for all our mobile projects:
- End-to-end (client-to-server) manual tests
- Client-side only verification automated tests (TBD)
This wiki outlines a test plan for manual testing.
Frequency:
- upon request from devs, management
Who:
- SoftVision engineers
Ping Scenarios
Description
A "ping" refers to a JSON data blob sent from the client to the telemetry service. There are three basic types (scenarios):
In addition, custom pings (i.e. activation, sync pings) can also be defined using the pings.yaml and metrics.yaml files.
All test cases should be designed around these various "ping scenarios." This wiki will explain how to test each of these scenarios.
Ping Activities
Description
For each ping scenario, we will conduct a single end-to-end test. Each test can be broken down into two high-level steps:
- Data Collection & Submission
- Ping data is collected on the client (and stored locally on disk). Some data is recorded automatically (i.e. baseline and metrics pings). Other data is collected when certain specific actions are performed by the user. The latter are referred to as "events" (i.e. events pings).
- Our test coverage must ensure we verify that data collection and submission has taken place locally for each of the ping scenarios. We can monitor this local activity with the logcat tool (built into Android Studio).
- Using the Glean CLI tool, we'll tag and force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc.
- Data Integrity Verification
- Using the Glean Debug Dashboard, we must verify that our ping data arrives at the telemetry server without undue latency (within approx. 10-20 seconds).
- Drill down on the JSON data and verify the correct data has been submitted.
Test Execution
Summary
This section describes how to perform an end-to-end (client-to-server) test using the production telemetry service across the real network. Please note, for these tests we will not be a using the telemetry staging server nor a mock server on localhost to receive telemetry pings. Test execution will require installation of an application APK on a physical mobile device and use of a built-in CLI tool called Glean.
First, if necessary, we'll trigger telemetry data collection locally by emulating a specific set of user actions in the application (events ping). Next, we'll force immediate data submission to the telemetry server. Lastly, we will use the Glean Debug Dashboard to verify data integrity.
To summarize, for each ping scenario, we'll perform the following steps in sequence:
Test Demo
This demo video illustrates the basic steps for test setup and execution. This is not intended to illustrate a real test case! Additionally, you will want to run all tests on a physical (not an x86 device) and monitor all local activity using logcat on Android Studio.
Test Setup
Download Application
- Close and uninstall any previous instances of app-under-test
- Download the Fenix (or other) release apk file
NOTE:
- Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed)
Install Application
- Connect an Android (physical) device via USB
- make sure your device has Settings > Developer Options > USB debugging enabled
- Open a terminal
- Verify device is connected:
$ adb devices
- Install the apk:
$ adb install ~/Downloads/target.aarch64.apk
- Verify app has been installed on device
- NOTE: Do not open the application (Glean will do this for us)
Check Network
ping Glean Debug Dashboard: [1]
$ ping https://debug-ping-preview.firebaseapp.com
Data Collection & Submission
Summary
Next, we will need collect and submit data to the telemetry server:
- Tag our telemetry pings with a custom identifier
- Trigger ping collection (generation) on the client. Since this is done differently for each of the ping scenarios, we'll list the steps separately below
- Submit data to telemetry server
- Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait!
Baseline Ping
Launch Glean See test steps in TestRail - Fenix - telemetry
Metrics Ping
See test steps in TestRail - Fenix - telemetry
NOTE
- If metrics ping doesn't get submitted, go to Android Settings and change local time to 3:58 then launch Glean command
Events Ping
See test steps in TestRail - Fenix - telemetry
As of the writing of this test plan, there are currently 37 events defined in the fenix events.yaml (Look for "type: event").
data collection
- Pick 4-5 event types and perform those user actions in sequence (NOTE: we will later want to verify that they are recorded on the telemetry server in the correct sequence).
- Reason (per the events ping docs): "The events ping's purpose is to transport all of the event metric information"
Data Integrity
Now let's verify that data has arrived to telemetry service intact.
Verify JSON data per pings doc
Setup: Open Ping Data
- Open Glean Debug Dashboard: [2]
- Verify real-time receipt of tagged ping in dashboard
- Verify ping entry contains correct field data
- Drill down into ping
- Verify that "Recent pings..." entry contains correct ping types
- Drill down to JSON data
Verify Common Info
- Verify data in all fields per common fields doc
- NOTE: if common info is good on baseline ping, SHOULD BE good on others - no need to repeat for other ping scenarios! (All use same path of execution)
Verify Baseline Info
Verify the following fields:
- common fields info
- duration
- should roughly approximate difference between start_time, end_time
- Verify locale string
Verify Events Info
Verify the following fields:
Verify that specific events performed in the Data Collection & Submission section have arrived in Dashboard in the proper sequence.
NOTE: From just launching the application, you should see the “app_icon launch” event listed first
Verify Metrics Info
Verify the following fields:
- common fields info
- Review al fields in metrics.yaml not of type: event
Verify Custom Info
TBD
Test Tools
Summary
You will need the following tools to execute end-to-end telemetry tests:
- Android physical device
- Testing can be done on an x86 device but is less reliable
- Glean currently broken on x86: issue/2291
- Browser
- Test plan (this page)
- Taskcluster APK download page
- Glean Debug Dashboard
- to verify server-side activity
- Terminal
- You will run all your ADB, Glean commands from here
- Logcat
- Open Android Studio to monitor all client-side activity
Glean CLI
Below is a brief summary of the Glean CLI features documented here: Debugging products using Glean
logPings
- To dump Glean logs to logcat
-
adb shell am start -n ...GleanDebugActivity --ez logPings true
-
sendPing
- To force sending a ping immediately use:
-
adb shell am start -n ...GleanDebugActivity --es sendPing metrics
- (or instead of metrics other ping names)
-
- Look at Glean frontend to verify data is landing
tagPings
- To tag a ping with a user label for easy query in Glean Debug dashboard
-
adb shell am start -n ...GleanDebugActivity --es tagPings rpapa-test1
, then spotrpapa-test1
in the frontend to identify your pings.
-
Examples
For running the commands with any Firefox Android app, adjust the application ID part in the command:
Fenix
-
adb shell am start -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1
Android Components (sample apps)
-
adb shell am start -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1
Reference Browser
-
adb shell am start -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1
- NOTES
- even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the baseline ping.
- suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data:
- NOTES
Reference
Automated Tests (Firefox Mobile) - TBD
Automated Tests (Firefox Desktop)
- Automated tests for Firefox desktop do "client verification only test" also rely on a local HTTP client as a kind of simple mock telemetry server, used to capture, read and verify the outbound JSON payload locally
- Events are triggered, data integrity is verified
- see source: mozilla-central
NOTES
- Bugzilla: Enable tagging glean pings for the "debug view"
- glean on iOS platform [3]