QA/Mobile/TelemetryTestPlan

Summary

Product Coverage

This is the test plan for verification of telemetry service clients for the following Mozilla Android products:

Fenix

Frequency:

• thorough first pass and on releases, spot check w/ sprints (to be verified w/ kbrosnan)

Issues:

• labeled QA-Needed

Reference Browser

Frequency:

• upon request from dev, management

android-components - sample apps

Frequency:

• upon request from dev, management

Firefox for FireTV

Frequency:

• TBD - pending Glean implementation

Test Demo

This demo video illustrates the basic steps for test setup and execution. This is not intended to illustrate a real test case. Additionally, you will want to monitor all local activity using logcat on Android Studio.

In this demo, you can observe:

1. Telemetry being sent from client to the server
2. Telemetry arrives at the server without excessive latency (can be viewed from the Glean Debug Dashboard)
3. Telemetry is complete and data integrity maintained

NOTES:

• For ease of demonstration, an X86 device was used, but all testing should be done using physical devices.
• Test steps in video are outlined in wiki below

Test Strategy

Test Types

The Mobile QA team will perform two test types of telemetry verification for all our mobile projects:

1. End-to-end (client-to-server) manual tests
2. Client-side only verification automated tests (TBD)

This wiki outlines a test plan for manual testing.

Manual Tests

Frequency:

• upon request from devs, management

Who:

• SoftVision engineers

Tools:

• Glean CLI, Glean Debug Dashboard

Automated Tests

Frequency:

• github trigger on merge-to-master

Who:

• No-Jun's team

Tools:

• see discussion: discussion

Ping Scenarios

Summary

A "ping" refers to a JSON data blob sent from the client to the telemetry service. There are three basic types (scenarios):

1. baseline ping
2. events ping
3. metrics pings

In addition, custom pings (i.e. activation, sync pings) can also be defined using the pings.yaml and metrics.yaml files. Test cases should be designed around these various "ping scenarios."

This wiki will explain how to test each of these scenarios.

Ping Activities

Summary

Testing ping integrity from end-to-end can be broken down into three high-level steps:

1. Data Collection
   • Ping data is collected on the client (and stored locally on disk) when certain specific actions are performed.
   • Our test coverage must ensure we trigger data collection for each of the ping scenarios by performing those actions in the proper sequence.
   • Generated data will first be verified locally via logcat
2. Data Submission
   • Using the Glean CLI tool, we'll force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc.
3. Data Integrity
   • Using the Glean Debug Dashboard, we must verify that the ping data we've tagged and forced to be submitted (using the Glean CLI tool), arrives without undue latency (within approx. 10-20 seconds).
   • Drill down on the JSON data and verify the correct data has been submitted.

Test Execution

Summary

This will be a "client-to-server (end-2-end) verification" test using the production telemetry service across the real network (not a mock service on localhost). Execution requires manually triggering a telemetry ping to be sent to the telemetry service by emulating a specific set of user activities in the application.

Pings can be successfully verified by ensuring they have arrived in the telemetry server - viewable in the Glean Debug Dashboard.

Dashboard Glean Dashboard

• Authentication requires SSO login
• see: Debugging products using Glean
• Features
  • Glean Debug Dashboard makes data received by the pipeline from some devices visible in real-time
  • Enabling debug mode either returns a unique identifier, or the ID you want to use (see tagPings parameter below), enabling immediate query of test pings

Setup

Download Application

1. Download the Fenix (or other) release apk file
   • choose: public/target.x86.apk or public/target.aarch64.apk

NOTE:

• Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed)

Install Application

1. Connect an Android (physical) device via USB
2. Open terminal
3. Verify device is connected: $ adb devices
4. Install the apk: $ adb install ~/Downloads/target.aarch64.apk
5. Verify app has been installed on device
6. NOTE: Do not open the application, Glean will do this for us

Check Network

1. ping Glean Debug Dashboard: [1]
   • $ ping https://debug-ping-preview.firebaseapp.com

Data Collection

Next, we will do three things (see "Examples" below):

1. Tag our telemetry pings with a custom identifier
2. Trigger ping generation on client

Data Submission

Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait!

Perform the following steps to force immediate data submission for ALL scenarios:

1. Close and uninstall any previous instances of app-under-test
2. Re-install a fresh instance of app on device
3. Launch app with Glean CLI command (not manually)
4. Move app to the background (this should immediately trigger send)

Data Integrity

Now let's verify that data has arrived to telemetry service intact.

see: [# Verify JSON data per pings doc

Open Ping

1. Open Glean Debug Dashboard: [2]
2. Verify real-time receipt of tagged ping in dashboard
3. Verify ping entry contains correct field data
4. Drill down into ping
5. Verify that "Recent pings..." entry contains correct ping types
6. Drill down to JSON data

Verify Common Info

• Verify data in all fields per common fields doc
• NOTE: if common info is good on baseline ping, SHOULD BE good on others - no need to repeat for other ping scenarios! (All use same path of execution)

Verify Baseline Info

Verify the following fields:

• common fields info
• duration
  • should roughly approximate difference between start_time, end_time
• Verify locale string

Verify Events Info

Verify the following fields:

• common fields info

Verify events that were performed have arrived in Dashboard

Verify Metrics Info

Verify the following fields:

• common fields info
• default browser

Review fields in metrics.yaml

Verify Custom Info

[Reference]

Test Tools

Summary

You will need the following tools to execute end-to-end telemetry tests:

1. Android physical device
   • Testing can be done on an x86 device but is less reliable
   • Glean currently broken on x86: issue/2291
2. Browser
   • Test plan (this page)
   • Taskcluster APK download page
   • Glean Debug Dashboard
     • to verify server-side activity
3. Terminal
   • You will run all your ADB, Glean commands from here
4. Logcat
   • Open Android Studio to monitor all client-side activity

Glean CLI

Below is a brief summary of the Glean CLI features documented here: Debugging products using Glean

logPings

1. To dump Glean logs to logcat
   • adb shell -n ...GleanDebugActivity --ez logPings true

sendPing

1. To force sending a ping immediately use:
   • adb shell -n ...GleanDebugActivity --es sendPing metrics
   • (or instead of metrics other ping names)
2. Look at Glean frontend to verify data is landing

tagPings

1. To tag a ping with a user label for easy query in Glean Debug dashboard
   • adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1, then spot rpapa-test1 in the frontend to identify your pings.

Examples

For running the commands with any Firefox Android app, adjust the application ID part in the command:

Fenix

• adb shell -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1

Android Components (sample apps)

• adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1

Reference Browser

• adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1
  • NOTES
    • even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the baseline ping.
    • suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data:

Reference

Automated Tests (Firefox Desktop)

DESCRIPTION

• Automated tests for Firefox desktop do "client verification only test" also rely on a local HTTP client as a kind of simple mock telemetry server, used to capture, read and verify the outbound JSON payload locally
• Events are triggered, data integrity is verified
• see source: mozilla-central

NOTES

• Bugzilla: Enable tagging glean pings for the "debug view"