QA/Mobile/TelemetryTestPlan

Summary

Product Coverage

This is the test plan for verification of telemetry service clients for the following Mozilla Android products:

Fenix

Frequency:

• thorough first pass and on releases, spot check w/ sprints (to be verified w/ kbrosnan)

Issues:

• labeled QA-Needed

Reference Browser

Frequency:

• upon request from dev, management

android-components - sample apps

Frequency:

• upon request from dev, management

Firefox for FireTV

Frequency:

• TBD - pending Glean implementation

Testing Frequency

Ad-Hoc Testing

First Pass

As telemetry is now being tested for the first time on mobile, we'll do a more thorough "first pass" to verify all data points one time for each ping scenario. This will be a one-time full verification (which will only be repeated on request) Currently: baseline, metrics, events TBD: activation, later, sync

New Ping Types

End-to-end verification of all data points will only be performed for new ping data in new scenarios.

Per Management / dev request

Upon request from management, fenix or telemetry dev engineers.

Scheduled Testing

Sprint

Spot checking = verify each ping scenario is successfully transmitting data

NOTE:

not verifying every single data point, just eye-balling that data is being transmitted and arrives at telemetry service.

Releases

Spot checking = verify each ping scenario is successfully transmitting data

NOTE:

not verifying every single point, just eye-balling that data is being transmitted and arrives at telemetry service.

Test Strategy

Summary

The Mobile QA team will perform two test types of telemetry verification for all our mobile projects:

1. End-to-end (client-to-server) manual tests
2. Client-side only verification automated tests (TBD)

This wiki outlines a test plan for manual testing.

Frequency:

• upon request from devs, management

Who:

• SoftVision engineers

Ping Scenarios

Description

A "ping" refers to a JSON data blob sent from the client to the telemetry service. There are three basic types (scenarios):

1. baseline ping
2. events ping
3. metrics pings

In addition, custom pings (i.e. activation, sync pings) can also be defined using the pings.yaml and metrics.yaml files.

All test cases should be designed around these various "ping scenarios." This wiki will explain how to test each of these scenarios.

Ping Activities

Description

For each ping scenario, we will conduct a single end-to-end test. Each test can be broken down into two high-level steps:

1. Data Collection & Submission
   • Ping data is collected on the client (and stored locally on disk). Some data is recorded automatically (i.e. baseline and metrics pings). Other data is collected when certain specific actions are performed by the user. The latter are referred to as "events" (i.e. events pings).
   • Our test coverage must ensure we verify that data collection and submission has taken place locally for each of the ping scenarios. We can monitor this local activity with the logcat tool (built into Android Studio).
   • Using the Glean CLI tool, we'll tag and force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc.
2. Data Integrity Verification
   • Using the Glean Debug Dashboard, we must verify that our ping data arrives at the telemetry server without undue latency (within approx. 10-20 seconds).
   • Drill down on the JSON data and verify the correct data has been submitted.

Test Execution

Summary

This section describes how to perform an end-to-end (client-to-server) test using the production telemetry service across the real network. Please note, for these tests we will not be a using the telemetry staging server nor a mock server on localhost to receive telemetry pings. Test execution will require installation of an application APK on a physical mobile device and use of a built-in CLI tool called Glean.

First, if necessary, we'll trigger telemetry data collection locally by emulating a specific set of user actions in the application (events ping). Next, we'll force immediate data submission to the telemetry server. Lastly, we will use the Glean Debug Dashboard to verify data integrity.

To summarize, for each ping scenario, we'll perform the following steps in sequence:

1. Test Setup
2. Data Collection & Submission
3. Data Integrity Verification

Test Demo

This demo video illustrates the basic steps for test setup and execution. This is not intended to illustrate a real test case! Additionally, you will want to run all tests on a physical (not an x86 device) and monitor all local activity using logcat on Android Studio.

Test Setup

Download Application

1. Close and uninstall any previous instances of app-under-test
2. Download the Fenix (or other) release apk file
   • public/target.arm.apk
   • public/target.aarch64.apk

NOTE:

• Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed)

Install Application

1. Connect an Android (physical) device via USB
   • make sure your device has Settings > Developer Options > USB debugging enabled
2. Open a terminal
3. Verify device is connected: $ adb devices
4. Install the apk: $ adb install ~/Downloads/target.aarch64.apk
5. Verify app has been installed on device
   • NOTE: Do not open the application (Glean will do this for us)

Check Network

ping Glean Debug Dashboard: [1]

$ ping https://debug-ping-preview.firebaseapp.com

Data Collection & Submission

Summary

Next, we will need collect and submit data to the telemetry server:

1. Tag our telemetry pings with a custom identifier
2. Trigger ping collection (generation) on the client. Since this is done differently for each of the ping scenarios, we'll list the steps separately below
3. Submit data to telemetry server
   • Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait!

Baseline Ping

Launch Glean See test steps in TestRail - Fenix - telemetry

Metrics Ping

See test steps in TestRail - Fenix - telemetry

NOTE

• If metrics ping doesn't get submitted, go to Android Settings and change local time to 3:58 then launch Glean command

Events Ping

See test steps in TestRail - Fenix - telemetry

As of the writing of this test plan, there are currently 37 events defined in the fenix events.yaml (Look for "type: event").

data collection

• Pick 4-5 event types and perform those user actions in sequence (NOTE: we will later want to verify that they are recorded on the telemetry server in the correct sequence).
• Reason (per the events ping docs): "The events ping's purpose is to transport all of the event metric information"

Data Integrity

Now let's verify that data has arrived to telemetry service intact.

Verify JSON data per pings doc

Setup: Open Ping Data

1. Open Glean Debug Dashboard: [2]
2. Verify real-time receipt of tagged ping in dashboard
3. Verify ping entry contains correct field data
4. Drill down into ping
5. Verify that "Recent pings..." entry contains correct ping types
6. Drill down to JSON data

Verify Common Info

• Verify data in all fields per common fields doc
• NOTE: if common info is good on baseline ping, SHOULD BE good on others - no need to repeat for other ping scenarios! (All use same path of execution)

Verify Baseline Info

Verify the following fields:

• common fields info
• duration
  • should roughly approximate difference between start_time, end_time
• Verify locale string

Verify Events Info

Verify the following fields:

• common fields info

Verify that specific events performed in the Data Collection & Submission section have arrived in Dashboard in the proper sequence.

NOTE: From just launching the application, you should see the “app_icon launch” event listed first

Verify Metrics Info

Verify the following fields:

• common fields info
• Review al fields in metrics.yaml not of type: event

Verify Custom Info

TBD

Test Tools

Summary

You will need the following tools to execute end-to-end telemetry tests:

1. Android physical device
   • Testing can be done on an x86 device but is less reliable
   • Glean currently broken on x86: issue/2291
2. Browser
   • Test plan (this page)
   • Taskcluster APK download page
   • Glean Debug Dashboard
     • to verify server-side activity
3. Terminal
   • You will run all your ADB, Glean commands from here
4. Logcat
   • Open Android Studio to monitor all client-side activity

Glean CLI

Below is a brief summary of the Glean CLI features documented here: Debugging products using Glean

logPings

1. To dump Glean logs to logcat
   • adb shell am start -n ...GleanDebugActivity --ez logPings true

sendPing

1. To force sending a ping immediately use:
   • adb shell am start -n ...GleanDebugActivity --es sendPing metrics
   • (or instead of metrics other ping names)
2. Look at Glean frontend to verify data is landing

tagPings

1. To tag a ping with a user label for easy query in Glean Debug dashboard
   • adb shell am start -n ...GleanDebugActivity --es tagPings rpapa-test1, then spot rpapa-test1 in the frontend to identify your pings.

Examples

For running the commands with any Firefox Android app, adjust the application ID part in the command:

Fenix

• adb shell am start -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1

Android Components (sample apps)

• adb shell am start -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1

Reference Browser

• adb shell am start -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1
  • NOTES
    • even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the baseline ping.
    • suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data:

Reference

Automated Tests (Firefox Mobile) - TBD

see notes: https://groups.google.com/a/mozilla.com/forum/?utm_medium=email&utm_source=footer#!msg/android-product-team/7XpVCA2Be80/tpWsuaXoEwAJ]

Automated Tests (Firefox Desktop)

• Automated tests for Firefox desktop do "client verification only test" also rely on a local HTTP client as a kind of simple mock telemetry server, used to capture, read and verify the outbound JSON payload locally
• Events are triggered, data integrity is verified
• see source: mozilla-central

NOTES

• Bugzilla: Enable tagging glean pings for the "debug view"
• glean on iOS platform [3]