QA/Mobile/TelemetryTestPlan

From MozillaWiki
< QA
Revision as of 20:35, 10 May 2019 by Rpappalardo (talk | contribs) (change one word)
Jump to navigation Jump to search

Summary

Product Coverage

This is the test plan for verification of telemetry service clients for the following Mozilla Android products:

Fenix

Frequency:

  • thorough first pass and on releases, spot check w/ sprints (to be verified w/ kbrosnan)

Issues:

Reference Browser

Frequency:

  • upon request from dev, management

android-components - sample apps

Frequency:

  • upon request from dev, management

Firefox for FireTV

Frequency:

  • TBD - pending Glean implementation

Test Demo

This demo video illustrates the basic steps for test setup and execution. This is not intended to illustrate a real test case! Additionally, you will want to run all tests on a physical (not an x86 device) and monitor all local activity using logcat on Android Studio.

Test Strategy

Summary

The Mobile QA team will perform two test types of telemetry verification for all our mobile projects:

  1. End-to-end (client-to-server) manual tests
  2. Client-side only verification automated tests (TBD)

This wiki outlines a test plan for manual testing.

Frequency:

  • upon request from devs, management

Who:

  • SoftVision engineers

Ping Scenarios

Description

A "ping" refers to a JSON data blob sent from the client to the telemetry service. There are three basic types (scenarios):

  1. baseline ping
  2. events ping
  3. metrics pings

In addition, custom pings (i.e. activation, sync pings) can also be defined using the pings.yaml and metrics.yaml files.

All test cases should be designed around these various "ping scenarios." This wiki will explain how to test each of these scenarios.

Ping Activities

Description

For each ping scenario, testing ping behavior from end-to-end can be broken down into three high-level steps:

  1. Data Collection
    • Ping data is collected on the client (and stored locally on disk). Some data is recorded automatically (i.e. baseline and metrics pings). Other data is collected when certain specific actions are performed by the user. The latter are referred to as "events" (i.e. events pings).
    • Our test coverage must ensure we verify that data collection and submission has taken place locally for each of the ping scenarios. We can monitor this activity with logcat.
  2. Data Submission
    • Using the Glean CLI tool, we'll tag and force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc.
  3. Data Integrity Verication
    • Using the Glean Debug Dashboard, we must verify that our ping data arrives at the telemetry server without undue latency (within approx. 10-20 seconds).
    • Drill down on the JSON data and verify the correct data has been submitted.

Test Execution

Summary

This section describes how to perform an end-to-end (client-to-server) test using the production telemetry service across the real network. Please note, for these tests we will not be a using the telemetry staging server nor a mock server on localhost to receive telemetry pings. Test execution will require installation of an application APK on a physical mobile device and use of a built-in CLI tool called Glean.


First, we will trigger telemetry data collection locally by emulating a specific set of user actions in the application. Next, we will force immediate data submission to the telemetry server. Lastly, we will use the Glean Debug Dashboard to verify data integrity.


To summarize, for each ping scenario, we will perform the following steps in sequence:

  1. Test Setup
  2. Data Collection
  3. Data Submission
  4. Data Integrity Verification


NOTES on Dashboard Glean Dashboard

Test Setup

Download Application

  1. Download the Fenix (or other) release apk file

NOTE:

  • Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed)

Install Application

  1. Connect an Android (physical) device via USB
    • make sure your device has Settings > Developer Options > USB debugging enabled
  2. Open a terminal
  3. Verify device is connected: $ adb devices
  4. Install the apk: $ adb install ~/Downloads/target.aarch64.apk
  5. Verify app has been installed on device
  6. NOTE: Do not open the application (Glean will do this for us)

Check Network

ping Glean Debug Dashboard: [1]

$ ping https://debug-ping-preview.firebaseapp.com

Data Collection

Summary

Next, we will need to do two things:

  1. Tag our telemetry pings with a custom identifier
  2. Trigger ping collection (generation) on the client. Since this is done differently for each of the ping scenarios, we'll list the steps separately below

Baseline Ping

No specific user actions are required.

Reason (per the baseling ping docs): "This ping is intended to provide metrics that are managed by the library itself, and not explicitly set by the application"

Metrics Ping

No specific user actions are required.

Reason (per the metrics ping docs): "The metrics ping is intended for all of the metrics that are explicitly set by the application or are included in the application's metrics.yaml file (except events)"

Events Ping

As of the writing of this test plan, there are currently 37 events defined in the fenix metrics.yaml (Look for "type: event").

Pick 4-5 event types and perform those user actions in sequence (NOTE: we will later want to verify that they are recorded on the telemetry server in the correct sequence).

Reason (per the events ping docs): "The events ping's purpose is to transport all of the event metric information"

Data Submission

Once we have performed the above steps to locally collect data for the desired scenario, we will need to force it to be submitted immediately. By default, data is only submitted at scheduled times or when certain data thresholds are reached. We don't want to wait!

Repeat the following steps on every test to force immediate data submission for ALL scenarios:

  1. Close and uninstall any previous instances of app-under-test
  2. Re-install a fresh instance of app on device
  3. Launch app with Glean CLI command (not manually)
  4. Move app to the background (this should immediately trigger send)

Data Integrity

Now let's verify that data has arrived to telemetry service intact.

Verify JSON data per pings doc

Setup: Open Ping Data

  1. Open Glean Debug Dashboard: [2]
  2. Verify real-time receipt of tagged ping in dashboard
  3. Verify ping entry contains correct field data
  4. Drill down into ping
  5. Verify that "Recent pings..." entry contains correct ping types
  6. Drill down to JSON data

Verify Common Info

  • Verify data in all fields per common fields doc
  • NOTE: if common info is good on baseline ping, SHOULD BE good on others - no need to repeat for other ping scenarios! (All use same path of execution)

Verify Baseline Info

Verify the following fields:

  • common fields info
  • duration
    • should roughly approximate difference between start_time, end_time
  • Verify locale string

Verify Events Info

Verify the following fields:

Verify events that were performed in the Data Collection section have arrived in Dashboard

NOTE: From just launching the application, you should see the “app_icon launch” event listed first

Verify Metrics Info

Verify the following fields:

TBD TBD TBD

Review fields in metrics.yaml

Verify Custom Info

[Reference]

Test Tools

Summary

You will need the following tools to execute end-to-end telemetry tests:

  1. Android physical device
    • Testing can be done on an x86 device but is less reliable
    • Glean currently broken on x86: issue/2291
  2. Browser
  3. Terminal
    • You will run all your ADB, Glean commands from here
  4. Logcat
    • Open Android Studio to monitor all client-side activity

Glean CLI

Below is a brief summary of the Glean CLI features documented here: Debugging products using Glean

logPings

  1. To dump Glean logs to logcat
    • adb shell -n ...GleanDebugActivity --ez logPings true

sendPing

  1. To force sending a ping immediately use:
    • adb shell -n ...GleanDebugActivity --es sendPing metrics
    • (or instead of metrics other ping names)
  2. Look at Glean frontend to verify data is landing

tagPings

  1. To tag a ping with a user label for easy query in Glean Debug dashboard
    • adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1, then spot rpapa-test1 in the frontend to identify your pings.

Examples

For running the commands with any Firefox Android app, adjust the application ID part in the command:

Fenix

  • adb shell -n org.mozilla.fenix/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1

Android Components (sample apps)

  • adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1

Reference Browser

  • adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing baseline --es tagPings rpapa-test1
    • NOTES
      • even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the baseline ping.
      • suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data:

Reference

Automated Tests (Firefox Mobile) - TBD

see notes: https://groups.google.com/a/mozilla.com/forum/?utm_medium=email&utm_source=footer#!msg/android-product-team/7XpVCA2Be80/tpWsuaXoEwAJ]

Automated Tests (Firefox Desktop)

  • Automated tests for Firefox desktop do "client verification only test" also rely on a local HTTP client as a kind of simple mock telemetry server, used to capture, read and verify the outbound JSON payload locally
  • Events are triggered, data integrity is verified
  • see source: mozilla-central

NOTES

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Mobile/TelemetryTestPlan&oldid=1212173"