MozillaWiki

QA/Mobile/TelemetryTestPlan

< QA
Revision as of 23:00, 8 May 2019 by Rpappalardo (talk | contribs) (testing)

Summary

Product Coverage

This is the test plan for verification of telemetry service clients for the following Mozilla Android products:

Fenix

Frequency:

  • thorough first pass and on releases, spot check w/ sprints (to be verified w/ kbrosnan)

Issues:

Reference Browser

Frequency:

  • upon request from dev, management

android-components - sample apps

Frequency:

  • upon request from dev, management

Firefox for FireTV

Frequency:

  • TBD - pending Glean implementation

Test Demo

This demo video illustrates the basic steps for test setup and execution. This is not intended to illustrate a real test case. Additionally, you will want to monitor all local activity using logcat on Android Studio.

In this demo, you can observe:

  1. Telemetry being sent from client to the server
  2. Telemetry arrives at the server without excessive latency (can be viewed from the Glean Debug Dashboard)
  3. Telemetry is complete and data integrity maintained

NOTES:

  • For ease of demonstration, an X86 device was used, but all testing should be done using physical devices.
  • Test steps in video are outlined in wiki below

Test Strategy

Test Types

The Mobile QA team will perform two test types of telemetry verification for all our mobile projects:


End-2-end (client-to-server) tests

Type:

  • manual

Frequency:

  • upon request from devs, management

Who:

  • SoftVision engineers

Tools:

  • Glean CLI, Glean Debug Dashboard


Client-side only verification tests (TBD)

Type:

  • automated

Frequency:

  • github trigger on merge-to-master

Who:

  • No-Jun's team

Tools:

Ping Testing

Summary

A "ping" refers to a JSON blog of data sent from the client to the telemetry service. There are three basic types: baseline, events and metrics pings. In addition, custom pings (i.e. activities, sync pings) can be defined using the pings.yaml and metrics.yaml files. Test cases should be designed around these various "ping scenarios."

Ping Activities

Summary

Testing ping integrity from end-to-end can be broken down into three high-level steps:

  1. Data Collection
    • Ping data is collected on the client (and stored locally on disk) when certain specific actions are performed.
    • Our test coverage must ensure we trigger data collection for each of the ping scenarios by performing those actions in the proper sequence.
    • Generated data will first be verified locally via logcat
  2. Data Submission
    • Using the Glean CLI tool, we'll force data to be submitted to the telemetry server immediately, rather than waiting for a scheduled trigger event(s) or for thresholds to be met, etc.
  3. Data Integrity
    • Using the Glean Debug Dashboard, we must verify that the ping data we've tagged and forced to be submitted (using the Glean CLI tool), arrives without undue latency (within approx. 10-20 seconds).
    • Drill down on the JSON data and verify the correct data has been submitted.

Ping Scenarios

Baseline ping

  • trigger(s) / scheduling:
    • sent when application is moved to background
  • contents: includes the following fields....
    • duration, locale
    • common ping sections found in all pings

Events ping

  • trigger(s) / scheduling:
    • sent when application goes into background (if any recorded events to send)
    • when queue of events exceeds Glean.configuration.maxEvents (default 500)
    • if any unsent events found on disk when starting application
  • contents:
    • array of all of the events that have occurred since the last time the events ping was sent
    • common ping sections found in all pings

Metrics ping

  • trigger(s) / scheduling:
    • sent at first available opportunity after 4AM local time on a new calendar day
  • contents:

Manual Tests

End-2-end client-to-server tests

SUMMARY

Description

This will be a "client-to-server (end-2-end) verification" test using the production telemetry service across the real network (not a mock service on localhost). Execution requires manually triggering a telemetry ping to be sent to the telemetry service by emulating a specific set of user activities in the application.

Pings can be successfully verified by ensuring they have arrived in the telemetry server - viewable in the Glean Debug Dashboard.


Dashboard Glean Dashboard

  • Authentication requires SSO login
  • see: Debugging products using Glean
  • Features
    • Glean Debug Dashboard makes data received by the pipeline from some devices visible in real-time
    • Enabling debug mode either returns a unique identifier, or the ID you want to use (see tagPings parameter below), enabling immediate query of test pings

TEST EXECUTION

Download Application

  1. Download the Fenix (or other) release apk file

NOTE:

  • Debug builds won't work, but test build can otherwise be custom build or from PlayStore (no special build needed)

Install Application

  1. Connect an Android (physical) device via USB
  2. Open terminal
  3. Verify device is connected: $ adb devices
  4. Install the apk: $ adb install ~/Downloads/target.aarch64.apk
  5. Verify app has been installed on device
  6. NOTE: Do not open the application, Glean will do this for us

Check Network

  1. ping Glean Debug Dashboard: [1]

Send Telemetry Pings

Next, we will do three things (see "Examples" below):

  1. Tag our telemetry pings with a custom identifier
  2. Trigger ping generation on client
  3. Force immediate send of ping to telemetry server (i.e. we won't wait for scheduled send)

Verify Ping Success

  1. Open Glean Debug Dashboard: [2]
  2. Verify real-time receipt of tagged ping in dashboard
  3. Verify ping entry contains correct field data
  4. Drill down into ping
  5. Verify that "Recent pings..." entry contains correct ping types
  6. Drill down to JSON data
  7. Verify JSON data per glean pings doc

EXAMPLES

For running the commands with any Firefox Android app, adjust the application ID part in the command:

Fenix

  • adb shell -n org.mozilla.fenix.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1

Android Components (sample apps)

  • adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1

Reference Browser

  • adb shell -n org.mozilla.reference.browser.debug/mozilla.components.service.glean.debug.GleanDebugActivity --ez logPings true --es sendPing metrics --es tagPings rpapa-test1
  • NOTE
    • even with telemetry enabled, reference-browser doesn't define any metric outside of the ones we provide out of the box, which are sent in the baseline ping.
    • suggestion: run commands against a version of reference-browser that has telemetry enabled (e.g. the one from the play store), and send a ping for which we're sure to have data:

PARAMETERS

logPings

  1. To dump Glean logs to logcat
    • adb shell -n ...GleanDebugActivity --ez logPings true

sendPing

  1. To force sending a ping right away use:
    • adb shell -n ...GleanDebugActivity --es sendPing metrics
    • (or instead of metrics other ping names)
  2. Look at Glean frontend to verify data is landing

tagPings

  1. To tag a ping with a user label for easy query in Glean dashboard
    • adb shell -n ...GleanDebugActivity --es tagPings rpapa-test1, then spot rpapa-test1 in the frontend to identify your pings.

NOTES

Automated Tests (TBD)

Client-side only verification test

SUMMARY

This will be a "client-side only verification" test which will rely on a local HTTP client (probably the A-C fetch component) as a kind of simple mock telemetry server, which we will use to capture, read and verify the outbound JSON payload (jailed on localhost).

  • priority #1: verify that our telemetry object has been called as expected using assertion functions
  • priority #2: unpack and verify integrity of payload contents
  • frequency: Test execution frequency will be determined by the CI (Taskcluster or Bitrise) config file

SCHEDULE Targeting for 2019, H2

Reference

Automated Tests (Firefox Desktop)

DESCRIPTION

  • Automated tests for Firefox desktop do "client verification only test" also rely on a local HTTP client as a kind of simple mock telemetry server, used to capture, read and verify the outbound JSON payload locally
  • Events are triggered, data integrity is verified
  • see source: mozilla-central
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Mobile/TelemetryTestPlan&oldid=1212033"