QA/Mobile/TelemetryTestPlan

From MozillaWiki
< QA
Jump to: navigation, search
--------------->>> THIS IS A DRAFT! <<<---------------  

This is the test plan for verification of the telemetry service on Mozilla mobile platforms

Test Strategy

Two types of telemetry verification will take place for all our mobile projects:

  • End-2-End (Networked) Test
    • type: manual
    • frequency: upon request from devs
  • Client-verification tests
    • type: automated
    • frequency: github trigger on PR (new, updated)
  • For more information regarding implementation details, see discussion: discussion

Manual Tests

End-2-End (Networked) Test

SUMMARY

This will be a "client-to-server verification" test which will require manually triggering specific ping activities and verifying them in the debug-view of the Glean dashboard (TBD)


TEST STEPS

  1. launch the app
  2. "tag" the data
    • This will enable the data to be easily queried from within the Glean datastream
    • Example: adb shell -n org.mozilla.reference.browser/mozilla.components.service.glean.debug.GleanDebugActivity --es userTag georg-test1
  3. When you want to force sending a ping right away use:
    • adb shell -n ...GleanDebugActivity --es sendPing metrics
    • (or instead of metrics other ping names)
  4. Look at Glean frontend to verify data is landing


userTag (TBD)

  • This is a proposed feature
  • see: Bugzilla bug 1522428
  • --es userTag georg-test1, then spot georg-test1 in the frontend to identify your pings.


Sample App Usage

For running the commands with a [android-components https://github.com/mozilla-mobile/android-components] sample app, you'd adjust the application ID part in the command:

adb shell -n org.mozilla.samples.glean/mozilla.components.service.glean.debug.GleanDebugActivity --es userTag georg-test1


GLEAN DASHBOARD (DEBUG MODE)

  • NOTE: This dashboard is currently scheduled for deliver ~April 15 (depending on Fenix project planning)
  • Debug view makes data received by the pipeline from some devices visible in real-time
  • Enabling debug mode would either return a unique identifier, or ask for the ID you want to use (see userTag above)
  • userTags would be used to organize data in the web UI


NOTES

Automated Tests

Client Verification Test

SUMMARY

This will be a "client verification only" test which will rely on a local HTTP client (probably the A-C fetch component) as a kind of simple mock telemetry server, which we will use to capture, read and verify the outbound JSON payload locally.

  • priority #1: verify that our telemetry object has been called as expected using assertion functions
  • priority #2: unpack and verify integrity of payload contents
  • frequency: Test execution frequency will be determined by the CI (Taskcluster or Bitrise) config file

Reference

Automated Tests (Firefox Desktop)

SUMMARY

  • Automated tests for Firefox desktop do "client verification only test" also rely on a local HTTP client as a kind of simple mock telemetry server, used to capture, read and verify the outbound JSON payload locally
  • Events are triggered, data integrity is verified
  • see source: mozilla-central


WITHIN SCOPE

  • client-side data integrity test


OUTSIDE SCOPE

  • end-to-end data integrity test


STEPS

  • Creates trigger event(s)
    • i.e. open tab(s), restart
  • Checks ping contents (one or more of scalars)


FUTURE

  • re-use a user profile (rather than only fresh profiles)