Firefox/Stub Attribution/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 15: Line 15:


== Acceptance Criteria ==
== Acceptance Criteria ==
In order to begin testing, the following are needed:
In order to fully test, the following are needed:
<bugzilla>
<bugzilla>
{
{
"id": "1306457, 1273940, 1279291, 1278981, 1318456, 1324692"
"id": "1306457, 1273940, 1279291, 1278981, 1318456, 1324692, 1320773"
}
}
</bugzilla>
</bugzilla>

Revision as of 19:11, 20 December 2016

Stub Attribution Test Plan

Dependencies (tools/systems):

  1. Firefox client code (Stub Installer)
  2. Bouncer - GitHub
    1. Staging: http://bouncer-bouncer.stage.mozaws.net/
  3. Mozilla.org (Bedrock) - GitHub
  4. Stub Downloader service - GitHub
  5. Telemetry
    • Caveat: Telemetry data lags 3 weeks, even after the switches have been flipped

Acceptance Criteria

In order to fully test, the following are needed:

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 406 [1] => Not Acceptable ) ) 1

  1. a staged or dark-launched Mozilla.org instance ready with Download-button logic and passed-in attribution_code (from bug 1279291), which:
  2. ...uses the AJAX service to sign Stub-Attribution URLs with a SHA-246 HMAC hash (bug 1278981)
  3. Bouncer (download.mozilla.org) logic updated to ignore attribution_code for XP+Vista users
  4. Firefox stub installer binaries - PENDING in Firefox 50 builds shipping November 15th - which include the stub attribution_code's post-signing data capability available and pointed to...
  5. production-ready stubdownloader.prod.mozaws.net service/instance

Latest Testing Status

  1. verified that https://stubattribution-default.stage.mozaws.net/?product=test-stub&os=win&lang=en-US returns test-stub.exe from https://download-installer.cdn.mozilla.net/pub/firefox/nightly/experimental/bug1318456/test-stub.exe
  2. verified that https://bouncer-bouncer.stage.mozaws.net/?product=test-stub&os=win&lang=en-US redirects and returns test-stub.exe from https://download-installer.cdn.mozilla.net/pub/firefox/nightly/experimental/bug1318456/test-stub.exe

Open Issues

TODO

  • determine what additional test-automation coverage we might need to add to the end-to-end Bouncer tests
    • assert SHA-1 only served to Chrome, IE 6, 7, 8 on XP, Vista
    • assert SHA-256 is served to IE 8, Windows 7
  • confirm and note who checks the Telemetry pings
  • confirm and note load-testing strategy
  • confirm and note performance-testing strategy
    • duration from Mozilla.org Download-button click to being served the correct stub-installer binary

Questions:

  1. How (or can we even?) establish a reasonable performance metric around downloading the stub installer, currently?
    1. ...so we can measure against this baseline when we test the dynamic vs. cached-downloads Stub-Attribution Service
  2. How to test (all five?) codes/fields?
    1. Source
    2. Medium
    3. Campaign
    4. Content
    5. "Referrer" alone?
      1. Which specific browsers (vendors + versions) properly support it/if have it enabled, we'll honor the setting?
  3. Which locale(s)?
  4. Entry-points on Mozilla.com. Which page(s)? Or all download pages?
    1. https://www.mozilla.org/en-US/firefox/all/
    2. https://www.mozilla.org/en-US/firefox/new/?scene=2 (and do the ?scene=[] parameters matter?)
  5. Do we need to cover the upgrade-path scenario? i.e.:
    1. user downloads and installs using the special stub installer w/tracking code
    2. we verify correct pings, etc.
    3. user upgrades later to a newer version of Firefox (pave-over install)
      1. do we still check for this ping?
  6. How about the successfully installed, then uninstalled case: check to see that client no longer sends ping?
  7. Should we test/what should happen in the double-install case? (Install, then install again w+w/o uninstalling the 1st binary.)
    1. Similarly, what about the a) install stub w/attribution b) upgrade to a later version of Firefox case?

Testing Approaches

Manual Tests:

  1. Positive (happy-path) Tests:
  2. Ensure that https://stubattribution-default.stage.mozaws.net/?product=test-stub&os=win&lang=en-US works and returns a Telemetry-enabled (with Stub Attribution params) build
    1. Do Not Track disabled
      • Click an ad banner or link which has an attribution (source? medium? campaign? content? referrer?) code
      • Verify that the URL which takes you to a Mozilla.org download page contains a stub_attcode with a valid param
      • Verify that the same valid stub_attcode param/value gets passed to the Mozilla.org Download Firefox button
      • Download and install the stub installer
      • Verify (how?) that upon successful installation, the stub installer sends a "success!" ping with the same stub_attcode param/value

Regression Testing

Goal: Ensure that Bouncer + Mozilla.org continue to deliver the correct builds to the appropriate users

  • All browsers available on Windows XP/Vista (test IE 6, IE 7, Chrome) except for Firefox should still get the SHA-1 stub installer

Negative Testing

  1. Ensure other installers/binaries don't have/pass on the URL param/stub code
    1. e.g. Mac, Linux, full Firefox for Windows installers

Performance Testing

Goal: Understand, mitigate, and quantify, if possible, the performance impact that adding a dynamic Stub Attribution service has on downloading stub-installer builds on Windows clients (and without regressing the delivery performance of builds for other platforms?)

Load/Soak Tests

Goals:

  1. In order to be able to more fully test the Mozilla.org -> Bouncer -> Stub Attribution dynamic stub-installer build scenario generate enough traffic/load such that we try to break through server-side caching
  2. Help establish baselines and thresholds for server-side performance/availability

Fuzz/Security Testing

  • Purpose:
    • Surface potential incorrectly-handled errors (tracebacks/HTTP 5xx, etc.) and potential security issues
  • Likely using OWASP ZAP, either manually and/or via the CLI, using Docker/Jenkins
    • Against Bouncer
    • Against Stub Downloader service
    • Against Mozilla.org

Test Automation Coverage:

  • What will the Bedrock unit tests cover?
    • Where/how frequently will they run? With each pull request/commit/release?
  • What will the Bouncer tests cover?
    • Where/how frequently will they run? With each pull request/commit/release? On a cron?
  • What will the Stub Attribution unit tests cover?
    • Where/how frequently will they run? With each pull request/commit/release? On a cron?

References

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox/Stub_Attribution/Test_Plan&oldid=1157546"