Confirmed users
9,511
edits
Firefox/Stub Attribution/Test Plan: Difference between revisions
Firefox/Stub Attribution/Test Plan (view source)
Revision as of 22:38, 5 January 2017
, 5 January 2017→References
(→TODO) |
|||
| (39 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
== Dependencies (tools/systems): == | == Dependencies (tools/systems): == | ||
# Firefox client code (Stub Installer) | |||
# [https://github.com/mozilla-services/go-bouncer Bouncer] - GitHub | |||
## Staging: http://bouncer-bouncer.stage.mozaws.net/ | |||
# Mozilla.org ([https://github.com/mozilla/bedrock Bedrock]) - GitHub | |||
# [https://github.com/mozilla-services/stub_attribution Stub Downloader service] - GitHub | |||
#* Staging: https://stubattribution-default.stage.mozaws.net/ | |||
#** Sentry URL: https://sentry.prod.mozaws.net/operations/stub_attribution-stage/ | |||
#* Production: https://stubdownloader.services.mozilla.com/ | |||
#** Sentry URL: https://sentry.prod.mozaws.net/operations/stub_attribution-prod/ | |||
# [https://wiki.mozilla.org/Telemetry Telemetry] | |||
#* Caveat: Telemetry data lags 3 weeks, even after the switches have been flipped | |||
== Acceptance Criteria == | == Acceptance Criteria == | ||
In order to | In order to fully test, the following are needed: | ||
<bugzilla> | <bugzilla> | ||
{ | { | ||
"id": "1306457, 1273940, 1279291, 1278981" | "id": "1306457, 1273940, 1279291, 1278981, 1318456, 1324692, 1320773" | ||
} | } | ||
</bugzilla> | </bugzilla> | ||
# a staged or dark-launched Mozilla.org instance ready with Download-button logic and passed-in attribution_code (from [https://bugzilla.mozilla.org/show_bug.cgi?id=1279291 bug 1279291]), which: | # a staged or dark-launched Mozilla.org instance ready with [https://github.com/mozilla/bedrock/pull/4456#issue-188016683 Download-button logic] and passed-in attribution_code (from [https://bugzilla.mozilla.org/show_bug.cgi?id=1279291 bug 1279291]), which: | ||
# ...uses the [https://github.com/mozilla/bedrock/pull/4253 AJAX service to sign Stub-Attribution URLs] with a SHA-246 HMAC hash ([https://bugzilla.mozilla.org/show_bug.cgi?id=1278981 bug 1278981]) | # ...uses the [https://github.com/mozilla/bedrock/pull/4253 AJAX service to sign Stub-Attribution URLs] with a SHA-246 HMAC hash ([https://bugzilla.mozilla.org/show_bug.cgi?id=1278981 bug 1278981]) | ||
# [https://github.com/mozilla-services/go-bouncer/pull/36 Bouncer (download.mozilla.org) logic updated] to ignore attribution_code for XP users | # [https://github.com/mozilla-services/go-bouncer/pull/36 Bouncer (download.mozilla.org) logic updated] to ignore attribution_code for XP+Vista users | ||
# [https://bugzilla.mozilla.org/show_bug.cgi?id=1273940#c35 Firefox stub installer binaries] - PENDING in [https://wiki.mozilla.org/RapidRelease/Calendar Firefox 50 builds] shipping November 15th - which include the stub attribution_code's post-signing data capability available and pointed to... | # [https://bugzilla.mozilla.org/show_bug.cgi?id=1273940#c35 Firefox stub installer binaries] - PENDING in [https://wiki.mozilla.org/RapidRelease/Calendar Firefox 50 builds] shipping November 15th - which include the stub attribution_code's post-signing data capability available and pointed to... | ||
# production-ready [https://bugzilla.mozilla.org/show_bug.cgi?id=1273940 stubdownloader.prod.mozaws.net] service/instance | # production-ready [https://bugzilla.mozilla.org/show_bug.cgi?id=1273940 stubdownloader.prod.mozaws.net] service/instance | ||
= Dark Launch Criteria = | |||
# Bedrock's [https://github.com/mozilla/bedrock/pull/4456 front-end stub-attribution button code] merged to master, pushed to prod | |||
# [https://github.com/mozilla-services/stubattribution Stub Attribution service] (backend) | |||
## pushed to prod: https://stubdownloader.services.mozilla.com/ | |||
## monitoring in place (with alerting) for: | |||
### expected cache misses | |||
### server error codes, i.e. HTTP 4xx, 5xxx | |||
# RelEng: all changes listed in [https://bugzilla.mozilla.org/show_bug.cgi?id=1320773#c22 bug 1320773] | |||
= Latest Testing Status = | |||
# verified that https://stubattribution-default.stage.mozaws.net/?product=test-stub&os=win&lang=en-US returns test-stub.exe from https://download-installer.cdn.mozilla.net/pub/firefox/nightly/experimental/bug1318456/test-stub.exe | |||
# verified that https://bouncer-bouncer.stage.mozaws.net/?product=test-stub&os=win&lang=en-US redirects and returns test-stub.exe from https://download-installer.cdn.mozilla.net/pub/firefox/nightly/experimental/bug1318456/test-stub.exe | |||
= Open Issues = | = Open Issues = | ||
== TODO == | == TODO == | ||
* find Python help sourcing raw, incoming/already-collected (server-side) attribution.* data, using http://reports.telemetry.mozilla.org/render?markdown=tutorials/telemetry_hello_world.kp (or the like) | |||
** see also: https://wiki.mozilla.org/Telemetry/Custom_analysis_with_spark | |||
* add more sources, like: | |||
** www.getfirefox.com (which 301 -> https://www.mozilla.org/en-US/firefox/new/?utm_medium=referral&utm_source=getfirefox-com) | |||
* add Twitter | |||
* which other social-media sources? | |||
* determine what additional test-automation coverage we might need to add to the [https://github.com/mozilla-services/go-bouncer/tree/209728e2886cb9339f2d9f848c9fedf73e01c544/tests/e2e end-to-end Bouncer tests] | * determine what additional test-automation coverage we might need to add to the [https://github.com/mozilla-services/go-bouncer/tree/209728e2886cb9339f2d9f848c9fedf73e01c544/tests/e2e end-to-end Bouncer tests] | ||
** assert SHA-1 only served to Chrome, IE 6, 7, 8 on XP, Vista | |||
** assert SHA-256 is served to IE 8, Windows 7 | |||
* confirm and note who checks the Telemetry pings | * confirm and note who checks the Telemetry pings | ||
* confirm and note load-testing strategy | |||
* confirm and note performance-testing strategy | |||
** duration from Mozilla.org Download-button click to being served the correct stub-installer binary | |||
== Questions: == | == Notes + Questions: == | ||
# How (or can we even?) establish a reasonable performance metric around downloading the stub installer, currently? | * From https://github.com/mozilla-services/go-bouncer/issues/34: "If the attribution_code parameter is set, redirect to the stub downloader service with the product, os, lang and attribution_code parameters set." | ||
## ...so we can measure against this baseline when we test the dynamic vs. cached-downloads Stub-Attribution Service | ** Additionally, from https://github.com/mozilla-services/go-bouncer/pull/36#issuecomment-254936680: "This looks fine to me. It should include some sort of signature or hmac parameter that will contain the hmac signature that verifies that the request came from www.mozilla.org, but I'm very pleased that bouncer will handle the redirect." | ||
*** And, finally, from https://github.com/mozilla-services/go-bouncer/pull/36#issuecomment-255865245: "New commit passes through an attribution_sig parameter." | |||
* From https://github.com/mozilla-services/stubattribution/pull/20/files: "If set, the `attribution_code` parameter will be verified by validating that the `attribution_sig` parameter matches the hex-encoded sha256 hmac of `attribution_code` using`HMAC_KEY`." | |||
* From oremj: "we want every unique group of four attribution code parameters to point at the same url; so for source=google, campaign=foo, source=foo, medium=foo... attribution_code and attribution_sig should always be the same" | |||
* From me: "and right now, looks like attribution_sig is a timestamped + more hashed value" | |||
#Notes on caching/timeout values: | |||
## (Stub Attribution) HMAC_TIMEOUT is 10 minutes: https://github.com/mozilla-services/stubattribution/commit/8a6cc83547705d35451fb3ad08e3f82035f04abb | |||
## (Stub Attribution) unaltered stub-installer binary w/cert is cached for 5 minutes: https://github.com/mozilla-services/stubattribution/pull/38/files | |||
## (Bedrock) stub-attribution cookie is now 24 hours: https://github.com/mozilla/bedrock/pull/4456/commits/4e8d5bb6bd6506d9cd2dfabef67ff2c780c8bbd9 | |||
#How can we test through cookied-Bedrock flow, at scale, for unique builds? | |||
#How can we test (and try to break) Jeremy's 10-minute cache. And what does it cover, binary/condition-wise? (i.e. what's the driving logic/algorithm for caching vs. serving fresh?) | |||
#How/can we performance-test the UI experience? | |||
#What can we drive, using WebDriver? | |||
#Can we make two identical requests using the go-bouncer e2e tests, and check for/ensure we get a cached binary? | |||
##Likewise, two different requests (i.e. with just one unique key attribute), and get fresh, unique binaries in that case? | |||
#How (or can we even?) establish a reasonable performance metric around downloading the stub installer, currently? | |||
##...so we can measure against this baseline when we test the dynamic vs. cached-downloads Stub-Attribution Service | |||
# How to test (all five?) codes/fields? | # How to test (all five?) codes/fields? | ||
## Source | ## Source | ||
| Line 50: | Line 87: | ||
## https://www.mozilla.org/en-US/firefox/all/ | ## https://www.mozilla.org/en-US/firefox/all/ | ||
## https://www.mozilla.org/en-US/firefox/new/?scene=2 (and do the ?scene=[] parameters matter?) | ## https://www.mozilla.org/en-US/firefox/new/?scene=2 (and do the ?scene=[] parameters matter?) | ||
# Do we need to cover the upgrade-path scenario? i.e.: | # Do we need to cover the upgrade-path scenario? i.e.: | ||
## user downloads and installs using the special stub installer w/tracking code | ## user downloads and installs using the special stub installer w/tracking code | ||
| Line 65: | Line 100: | ||
=== Manual Tests: === | === Manual Tests: === | ||
# Positive (happy-path) Tests: | # Positive (happy-path) Tests: | ||
# Ensure that https://stubattribution-default.stage.mozaws.net/?product=test-stub&os=win&lang=en-US works and returns a Telemetry-enabled (with Stub Attribution params) build | |||
## Do Not Track disabled | ## Do Not Track disabled | ||
##* Click an ad banner or link which has an attribution (source? medium? campaign? content? referrer?) code | ##* Click an ad banner or link which has an attribution (source? medium? campaign? content? referrer?) code | ||
| Line 73: | Line 109: | ||
=== Regression Testing === | === Regression Testing === | ||
'''Goal:''' Ensure that Bouncer + Mozilla.org continue to deliver the correct builds to the appropriate users | |||
* All browsers available on Windows XP/Vista (test IE 6, IE 7, Chrome) except for Firefox should still get the SHA-1 stub installer | * All browsers available on Windows XP/Vista (test IE 6, IE 7, Chrome) except for Firefox should still get the SHA-1 stub installer | ||
| Line 80: | Line 117: | ||
=== Performance Testing === | === Performance Testing === | ||
'''Goal:''' Understand, mitigate, and quantify, if possible, the performance impact that adding a dynamic Stub Attribution service has on downloading stub-installer builds on Windows clients (and without regressing the delivery performance of builds for other platforms?) | |||
=== Load/Soak Tests === | === Load/Soak Tests === | ||
'''Goals:''' | |||
# In order to be able to more fully test the Mozilla.org -> Bouncer -> Stub Attribution dynamic stub-installer build scenario generate enough traffic/load such that we try to break through server-side caching | |||
# Help establish baselines and thresholds for server-side performance/availability | |||
* Against Bouncer | * Against Bouncer | ||
* Against Stub Downloader service | * Against Stub Downloader service | ||
| Line 102: | Line 143: | ||
** Where/how frequently will they run? With each pull request/commit/release? | ** Where/how frequently will they run? With each pull request/commit/release? | ||
* What will the Bouncer tests cover? | * What will the Bouncer tests cover? | ||
** Where/how frequently will they run? With each pull request/commit/release? On a cron? | |||
* What will the Stub Attribution unit tests cover? | |||
** Where/how frequently will they run? With each pull request/commit/release? On a cron? | ** Where/how frequently will they run? With each pull request/commit/release? On a cron? | ||
{{VerifiedUser}} | {{VerifiedUser}} | ||
= References = | |||
* Source, medium, campaign, and content values from in-the-wild: | |||
** https://docs.google.com/spreadsheets/d/1U-0JHpc3INJnBwTFkdrPqpk7yqvpEn16DJtp6d6TRiY/edit#gid=1681956968 | |||
* Whiteboard flows/diagrams: | * Whiteboard flows/diagrams: | ||
** [https://wiki.mozilla.org/File:Stub-attribution-organic.JPG Organic] | ** [https://wiki.mozilla.org/File:Stub-attribution-organic.JPG Organic] | ||