Services/TrackingProtection/Shavar Server - Testing: Difference between revisions
Jump to navigation
Jump to search
Rpappalardo (talk | contribs) m (Add image) |
Rpappalardo (talk | contribs) m (Fix broken link) |
||
| Line 313: | Line 313: | ||
* ensure that the resource is blocked: | * ensure that the resource is blocked: | ||
* You should see: | * You should see: | ||
** A black cat bearing its claws [[ | ** A black cat bearing its claws [[https://wiki.mozilla.org/File:Tracking_protection_off.png|sad cat]] | ||
** A tracking protection shield icon in the URL bar | ** A tracking protection shield icon in the URL bar | ||
** A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled." | ** A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled." | ||
| Line 336: | Line 336: | ||
** Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112) | ** Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112) | ||
** NOTE: if line #112 failure, rest of test will likely not work! | ** NOTE: if line #112 failure, rest of test will likely not work! | ||
** If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [[ | ** If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [[https://wiki.mozilla.org/File:Tracking_protection_off.png|sad cat]] | ||
[[#top]] | [[#top]] | ||
Revision as of 04:13, 14 September 2015
Testplan (Overview)
SUMMARY
- The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments
TEST RESULTS
- Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
- https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599
SIGNOFF CRITERIA
Cloud Services QA team will verify that:
- STAGE and PRODUCTION stacks are correctly deployed
- List creation scripts are correct
- Shavar server is working properly (delivering content to the client)
- Sample / Test pages are blocking / unblocking desired content
CACHE STATE
- for server-side tests, clear cache for each test so we can focus on the server's behavior
- note: we may want to add a future client-side test to verify proper cache behavior
FUTURE
Identify and test failure cases
- Server outage
- Partial responses
- Erroneous entry on black/whitelist, etc...
- Accidental blockage of Firefox updates?
REFERENCE
UI for Tracking protection
Test Pages
- http://itisatrap.org/firefox/its-a-tracker.html
- client-only test! (shavar-server could be completely broken and we wouldn't know)
- good test to run first, because if its broken, no reason to proceed
- http://people.mozilla.org/~fmarier/tracking-test/
- page checks if list delivery/generation is broken
- Uses: http://extremetracking.com/ (essentially within an iframe)
Staging Environment
SHAVAR URLs
Stage:
- tracking.stage.mozaws.net - e2e test URL
- tracking-protection.stage.mozaws.net - CDN for lists
- ckshavar.stage.mozaws.net - loadtest URL
Production:
- tracking.services.mozilla.com
- tracking.prod.mozaws.net
SHAVAR LISTS
We'll use 6 different lists in testing
- shumway list,
- abtest list,
2 production lists
- mozpub-track-digest256
- mozpub-trackwhite-digest256
2 stage testing lists
- moztestpub-track-digest256 **
- moztestpub-trackwhite-digest256 **
see: https://github.com/mozilla-services/shavar-test-lists
NOTE:
- In staging environment, we use the same shavar code, just using different lists.
Test Types
Server tests can be roughly broken into the following categories:
Stack Check
- Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.
List Delivery and Creation Test
Additions/Deletions Test
- follow Setup #1 instructions (create new prof, set different prefs)
- do cache file sanity checks (TEST #0)
- TEST #1
- TEST #2
Loadtesting
TBD
NOTE
- ideally we'd be using the real list
- we can do that for almost all the e2e test and validation test, but we can't do that for the "update" test
- TEST #0 gets executed twice
Test Cases
Stack-Check
PURPOSE
- test that the server is configured properly.
LIST THE LISTS
$ curl -d" " 'https://tracking.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'
Should return all of these lists:
mozpub-shumway-digest256
mozpub-track-digest256
mozpub-trackwhite-digest256
mozpub2-track-digest256
moztestpub-track-digest256
moztestpub-trackwhite-digest256
Next:
$ curl -d"mozpub-track-digest256;" 'https://tracking.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
Should return this:
n:3600
i:mozpub-track-digest256
u:tracking-protection.stage.mozaws.net/mozpub-track-digest256/1441146041
where n = 3600.
List Delivery & Creation Tests
Setup #1
SUMMARY
- Ensure the client is working
- Create a fresh new Firefox profile with custom prefs
SET CUSTOM PREFS
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
Replace production URL with stage URL for the following prefs:
- browser.trackingprotection.updateURL
- browser.trackingprotection.gethashURL
Set pref for debugging:
- browser.safebrowsing.debug = true
- this will give you some will give you some updates on the console
Remove Google malware & phishing protection:
- browser.safebrowsing.enabled = false
- browser.safebrowsing.malware.enabled = false
- privacy.trackingprotection.enabled = true (no reboot necessary)
RESET BROWSER
- !!!!! restart Firefox !!!!!!
TEST #0: verify downloaded (cached) mozpub list files
We're looking at:
- all 6 mozpub files should be present
- verify filesize:
- mozpub-track-digest256 files < 100K
- mozpub-trackwhite-digest256 is < 400k
Once the list updates are done, look into the cache directory:
~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac
After Setup #1, you should see something like this:
-rw-r--r-- 1 johnnyquest johnnyquest 68K Sep 2 12:57 mozpub-track-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 mozpub-track-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 76K Sep 2 12:57 mozpub-track-digest256.sbstore
-rw-r--r-- 1 johnnyquest johnnyquest 315K Sep 2 12:57 mozpub-trackwhite-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 mozpub-trackwhite-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 354K Sep 2 12:57 mozpub-trackwhite-digest256.sbstore
List Additions / Deletions Tests
Setup #2
SUMMARY
- We start with the same custom prefs we set in Setup #1
SET CUSTOM PREFS
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "moztestpub-track-digest256,moztestpub-trackwhite-digest256");
Additional prefs to append / modify
- urlclassifier.trackingTable = "test-track-simple,moztestpub-track-digest256"
- urlclassifier.trackingWhitelistTable = "test-trackwhite-simple,moztestpub-trackwhite-digest256"
- urlclassifier.disallow_completions += ",moztestpub-track-digest256,moztestpub-trackwhite-digest256"
- add these to existing entries! separated by commas
- for example, in Nightly: test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256
- !!!!! restart Firefox !!!!!!
TEST #0: verify downloaded (cached) moztestpub list files
We're looking at:
- all 6 moztestpub files should be present
- verify filesize:
- moztestpub-track-digest256 files < 100K
- moztestpub-trackwhite-digest256 is < 400k
Once the list updates are done, look into the cache directory:
~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac
After Setup #2, you should see something like this:
-rw-r--r-- 1 johnnyquest johnnyquest 68K Sep 2 12:57 moztestpub-track-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 moztestpub-track-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 76K Sep 2 12:57 moztestpub-track-digest256.sbstore
-rw-r--r-- 1 johnnyquest johnnyquest 315K Sep 2 12:57 moztestpub-trackwhite-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 moztestpub-trackwhite-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 354K Sep 2 12:57 moztestpub-trackwhite-digest256.sbstore
VERIFY ENABLED
its-a-tracker page
- Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
- Ensure that it is all green (blacklist and whitelist are working)
- This page doesn't hit shavar server at all
- If not working, then either about:config issue or client issue - repeat above!
CNN page
- Pull up CNN: http://www.cnn.com
- make sure that the Shield shows up
- the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking
YouTube page
- Pull up Youtube: https://www.youtube.com
- make sure that googlesyndication.com is NOT blocked by tracking protection on that site
- if the whitelist doesn't work, the devtools should indicate that googlesyndicaiton.com was blocked by TP
- if you see in console tab that googlesyndication.com is blocked, that's a sign that TP isn't great
TEST #1: verify blacklist deletions
VERIFY ENABLED
- Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
- ensure that the resource is blocked:
- You should see:
- A black cat bearing its claws [cat]
- A tracking protection shield icon in the URL bar
- A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."
DELETE FROM LIST
- Remove the extremetracking.com json blob from the blacklist on https://github.com/mozilla-services/shavar-test-lists
- This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
VERIFY DISABLED
Start Debuggging
- Restart Firefox from terminal to force list updates on client
- The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
- Debug output will start in terminal 10 - 30 secs after you start browser
Verify:
- update URL has registered correctly (line #8)
- Registering whitelist- mozpub-trackwhite-digest256 (line #10)
- Checking for updates on shavar server: "checkForUpdates with https://tracking.services.mozilla.com" (line #61)
- Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223" (line #108)
- Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112)
- NOTE: if line #112 failure, rest of test will likely not work!
- If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [cat]
TEST #2: verify list additions
ADD TO LIST
- add extremetracking.com back to the list
- update the list
VERIFY RE-ENABLED
- check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/
TEST #3: verify whitelist updates
NOTE:
- this is very much like TEST #1, but just verifying something different
- might be able to merge TEST #1, #3
STEPS
- remove googlesyndication.com from the "Google" organisation in the whitelist (github/shavar-test-lists)
- wait for the list to be updated in S3
- update the list on the client (if you restart Firefox, that's one way to force it)
- restart Firefox
- wait for the whitelist to be updated in Firefox:
- i.e.: https://pastebin.mozilla.org/8844272 line #112 - success for both of these lists for shavar server
- visit youtube.com
- you should see the shield in the url bar
- in the devtool console tab, you should see that googlesyndication.com was blocked because of TP