Services/TrackingProtection/Shavar Server - Testing
Jump to navigation
Jump to search
Testplan (Overview)
SUMMARY
- The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments
TEST RESULTS
- Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
- https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599
SIGNOFF CRITERIA
Cloud Services QA team will verify that:
- STAGE and PRODUCTION stacks are correctly deployed
- List creation scripts are correct
- Shavar server is working properly (delivering content to the client)
- Sample / Test pages are blocking / unblocking desired content
CACHE STATE
- for server-side tests, clear cache for each test so we can focus on the server's behavior
- note: we may want to add a future client-side test to verify proper cache behavior
FUTURE
Identify and test failure cases
- Server outage
- Partial responses
- Erroneous entry on black/whitelist, etc...
- Accidental blockage of Firefox updates?
REFERENCE
UI for Tracking protection
Test Pages
- http://itisatrap.org/firefox/its-a-tracker.html
- client-only test! (shavar-server could be completely broken and we wouldn't know)
- good test to run first, because if its broken, no reason to proceed
- http://people.mozilla.org/~fmarier/tracking-test/
- page checks if list delivery/generation is broken
- Uses: http://extremetracking.com/ (essentially within an iframe)
Staging Environment
SHAVAR URLs
Stage:
- tracking.stage.mozaws.net - e2e test URL
- tracking-protection.stage.mozaws.net - CDN for lists
- ckshavar.stage.mozaws.net - loadtest URL
Production:
- tracking.services.mozilla.com
- tracking.prod.mozaws.net
SHAVAR LISTS
We'll use 6 different lists in testing
- shumway list,
- abtest list,
2 production lists
- mozpub-track-digest256
- mozpub-trackwhite-digest256
2 stage testing lists
- moztestpub-track-digest256 **
- moztestpub-trackwhite-digest256 **
see: https://github.com/mozilla-services/shavar-test-lists
NOTE:
- In staging environment, we use the same shavar code, just using different lists.
Test Types
Server tests can be roughly broken into the following categories:
Stack Check
- Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.
List Delivery and Creation Test
Additions/Deletions Test
- follow Setup #1 instructions (create new prof, set different prefs)
- do cache file sanity checks (TEST #0)
- TEST #1
- TEST #2
Loadtesting (notes to follow)
NOTE
- ideally we'd be using the real list
- we can do that for almost all the e2e test and validation test, but we can't do that for the "update" test
- TEST #0 gets executed twice
Test Cases
Stack-Check
PURPOSE
- test that the server is configured properly.
LIST THE LISTS
$ curl -d" " 'https://tracking.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'
Should return all of these lists:
mozpub-shumway-digest256
mozpub-track-digest256
mozpub-trackwhite-digest256
mozpub2-track-digest256
moztestpub-track-digest256
moztestpub-trackwhite-digest256
Next:
$ curl -d"mozpub-track-digest256;" 'https://tracking.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
Should return this:
n:3600
i:mozpub-track-digest256
u:tracking-protection.stage.mozaws.net/mozpub-track-digest256/1441146041
where n = 3600.
List Delivery & Creation Tests
Setup #1
SUMMARY
- Ensure the client is working
- Create a fresh new Firefox profile with custom prefs
SET CUSTOM PREFS
- like so: https://pastebin.mozilla.org/8845647
Replace production URL with stage URL for the following prefs:
- browser.trackingprotection.updateURL
- browser.trackingprotection.gethashURL
Set pref for debugging:
- browser.safebrowsing.debug = true
- this will give you some will give you some updates on the console
Remove Google malware & phishing protection:
- browser.safebrowsing.enabled = false
- browser.safebrowsing.malware.enabled = false
- privacy.trackingprotection.enabled = true (no reboot necessary)
RESET BROWSER
- !!!!! restart Firefox !!!!!!
TEST #0: verify downloaded (cached) mozpub list files
We're looking at:
- all 6 mozpub files should be present
- verify filesize:
- -track-digest256 files < 100K
- -trackwhite-digest256 is < 400k
Once the list updates are done, look into the cache directory:
~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac
After Setup #1, you should see something like this:
-rw-r--r-- 1 johnnyquest johnnyquest 68K Sep 2 12:57 mozpub-track-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 mozpub-track-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 76K Sep 2 12:57 mozpub-track-digest256.sbstore
-rw-r--r-- 1 johnnyquest johnnyquest 315K Sep 2 12:57 mozpub-trackwhite-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 mozpub-trackwhite-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 354K Sep 2 12:57 mozpub-trackwhite-digest256.sbstore
List Additions / Deletions Tests
Setup #2
SET CUSTOM PREFS set prefs: https://pastebin.mozilla.org/8846245
We start with the same custom prefs we set in Setup #1
Additional prefs to append / modify
- urlclassifier.trackingTable = "test-track-simple,moztestpub-track-digest256"
- urlclassifier.trackingWhitelistTable = "test-trackwhite-simple,moztestpub-trackwhite-digest256"
- urlclassifier.disallow_completions += ",moztestpub-track-digest256,moztestpub-trackwhite-digest256"
- add these to existing entries! separated by commas
- for example, in Nightly: test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256
- !!!!! restart Firefox !!!!!!
TEST #0: verify downloaded (cached) moztestpub list files
This is more of a 'sanity' check
VERIFY CACHED moztestpub FILES / FILE SIZES
VERIFY CACHED moztestpub FILES
We're looking at:
- all 6 mozpub files should be present
- verify filesize:
- -track-digest256 files < 100K
- -trackwhite-digest256 is < 400k
Once the list updates are done, look into the cache directory:
~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac
After Setup #2, you should see something like this:
-rw-r--r-- 1 johnnyquest johnnyquest 68K Sep 2 12:57 moztestpub-track-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 moztestpub-track-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 76K Sep 2 12:57 moztestpub-track-digest256.sbstore
-rw-r--r-- 1 johnnyquest johnnyquest 315K Sep 2 12:57 moztestpub-trackwhite-digest256.cache
-rw-r--r-- 1 johnnyquest johnnyquest 16 Sep 2 12:57 moztestpub-trackwhite-digest256.pset
-rw-r--r-- 1 johnnyquest johnnyquest 354K Sep 2 12:57 moztestpub-trackwhite-digest256.sbstore
VERIFY ENABLED
its-a-tracker page
- Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
- Ensure that it is all green (blacklist and whitelist are working)
- This page doesn't hit shavar server at all
- If not working, then either about:config issue or client issue - repeat above!
CNN page
- Pull up CNN: http://www.cnn.com
- make sure that the Shield shows up
- the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking
YouTube page
- Pull up Youtube: https://www.youtube.com
- make sure that googlesyndication.com is NOT blocked by tracking protection on that site
- if the whitelist doesn't work, the devtools should indicate that googlesyndicaiton.com was blocked by TP
- if you see in console tab that googlesyndication.com is blocked, that's a sign that TP isn't great
TEST #1: verify blacklist deletions
VERIFY ENABLED
- Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
- ensure that the resource is blocked:
- should be a happy cat with a stop sign
- it shows the Shield
- MESSAGE READS: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled." (in the devtool cons)
only in a 3rd party context (load an image, script from that page)
DELETE FROM LIST
- remove extremetracking.com from the blacklist on https://github.com/mozilla-services/shavar-test-lists
- trigger the list creation script (rpapa check on Jenkins or ask ckolos / rtilder)
VERIFY DISABLED
- update the list on the client
- if you restart Firefox, that's one way to force it)
- pref we set in STEP #0: browser.safebrowsing.debug will give us a bu
- for example, https://pastebin.mozilla.org/8844272
- * this output will start 10 - 30 secs after you start browser
- important part - line #8 update URL has registered correctly
- line #10 - whitelist
- line #61 - checking for updates on shavar server
- line #108 - this is where the request starts (to shavar server)
- line #112 - success for both of these lists for shavar server
- if line #112 failure, rest of test will likely not work
- now check if it is loaded (should be a sad cat)
TEST #2: verify list additions
ADD TO LIST
- add extremetracking.com back to the list
- update the list
VERIFY RE-ENABLED
- check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/
TEST #3: verify whitelist updates
NOTE:
- this is very much like TEST #1, but just verifying something different
- might be able to merge TEST #1, #3
STEPS
- remove googlesyndication.com from the "Google" organisation in the whitelist (github/shavar-test-lists)
- wait for the list to be updated in S3
- update the list on the client (if you restart Firefox, that's one way to force it)
- restart Firefox
- wait for the whitelist to be updated in Firefox:
- i.e.: https://pastebin.mozilla.org/8844272 line #112 - success for both of these lists for shavar server
- visit youtube.com
- you should see the shield in the url bar
- in the devtool console tab, you should see that googlesyndication.com was blocked because of TP