Services/TrackingProtection/Shavar Server - Testing: Difference between revisions
Jump to navigation
Jump to search
Rpappalardo (talk | contribs) m (formatting) |
Rpappalardo (talk | contribs) m (Update to test pages) |
||
| Line 43: | Line 43: | ||
** page checks if list delivery/generation is broken | ** page checks if list delivery/generation is broken | ||
** Uses: http://extremetracking.com/ (essentially within an iframe) | ** Uses: http://extremetracking.com/ (essentially within an iframe) | ||
** TP OFF: displays black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat] | |||
** TP ON: A fox with a stop sign: [https://wiki.mozilla.org/File:Tracking_protection_on.png Fox] | |||
[[#top]] | [[#top]] | ||
| Line 74: | Line 76: | ||
STAGE & PRODUCTION: | STAGE & PRODUCTION: | ||
* mozpub-track-digest256 | |||
* mozpub-trackwhite-digest256 | |||
* mozstd-track-digest256 | * mozstd-track-digest256 | ||
* mozstd-trackwhite-digest256 | * mozstd-trackwhite-digest256 | ||
Revision as of 20:29, 17 September 2015
Testplan (Overview)
SUMMARY
- The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments
TEST RESULTS
- Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
- https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599
SIGNOFF CRITERIA
Cloud Services QA team will verify that:
- STAGE and PRODUCTION stacks are correctly deployed
- List creation scripts are correct
- Shavar server is working properly (delivering content to the client)
- Sample / Test pages are blocking / unblocking desired content
CACHE STATE
- for server-side tests, clear cache for each test so we can focus on the server's behavior
- note: we may want to add a future client-side test to verify proper cache behavior
FUTURE
Identify and test failure cases
- Server outage
- Partial responses
- Erroneous entry on black/whitelist, etc...
- Accidental blockage of Firefox updates?
REFERENCE
UI for Tracking protection
Test Pages
- http://itisatrap.org/firefox/its-a-tracker.html
- client-only test! (shavar-server could be completely broken and we wouldn't know)
- good test to run first, because if its broken, no reason to proceed
- http://people.mozilla.org/~fmarier/tracking-test/
- page checks if list delivery/generation is broken
- Uses: http://extremetracking.com/ (essentially within an iframe)
- TP OFF: displays black cat bearing its claws: Cat
- TP ON: A fox with a stop sign: Fox
Staging Environment
SHAVAR URLs
Stage:
- shavar.stage.mozaws.net - e2e test URL
- tracking.stage.mozaws.net - diary study URL
- tracking-protection.stage.mozaws.net - CDN for lists
- ckshavar.stage.mozaws.net - loadtest URL
Production:
- shavar.services.mozilla.com
- tracking.services.mozilla.com (DEPRECATED)
- tracking-protection.cdn.mozilla.net - CDN for lists
SHAVAR LISTS
We'll use 5 different lists in testing.
NOTE:
- there is currently no whitelist for diary testing (mozfull)
- staging and production list are the same (except moztestpub is only used in staging tests)
- In staging environment, we use the same shavar code, just using different lists.
- moztespub lists are generated on stage by committing changes to: https://github.com/mozilla-services/shavar-test-lists
STAGE & PRODUCTION:
- mozpub-track-digest256
- mozpub-trackwhite-digest256
- mozstd-track-digest256
- mozstd-trackwhite-digest256
- mozfull-track-digest256 (diary testing list)
STAGE ONLY
- moztestpub-track-digest256
- moztestpub-trackwhite-digest256
Test Types
Server tests can be roughly broken into the following categories:
Stack Check
- Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.
Loadtesting
- Once a stack has been deployed and verified, shavar server can be loadtested to ensure that a release is scalable.
- Instance sizing should be performed to identify ideal size to withstand maximum load while maintaining cost-effectiveness of deployment
- Shavar server should only be loadtested in staging environment
List Delivery and Creation Test
- Follow Setup instructions (create new profile, set prefs!)
- Do cache file sanity checks (TEST #0) with unchanged black/white lists.
Additions/Deletions Test
- Follow Setup instructions (create new profile, set different prefs)
- Do cache file sanity checks (TEST #0)
- Verify that blacklist and whitelist updates get properly delivered to client
NOTE
- ideally we'd be using the real list
- we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test
- TEST #0 gets executed twice
Server-side Test Cases
Stack-Check
PURPOSE
- test that the server is configured properly.
LIST THE LISTS
$ curl -d" " 'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'
Should return all of these lists:
mozfull-track-digest256
mozpub-shumway-digest256
mozpub-track-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
moztestpub-track-digest256
moztestpub-trackwhite-digest256
Next:
mozstd lists:
$ curl -d"mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
Should return this:
n:3600
i:mozstd-track-digest256
u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596
mozfull lists:
$ curl -d"mozfull-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
Should return this:
n:3600
i:mozfull-track-digest256
u:tracking-protection.stage.mozaws.net/mozfull-track-digest256/1442275596
Loadtesting
SUMMARY Shavar server can be loadtested using a tool like beeswithmachineguns or seige.
EXAMPLE
siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < blist.txt'
- Increase -c and -r from 10 -> 100 -> 500 -> 1000
- Monitor the server resources (cpu, memory, network, disk)
- Verify the server can handle the load you are applying
- Once you reach 60% on any one of those resources, halt test
- note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
- Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle
Client-side Test Cases
Setup
SUMMARY
- Setup for client-side (end-2-end) verification of shavar server largely consists of:
- creating a new Firefox profile
- setting custom prefs (one set of custom prefs per list-file-under-test)
- verifying some server data is delivered / activated in the client
- Below are the custom prefs needed for each of the data lists
- Not all data lists (and corresponding prefs) are needed for each test
NOTE:
- use mozstd, til patch lands
SET CUSTOM PREFS (mozpub)
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,mozpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozpub-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "mozpub-track-digest256");
SET CUSTOM PREFS (moztestpub)
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "moztestpub-track-digest256,moztestpub-trackwhite-digest256");
SET CUSTOM PREFS (mozstd)
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "mozstd-track-digest256,mozstd-trackwhite-digest256");
SET CUSTOM PREFS (mozfull)
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozfull-track-digest256,mozstd-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "mozfull-track-digest256,mozstd-trackwhite-digest256");
- !!!!! restart Firefox !!!!!!
Replace production URL with stage URL for the following prefs:
- browser.trackingprotection.updateURL
- browser.trackingprotection.gethashURL
Set pref for debugging:
- browser.safebrowsing.debug = true
- this will give you some will give you some updates on the console
Remove Google malware & phishing protection:
- browser.safebrowsing.enabled = false
- browser.safebrowsing.malware.enabled = false
- privacy.trackingprotection.enabled = true (no reboot necessary)
RESET BROWSER
- !!!!! restart Firefox !!!!!!
List Delivery & Creation Tests (mozpub, moztestpub, mozstd, mozfull)
list delivery
TEST #0
Verify that all data lists have been delivered and have a reasonable filesize. Execute the following tests, once for each data list.
- mozpub
- moztestpub
- mozstd
- mozfull
verify list files cache
SETUP Follow setup instructions for all three list files here: Setup
SUMMARY Once the list updates are done, look into the cache directory and verify: moztestpub, mozstd, mozfull list files should be present and with correct filesize(s).
FILESIZES
blacklists should be < 100K
- moztestpub-track-digest256
- mozpub-track-digest256
- mozstd-track-digest256
- mozfull-track-digest256
whitelists should be < 400K
- moztestpub-trackwhite-digest256
- mozstd-trackwhite-digest256
'CACHE LOCATION
For mozpub list:
~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track-digest256.* on Mac
For moztestpub lists:
~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac
For mozstd lists:
~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Mac
For mozfull list:
~/.cache/mozilla/firefox/XXXX/safebrowsing/mozfull-track-digest256.* on Linux
~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozfull-track-digest256.* on Mac
LIST FILES / SIZE
for mozpub lists:
-rw-r--r-- 1 johnnyquest staff 52620 Sep 15 10:55 mozpub-track-digest256.cache
-rw-r--r-- 1 johnnyquest staff 16 Sep 15 10:55 mozpub-track-digest256.pset
-rw-r--r-- 1 johnnyquest staff 59380 Sep 15 10:55 mozpub-track-digest256.sbstore
for moztestpub lists:
-rw-r--r-- 1 johnnyquest staff 52620 Sep 15 10:55 moztestpub-track-digest256.cache
-rw-r--r-- 1 johnnyquest staff 16 Sep 15 10:55 moztestpub-track-digest256.pset
-rw-r--r-- 1 johnnyquest staff 59380 Sep 15 10:55 moztestpub-track-digest256.sbstore
-rw-r--r-- 1 johnnyquest staff 321676 Sep 15 10:55 moztestpub-trackwhite-digest256.cache
-rw-r--r-- 1 johnnyquest staff 16 Sep 15 10:55 moztestpub-trackwhite-digest256.pset
-rw-r--r-- 1 johnnyquest staff 362068 Sep 15 10:55 moztestpub-trackwhite-digest256.sbstore
for mozstd lists:
-rw-r--r-- 1 johnnyquest staff 52620 Sep 15 10:55 mozstd-track-digest256.cache
-rw-r--r-- 1 johnnyquest staff 16 Sep 15 10:55 mozstd-track-digest256.pset
-rw-r--r-- 1 johnnyquest staff 59380 Sep 15 10:55 mozstd-track-digest256.sbstore
-rw-r--r-- 1 johnnyquest staff 321676 Sep 15 10:55 mozstd-trackwhite-digest256.cache
-rw-r--r-- 1 johnnyquest staff 16 Sep 15 10:55 mozstd-trackwhite-digest256.pset
-rw-r--r-- 1 johnnyquest staff 362068 Sep 15 10:55 mozstd-trackwhite-digest256.sbstore
for mozfull list:
-rw-r--r-- 1 johnnyquest staff 52620 Sep 15 10:55 mozfull-track-digest256.cache
-rw-r--r-- 1 johnnyquest staff 16 Sep 15 10:55 mozfull-track-digest256.pset
-rw-r--r-- 1 johnnyquest staff 59380 Sep 15 10:55 mozfull-track-digest256.sbstore
verify its-a-tracker page
Note: skip this test for mozpub-track-digest256.
- Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
- Ensure that it is all green
- blacklist and whitelist are working
- This page doesn't hit shavar server at all
- If not working, then either about:config issue or client issue - repeat above!
verify CNN page
- Pull up CNN: http://www.cnn.com
- make sure that the Shield shows up
- the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking
verify YouTube page
Note: skip this test for mozpub-track-digest256.
- Pull up Youtube: https://www.youtube.com
- Make sure that googlesyndication.com is NOT blocked by tracking protection on that site
- If the whitelist doesn't work, the Tools > WebDeveloper > BrowserConsole should indicate that googlesyndication.com was blocked by TP
- If you see in console tab that googlesyndication.com is blocked, that's a sign that TP is NOT Working
List Additions / Deletions Tests (moztestpub)
blacklist deletions
TEST #1
verify enabled
- Follow setup instructions for moztestpub file here: Setup
- Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
- ensure that the resource is blocked:
- You should see:
- A fox with a stop sign: Fox
- A tracking protection shield icon in the URL bar
- A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."
delete from blacklist
- Follow setup instructions for moztestpub file here: Setup
- Remove the extremetracking.com json blob from the blacklist on https://github.com/mozilla-services/shavar-test-lists
- This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
verify disabled
- Start Debuggging
- Restart Firefox from terminal to force list updates on client
- The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
- Debug output will start in terminal 10 - 30 secs after you start browser
- update URL has registered correctly (line #8)
- Registering whitelist- mozpub-trackwhite-digest256 (line #10)
- Checking for updates on shavar server: "checkForUpdates with https://tracking.services.mozilla.com" (line #61)
- Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223" (line #108)
- Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112)
- NOTE: if line #112 failure, rest of test will likely not work!
- If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: Cat
blacklist additions
TEST #2
add to blacklist
- Add the extremetracking.com json blob back to the blacklist on https://github.com/mozilla-services/shavar-test-lists
- This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
verify enabled
- check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/
- If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: Fox
whitelist deletions
TEST #3
SUMMARY
- This test is similar to TEST #1, but now we verify the whitelist update (vs. blacklist update)
- Normally googlesyndication is whitelisted so that it doesn't get blocked from YouTube
- By removing it from whitelist, we should be able to verify that it now gets blocked!
verify enabled
Start Debuggging
- Restart Firefox from terminal to force list updates on client
- The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
- Debug output will start in terminal 10 - 30 secs after you start browser
delete from whitelist
- Remove googlesyndication.com from the "Google" organization in the resources section of the whitelist - https://github.com/mozilla-services/shavar-test-lists
- This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
verify disabled
- update URL has registered correctly (line #8)
- Registering whitelist- mozstd-trackwhite-digest256 (line #10) - or mozfull
- Checking for updates on shavar server: "checkForUpdates with https://shavar.services.mozilla.com" (line #61)
- Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozstd-track-digest256;a:1440776223" (line #108)
- Update success for both lists: "update success for mozstd-track-digest256,mozstd-trackwhite-digest256" (line #112)
- NOTE: if line #112 failure, rest of test will likely not work!
- visit youtube.com
- you should see the shield in the url bar
- In the: Tools > Web Developer > Browser Console, you should see that googlesyndication.com was blocked because of TrackingProtection