Services/TrackingProtection/Shavar Server - Testing

From MozillaWiki
< Services‎ | TrackingProtection
Revision as of 22:43, 14 September 2015 by Rpappalardo (talk | contribs) (Fix broken link)
Jump to navigation Jump to search

Testplan (Overview)

SUMMARY

  • The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments

TEST RESULTS


SIGNOFF CRITERIA

Cloud Services QA team will verify that:

  • STAGE and PRODUCTION stacks are correctly deployed
  • List creation scripts are correct
  • Shavar server is working properly (delivering content to the client)
  • Sample / Test pages are blocking / unblocking desired content

CACHE STATE

  • for server-side tests, clear cache for each test so we can focus on the server's behavior
  • note: we may want to add a future client-side test to verify proper cache behavior

FUTURE

Identify and test failure cases

  • Server outage
  • Partial responses
  • Erroneous entry on black/whitelist, etc...
  • Accidental blockage of Firefox updates?

REFERENCE

UI for Tracking protection

Test Pages

#top


Staging Environment

SHAVAR URLs

Stage:

  • shavar.stage.mozaws.net - e2e test URL
  • tracking.stage.mozaws.net - diary study URL
  • tracking-protection.stage.mozaws.net - CDN for lists
  • ckshavar.stage.mozaws.net - loadtest URL

Production:

  • tracking.services.mozilla.com
  • tracking.prod.mozaws.net


SHAVAR LISTS

We'll use 6 different lists in testing

  • shumway list,
  • abtest list,

2 production lists

  • mozpub-track-digest256
  • mozpub-trackwhite-digest256

2 stage testing lists

  • moztestpub-track-digest256 **
  • moztestpub-trackwhite-digest256 **

see: https://github.com/mozilla-services/shavar-test-lists


NOTE:

  • In staging environment, we use the same shavar code, just using different lists.

#top

Test Types

Server tests can be roughly broken into the following categories:

Stack Check

  • Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

Loadtesting

  • Once a stack has been deployed and verified, shavar server can be loadtested to ensure that a release is scalable.
  • Instance sizing should be performed to identify ideal size to withstand maximum load while maintaining cost-effectiveness of deployment
  • Shavar server should only be loadtested in staging environment

List Delivery and Creation Test

  • follow Setup #1 instructions (create new profile, set prefs!)
  • do cache file sanity checks (TEST #0) with unchanged black/white lists.

Additions/Deletions Test

  • follow Setup #1 instructions (create new prof, set different prefs)
  • do cache file sanity checks (TEST #0)
  • Verify that blacklist and whitelist updates get properly delivered to client


NOTE

  • ideally we'd be using the real list
    • we can do that for almost all the e2e test and validation test, but we can't do that for the "update" test
  • TEST #0 gets executed twice
    • once w/ Setup #1 to verify mozpub cache files
    • once w/ Setup #2 to verify moztestpub cache files

#top

Test Cases

Stack-Check

PURPOSE

  • test that the server is configured properly.

LIST THE LISTS 

 $ curl -d" "  'https://tracking.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists: 

 mozpub-shumway-digest256
 mozpub-track-digest256
 mozpub-trackwhite-digest256
 mozpub2-track-digest256
 moztestpub-track-digest256
 moztestpub-trackwhite-digest256

Next: 

 $ curl -d"mozpub-track-digest256;" 'https://tracking.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return this: 

 n:3600
 i:mozpub-track-digest256
 u:tracking-protection.stage.mozaws.net/mozpub-track-digest256/1441146041
 where n = 3600.

#top

Loadtesting

SUMMARY Shavar server can be loadtested using a tool like beeswithmachineguns or seige.

EXAMPLE 

siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < blist.txt'

  • Increase -c and -r from 10 -> 100 -> 500 -> 1000
  • Monitor the server resources (cpu, memory, network, disk)
  • Verify the server can handle the load you are applying
  • Once you reach 60% on any one of those resources, halt test
    • note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
    • Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle

#top

List Delivery & Creation Tests

Setup #1

SUMMARY

  • Ensure the client is working
  • Create a fresh new Firefox profile with custom prefs

SET CUSTOM PREFS 

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);

Replace production URL with stage URL for the following prefs:

  • browser.trackingprotection.updateURL
  • browser.trackingprotection.gethashURL

Set pref for debugging:

  • browser.safebrowsing.debug = true
    • this will give you some will give you some updates on the console

Remove Google malware & phishing protection:

  • browser.safebrowsing.enabled = false
  • browser.safebrowsing.malware.enabled = false
  • privacy.trackingprotection.enabled = true (no reboot necessary)

RESET BROWSER

  • !!!!! restart Firefox !!!!!!

#top

TEST #0: verify downloaded (cached) mozpub list files

We're looking at:

  • all 6 mozpub files should be present
  • verify filesize:
    • mozpub-track-digest256 files < 100K
    • mozpub-trackwhite-digest256 is < 400k

Once the list updates are done, look into the cache directory: 

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac

After Setup #1, you should see something like this: 

 -rw-r--r-- 1 johnnyquest johnnyquest  68K Sep  2 12:57 mozpub-track-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 mozpub-track-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest  76K Sep  2 12:57 mozpub-track-digest256.sbstore
 -rw-r--r-- 1 johnnyquest johnnyquest 315K Sep  2 12:57 mozpub-trackwhite-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 mozpub-trackwhite-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest 354K Sep  2 12:57 mozpub-trackwhite-digest256.sbstore

#top

List Additions / Deletions Tests

Setup #2

SUMMARY

  • We start with the same custom prefs we set in Setup #1

SET CUSTOM PREFS 

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://tracking.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://tracking.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "moztestpub-track-digest256,moztestpub-trackwhite-digest256");

Additional prefs to append / modify

  • urlclassifier.trackingTable = "test-track-simple,moztestpub-track-digest256"
  • urlclassifier.trackingWhitelistTable = "test-trackwhite-simple,moztestpub-trackwhite-digest256"
  • urlclassifier.disallow_completions += ",moztestpub-track-digest256,moztestpub-trackwhite-digest256"
  • add these to existing entries! separated by commas
    • for example, in Nightly: test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256
  • !!!!! restart Firefox !!!!!!

#top

TEST #0: verify downloaded (cached) moztestpub list files

We're looking at:

  • all 6 moztestpub files should be present
  • verify filesize:
    • moztestpub-track-digest256 files < 100K
    • moztestpub-trackwhite-digest256 is < 400k

Once the list updates are done, look into the cache directory: 

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac

After Setup #2, you should see something like this: 

 -rw-r--r-- 1 johnnyquest johnnyquest  68K Sep  2 12:57 moztestpub-track-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 moztestpub-track-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest  76K Sep  2 12:57 moztestpub-track-digest256.sbstore
 -rw-r--r-- 1 johnnyquest johnnyquest 315K Sep  2 12:57 moztestpub-trackwhite-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 moztestpub-trackwhite-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest 354K Sep  2 12:57 moztestpub-trackwhite-digest256.sbstore

VERIFY ENABLED

its-a-tracker page

  • Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
  • Ensure that it is all green (blacklist and whitelist are working)
  • This page doesn't hit shavar server at all
  • If not working, then either about:config issue or client issue - repeat above!

CNN page

  • Pull up CNN: http://www.cnn.com
  • make sure that the Shield shows up
  • the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

YouTube page

  • Pull up Youtube: https://www.youtube.com
  • make sure that googlesyndication.com is NOT blocked by tracking protection on that site
  • if the whitelist doesn't work, the devtools should indicate that googlesyndicaiton.com was blocked by TP
  • if you see in console tab that googlesyndication.com is blocked, that's a sign that TP isn't great

#top

TEST #1: verify blacklist deletions

VERIFY ENABLED

DELETE FROM LIST

VERIFY DISABLED

Start Debuggging

  • Restart Firefox from terminal to force list updates on client
  • The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
  • Debug output will start in terminal 10 - 30 secs after you start browser

Verify:

  • update URL has registered correctly (line #8)
    • Registering whitelist- mozpub-trackwhite-digest256 (line #10)
    • Checking for updates on shavar server: "checkForUpdates with https://tracking.services.mozilla.com" (line #61)
    • Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223" (line #108)
    • Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112)
    • NOTE: if line #112 failure, rest of test will likely not work!
    • If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: Cat

#top

TEST #2: verify list additions

ADD TO LIST


VERIFY RE-ENABLED

#top

TEST #3: verify whitelist deletions

SUMMARY

  • This test is similar to TEST #1, but now we verify the whitelist update (vs. blacklist update)
  • Normally googlesyndication is whitelisted so that it doesn't get blocked from YouTube
  • By removing it from whitelist, we should be able to verify that it now gets blocked!

VERIFY ENABLED

Start Debuggging

  • Restart Firefox from terminal to force list updates on client
  • The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
  • Debug output will start in terminal 10 - 30 secs after you start browser

REMOVE FROM WHITELIST

  • Remove googlesyndication.com from the "Google" organization in the resources section of the whitelist - https://github.com/mozilla-services/shavar-test-lists
  • This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

VERIFY DISABLED

  • update URL has registered correctly (line #8)
    • Registering whitelist- mozpub-trackwhite-digest256 (line #10)
    • Checking for updates on shavar server: "checkForUpdates with https://tracking.services.mozilla.com" (line #61)
    • Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223" (line #108)
    • Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112)
    • NOTE: if line #112 failure, rest of test will likely not work!
  • visit youtube.com
    • you should NOT see the shield in the url bar
  • In the: Tools > Web Developer > Browser Console, you should see that googlesyndication.com was blocked because of TrackingProtection


#top

Retrieved from "https://wiki.mozilla.org/index.php?title=Services/TrackingProtection/Shavar_Server_-_Testing&oldid=1095266"