Services/TrackingProtection/Shavar Server - Testing

Testplan (Overview)

SUMMARY

• The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments

TEST RESULTS

• Test results will be posted in Bugzilla in the Shavar server STAGE deployment tickets
• https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599

SIGNOFF CRITERIA

Cloud Services QA team will verify that:

• STAGE and PRODUCTION stacks are correctly deployed
• list creation scripts are correct
• Shavar server is working properly (delivering content to the client)
• Sample / Test pages are blocking / unblocking desired content

FUTURE

• looking at failure cases
• what if something is broken? (server down, partial responses, erroneous entry on black/whitelist, etc...)
• could we accidentally block Firefox updates?

REFERENCE

UI for Tracking protection

• [for tracking protection]

Test Pages

• its a tracker.org only client test (shavar could be completely broken and we wouldn't know)
• good test to run first, because if its broken, no reason to proceed
• trackingtest on fmarier page checks if list delivery/generation is broken
• just using extremetracking.com (essentially within an iframe)

#top

Staging Environment

SHAVAR URLs

Stage:

• tracking.stage.mozaws.net - e2e test URL
• tracking-protection.stage.mozaws.net - CDN for lists
• ckshavar.stage.mozaws.net - loadtest URL

Production:

• tracking.services.mozilla.com
• tracking.prod.mozaws.net

SHAVAR LISTS

6 different lists

• shumway list,
• abtest list,
• 2 prod lists
• mozpub-track-digest256
• mozpub-trackwhite-digest256
• 2 stage testing lists
• moztestpub-track-digest256 **
• moztestpub-trackwhite-digest256 **

NOTE:

• on stage always using the same shavar code, just using different lists

#top

Test Types

Server tests can be roughly broken into the following categories:

Stack Check

• Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

List Delivery and Creation Test

• follow SETUP 1 instructions (create new profile, set prefs!)
• do the sanity checks (TEST #0)

Additions/Deletions Test

• follow SETUP 2 instructions (create new prof, set different prefs)
• do the sanity checks (TEST #0)
• TEST #1
• TEST #2

Loadtesting (notes to follow)

NOTE

• ideally we'd be using the real list
  • we can do that for almost all the e2e test and validation test, but we can't do that for the "update" test
• TEST #0 gets executed twice
  • once w/ SETUP 1 to verify mozpub cache files
  • once w/ SETUP 2 to verify moztestpub cache files

#top

Test Cases

Stack-Check

PURPOSE

• test that the server is configured properly.

LIST THE LISTS

 $ curl -d" "  'https://tracking.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

 mozpub-shumway-digest256
 mozpub-track-digest256
 mozpub-trackwhite-digest256
 mozpub2-track-digest256
 moztestpub-track-digest256
 moztestpub-trackwhite-digest256

Next:

 $ curl -d"mozpub-track-digest256;" 'https://tracking.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozpub-track-digest256
 u:tracking-protection.stage.mozaws.net/mozpub-track-digest256/1441146041
 where n = 3600.

#top

List Delivery & Creation Tests

SETUP 1

SUMMARY

• to ensure the client is working, we'll create a fresh new Firefox profile w/ custom prefs

SET CUSTOM PREFS

• like so: https://pastebin.mozilla.org/8845647

ADD STAGE URL

• browser.trackingprotection.updateURL
• browser.trackingprotection.gethashURL

DEBUG

• browser.safebrowsing.debug = true
  • this will give you some will give you some updates on the console

REMOVE MALWARE & PHISHING PROTECTION FROM GOOGLE

• browser.safebrowsing.enabled = false
• browser.safebrowsing.malware.enabled = false
• privacy.trackingprotection.enabled = true (no reboot necessary)

RESET BROWSER

• • !!!!! restart Firefox !!!!!!

#top

TEST #0: sanity check on the downloaded list

VERIFY CACHED mozpub FILES

We're looking at:

• the existence of all 6 of these files
• the file size should be similar to the above:
• if -track-digest256 is > 100k, something is wrong
• if -trackwhite-digest256 is > 400k, something is wrong

Once the list updates are done, look into the cache directory:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac 

After SETUP 1, you should see something like this:

 -rw-r--r-- 1 johnnyquest johnnyquest  68K Sep  2 12:57 mozpub-track-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 mozpub-track-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest  76K Sep  2 12:57 mozpub-track-digest256.sbstore
 -rw-r--r-- 1 johnnyquest johnnyquest 315K Sep  2 12:57 mozpub-trackwhite-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 mozpub-trackwhite-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest 354K Sep  2 12:57 mozpub-trackwhite-digest256.sbstore

#top

List Additions / Deletions Tests

SETUP 2

START W/ SETUP 1

• Follow steps for SETUP 1 w/ fresh profile

SET CUSTOM PREFS set prefs: https://pastebin.mozilla.org/8845648

• Additional prefs to append / modify (to SETUP 1 prefs.js):
• urlclassifier.trackingTable = "test-track-simple,moztestpub-track-digest256"
• urlclassifier.trackingWhitelistTable = "test-trackwhite-simple,moztestpub-trackwhite-digest256"
• urlclassifier.disallow_completions += ",moztestpub-track-digest256,moztestpub-trackwhite-digest256"
• add these to existing entries! separated by commas
  • for example, in Nightly: test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256
• !!!!! restart Firefox !!!!!!

#top

TEST #0: verify downloaded list (filesize)

This is more of a 'sanity' check

VERIFY CACHED moztestpub FILES / FILE SIZES

We're looking at:

• the existence of all 6 of these files
• the file size should be similar to the above:
• if -track-digest256 is > 100k, something is wrong
• if -trackwhite-digest256 is > 400k, something is wrong

Once the list updates are done, look into the cache directory:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac 

After SETUP 2, you should see something like this:

 -rw-r--r-- 1 johnnyquest johnnyquest  68K Sep  2 12:57 moztestpub-track-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 moztestpub-track-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest  76K Sep  2 12:57 moztestpub-track-digest256.sbstore
 -rw-r--r-- 1 johnnyquest johnnyquest 315K Sep  2 12:57 moztestpub-trackwhite-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 moztestpub-trackwhite-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest 354K Sep  2 12:57 moztestpub-trackwhite-digest256.sbstore

VERIFY ENABLED

its-a-tracker page

• Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
• ensure that it is all green (blacklist and whitelist are working)
• this page doesn't hit shavar at all - if not working, then either about:config issue or client issue - repeat above or ping francois!

CNN page

• Pull up CNN: http://www.cnn.com
• make sure that the Shield shows up
• the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

YouTube page

• Pull up Youtube: https://www.youtube.com
• make sure that googlesyndication.com is NOT blocked by tracking protection on that site
• if the whitelist doesn't work, the devtools should indicate that googlesyndicaiton.com was blocked by TP
• if you see in console tab that googlesyndication.com is blocked, that's a sign that TP isn't great

#top

TEST #1: verify blacklist deletions

VERIFY ENABLED

• Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
• ensure that the resource is blocked:
• should be a happy cat with a stop sign
• it shows the Shield
• MESSAGE READS: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled." (in the devtool cons)

   only in a 3rd party context (load an image, script from that page)

DELETE FROM LIST

• remove extremetracking.com from the blacklist on https://github.com/mozilla-services/shavar-test-lists
• trigger the list creation script (rpapa check on Jenkins or ask ckolos / rtilder)

VERIFY DISABLED

• update the list on the client
  • if you restart Firefox, that's one way to force it)
• pref we set in STEP #0: browser.safebrowsing.debug will give us a bu

• for example, https://pastebin.mozilla.org/8844272
• * this output will start 10 - 30 secs after you start browser
  • important part - line #8 update URL has registered correctly
  • line #10 - whitelist
  • line #61 - checking for updates on shavar server
  • line #108 - this is where the request starts (to shavar server)
  • line #112 - success for both of these lists for shavar server
  • if line #112 failure, rest of test will likely not work
• now check if it is loaded (should be a sad cat)

#top

TEST #2: verify list additions

ADD TO LIST

• add extremetracking.com back to the list
• update the list

VERIFY RE-ENABLED

• check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/

#top

TEST #3: verify whitelist updates

NOTE:

• this is very much like TEST #1, but just verifying something different
• might be able to merge TEST #1, #3

STEPS

• remove googlesyndication.com from the "Google" organisation in the whitelist (github/shavar-test-lists)
• wait for the list to be updated in S3
• update the list on the client (if you restart Firefox, that's one way to force it)
• restart Firefox
• wait for the whitelist to be updated in Firefox:
• i.e.: https://pastebin.mozilla.org/8844272 line #112 - success for both of these lists for shavar server
• visit youtube.com
• you should see the shield in the url bar
• in the devtool console tab, you should see that googlesyndication.com was blocked because of TP

#top