TESTPLAN - OVERVIEW

SUMMARY

below we're going to outline some manual tests for testing all shavar features (including notes for automating).

TEST RESULTS

Test results will be posted in Bugzilla in the Shavar server STAGE deployment tickets
https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599

SIGNOFF CRITERIA

we need to verify:

STAGE and PRODUCTION stacks are correctly deployed
list creation scripts are correct
shavar server is working (delivering content to the client)
Sample / Test pages are blocking / unblocking desired content

FUTURE

looking at failure cases
what if something is broken? (server down, partial responses, erroneous entry on black/whitelist, etc...)
could we accidentally block Firefox updates?

VERIFICATION PAGES

its a tracker.org only client test (shavar could be completely broken and we wouldn't know)
good test to run first, because if its broken, no reason to proceed
trackingtest on fmarier page checks if list delivery/generation is broken
just using extremetracking.com (essentially within an iframe)

REFERENCE

[[1]]

#top

STAGE ENV

SHAVAR URLs

Stage:

tracking.stage.mozaws.net - e2e test URL
tracking-protection.stage.mozaws.net - CDN for lists
ckshavar.stage.mozaws.net - loadtest URL

Production:

tracking.services.mozilla.com
tracking.prod.mozaws.net

SHAVAR LISTS

6 different lists

shumway list,
abtest list,
2 prod lists
mozpub-track-digest256
mozpub-trackwhite-digest256
2 stage testing lists
moztestpub-track-digest256 **
moztestpub-trackwhite-digest256 **

NOTE:

on stage always using the same shavar code, just using different lists

#top

TEST TYPES

e2e tests **

have to use the STAGE test lists (real lists from shavar PROD repo)

validation test

need to make sure that the real lists look reasonable

#top

TEST STRATEGY

prob need to split Testing into 2 parts

Stack Check

Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

List Delivery and Creation Test

follow SETUP 1 instructions (create new profile, set prefs!)
do the sanity checks (TEST #0)

Additions/Deletions Test

follow SETUP 2 instructions (create new prof, set different prefs)
do the sanity checks (TEST #0)
TEST #1
TEST #2

NOTE

ideally we'd be using the real list
- we cna do that for almost all the e2e test and validation test, but we can't do that for the "update" test
TEST #0 gets executed twice
- once w/ SETUP 1 to verify mozpub cache files
- once w/ SETUP 2 to verify moztestpub cache files

#top

TEST-TYPES

STACK-CHECK

PURPOSE

test that the server is configured properly.

LIST THE LISTS

 $ curl -d" "  'https://tracking.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

 mozpub-shumway-digest256
 mozpub-track-digest256
 mozpub-trackwhite-digest256
 mozpub2-track-digest256
 moztestpub-track-digest256
 moztestpub-trackwhite-digest256

 $ curl -d"mozpub-track-digest256;" 'https://tracking.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozpub-track-digest256
 u:tracking-protection.stage.mozaws.net/mozpub-track-digest256/1441146041
 where n = 3600.

#top

LIST-DELIVERY & CREATION TEST

SETUP 1

SUMMARY

to ensure the client is working, we'll create a fresh new Firefox profile w/ custom prefs

SET CUSTOM PREFS

like so: https://pastebin.mozilla.org/8845647

ADD STAGE URL

browser.trackingprotection.updateURL
browser.trackingprotection.gethashURL

DEBUG

browser.safebrowsing.debug = true
- this will give you some will give you some updates on the console

REMOVE MALWARE & PHISHING PROTECTION FROM GOOGLE

browser.safebrowsing.enabled = false
browser.safebrowsing.malware.enabled = false
privacy.trackingprotection.enabled = true (no reboot necessary)

RESET BROWSER

- !!!!! restart Firefox !!!!!!

#top

TEST #0 - sanity check on the downloaded list

VERIFY CACHED mozpub FILES

We're looking at:

the existence of all 6 of these files
the file size should be similar to the above:
if -track-digest256 is > 100k, something is wrong
if -trackwhite-digest256 is > 400k, something is wrong

Once the list updates are done, look into the cache directory:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac

After SETUP 1, you should see something like this:

 -rw-r--r-- 1 johnnyquest johnnyquest  68K Sep  2 12:57 mozpub-track-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 mozpub-track-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest  76K Sep  2 12:57 mozpub-track-digest256.sbstore
 -rw-r--r-- 1 johnnyquest johnnyquest 315K Sep  2 12:57 mozpub-trackwhite-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 mozpub-trackwhite-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest 354K Sep  2 12:57 mozpub-trackwhite-digest256.sbstore

#top

ADDITIONS / DELETIONS TEST

SETUP 2

START W/ SETUP 1

Follow steps for SETUP 1 w/ fresh profile

SET CUSTOM PREFS set prefs: https://pastebin.mozilla.org/8845648

Additional prefs to append / modify (to SETUP 1 prefs.js):
urlclassifier.trackingTable = "test-track-simple,moztestpub-track-digest256"
urlclassifier.trackingWhitelistTable = "test-trackwhite-simple,moztestpub-trackwhite-digest256"
urlclassifier.disallow_completions += ",moztestpub-track-digest256,moztestpub-trackwhite-digest256"
add these to existing entries! separated by commas
- for example, in Nightly: test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256
!!!!! restart Firefox !!!!!!

#top

TEST #0 - sanity check on the downloaded list

VERIFY CACHED moztestpub FILES / FILE SIZES

We're looking at:

the existence of all 6 of these files
the file size should be similar to the above:
if -track-digest256 is > 100k, something is wrong
if -trackwhite-digest256 is > 400k, something is wrong

Once the list updates are done, look into the cache directory:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozpub-track{,white}-digest256.* on Mac

After SETUP 2, you should see something like this:

 -rw-r--r-- 1 johnnyquest johnnyquest  68K Sep  2 12:57 moztestpub-track-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 moztestpub-track-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest  76K Sep  2 12:57 moztestpub-track-digest256.sbstore
 -rw-r--r-- 1 johnnyquest johnnyquest 315K Sep  2 12:57 moztestpub-trackwhite-digest256.cache
 -rw-r--r-- 1 johnnyquest johnnyquest   16 Sep  2 12:57 moztestpub-trackwhite-digest256.pset
 -rw-r--r-- 1 johnnyquest johnnyquest 354K Sep  2 12:57 moztestpub-trackwhite-digest256.sbstore

VERIFY ENABLED

its-a-tracker page

Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
ensure that it is all green (blacklist and whitelist are working)
this page doesn't hit shavar at all - if not working, then either about:config issue or client issue - repeat above or ping francois!

CNN page

Pull up CNN: http://www.cnn.com
make sure that the Shield shows up
the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

YouTube page

Pull up Youtube: https://www.youtube.com
make sure that googlesyndication.com is NOT blocked by tracking protection on that site
if the whitelist doesn't work, the devtools should indicate that googlesyndicaiton.com was blocked by TP
if you see in console tab that googlesyndication.com is blocked, that's a sign that TP isn't great

#top

TEST #1 - testing list deletion

focus on testing the blacklist for now (we'll do whitelist later)

VERIFY ENABLED

Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
ensure that the resource is blocked:
should be a happy cat with a stop sign
it shows the Shield
MESSAGE READS: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled." (in the devtool cons)

   only in a 3rd party context (load an image, script from that page)

DELETE FROM LIST

remove extremetracking.com from the blacklist on https://github.com/mozilla-services/shavar-test-lists
trigger the list creation script (rpapa check on Jenkins or ask ckolos / rtilder)

VERIFY DISABLED

update the list on the client
- if you restart Firefox, that's one way to force it)
pref we set in STEP #0: browser.safebrowsing.debug will give us a bu

for example, https://pastebin.mozilla.org/8844272
* this output will start 10 - 30 secs after you start browser
- important part - line #8 update URL has registered correctly
- line #10 - whitelist
- line #61 - checking for updates on shavar server
- line #108 - this is where the request starts (to shavar server)
- line #112 - success for both of these lists for shavar server
- if line #112 failure, rest of test will likely not work
now check if it is loaded (should be a sad cat)

#top

TEST #2 - testing list addition

ADD TO LIST

add extremetracking.com back to the list
update the list

VERIFY RE-ENABLED

check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/

#top

TEST #3 - ensure that the whitelist updates are happening

NOTE:

this is very much like TEST #1, but just verifying something different
might be able to merge TEST #1, #3

STEPS

remove googlesyndication.com from the "Google" organisation in the whitelist (github/shavar-test-lists)
wait for the list to be updated in S3
update the list on the client (if you restart Firefox, that's one way to force it)
restart Firefox
wait for the whitelist to be updated in Firefox:
i.e.: https://pastebin.mozilla.org/8844272 line #112 - success for both of these lists for shavar server
visit youtube.com
you should see the shield in the url bar
in the devtool console tab, you should see that googlesyndication.com was blocked because of TP

#top

Services/TrackingProtection/Shavar Server - Testing

Contents

TESTPLAN - OVERVIEW

STAGE ENV

TEST TYPES

TEST STRATEGY

TEST-TYPES

STACK-CHECK

LIST-DELIVERY & CREATION TEST

SETUP 1

TEST #0 - sanity check on the downloaded list

ADDITIONS / DELETIONS TEST

SETUP 2

TEST #0 - sanity check on the downloaded list

TEST #1 - testing list deletion

TEST #2 - testing list addition

TEST #3 - ensure that the whitelist updates are happening

Navigation menu

Services/TrackingProtection/Shavar Server - Testing

TESTPLAN - OVERVIEW

STAGE ENV

TEST TYPES

TEST STRATEGY

TEST-TYPES

STACK-CHECK

LIST-DELIVERY & CREATION TEST

SETUP 1

TEST #0 - sanity check on the downloaded list

ADDITIONS / DELETIONS TEST

SETUP 2

TEST #0 - sanity check on the downloaded list

TEST #1 - testing list deletion

TEST #2 - testing list addition

TEST #3 - ensure that the whitelist updates are happening

Navigation menu

Search