Testplan (Overview)

SUMMARY

The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments

TEST RESULTS

Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599

SIGNOFF CRITERIA

Cloud Services QA team will verify that:

STAGE and PRODUCTION stacks are correctly deployed
List creation scripts are correct
Shavar server is working properly (delivering content to the client)
Sample / Test pages are blocking / unblocking desired content

CACHE STATE

for server-side tests, clear cache for each test so we can focus on the server's behavior
note: we may want to add a future client-side test to verify proper cache behavior

FUTURE

Identify and test failure cases

Server outage
Partial responses
Erroneous entry on black/whitelist, etc...
Accidental blockage of Firefox updates?

REFERENCE

UI for Tracking protection

[for tracking protection]

Test Pages

http://itisatrap.org/firefox/its-a-tracker.html
- client-only test! (shavar-server could be completely broken and we wouldn't know)
- good test to run first, because if its broken, no reason to proceed
http://people.mozilla.org/~fmarier/tracking-test/
- page checks if list delivery/generation is broken
- Uses: http://extremetracking.com/ (essentially within an iframe)
- TP OFF: displays black cat bearing its claws: Cat
- TP ON: A fox with a stop sign: Fox

#top

Staging Environment

SHAVAR URLs

Stage:

shavar.stage.mozaws.net - e2e test URL
tracking.stage.mozaws.net - diary study URL
tracking-protection.stage.mozaws.net - CDN for lists
ckshavar.stage.mozaws.net - loadtest URL

Production:

shavar.services.mozilla.com
tracking.services.mozilla.com (TO BE DEPRECATED)
tracking-protection.cdn.mozilla.net - CDN for lists

SHAVAR LISTS

We'll use 7 different lists in testing.

NOTE:

there is currently no whitelist for diary testing (mozfull)
staging and production list are the same (except moztestpub is only used in staging tests)
In staging environment, we use the same shavar code, just using different lists.
moztespub lists are generated on stage by committing changes to: https://github.com/mozilla-services/shavar-test-lists

STAGE & PRODUCTION:

mozpub-track-digest256
mozpub-trackwhite-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozfull-track-digest256 (diary testing list)

STAGE ONLY

moztestpub-track-digest256
moztestpub-trackwhite-digest256

#top

Test Types

Server tests can be roughly broken into the following categories:

Stack Check

Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

Loadtesting

Once a stack has been deployed and verified, shavar server can be loadtested to ensure that a release is scalable.
Instance sizing should be performed to identify ideal size to withstand maximum load while maintaining cost-effectiveness of deployment
Shavar server should only be loadtested in staging environment

List Delivery and Creation Test

Follow Setup instructions (create new profile, set prefs!)
Do cache file sanity checks (TEST #0) with unchanged black/white lists.

Additions/Deletions Test

Follow Setup instructions (create new profile, set different prefs)
Do cache file sanity checks (TEST #0)
Verify that blacklist and whitelist updates get properly delivered to client

NOTE

ideally we'd be using the real list
- we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test

#top

Server-side Test Cases

Stack-Check

PURPOSE

test that the server is configured properly.

LIST THE LISTS

STAGE

 $ curl -d" "  'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

mozfull-track-digest256
mozpub-shumway-digest256
mozpub-track-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
moztestpub-track-digest256
moztestpub-trackwhite-digest256

 $ curl -d"mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596

mozfull lists:

 $ curl -d"mozfull-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return this:

n:3600
i:mozfull-track-digest256
u:tracking-protection.stage.mozaws.net/mozfull-track-digest256/1442275596

#top

PRODUCTION

 # shavar URL
 $ curl -d" "  'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d" "  'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

mozfull-track-digest256
mozpub-shumway-digest256
mozpub-track-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256

Curl for the mozpub list:

 # shavar URL
 $ curl -d"mozpub-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozpub-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozpub-track-digest256
 u:tracking-protection.cdn.mozilla.net/mozpub-track-digest256/1442264823

Curl for the mozstd list:

 # shavar URL 
 $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596

Curl for the mozfull lists:

 # shavar URL
 $ curl -d"mozfull-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozfull-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozfull-track-digest256
 u:tracking-protection.services.mozilla.com/mozfull-track-digest256/1442275596

#top

Loadtesting

SUMMARY Shavar server can be loadtested using a tool like beeswithmachineguns or seige.

EXAMPLE

siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < blist.txt'

Increase -c and -r from 10 -> 100 -> 500 -> 1000
Monitor the server resources (cpu, memory, network, disk)
Verify the server can handle the load you are applying
Once you reach 60% on any one of those resources, halt test
- note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
- Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle

#top

Client-side Test Cases

Setup

SUMMARY

Setup for client-side (end-2-end) verification of shavar server largely consists of:
- creating a new Firefox profile
- setting custom prefs (one set of custom prefs per list-file-under-test)
- verifying some server data is delivered / activated in the client
Below are the custom prefs needed for each of the data lists
Not all data lists (and corresponding prefs) are needed for each test

NOTE:

use mozstd, til patch lands

SET CUSTOM PREFS

NOTE

Prefs are listed below for stage testing.
To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com)
all list files are tested in both stage and production (except moztestpub - stage only)

Prefs for: mozpub list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,mozpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple");
user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozpub-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "mozpub-track-digest256");

Prefs for: moztestpub list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "moztestpub-track-digest256,moztestpub-trackwhite-digest256");

Prefs for: mozstd list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "mozstd-track-digest256,mozstd-trackwhite-digest256");

Prefs for: mozfull list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozfull-track-digest256,mozstd-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "mozfull-track-digest256,mozstd-trackwhite-digest256");

!!!!! restart Firefox !!!!!!

Replace production URL with stage URL for the following prefs:

browser.trackingprotection.updateURL
browser.trackingprotection.gethashURL

Set pref for debugging:

browser.safebrowsing.debug = true
- this will give you some will give you some updates on the console

Remove Google malware & phishing protection:

browser.safebrowsing.enabled = false
browser.safebrowsing.malware.enabled = false
privacy.trackingprotection.enabled = true (no reboot necessary)

RESET BROWSER

!!!!! restart Firefox !!!!!!

#top

List Delivery & Creation Tests (mozpub, moztestpub, mozstd, mozfull)

list delivery

Verify that all data lists have been delivered and have a reasonable filesize. Execute the following tests, once for each data list.

mozpub
moztestpub
mozstd
mozfull

verify list files cache

SETUP Follow setup instructions for all three list files here: Setup

SUMMARY Once the list updates are done, look into the cache directory and verify: moztestpub, mozstd, mozfull list files should be present and with correct filesize(s).

FILESIZES

blacklists should be < 100K

moztestpub-track-digest256
mozpub-track-digest256
mozstd-track-digest256
mozfull-track-digest256

whitelists should be < 400K

moztestpub-trackwhite-digest256
mozstd-trackwhite-digest256

'CACHE LOCATION

For mozpub list:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track-digest256.* on Mac

For moztestpub lists:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac

For mozstd lists:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Mac

For mozfull list:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozfull-track-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozfull-track-digest256.* on Mac

LIST FILES / SIZE

for mozpub lists:

-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozpub-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozpub-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozpub-track-digest256.sbstore

for moztestpub lists:

-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 moztestpub-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 moztestpub-track-digest256.sbstore
-rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 moztestpub-trackwhite-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-trackwhite-digest256.pset
-rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 moztestpub-trackwhite-digest256.sbstore

for mozstd lists:

-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozstd-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozstd-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozstd-track-digest256.sbstore
-rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 mozstd-trackwhite-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozstd-trackwhite-digest256.pset
-rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 mozstd-trackwhite-digest256.sbstore

for mozfull list:

-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozfull-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozfull-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozfull-track-digest256.sbstore

verify its-a-tracker page

Note: skip this test for mozpub-track-digest256.

Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
Ensure that it is all green
- blacklist and whitelist are working
This page doesn't hit shavar server at all
If not working, then either about:config issue or client issue - repeat above!

verify CNN page

Pull up CNN: http://www.cnn.com
make sure that the Shield shows up
the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

verify YouTube page

Note: skip this test for mozpub-track-digest256.

Pull up Youtube: https://www.youtube.com
Make sure that googlesyndication.com is NOT blocked by tracking protection on that site
If the whitelist doesn't work, the Tools > WebDeveloper > BrowserConsole should indicate that googlesyndication.com was blocked by TP
If you see in console tab that googlesyndication.com is blocked, that's a sign that TP is NOT Working

#top

List Additions / Deletions Tests (moztestpub)

blacklist deletions

TEST #1

verify enabled

Follow setup instructions for moztestpub file here: Setup
Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
ensure that the resource is blocked:
You should see:
- A fox with a stop sign: Fox
- A tracking protection shield icon in the URL bar
- A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."

delete from blacklist

Follow setup instructions for moztestpub file here: Setup
Remove the extremetracking.com json blob from the blacklist on https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify disabled

Start Debuggging
- Restart Firefox from terminal to force list updates on client
- The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
- Debug output will start in terminal 10 - 30 secs after you start browser
update URL has registered correctly (line #8)
- Registering whitelist- mozpub-trackwhite-digest256 (line #10)
- Checking for updates on shavar server: "checkForUpdates with https://tracking.services.mozilla.com" (line #61)
- Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223" (line #108)
- Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112)
- NOTE: if line #112 failure, rest of test will likely not work!
- If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: Cat

#top

blacklist additions

TEST #2

add to blacklist

Add the extremetracking.com json blob back to the blacklist on https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify enabled

check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/
If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: Fox

#top

whitelist deletions

TEST #3

SUMMARY

This test is similar to TEST #1, but now we verify the whitelist update (vs. blacklist update)
Normally googlesyndication is whitelisted so that it doesn't get blocked from YouTube
By removing it from whitelist, we should be able to verify that it now gets blocked!

verify enabled

Start Debuggging

Restart Firefox from terminal to force list updates on client
The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
Debug output will start in terminal 10 - 30 secs after you start browser

delete from whitelist

Remove googlesyndication.com from the "Google" organization in the resources section of the whitelist - https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify disabled

update URL has registered correctly (line #8)
- Registering whitelist- mozstd-trackwhite-digest256 (line #10) - or mozfull
- Checking for updates on shavar server: "checkForUpdates with https://shavar.services.mozilla.com" (line #61)
- Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozstd-track-digest256;a:1440776223" (line #108)
- Update success for both lists: "update success for mozstd-track-digest256,mozstd-trackwhite-digest256" (line #112)
- NOTE: if line #112 failure, rest of test will likely not work!
visit youtube.com
- you should see the shield in the url bar
In the: Tools > Web Developer > Browser Console, you should see that googlesyndication.com was blocked because of TrackingProtection

#top

Services/TrackingProtection/Shavar Server - Testing

Contents

Testplan (Overview)

Staging Environment

Test Types

Server-side Test Cases

Stack-Check

STAGE

PRODUCTION

Loadtesting

Client-side Test Cases

Setup

SUMMARY

SET CUSTOM PREFS

List Delivery & Creation Tests (mozpub, moztestpub, mozstd, mozfull)

list delivery

verify list files cache

verify its-a-tracker page

verify CNN page

verify YouTube page

List Additions / Deletions Tests (moztestpub)

blacklist deletions

verify enabled

delete from blacklist

verify disabled

blacklist additions

add to blacklist

verify enabled

whitelist deletions

verify enabled

delete from whitelist

verify disabled

Navigation menu

Services/TrackingProtection/Shavar Server - Testing

Testplan (Overview)

Staging Environment

Test Types

Server-side Test Cases

Stack-Check

STAGE

PRODUCTION

Loadtesting

Client-side Test Cases

Setup

SUMMARY

SET CUSTOM PREFS

List Delivery & Creation Tests (mozpub, moztestpub, mozstd, mozfull)

list delivery

verify list files cache

verify its-a-tracker page

verify CNN page

verify YouTube page

List Additions / Deletions Tests (moztestpub)

blacklist deletions

verify enabled

delete from blacklist

verify disabled

blacklist additions

add to blacklist

verify enabled

whitelist deletions

verify enabled

delete from whitelist

verify disabled

Navigation menu

Search