Services/TrackingProtection/Shavar Server - Testing

This page is deprecated!!!!!! Please see: TestRail testcase repository

(for TestRail permissions, file an issue here: https://github.com/mozilla-services/shavar-e2e-tests )

Testplan (Overview)

SUMMARY

• The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments

TEST RESULTS

• Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
• http://mzl.la/1m7oHJH

SIGNOFF CRITERIA

Cloud Services QA team will verify that:

• STAGE and PRODUCTION stacks are correctly deployed
• List creation scripts are correct
• Shavar server is working properly (delivering content to the client)
• Sample / Test pages are blocking / unblocking desired content

CACHE STATE

• for server-side tests, clear cache for each test so we can focus on the server's behavior
• note: we may want to add a future client-side test to verify proper cache behavior

FUTURE

Identify and test failure cases

• Server outage
• Partial responses
• Erroneous entry on black/whitelist, etc...
• Accidental blockage of Firefox updates?

REFERENCE

Shavar Lists

• Services/Shavar/Lists

UI for Tracking protection

• [for tracking protection]

Test Pages

• http://itisatrap.org/firefox/its-a-tracker.html
  • client-only test! (shavar-server could be completely broken and we wouldn't know)
  • good test to run first, because if its broken, no reason to proceed
• https://mozilla.github.io/tracking-test/
• https://mozilla.github.io/tracking-test/ads.html
• https://mozilla.github.io/tracking-test/analytics.html
• https://mozilla.github.io/tracking-test/social.html
  • page checks if list delivery/generation is broken
  • Uses: http://extremetracking.com/ (essentially within an iframe)
  • TP OFF: displays black cat bearing its claws: Cat
  • TP ON: A fox with a stop sign: Fox

#top

Staging Environment

SHAVAR URLs

STAGE:

• shavar.stage.mozaws.net - e2e test / loadtest URL
• tracking.stage.mozaws.net - diary study URL
• tracking-protection.stage.mozaws.net - CDN for lists

PRODUCTION:

• shavar.services.mozilla.com - e2e test URL (no loadtesting on PRODUCTION)
• tracking-protection.cdn.mozilla.net - CDN for lists

SHAVAR LISTS

There are a number of lists served by the Shavar server that require testing. see: Services/Shavar/Lists

#top

Test Types

Server tests can be roughly broken into the following categories:

Stack Check

• Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

Loadtesting

• Once a stack has been deployed and verified, shavar server can be loadtested to ensure that a release is scalable.
• Instance sizing should be performed to identify ideal size to withstand maximum load while maintaining cost-effectiveness of deployment
• Shavar server should only be loadtested in staging environment

List Delivery and Creation Test

• Follow Setup instructions (create new profile, set prefs!)
• Do cache file sanity checks (TEST #0) with unchanged black/white lists.

Additions/Deletions Test

• Follow Setup instructions (create new profile, set different prefs)
• Do cache file sanity checks (TEST #0)
• Verify that blacklist and whitelist updates get properly delivered to client

NOTE

• ideally we'd be using the real list
  • we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test

#top

Server-side Test Cases

Stack-Check

PURPOSE

• test that the server is configured properly.

STAGE

1) Verify the download of all lists:

 $ curl -d" "  'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

 ads-track-digest256
 allow-flashallow-digest256
 analytics-track-digest256
 base-track-digest256
 baseeff-track-digest256
 basew3c-track-digest256
 block-flash-digest256
 block-flashsubdoc-digest256
 content-track-digest256
 contenteff-track-digest256
 contentw3c-track-digest256
 except-flash-digest256
 except-flashallow-digest256
 except-flashinfobar-digest256
 except-flashsubdoc-digest256
 except-infobars-digest256
 mozfull-track-digest256
 mozfullstaging-track-digest256
 mozplugin-block-digest256
 mozpub-track-digest256
 mozstd-track-digest256
 mozstd-trackwhite-digest256
 mozstdstaging-track-digest256
 mozstdstaging-trackwhite-digest256
 moztestpub-track-digest256
 moztestpub-trackwhite-digest256
 social-track-digest256

2) Curl each of the list groups.

Example for MOZSTD:

   $ curl -d "mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return a response similar to this:

 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596

#top

PRE-PRODUCTION

Stack checks are run on pre-prod to verify that a new app stack returns the same results as the old stack. It verifies the server response. This environment is only up when the tests need to be run, which is only when there is a new stack.

$ curl -k -d" "  'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists: [* update with next verification]

base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256

#top

PRODUCTION

 # shavar URL
 $ curl -d" "  'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d" "  'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

ads-track-digest256
allow-flashallow-digest256
analytics-track-digest256
base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
block-flash-digest256
block-flashsubdoc-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
except-flash-digest256
except-flashallow-digest256
except-flashinfobar-digest256
except-flashsubdoc-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozpub-track-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256
social-track-digest256

Curl for the mozstd list:

 # shavar URL 
 $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596

#top

Loadtesting

SUMMARY

Going forward we will standardize our loadtesting of Shavar server using a fork of the beeswithmachineguns repository which is optimized for our staging environment.

NOTE:

We will also being investigating using Docker containers to test Shavar server on stage. https://hub.docker.com/r/ckolos/bees/tags/

#top

INSTALLATION

• Clone the test tool repo: https://github.com/ckolos/beeswithmachineguns/beeswithmachineguns
• Follow instructions on README for installation

#top

TEST STRATEGY

In general, we break loadtesting strategy into 2 blocks:

1. Standalone instance test
   1. Find max RPS (requests / second) that 1 EC2 instance can support at desired CPU capacity (and without throwing errors)
   2. Result might be recorded as: "standalone instance capacity: X RPS @ Y% CPU"
2. ELB test
   1. Setup auto-scaling parameters in AWS
   2. Warm up ELB (see notes below)
   3. Kickoff loadtest and allow it to run until scaled to desired number of instances
   4. Result might be recorded as: "ELB capacity: X RPS across Y number of instances" (CPU percentage is implicit in auto-scaling parameters)
   5. Also make a note of number of concurrent connections needed, 'bee'-size used etc.
   6. For our purposes, we will use the largest (HVM instance) bee-size possible to ensure that network is optimized from the bee end (and won't risk bottle-necking there)

NOTES:

• EC2 instances must be "warmed-up" prior to a test run
• This can be accomplished by throwing a nominal load against test stack for at least 5 minutes just prior to initiating a loadtest run.

#top

EXAMPLE TEST RUN

The following test will:

• instantiate -s [# of bees]
• of -t [AWS instance size]
• using -k <SSH keypair (pem file) name>

 $ cd beeswithmachineguns
 $ ./bees up -s 20 -k loads -t t2.medium 

once bees are running, the following test will create:

• -c [# of concurrent users]
• for a test of duration -w [seconds]
• using siege engine --use-siege

./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST mozpub-track-digest256;a:1'

or with multiple files:

./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST < filelist.txt'

or curl statement to call multiple files:

#!/bin/sh
curl --data-binary @- https://<shavar server>/downloads?pver=2.2 <<EOF
mozstd-track-digest256;a:1
mozstd-trackwhite-digest256;a:1
mozplugin-block-digest256;a:1
EOF

• Increase -c and -r from 10 -> 100 -> 500 -> 1000
• Monitor the server resources (cpu, memory, network, disk)
• Verify the server can handle the load you are applying
• Once you reach 60% on any one of those resources, halt test
  • note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
  • Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle

Example test run:

https://bugzilla.mozilla.org/show_bug.cgi?id=1204668#c6

#top

TESTING WITH SEIGE

In Cloud Services, we we'll only use beeswithmachineguns for consistency and optimization around our test environemnt. However, loadtesting of Shavar server can also be done using: seige. See the example below.

The following test will:

• create -c [# of concurrent users]
• execute -r [# of repetitions (hits)]

siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < filelist.txt'

#top

Client-side Test Cases

Setup

SUMMARY

• Setup for client-side (end-2-end) verification of shavar server largely consists of:
  • creating a new Firefox profile
  • setting custom prefs (one set of custom prefs per list-file-under-test)
  • verifying some server data is delivered / activated in the client
• Below are the custom prefs needed for each of the data lists
• Not all data lists (and corresponding prefs) are needed for each test

SET CUSTOM PREFS

NOTE

• Prefs info is permanently located at (https://github.com/mozilla-services/services-test/blob/dev/shavar/e2e-test/prefs.ini)
• Prefs are listed below for stage testing.
• To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com)
• all list files are tested in both stage and production (except moztestpub - stage only)

Prefs for: base list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Create a Strict list

1. Create a new Nightly profile.
2. Go into about:preferences#privacy and set Tracking Protection to Always.
3. Click on Change Block List and select strict protection.
4. Make these edits in about:config
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");

Create a Cryptomining Fingerprinting list

1. Create a new profile
2. Make these edits in about:config
browser.safebrowsing.provider.mozilla.gethashURL;https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2
browser.safebrowsing.provider.mozilla.updateURL;https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2
browser.safebrowsing.provider.mozilla.lists;base-track-digest256,mozstd-trackwhite-digest256,content-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,block-flash-digest256,except-flash-digest256,allow-flashallow-digest256,except-flashallow-digest256,block-flashsubdoc-digest256,except-flashsubdoc-digest256,except-flashinfobar-digest256,ads-track-digest256,social-track-digest256,analytics-track-digest256,fastblock1-track-digest256,fastblock1-trackwhite-digest256,fastblock2-track-digest256,fastblock2-trackwhite-digest256,fastblock3-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256
urlclassifier.trackingTable;test-track-simple,base-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256
browser.safebrowsing.provider.mozilla.nextupdatetime;0
3. Restart browser
4. Open a Private Browsing window

Prefs for: moztestpub list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");           
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");

Prefs for: mozstd list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");           
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: mozfull list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");            
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: mozstaging list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstdstaging-track-digest256,mozfullstaging-track-digest256");                   
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256");

#top

List Delivery & Creation Tests: moztestpub, mozstd, mozfull, mozstaging, base

NOTE:

• run tests on both STAGE & PROD (w/ the exception of moztestpub which only exists on STAGE)

list delivery

Verify that all data lists have been delivered and have a reasonable filesize. Execute the following tests, once for each data list.

LIST GROUPS

• moztestpub [STAGE ONLY]
• mozstd
• mozfull
• mozstaging
• base
• strict
• fastblock

verify list files cache

SETUP

• Follow setup instructions for all three list files here: Setup

SUMMARY

• Once the list updates are done, look into the cache directory and verify: moztestpub, mozstd, mozfull list files should be present and with correct filesize(s).
• List updates may not happen immediately after browser restart (see note below).

NOTE:

• "The first request for data MUST happen at a random interval between 0 and 5 minutes after the browser starts."
• see: https://bugzilla.mozilla.org/show_bug.cgi?id=1045163#c12

FILESIZES

whitelists should be < 400K

• mozstd-trackwhite-digest256
• mozstdstaging-trackwhite-digest256 [NEW]
• moztestpub-trackwhite-digest256

blacklists should be < 100K

• mozfull-track-digest256
• mozfullstaging-track-digest256
• mozstd-track-digest256
• mozstdstaging-track-digest256
• moztestpub-track-digest256
• base-track-digest256

content list < 30k

• content-track-digest256

DNT lists < 1k

• basew3c-track-digest256
• baseeff-track-digest256
• contentw3c-track-digest256
• contenteff-track-digest256

plugin lists should be > 0

• mozplugin-block-digest256
• mozplugin2-block-digest256

'CACHE LOCATION The cache has a different location for different operating systems.

Example for moztestpub lists:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac 

LIST FILES / SIZE All files in the safebrowsing folder created for the profile should have full permissions.

Example for moztestpub lists:

-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 moztestpub-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 moztestpub-track-digest256.sbstore
-rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 moztestpub-trackwhite-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-trackwhite-digest256.pset
-rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 moztestpub-trackwhite-digest256.sbstore

verify list pages

repeat test 1 x for each of these list groups:

• STAGE: mozstd, mozfull, moztestpub, base
• PROD: mozstd, mozfull, moztestpub, base

STEPS:

• Set up list
• Verify with matching test page:

https://mozilla.github.io/tracking-test/ads.html https://mozilla.github.io/tracking-test/analytics.html https://mozilla.github.io/tracking-test/social.html https://mozilla.github.io/tracking-test/fastblock.html https://mozilla.github.io/tracking-test/disconnect.html https://mozilla.github.io/tracking-test/full.html - for the Strict list

verify CNN page

repeat test 1 x for each of:

• STAGE: moztestpub, mozstd, mozfull, mozstaging, base
• PROD: mozstd, mozfull, mozstaging, base

STEPS:

• Pull up CNN: http://www.cnn.com
• make sure that the Shield shows up
• the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

verify YouTube page

repeat test 1 x for each of:

• STAGE: mozstd, mozfull, mozstaging, base
• PROD: mozstd, mozfull, mozstaging, base

Note: skip this test for moztestpub-track-digest256.

STEPS:

• Pull up Youtube: https://www.youtube.com
• Shouldn't see the shield
• Make sure that doubleclick.net is NOT blocked by tracking protection on that site
• If the whitelist doesn't work, the Tools > WebDeveloper > BrowserConsole should indicate that doubleclick.net was blocked by TP
• If you see in console (network) tab that doubleclick.net is blocked, that's a sign that TP is NOT Working

#top

List Additions / Deletions Tests: moztestpub

NOTE:

• STAGE only

blacklist deletions

verify enabled

• Follow setup instructions for moztestpub file here: Setup
• Pull up this test page: https://mozilla.github.io/tracking-test/
• ensure that the resource is blocked:
• You should see:
  • A fox with a stop sign: Fox
  • A tracking protection shield icon in the URL bar
  • A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."

delete from blacklist

• Follow setup instructions for moztestpub file here: Setup
• Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists
• This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
• Manually trigger an update via the mozilla button on the about:url-classifier page

verify disabled

• Restart Firefox from terminal to force list updates on client
• Start Debugging

Verify debugging log (in terminal where you launched Firefox).

Here are some examples of what you might see:

update URL has registered correctly

• i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Registering whitelist- mozstd-trackwhite-digest256

• listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Checking for updates on shavar server

• listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Request to shavar server begins

• listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request moztestpub-track-digest256;a:1440776223

Update success for both lists

• listmanager: 09:44:07 GMT-0700 (PDT): update success for mozpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700

VERIFY

• check that it's no longer blocked by visiting https://mozilla.github.io/tracking-test/
• If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: Cat

#top

blacklist additions

add to blacklist

• Add the extremetracking.com json blob back to the blacklist on https://github.com/mozilla-services/shavar-test-lists
• This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify enabled

• check that it's blocked again by visiting https://mozilla.github.io/tracking-test/
• If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: Fox

#top

whitelist deletions

SUMMARY

• This test is similar to blacklist deletions, but now we verify the whitelist update (vs. blacklist update)
• Normally googlesyndication.com and doubleclick.net are whitelisted so that they don't get blocked from YouTube
• By removing them from whitelist, we should be able to verify that they now get blocked!

verify enabled

Start Debuggging

• Restart Firefox from terminal to force list updates on client
• The pref set in list delivery (browser.safebrowsing.debug) will begin outputting debug messages 10 - 30 secs after you start browser
• see: https://pastebin.mozilla.org/8844272

delete from whitelist

• Remove doubleclick.net and googlesyndication.com from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists
• This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
• Manually trigger an update via the mozilla button on the about:url-classifier page

verify disabled

Verify debugging log (from stdout where you launched Firefox from terminal):

update URL has registered correctly

• i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Registering whitelist- moztestpub-trackwhite-digest256

• listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Checking for updates on shavar server

• listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Request to shavar server begins

• listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223

Update success for both lists

• listmanager: 09:44:07 GMT-0700 (PDT): update success for moztestpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700

NOTE

• if update success fails, rest of test will likely not work!

VERIFY

• visit youtube.com
• you should see the shield in the url bar (whitelist isn't blocking because deleted)
• In the: Tools > Web Developer > Browser Console, you should see that doubleclick.net was blocked because of TrackingProtection

#top