Latest revision as of 17:42, 11 January 2019

This page is deprecated!!!!!! Please see: TestRail testcase repository

(for TestRail permissions, file an issue here: https://github.com/mozilla-services/shavar-e2e-tests )

Testplan (Overview)

SUMMARY

The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments

TEST RESULTS

Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
http://mzl.la/1m7oHJH

SIGNOFF CRITERIA

Cloud Services QA team will verify that:

STAGE and PRODUCTION stacks are correctly deployed
List creation scripts are correct
Shavar server is working properly (delivering content to the client)
Sample / Test pages are blocking / unblocking desired content

CACHE STATE

for server-side tests, clear cache for each test so we can focus on the server's behavior
note: we may want to add a future client-side test to verify proper cache behavior

FUTURE

Identify and test failure cases

Server outage
Partial responses
Erroneous entry on black/whitelist, etc...
Accidental blockage of Firefox updates?

REFERENCE

Shavar Lists

Services/Shavar/Lists

UI for Tracking protection

[for tracking protection]

Test Pages

http://itisatrap.org/firefox/its-a-tracker.html
- client-only test! (shavar-server could be completely broken and we wouldn't know)
- good test to run first, because if its broken, no reason to proceed
https://mozilla.github.io/tracking-test/
https://mozilla.github.io/tracking-test/ads.html
https://mozilla.github.io/tracking-test/analytics.html
https://mozilla.github.io/tracking-test/social.html
- page checks if list delivery/generation is broken
- Uses: http://extremetracking.com/ (essentially within an iframe)
- TP OFF: displays black cat bearing its claws: Cat
- TP ON: A fox with a stop sign: Fox

#top

Staging Environment

SHAVAR URLs

STAGE:

shavar.stage.mozaws.net - e2e test / loadtest URL
tracking.stage.mozaws.net - diary study URL
tracking-protection.stage.mozaws.net - CDN for lists

PRODUCTION:

shavar.services.mozilla.com - e2e test URL (no loadtesting on PRODUCTION)
tracking-protection.cdn.mozilla.net - CDN for lists

SHAVAR LISTS

There are a number of lists served by the Shavar server that require testing. see: Services/Shavar/Lists

#top

Test Types

Server tests can be roughly broken into the following categories:

Stack Check

Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

Loadtesting

Once a stack has been deployed and verified, shavar server can be loadtested to ensure that a release is scalable.
Instance sizing should be performed to identify ideal size to withstand maximum load while maintaining cost-effectiveness of deployment
Shavar server should only be loadtested in staging environment

List Delivery and Creation Test

Follow Setup instructions (create new profile, set prefs!)
Do cache file sanity checks (TEST #0) with unchanged black/white lists.

Additions/Deletions Test

Follow Setup instructions (create new profile, set different prefs)
Do cache file sanity checks (TEST #0)
Verify that blacklist and whitelist updates get properly delivered to client

NOTE

ideally we'd be using the real list
- we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test

#top

Server-side Test Cases

Stack-Check

PURPOSE

test that the server is configured properly.

STAGE

1) Verify the download of all lists:

 $ curl -d" "  'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

 ads-track-digest256
 allow-flashallow-digest256
 analytics-track-digest256
 base-track-digest256
 baseeff-track-digest256
 basew3c-track-digest256
 block-flash-digest256
 block-flashsubdoc-digest256
 content-track-digest256
 contenteff-track-digest256
 contentw3c-track-digest256
 except-flash-digest256
 except-flashallow-digest256
 except-flashinfobar-digest256
 except-flashsubdoc-digest256
 except-infobars-digest256
 mozfull-track-digest256
 mozfullstaging-track-digest256
 mozplugin-block-digest256
 mozpub-track-digest256
 mozstd-track-digest256
 mozstd-trackwhite-digest256
 mozstdstaging-track-digest256
 mozstdstaging-trackwhite-digest256
 moztestpub-track-digest256
 moztestpub-trackwhite-digest256
 social-track-digest256

2) Curl each of the list groups.

Example for MOZSTD:

   $ curl -d "mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return a response similar to this:

 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596

#top

PRE-PRODUCTION

Stack checks are run on pre-prod to verify that a new app stack returns the same results as the old stack. It verifies the server response. This environment is only up when the tests need to be run, which is only when there is a new stack.

$ curl -k -d" "  'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists: [* update with next verification]

base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256

#top

PRODUCTION

 # shavar URL
 $ curl -d" "  'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d" "  'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:

ads-track-digest256
allow-flashallow-digest256
analytics-track-digest256
base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
block-flash-digest256
block-flashsubdoc-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
except-flash-digest256
except-flashallow-digest256
except-flashinfobar-digest256
except-flashsubdoc-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozpub-track-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256
social-track-digest256

Curl for the mozstd list:

 # shavar URL 
 $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:

 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596

#top

Loadtesting

SUMMARY

Going forward we will standardize our loadtesting of Shavar server using a fork of the beeswithmachineguns repository which is optimized for our staging environment.

NOTE:

We will also being investigating using Docker containers to test Shavar server on stage. https://hub.docker.com/r/ckolos/bees/tags/

#top

INSTALLATION

Clone the test tool repo: https://github.com/ckolos/beeswithmachineguns/beeswithmachineguns
Follow instructions on README for installation

#top

TEST STRATEGY

In general, we break loadtesting strategy into 2 blocks:

Standalone instance test
1. Find max RPS (requests / second) that 1 EC2 instance can support at desired CPU capacity (and without throwing errors)
2. Result might be recorded as: "standalone instance capacity: X RPS @ Y% CPU"
ELB test
1. Setup auto-scaling parameters in AWS
2. Warm up ELB (see notes below)
3. Kickoff loadtest and allow it to run until scaled to desired number of instances
4. Result might be recorded as: "ELB capacity: X RPS across Y number of instances" (CPU percentage is implicit in auto-scaling parameters)
5. Also make a note of number of concurrent connections needed, 'bee'-size used etc.
6. For our purposes, we will use the largest (HVM instance) bee-size possible to ensure that network is optimized from the bee end (and won't risk bottle-necking there)

NOTES:

EC2 instances must be "warmed-up" prior to a test run
This can be accomplished by throwing a nominal load against test stack for at least 5 minutes just prior to initiating a loadtest run.

#top

EXAMPLE TEST RUN

The following test will:

instantiate -s [# of bees]
of -t [AWS instance size]
using -k <SSH keypair (pem file) name>

 $ cd beeswithmachineguns
 $ ./bees up -s 20 -k loads -t t2.medium

once bees are running, the following test will create:

-c [# of concurrent users]
for a test of duration -w [seconds]
using siege engine --use-siege

./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST mozpub-track-digest256;a:1'

or with multiple files:

./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST < filelist.txt'

or curl statement to call multiple files:

#!/bin/sh
curl --data-binary @- https://<shavar server>/downloads?pver=2.2 <<EOF
mozstd-track-digest256;a:1
mozstd-trackwhite-digest256;a:1
mozplugin-block-digest256;a:1
EOF

Increase -c and -r from 10 -> 100 -> 500 -> 1000
Monitor the server resources (cpu, memory, network, disk)
Verify the server can handle the load you are applying
Once you reach 60% on any one of those resources, halt test
- note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
- Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle

Example test run:

https://bugzilla.mozilla.org/show_bug.cgi?id=1204668#c6

#top

TESTING WITH SEIGE

In Cloud Services, we we'll only use beeswithmachineguns for consistency and optimization around our test environemnt. However, loadtesting of Shavar server can also be done using: seige. See the example below.

The following test will:

create -c [# of concurrent users]
execute -r [# of repetitions (hits)]

siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < filelist.txt'

#top

Client-side Test Cases

Setup

SUMMARY

Setup for client-side (end-2-end) verification of shavar server largely consists of:
- creating a new Firefox profile
- setting custom prefs (one set of custom prefs per list-file-under-test)
- verifying some server data is delivered / activated in the client
Below are the custom prefs needed for each of the data lists
Not all data lists (and corresponding prefs) are needed for each test

SET CUSTOM PREFS

NOTE

Prefs info is permanently located at (https://github.com/mozilla-services/services-test/blob/dev/shavar/e2e-test/prefs.ini)
Prefs are listed below for stage testing.
To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com)
all list files are tested in both stage and production (except moztestpub - stage only)

Prefs for: base list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Create a Strict list

1. Create a new Nightly profile.
2. Go into about:preferences#privacy and set Tracking Protection to Always.
3. Click on Change Block List and select strict protection.
4. Make these edits in about:config
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");

Create a Cryptomining Fingerprinting list

1. Create a new profile
2. Make these edits in about:config
browser.safebrowsing.provider.mozilla.gethashURL;https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2
browser.safebrowsing.provider.mozilla.updateURL;https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2
browser.safebrowsing.provider.mozilla.lists;base-track-digest256,mozstd-trackwhite-digest256,content-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,block-flash-digest256,except-flash-digest256,allow-flashallow-digest256,except-flashallow-digest256,block-flashsubdoc-digest256,except-flashsubdoc-digest256,except-flashinfobar-digest256,ads-track-digest256,social-track-digest256,analytics-track-digest256,fastblock1-track-digest256,fastblock1-trackwhite-digest256,fastblock2-track-digest256,fastblock2-trackwhite-digest256,fastblock3-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256
urlclassifier.trackingTable;test-track-simple,base-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256
browser.safebrowsing.provider.mozilla.nextupdatetime;0
3. Restart browser
4. Open a Private Browsing window

Prefs for: moztestpub list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");           
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");

Prefs for: mozstd list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");           
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: mozfull list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");            
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: mozstaging list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstdstaging-track-digest256,mozfullstaging-track-digest256");                   
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256");

#top

List Delivery & Creation Tests: moztestpub, mozstd, mozfull, mozstaging, base

NOTE:

run tests on both STAGE & PROD (w/ the exception of moztestpub which only exists on STAGE)

list delivery

Verify that all data lists have been delivered and have a reasonable filesize. Execute the following tests, once for each data list.

LIST GROUPS

moztestpub [STAGE ONLY]
mozstd
mozfull
mozstaging
base
strict
fastblock

verify list files cache

SETUP

Follow setup instructions for all three list files here: Setup

SUMMARY

Once the list updates are done, look into the cache directory and verify: moztestpub, mozstd, mozfull list files should be present and with correct filesize(s).
List updates may not happen immediately after browser restart (see note below).

NOTE:

"The first request for data MUST happen at a random interval between 0 and 5 minutes after the browser starts."
see: https://bugzilla.mozilla.org/show_bug.cgi?id=1045163#c12

FILESIZES

whitelists should be < 400K

mozstd-trackwhite-digest256
mozstdstaging-trackwhite-digest256 [NEW]
moztestpub-trackwhite-digest256

blacklists should be < 100K

mozfull-track-digest256
mozfullstaging-track-digest256
mozstd-track-digest256
mozstdstaging-track-digest256
moztestpub-track-digest256
base-track-digest256

content list < 30k

content-track-digest256

DNT lists < 1k

basew3c-track-digest256
baseeff-track-digest256
contentw3c-track-digest256
contenteff-track-digest256

plugin lists should be > 0

mozplugin-block-digest256
mozplugin2-block-digest256

'CACHE LOCATION The cache has a different location for different operating systems.

Example for moztestpub lists:

    ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac

LIST FILES / SIZE All files in the safebrowsing folder created for the profile should have full permissions.

Example for moztestpub lists:

-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 moztestpub-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 moztestpub-track-digest256.sbstore
-rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 moztestpub-trackwhite-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-trackwhite-digest256.pset
-rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 moztestpub-trackwhite-digest256.sbstore

verify list pages

repeat test 1 x for each of these list groups:

STAGE: mozstd, mozfull, moztestpub, base
PROD: mozstd, mozfull, moztestpub, base

STEPS:

Set up list
Verify with matching test page:

https://mozilla.github.io/tracking-test/ads.html https://mozilla.github.io/tracking-test/analytics.html https://mozilla.github.io/tracking-test/social.html https://mozilla.github.io/tracking-test/fastblock.html https://mozilla.github.io/tracking-test/disconnect.html https://mozilla.github.io/tracking-test/full.html - for the Strict list

verify CNN page

repeat test 1 x for each of:

STAGE: moztestpub, mozstd, mozfull, mozstaging, base
PROD: mozstd, mozfull, mozstaging, base

STEPS:

Pull up CNN: http://www.cnn.com
make sure that the Shield shows up
the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

verify YouTube page

repeat test 1 x for each of:

STAGE: mozstd, mozfull, mozstaging, base
PROD: mozstd, mozfull, mozstaging, base

Note: skip this test for moztestpub-track-digest256.

STEPS:

Pull up Youtube: https://www.youtube.com
Shouldn't see the shield
Make sure that doubleclick.net is NOT blocked by tracking protection on that site
If the whitelist doesn't work, the Tools > WebDeveloper > BrowserConsole should indicate that doubleclick.net was blocked by TP
If you see in console (network) tab that doubleclick.net is blocked, that's a sign that TP is NOT Working

#top

List Additions / Deletions Tests: moztestpub

NOTE:

STAGE only

blacklist deletions

verify enabled

Follow setup instructions for moztestpub file here: Setup
Pull up this test page: https://mozilla.github.io/tracking-test/
ensure that the resource is blocked:
You should see:
- A fox with a stop sign: Fox
- A tracking protection shield icon in the URL bar
- A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."

delete from blacklist

Follow setup instructions for moztestpub file here: Setup
Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
Manually trigger an update via the mozilla button on the about:url-classifier page

verify disabled

Restart Firefox from terminal to force list updates on client
Start Debugging

Verify debugging log (in terminal where you launched Firefox).

Here are some examples of what you might see:

update URL has registered correctly

i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Registering whitelist- mozstd-trackwhite-digest256

listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Checking for updates on shavar server

listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Request to shavar server begins

listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request moztestpub-track-digest256;a:1440776223

Update success for both lists

listmanager: 09:44:07 GMT-0700 (PDT): update success for mozpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700

VERIFY

check that it's no longer blocked by visiting https://mozilla.github.io/tracking-test/
If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: Cat

#top

blacklist additions

add to blacklist

Add the extremetracking.com json blob back to the blacklist on https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify enabled

check that it's blocked again by visiting https://mozilla.github.io/tracking-test/
If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: Fox

#top

whitelist deletions

SUMMARY

This test is similar to blacklist deletions, but now we verify the whitelist update (vs. blacklist update)
Normally googlesyndication.com and doubleclick.net are whitelisted so that they don't get blocked from YouTube
By removing them from whitelist, we should be able to verify that they now get blocked!

verify enabled

Start Debuggging

Restart Firefox from terminal to force list updates on client
The pref set in list delivery (browser.safebrowsing.debug) will begin outputting debug messages 10 - 30 secs after you start browser
see: https://pastebin.mozilla.org/8844272

delete from whitelist

Remove doubleclick.net and googlesyndication.com from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
Manually trigger an update via the mozilla button on the about:url-classifier page

verify disabled

Verify debugging log (from stdout where you launched Firefox from terminal):

update URL has registered correctly

i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Registering whitelist- moztestpub-trackwhite-digest256

listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Checking for updates on shavar server

listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Request to shavar server begins

listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223

Update success for both lists

listmanager: 09:44:07 GMT-0700 (PDT): update success for moztestpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700

NOTE

if update success fails, rest of test will likely not work!

VERIFY

visit youtube.com
you should see the shield in the url bar (whitelist isn't blocking because deleted)
In the: Tools > Web Developer > Browser Console, you should see that doubleclick.net was blocked because of TrackingProtection

#top

Services/TrackingProtection/Shavar Server - Testing: Difference between revisions

Latest revision as of 17:42, 11 January 2019

Contents

Testplan (Overview)

Staging Environment

Test Types

Server-side Test Cases

Stack-Check

STAGE

PRE-PRODUCTION

PRODUCTION

Loadtesting

Client-side Test Cases

Setup

SUMMARY

SET CUSTOM PREFS

List Delivery & Creation Tests: moztestpub, mozstd, mozfull, mozstaging, base

list delivery

verify list files cache

verify list pages

verify CNN page

verify YouTube page

List Additions / Deletions Tests: moztestpub

blacklist deletions

verify enabled

delete from blacklist

verify disabled

blacklist additions

add to blacklist

verify enabled

whitelist deletions

verify enabled

delete from whitelist

verify disabled

@@ Line 1: / Line 1: @@
+<span style="color:#FF0000">This page is deprecated!!!!!! Please see:  [https://testrail.stage.mozaws.net/index.php?/suites/view/354&group_by=cases:section_id&group_order=asc TestRail testcase repository]</span>
+<span style="color:#FF0000">(for TestRail permissions, file an issue here:  https://github.com/mozilla-services/shavar-e2e-tests )</span>
+[[Category:Test Plans]]
 = Testplan (Overview) =
@@ Line 8: / Line 16: @@
 * Test results will be posted in Bugzilla within the Shavar server STAGE deployment tickets
-* https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=DEPLOY_SHAVAR&list_id=12537599
+* http://mzl.la/1m7oHJH
@@ Line 32: / Line 40: @@
 '''REFERENCE'''
+Shavar Lists
+* [[Services/Shavar/Lists]]
 UI for Tracking protection
@@ Line 40: / Line 51: @@
 ** client-only test! (shavar-server could be completely broken and we wouldn't know)
 ** good test to run first, because if its broken, no reason to proceed
-* http://people.mozilla.org/~fmarier/tracking-test/
+* https://mozilla.github.io/tracking-test/
+* https://mozilla.github.io/tracking-test/ads.html
+* https://mozilla.github.io/tracking-test/analytics.html
+* https://mozilla.github.io/tracking-test/social.html
 ** page checks if list delivery/generation is broken
 ** Uses: http://extremetracking.com/ (essentially within an iframe)
+** TP OFF: displays black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat]
+** TP ON: A fox with a stop sign: [https://wiki.mozilla.org/File:Tracking_protection_on.png Fox]
 [[#top]]
@@ Line 51: / Line 67: @@
 '''SHAVAR URLs'''
-Stage:
+STAGE:
-* shavar.stage.mozaws.net - e2e test URL
+* shavar.stage.mozaws.net - e2e test / loadtest URL
 * tracking.stage.mozaws.net - diary study URL
 * tracking-protection.stage.mozaws.net - CDN for lists
-* ckshavar.stage.mozaws.net - loadtest URL
-Production:
+PRODUCTION:
-* tracking.services.mozilla.com
+* shavar.services.mozilla.com - e2e test URL (no loadtesting on PRODUCTION)
-* tracking.prod.mozaws.net
+* tracking-protection.cdn.mozilla.net - CDN for lists
 '''SHAVAR LISTS'''
-We'll use 4 different lists in testing:
+There are a number of lists served by the Shavar server that require testing.
+see: [[Services/Shavar/Lists]]
-* mozstd-track-digest256
-* mozstd-trackwhite-digest256
-* mozfull-track-digest256 (diary testing list)
-NOTE:
-* there is currently no whitelist for diary testing (mozfull)
-LIST REPO:
-* https://github.com/mozilla-services/shavar-test-lists
-'''NOTE:'''
-* In staging environment, we use the same shavar code, just using different lists.
 [[#top]]
@@ Line 95: / Line 97: @@
 '''List Delivery and Creation Test'''
-* follow  [[#Setup_.231|Setup #1]]  instructions (create new profile, set prefs!)
+* Follow  [[#Setup|Setup]]  instructions (create new profile, set prefs!)
-* do cache file sanity checks ([[#TEST_.230:_verify_downloaded_.28cached.29_mozpub_list_files|TEST #0]]) with unchanged black/white lists.
+* Do cache file sanity checks ([[#TEST_.230:_verify_downloaded_.28cached.29_mozpub_list_files|TEST #0]]) with unchanged black/white lists.
 '''Additions/Deletions Test'''
-* follow  [[#Setup_.232|Setup #1]]  instructions  (create new prof, set different prefs)
+* Follow  [[#Setup|Setup]] instructions  (create new profile, set different prefs)
-* do cache file sanity checks ([[#TEST_.230:_verify_downloaded_.28cached.29_moztestpub_list_files|TEST #0]])
+* Do cache file sanity checks ([[#TEST_.230:_verify_downloaded_.28cached.29_moztestpub_list_files|TEST #0]])
 * Verify that blacklist and whitelist updates get properly delivered to client
@@ Line 107: / Line 109: @@
 * ideally we'd be using the real list
-** we can do that for almost all the e2e test and validation test, but we can't do that for the "update" test
+** we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test
-* TEST #0 gets executed twice
-** once w/  [[#Setup_.231|Setup #1]] to verify mozstd, mozfull cache files
-** once w/  [[#Setup_.232|Setup #2]] to verify mozstd, mozfull cache files
 [[#top]]
-= Test Cases =
+= Server-side Test Cases =
 == Stack-Check ==
@@ Line 121: / Line 119: @@
 * test that the server is configured properly.
-'''LIST THE LISTS'''
+=== STAGE ===
+) Verify the download of all lists:
-<code>
    $ curl -d" "  'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'
-</code>
 Should return all of these lists:
-<code>
+  ads-track-digest256
- mozfull-track-digest256
+  allow-flashallow-digest256
- mozpub-shumway-digest256
+  analytics-track-digest256
- mozpub-track-digest256
+  base-track-digest256
- mozstd-track-digest256
+  baseeff-track-digest256
- mozstd-trackwhite-digest256
+  basew3c-track-digest256
- moztestpub-track-digest256
+  block-flash-digest256
- moztestpub-trackwhite-digest256
+  block-flashsubdoc-digest256
-</code>
+  content-track-digest256
+  contenteff-track-digest256
+  contentw3c-track-digest256
+  except-flash-digest256
+  except-flashallow-digest256
+  except-flashinfobar-digest256
+  except-flashsubdoc-digest256
+  except-infobars-digest256
+  mozfull-track-digest256
+  mozfullstaging-track-digest256
+  mozplugin-block-digest256
+  mozpub-track-digest256
+  mozstd-track-digest256
+  mozstd-trackwhite-digest256
+  mozstdstaging-track-digest256
+  mozstdstaging-trackwhite-digest256
+  moztestpub-track-digest256
+  moztestpub-trackwhite-digest256
+  social-track-digest256
-Next:
+) Curl each of the list groups.
-mozstd lists:
+Example for MOZSTD:
-<code>
+    $ curl -d "mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
-  $ curl -d"mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
-</code>
-Should return this:
+Should return a response similar to this:
-<code>
    n:3600
    i:mozstd-track-digest256
    u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596
-</code>
-mozfull lists:
-<code>
+[[#top]]
-   $ curl -d"mozfull-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'
-</code>
+=== PRE-PRODUCTION ===
+Stack checks are run on pre-prod to verify that a new app stack returns the same results as the old stack. It verifies the server response. This environment is only up when the tests need to be run, which is only when there is a new stack.
+ $ curl -k -d" "  'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2'
+Should return all of these lists:
+[* update with next verification]
+ base-track-digest256
+ baseeff-track-digest256
+ basew3c-track-digest256
+ content-track-digest256
+ contenteff-track-digest256
+ contentw3c-track-digest256
+ mozfull-track-digest256
+ mozfullstaging-track-digest256
+ mozplugin-block-digest256
+ mozplugin2-block-digest256
+ mozstd-track-digest256
+ mozstd-trackwhite-digest256
+ mozstdstaging-track-digest256
+ mozstdstaging-trackwhite-digest256
+[[#top]]
+=== PRODUCTION ===
+  # shavar URL
+   $ curl -d" "  'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2'
+  # tracking URL (CNAME'd to shavar URL - should produce same output)
+  $ curl -d" "  'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2'
+Should return all of these lists:
+ ads-track-digest256
+ allow-flashallow-digest256
+ analytics-track-digest256
+ base-track-digest256
+ baseeff-track-digest256
+ basew3c-track-digest256
+ block-flash-digest256
+ block-flashsubdoc-digest256
+ content-track-digest256
+ contenteff-track-digest256
+ contentw3c-track-digest256
+ except-flash-digest256
+ except-flashallow-digest256
+ except-flashinfobar-digest256
+ except-flashsubdoc-digest256
+ mozfull-track-digest256
+ mozfullstaging-track-digest256
+ mozplugin-block-digest256
+ mozplugin2-block-digest256
+ mozpub-track-digest256
+ mozstd-track-digest256
+ mozstd-trackwhite-digest256
+ mozstdstaging-track-digest256
+ mozstdstaging-trackwhite-digest256
+ social-track-digest256
+Curl for the mozstd list:
+  # shavar URL
+  $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
+  # tracking URL (CNAME'd to shavar URL - should produce same output)
+  $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 Should return this:
+  n:3600
-<code>
+  i:mozstd-track-digest256
-n:3600
+  u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596
-i:mozfull-track-digest256
-u:tracking-protection.stage.mozaws.net/mozfull-track-digest256/1442275596
-</code>
 [[#top]]
@@ Line 174: / Line 243: @@
 '''SUMMARY'''
-Shavar server can be loadtested using a tool like [https://github.com/mostlygeek/beeswithmachineguns/ beeswithmachineguns] or [https://github.com/JoeDog/siege seige].
-'''EXAMPLE'''
+Going forward we will standardize our loadtesting of Shavar server using a fork of the [https://github.com/ckolos/beeswithmachineguns/beeswithmachineguns beeswithmachineguns] repository which is optimized for our staging environment.
-<code>
+NOTE:
- siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < blist.txt'
-</code>
-* Increase -c and -r from 10 -> 100 -> 500 -> 1000
+We will also being investigating using Docker containers to test Shavar server on stage.
-* Monitor the server resources (cpu, memory, network, disk)
+https://hub.docker.com/r/ckolos/bees/tags/
-* Verify the server can handle the load you are applying
-* Once you reach 60% on any one of those resources, halt test
-** note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
-** Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle
 [[#top]]
-== List Delivery & Creation Tests ==
-=== Setup #1 ===
+'''INSTALLATION'''
-'''SUMMARY'''
+* Clone the test tool repo: https://github.com/ckolos/beeswithmachineguns/beeswithmachineguns
+* Follow instructions on README for installation
-* Ensure the client is working
+[[#top]]
-* Create a fresh new Firefox profile with custom prefs
-'''SET CUSTOM PREFS'''
-<source lang="bash">
+'''TEST STRATEGY'''
-user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
-user_pref("browser.safebrowsing.debug", true);
-user_pref("browser.safebrowsing.enabled", false);
-user_pref("browser.safebrowsing.malware.enabled", false);
-user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
-user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
-user_pref("privacy.trackingprotection.enabled", true);
-</source>
-Replace production URL with stage URL for the following prefs:
+In general, we break loadtesting strategy into 2 blocks:
-* browser.trackingprotection.updateURL
+# Standalone instance test
-* browser.trackingprotection.gethashURL
+## Find max RPS (requests / second) that 1 EC2 instance can support at desired CPU capacity (and without throwing errors)
+## Result might be recorded as: "standalone instance capacity: X RPS @ Y% CPU"
+# ELB test
+## Setup auto-scaling parameters in AWS
+## Warm up ELB (see notes below)
+## Kickoff loadtest and allow it to run until scaled to desired number of instances
+## Result might be recorded as: "ELB capacity: X RPS across Y number of instances" (CPU percentage is implicit in auto-scaling parameters)
+## Also make a note of number of concurrent connections needed, 'bee'-size used etc.
+## For our purposes, we will use the largest (HVM instance) bee-size possible to ensure that network is optimized from the bee end (and won't risk bottle-necking there)
-Set pref for debugging:
+NOTES:
-* browser.safebrowsing.debug = true
+* EC2 instances must be "warmed-up" prior to a test run
-** this will give you some will give you some updates on the console
+* This can be accomplished by throwing a nominal load against test stack for at least 5 minutes just prior to initiating a loadtest run.
-Remove Google malware & phishing protection:
+[[#top]]
-* browser.safebrowsing.enabled = false
-* browser.safebrowsing.malware.enabled = false
-* privacy.trackingprotection.enabled = true (no reboot necessary)
-'''RESET BROWSER'''
-* !!!!!  restart Firefox !!!!!!
-[[#top]]
+'''EXAMPLE TEST RUN'''
-=== TEST #0: verify downloaded (cached) mozstd, mozfull list files ===
+The following test will:
+* instantiate -s [# of bees]
+* of -t [AWS instance size]
+* using -k <SSH keypair (pem file) name>
-All mozstd, mozfull files should be present and with correct filesize(s)
+<code>
+  $ cd beeswithmachineguns
+  $ ./bees up -s 20 -k loads -t t2.medium
+</code>
-blacklists should be < 100K
-* mozstd-track-digest256
-* mozfull-track-digest256
-whitelists should be < 400K
+once bees are running, the following test will create:
-* mozstd-trackwhite-digest256
+* -c [# of concurrent users]
-* mozfull-trackwhite-digest256
+* for a test of duration -w [seconds]
+* using siege engine --use-siege
-Once the list updates are done, look into the cache directory:
-For mozstd lists:
 <code>
-     ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Linux
+ ./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST mozpub-track-digest256;a:1'
-     ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Mac
 </code>
-For mozfull lists:
+or with multiple files:
 <code>
-     ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozfull-track{,white}-digest256.* on Linux
+ ./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST < filelist.txt'
-     ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozfull-track-digest256.* on Mac
 </code>
-After  [[#Setup_.232|Setup #2]], you should see something like this:
+or curl statement to call multiple files:
-for mozstd lists:
 <code>
-  -rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozstd-track-digest256.cache
+  #!/bin/sh
-  -rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozstd-track-digest256.pset
+  curl --data-binary @- https://<shavar server>/downloads?pver=2.2 <<EOF
-  -rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozstd-track-digest256.sbstore
+  mozstd-track-digest256;a:1
-  -rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 mozstd-trackwhite-digest256.cache
+  mozstd-trackwhite-digest256;a:1
-  -rw-r--r--   1 rpappalardo  staff      16 Sep 15 10:55 mozstd-trackwhite-digest256.pset
+  mozplugin-block-digest256;a:1
-  -rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 mozstd-trackwhite-digest256.sbstore
+  EOF
 </code>
-for mozfull lists:
-<code>
+* Increase -c and -r from 10 -> 100 -> 500 -> 1000
- -rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozfull-track-digest256.cache
+* Monitor the server resources (cpu, memory, network, disk)
- -rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozfull-track-digest256.pset
+* Verify the server can handle the load you are applying
- -rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozfull-track-digest256.sbstore
+* Once you reach 60% on any one of those resources, halt test
+** note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
+** Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle
+Example test run:
+https://bugzilla.mozilla.org/show_bug.cgi?id=1204668#c6
+[[#top]]
+'''TESTING WITH SEIGE'''
+In Cloud Services, we we'll only use beeswithmachineguns for consistency and optimization around our test environemnt.  However, loadtesting of Shavar server can also be done using: [https://github.com/JoeDog/siege seige].  See the example below.
+The following test will:
+* create -c [# of concurrent users]
+* execute -r [# of repetitions (hits)]
+<code>
+  siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < filelist.txt'
 </code>
 [[#top]]
-== List Additions / Deletions Tests ==
+= Client-side Test Cases =
+== Setup ==
+=== SUMMARY ===
+* Setup for client-side (end-2-end) verification of shavar server largely consists of:
+** creating a new Firefox profile
+** setting custom prefs (one set of custom prefs per list-file-under-test)
+** verifying some server data is delivered / activated in the client
+* Below are the custom prefs needed for each of the data lists
+* Not all data lists (and corresponding prefs) are needed for each test
+=== SET CUSTOM PREFS ===
+'''NOTE'''
+* Prefs info is permanently located at (https://github.com/mozilla-services/services-test/blob/dev/shavar/e2e-test/prefs.ini)
+* Prefs are listed below for stage testing.
+* To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com)
+* all list files are tested in both stage and production (except moztestpub - stage only)
+'''Prefs for: base list file'''
+<source lang="bash">
+user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
+user_pref("browser.safebrowsing.debug", true);
+user_pref("browser.safebrowsing.phishing.enabled", false);
+user_pref("browser.safebrowsing.malware.enabled", false);
+user_pref("privacy.trackingprotection.enabled", true);
+user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
+user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("urlclassifier.trackingTable", "test-track-simple,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
+</source>
-=== Setup #2 ===
+'''Create a Strict list'''
-'''SUMMARY'''
-* We start with the same custom prefs we set in [[#Setup_.231|Setup #1]]
+<source lang="bash">
+. Create a new Nightly profile.
+. Go into about:preferences#privacy and set Tracking Protection to Always.
+. Click on Change Block List and select strict protection.
+. Make these edits in about:config
+user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
+user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+</source>
+'''Create a Cryptomining Fingerprinting list'''
-'''SET CUSTOM PREFS (mozstd)'''
 <source lang="bash">
-user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
+. Create a new profile
+. Make these edits in about:config
+browser.safebrowsing.provider.mozilla.gethashURL;https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2
+browser.safebrowsing.provider.mozilla.updateURL;https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2
+browser.safebrowsing.provider.mozilla.lists;base-track-digest256,mozstd-trackwhite-digest256,content-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,block-flash-digest256,except-flash-digest256,allow-flashallow-digest256,except-flashallow-digest256,block-flashsubdoc-digest256,except-flashsubdoc-digest256,except-flashinfobar-digest256,ads-track-digest256,social-track-digest256,analytics-track-digest256,fastblock1-track-digest256,fastblock1-trackwhite-digest256,fastblock2-track-digest256,fastblock2-trackwhite-digest256,fastblock3-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256
+urlclassifier.trackingTable;test-track-simple,base-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256
+browser.safebrowsing.provider.mozilla.nextupdatetime;0
+. Restart browser
+. Open a Private Browsing window
+</source>
+'''Prefs for: moztestpub list file'''
+<source>
+user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
 user_pref("browser.safebrowsing.debug", true);
-user_pref("browser.safebrowsing.enabled", false);
+user_pref("browser.safebrowsing.phishing.enabled", false);
 user_pref("browser.safebrowsing.malware.enabled", false);
+user_pref("privacy.trackingprotection.enabled", true);
+user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
+user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
 user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
 user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");
+user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");
+</source>
+'''Prefs for: mozstd list file'''
+<source lang="bash">
+user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
+user_pref("browser.safebrowsing.debug", true);
+user_pref("browser.safebrowsing.phishing.enabled", false);
+user_pref("browser.safebrowsing.malware.enabled", false);
 user_pref("privacy.trackingprotection.enabled", true);
-user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");
+user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
+user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");
 user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
-user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256");
-user_pref("browser.safebrowsing.provider.mozilla.lists", "mozstd-track-digest256,mozstd-trackwhite-digest256");
 </source>
-'''SET CUSTOM PREFS (mozfull)'''
+'''Prefs for: mozfull list file'''
 <source lang="bash">
-user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html ");
+user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
 user_pref("browser.safebrowsing.debug", true);
-user_pref("browser.safebrowsing.enabled", false);
+user_pref("browser.safebrowsing.phishing.enabled", false);
 user_pref("browser.safebrowsing.malware.enabled", false);
+user_pref("privacy.trackingprotection.enabled", true);
+user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
+user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
+user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
 user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
 user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");
+user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");
+</source>
+'''Prefs for: mozstaging list file'''
+<source lang="bash">
+user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
+user_pref("browser.safebrowsing.debug", true);
+user_pref("browser.safebrowsing.phishing.enabled", false);
+user_pref("browser.safebrowsing.malware.enabled", false);
 user_pref("privacy.trackingprotection.enabled", true);
-user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");
+user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
-user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozfull-trackwhite-digest256");
+user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
-user_pref("urlclassifier.disallow_completions",   "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,mozfull-track-digest256,mozfull-trackwhite-digest256");
+user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
-user_pref("browser.safebrowsing.provider.mozilla.lists", "mozfull-track-digest256,mozfull-trackwhite-digest256");
+user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
+user_pref("urlclassifier.trackingTable", "test-track-simple,mozstdstaging-track-digest256,mozfullstaging-track-digest256");
+user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256");
 </source>
-* !!!!!  restart Firefox !!!!!!
+[[#top]]
-[[#top]]
+== List Delivery & Creation Tests:  moztestpub, mozstd, mozfull, mozstaging, base ==
+NOTE:
+* run tests on both STAGE & PROD (w/ the exception of moztestpub which only exists on STAGE)
+=== list delivery ===
+Verify that all data lists have been delivered and have a reasonable filesize.
+Execute the following tests, once for each data list.
+LIST GROUPS
+* moztestpub [STAGE ONLY]
+* mozstd
+* mozfull
+* mozstaging
+* base
+* strict
+* fastblock
+==== verify list files cache ====
+'''SETUP'''
+* Follow setup instructions for all three list files here: [[#Setup|Setup]]
+'''SUMMARY'''
+* Once the list updates are done, look into the cache directory and verify:  moztestpub, mozstd, mozfull list files should be present and with correct filesize(s).
+* List updates may not happen immediately after browser restart (see note below).
+NOTE:
+* "The first request for data MUST happen at a random interval between 0 and 5 minutes after the browser starts."
+* see: https://bugzilla.mozilla.org/show_bug.cgi?id=1045163#c12
-=== TEST #0: verify downloaded (cached) mozstd, mozfull list files ===
+'''FILESIZES'''
-All mozstd, mozfull files should be present and with correct filesize(s)
+whitelists should be < 400K
+* mozstd-trackwhite-digest256
+* mozstdstaging-trackwhite-digest256  [NEW]
+* moztestpub-trackwhite-digest256
 blacklists should be < 100K
+* mozfull-track-digest256
+* mozfullstaging-track-digest256
 * mozstd-track-digest256
-* mozfull-track-digest256
+* mozstdstaging-track-digest256
+* moztestpub-track-digest256
+* base-track-digest256
+content list < 30k
+* content-track-digest256
+DNT lists < 1k
+* basew3c-track-digest256
+* baseeff-track-digest256
+* contentw3c-track-digest256
+* contenteff-track-digest256
+plugin lists should be > 0
+* mozplugin-block-digest256
+* mozplugin2-block-digest256
-whitelists should be < 400K
-* mozstd-trackwhite-digest256
-Once the list updates are done, look into the cache directory:
+''''CACHE LOCATION'''
+The cache has a different location for different operating systems.
-For mozstd lists:
+Example for moztestpub lists:
 <code>
-      ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Linux
+      ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux
-      ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track{,white}-digest256.* on Mac
+      ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac
 </code>
-For mozfull lists:
+'''LIST FILES / SIZE'''
+All files in the safebrowsing folder created for the profile should have full permissions.
+Example for moztestpub lists:
 <code>
-      ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozfull-track{,white}-digest256.* on Linux
+ -rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 moztestpub-track-digest256.cache
-      ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozfull-track-digest256.* on Mac
+ -rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-track-digest256.pset
+ -rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 moztestpub-track-digest256.sbstore
+ -rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 moztestpub-trackwhite-digest256.cache
+ -rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-trackwhite-digest256.pset
+ -rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 moztestpub-trackwhite-digest256.sbstore
 </code>
-After  [[#Setup_.232|Setup #2]], you should see something like this:
+==== verify list pages ====
-for mozstd lists:
+repeat test 1 x for each of these list groups:
-<code>
+* STAGE: mozstd, mozfull, moztestpub, base
- -rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozstd-track-digest256.cache
+* PROD: mozstd, mozfull, moztestpub, base
- -rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozstd-track-digest256.pset
- -rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozstd-track-digest256.sbstore
- -rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 mozstd-trackwhite-digest256.cache
- -rw-r--r--   1 rpappalardo  staff      16 Sep 15 10:55 mozstd-trackwhite-digest256.pset
- -rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 mozstd-trackwhite-digest256.sbstore
-</code>
-for mozfull lists:
+STEPS:
-<code>
+* Set up list
- -rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 mozfull-track-digest256.cache
+* Verify with matching test page:
- -rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 mozfull-track-digest256.pset
+https://mozilla.github.io/tracking-test/ads.html
- -rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 mozfull-track-digest256.sbstore
+https://mozilla.github.io/tracking-test/analytics.html
-</code>
+https://mozilla.github.io/tracking-test/social.html
+https://mozilla.github.io/tracking-test/fastblock.html
+https://mozilla.github.io/tracking-test/disconnect.html
+https://mozilla.github.io/tracking-test/full.html  - for the Strict list
-'''VERIFY ENABLED'''
+==== verify CNN page ====
-'''its-a-tracker page'''
+repeat test 1 x for each of:
-* Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
+* STAGE: moztestpub, mozstd, mozfull, mozstaging, base
-* Ensure that it is all green (blacklist and whitelist are working)
+* PROD: mozstd, mozfull, mozstaging, base
-* This page doesn't hit shavar server at all
-* If not working, then either    about:config issue or client issue - repeat above!
-'''CNN page'''
+STEPS:
 * Pull up CNN: http://www.cnn.com
 * make sure that the Shield shows up
 * the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking
-'''YouTube page'''
+==== verify YouTube page ====
+repeat test 1 x for each of:
+* STAGE: mozstd, mozfull, mozstaging, base
+* PROD: mozstd, mozfull, mozstaging, base
+Note: skip this test for moztestpub-track-digest256.
+STEPS:
 * Pull up Youtube: https://www.youtube.com
-* make sure that googlesyndication.com is NOT blocked by tracking protection on that site
+* Shouldn't see the shield
-* if the whitelist doesn't work, the devtools should indicate that googlesyndicaiton.com was blocked by TP
+* Make sure that doubleclick.net is NOT blocked by tracking protection on that site
-* if you see in console tab that googlesyndication.com is blocked, that's a sign that TP isn't great
+* If the whitelist doesn't work, the Tools > WebDeveloper > BrowserConsole should indicate that doubleclick.net was blocked by TP
+* If you see in console (network) tab that doubleclick.net is blocked, that's a sign that TP is NOT Working
 [[#top]]
-=== TEST #1: verify blacklist deletions ===
+== List Additions / Deletions Tests: moztestpub ==
-'''VERIFY ENABLED'''
+NOTE:
+* STAGE only
-* Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
+=== blacklist deletions ===
+==== verify enabled ====
+* Follow setup instructions for moztestpub file here: [[#Setup|Setup]]
+* Pull up this test page: https://mozilla.github.io/tracking-test/
 * ensure that the resource is blocked:
 * You should see:
-** A black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat]
+** A fox with a stop sign: [https://wiki.mozilla.org/File:Tracking_protection_on.png Fox]
 ** A tracking protection shield icon in the URL bar
 ** A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."
-'''DELETE FROM LIST'''
+==== delete from blacklist ====
-* Remove the extremetracking.com json blob from the blacklist on https://github.com/mozilla-services/shavar-test-lists
+* Follow setup instructions for moztestpub file here: [[#Setup|Setup]]
+* Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists
 * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
+* Manually trigger an update via the mozilla button on the about:url-classifier page
+==== verify disabled ====
+* Restart Firefox from terminal to force list updates on client
+* Start Debugging
+Verify debugging log (in terminal where you launched Firefox).
+Here are some examples of what you might see:
+'''update URL has registered correctly'''
+* i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2
+'''Registering whitelist- mozstd-trackwhite-digest256'''
+* listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2
+'''Checking for updates on shavar server'''
+* listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2
-'''VERIFY DISABLED'''
+'''Request to shavar server begins'''
+* listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request moztestpub-track-digest256;a:1440776223
-Start Debuggging
+'''Update success for both lists'''
-* Restart Firefox from terminal  to force list updates on client
+* listmanager: 09:44:07 GMT-0700 (PDT): update success for mozpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700
-* The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
-* Debug output will start in terminal 10 - 30 secs after you start browser
-Verify:
+'''VERIFY'''
-* update URL has registered correctly (line #8)
+* check that it's no longer blocked by visiting https://mozilla.github.io/tracking-test/
-** Registering whitelist- mozpub-trackwhite-digest256 (line #10)
+* If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat]
-** Checking for updates on shavar server: "checkForUpdates with https://tracking.services.mozilla.com" (line #61)
-** Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223" (line #108)
-** Update success for both lists: "update success for mozpub-track-digest256,mozpub-trackwhite-digest256" (line #112)
-** NOTE: if line #112 failure, rest of test will likely not work!
-** If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat]
 [[#top]]
-=== TEST #2: verify list additions ===
+=== blacklist additions ===
-'''ADD TO LIST'''
+==== add to blacklist ====
 * Add the extremetracking.com json blob back to the blacklist on https://github.com/mozilla-services/shavar-test-lists
 * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
+==== verify enabled ====
-'''VERIFY RE-ENABLED'''
+* check that it's blocked again by visiting https://mozilla.github.io/tracking-test/
-* check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/
 * If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: [https://wiki.mozilla.org/File:Tracking_protection_on.png Fox]
 [[#top]]
-=== TEST #3: verify whitelist deletions  ===
+=== whitelist deletions  ===
 '''SUMMARY'''
-* This test is similar to TEST #1, but now we verify the whitelist update (vs. blacklist update)
+* This test is similar to [[#blacklist_deletions|blacklist deletions]], but now we verify the whitelist update (vs. blacklist update)
-* Normally googlesyndication is whitelisted so that it doesn't get blocked from YouTube
+* Normally <tt>googlesyndication.com</tt> and <tt>doubleclick.net</tt> are whitelisted so that they don't get blocked from YouTube
-* By removing it from whitelist, we should be able to verify that it now gets blocked!
+* By removing them from whitelist, we should be able to verify that they now get blocked!
-'''VERIFY ENABLED'''
+==== verify enabled ====
 Start Debuggging
 * Restart Firefox from terminal to force list updates on client
-* The pref set in STEP #0 (browser.safebrowsing.debug) will output debug messages like so: https://pastebin.mozilla.org/8844272
+* The pref set in [[#list_delivery|list delivery]] (browser.safebrowsing.debug) will begin outputting debug messages 10 - 30 secs after you start browser
-* Debug output will start in terminal 10 - 30 secs after you start browser
+* see: https://pastebin.mozilla.org/8844272
-'''REMOVE FROM WHITELIST'''
+==== delete from whitelist ====
-* Remove googlesyndication.com from the "Google" organization in the resources section of the whitelist - https://github.com/mozilla-services/shavar-test-lists
+* Remove <tt>doubleclick.net</tt> and <tt>googlesyndication.com</tt> from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists
 * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)
+* Manually trigger an update via the mozilla button on the about:url-classifier page
+==== verify disabled ====
+Verify debugging log (from stdout where you launched Firefox from terminal):
+'''update URL has registered correctly'''
+* i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2
+'''Registering whitelist- moztestpub-trackwhite-digest256'''
+* listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2
+'''Checking for updates on shavar server'''
+* listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2
+'''Request to shavar server begins'''
+* listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223
+'''Update success for both lists'''
+* listmanager: 09:44:07 GMT-0700 (PDT): update success for moztestpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700
+'''NOTE'''
+* if update success fails, rest of test will likely not work!
-'''VERIFY DISABLED'''
+'''VERIFY'''
-* update URL has registered correctly (line #8)
-** Registering whitelist- mozstd-trackwhite-digest256 (line #10) - or mozfull
-** Checking for updates on shavar server: "checkForUpdates with https://shavar.services.mozilla.com" (line #61)
-** Request to shavar server begins with: "makeUpdateRequestForEntry_: request mozstd-track-digest256;a:1440776223" (line #108)
-** Update success for both lists: "update success for mozstd-track-digest256,mozstd-trackwhite-digest256" (line #112)
-** NOTE: if line #112 failure, rest of test will likely not work!
 * visit youtube.com
-** you should NOT see the shield in the url bar
+* you should see the shield in the url bar (whitelist isn't blocking because deleted)
-* In the: Tools > Web Developer > Browser Console, you should see that googlesyndication.com was blocked because of TrackingProtection
+* In the: Tools > Web Developer > Browser Console, you should see that doubleclick.net was blocked because of TrackingProtection
 [[#top]]