Confirmed users
1,504
edits
| (13 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
<span style="color:#FF0000">This page is deprecated!!!!!! Please see: [https://testrail.stage.mozaws.net/index.php?/suites/view/354&group_by=cases:section_id&group_order=asc TestRail testcase repository]</span> | |||
<span style="color:#FF0000">(for TestRail permissions, file an issue here: https://github.com/mozilla-services/shavar-e2e-tests )</span> | |||
[[Category:Test Plans]] | [[Category:Test Plans]] | ||
= Testplan (Overview) = | = Testplan (Overview) = | ||
| Line 103: | Line 110: | ||
* ideally we'd be using the real list | * ideally we'd be using the real list | ||
** we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test | ** we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test | ||
[[#top]] | [[#top]] | ||
| Line 121: | Line 121: | ||
=== STAGE === | === STAGE === | ||
1) Verify the download of all lists: | 1) Verify the download of all lists: | ||
$ curl -d" " 'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2' | $ curl -d" " 'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2' | ||
Should return all of these lists: | Should return all of these lists: | ||
ads-track-digest256 | |||
allow-flashallow-digest256 | |||
analytics-track-digest256 | |||
base-track-digest256 | |||
baseeff-track-digest256 | |||
basew3c-track-digest256 | |||
block-flash-digest256 | |||
block-flashsubdoc-digest256 | |||
content-track-digest256 | |||
contenteff-track-digest256 | |||
contentw3c-track-digest256 | |||
except-flash-digest256 | |||
except-flashallow-digest256 | |||
except-flashinfobar-digest256 | |||
except-flashsubdoc-digest256 | |||
except-infobars-digest256 | |||
mozfull-track-digest256 | |||
mozfullstaging-track-digest256 | |||
mozplugin-block-digest256 | |||
mozpub-track-digest256 | |||
mozstd-track-digest256 | |||
mozstd-trackwhite-digest256 | |||
mozstdstaging-track-digest256 | |||
mozstdstaging-trackwhite-digest256 | |||
moztestpub-track-digest256 | |||
moztestpub-trackwhite-digest256 | |||
social-track-digest256 | |||
2) Curl each of the list groups. | 2) Curl each of the list groups. | ||
| Line 151: | Line 157: | ||
Example for MOZSTD: | Example for MOZSTD: | ||
$ curl -d "mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2' | |||
Should return a response similar to this: | Should return a response similar to this: | ||
n:3600 | n:3600 | ||
i:mozstd-track-digest256 | i:mozstd-track-digest256 | ||
u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596 | u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596 | ||
[[#top]] | [[#top]] | ||
=== PRE-PRODUCTION === | === PRE-PRODUCTION === | ||
Stack checks are run on pre-prod to verify that a new app stack returns the same results as the old stack. It verifies the server response. This environment is only up when the tests need to be run, which is only when there is a new stack. | |||
$ curl -k -d" " 'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2' | $ curl -k -d" " 'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2' | ||
Should return all of these lists: | Should return all of these lists: | ||
[* update with next verification] | |||
base-track-digest256 | base-track-digest256 | ||
baseeff-track-digest256 | baseeff-track-digest256 | ||
| Line 188: | Line 191: | ||
mozstdstaging-track-digest256 | mozstdstaging-track-digest256 | ||
mozstdstaging-trackwhite-digest256 | mozstdstaging-trackwhite-digest256 | ||
[[#top]] | [[#top]] | ||
=== PRODUCTION === | === PRODUCTION === | ||
# shavar URL | # shavar URL | ||
$ curl -d" " 'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | $ curl -d" " 'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | ||
# tracking URL (CNAME'd to shavar URL - should produce same output) | # tracking URL (CNAME'd to shavar URL - should produce same output) | ||
$ curl -d" " 'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | $ curl -d" " 'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | ||
Should return all of these lists: | Should return all of these lists: | ||
ads-track-digest256 | |||
allow-flashallow-digest256 | |||
analytics-track-digest256 | |||
base-track-digest256 | base-track-digest256 | ||
baseeff-track-digest256 | baseeff-track-digest256 | ||
basew3c-track-digest256 | basew3c-track-digest256 | ||
block-flash-digest256 | |||
block-flashsubdoc-digest256 | |||
content-track-digest256 | content-track-digest256 | ||
contenteff-track-digest256 | contenteff-track-digest256 | ||
contentw3c-track-digest256 | contentw3c-track-digest256 | ||
except-flash-digest256 | |||
except-flashallow-digest256 | |||
except-flashinfobar-digest256 | |||
except-flashsubdoc-digest256 | |||
mozfull-track-digest256 | mozfull-track-digest256 | ||
mozfullstaging-track-digest256 | mozfullstaging-track-digest256 | ||
mozplugin-block-digest256 | mozplugin-block-digest256 | ||
mozplugin2-block-digest256 | mozplugin2-block-digest256 | ||
mozpub-track-digest256 | |||
mozstd-track-digest256 | mozstd-track-digest256 | ||
mozstd-trackwhite-digest256 | mozstd-trackwhite-digest256 | ||
mozstdstaging-track-digest256 | mozstdstaging-track-digest256 | ||
mozstdstaging-trackwhite-digest256 | mozstdstaging-trackwhite-digest256 | ||
social-track-digest256 | |||
Curl for the mozstd list: | Curl for the mozstd list: | ||
# shavar URL | # shavar URL | ||
$ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | ||
# tracking URL (CNAME'd to shavar URL - should produce same output) | # tracking URL (CNAME'd to shavar URL - should produce same output) | ||
$ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | ||
Should return this: | Should return this: | ||
n:3600 | n:3600 | ||
i:mozstd-track-digest256 | i:mozstd-track-digest256 | ||
u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596 | u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596 | ||
[[#top]] | [[#top]] | ||
| Line 388: | Line 372: | ||
* To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com) | * To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com) | ||
* all list files are tested in both stage and production (except moztestpub - stage only) | * all list files are tested in both stage and production (except moztestpub - stage only) | ||
'''Prefs for: base list file''' | |||
<source lang="bash"> | |||
<source> | |||
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html"); | user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html"); | ||
user_pref("browser.safebrowsing.debug", true); | user_pref("browser.safebrowsing.debug", true); | ||
| Line 404: | Line 386: | ||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("urlclassifier.trackingTable", "test-track-simple, | user_pref("urlclassifier.trackingTable", "test-track-simple,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256"); | ||
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple, | user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256"); | ||
</source> | </source> | ||
'''Create a Strict list''' | |||
''' | |||
<source lang="bash"> | <source lang="bash"> | ||
1. Create a new Nightly profile. | |||
2. Go into about:preferences#privacy and set Tracking Protection to Always. | |||
3. Click on Change Block List and select strict protection. | |||
4. Make these edits in about:config | |||
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1); | user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1); | ||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
</source> | </source> | ||
''' | '''Create a Cryptomining Fingerprinting list''' | ||
<source lang="bash"> | <source lang="bash"> | ||
1. Create a new profile | |||
2. Make these edits in about:config | |||
browser.safebrowsing.provider.mozilla.gethashURL;https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2 | |||
browser.safebrowsing.provider.mozilla.updateURL;https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2 | |||
browser.safebrowsing.provider.mozilla.lists;base-track-digest256,mozstd-trackwhite-digest256,content-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,block-flash-digest256,except-flash-digest256,allow-flashallow-digest256,except-flashallow-digest256,block-flashsubdoc-digest256,except-flashsubdoc-digest256,except-flashinfobar-digest256,ads-track-digest256,social-track-digest256,analytics-track-digest256,fastblock1-track-digest256,fastblock1-trackwhite-digest256,fastblock2-track-digest256,fastblock2-trackwhite-digest256,fastblock3-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256 | |||
urlclassifier.trackingTable;test-track-simple,base-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256 | |||
browser.safebrowsing.provider.mozilla.nextupdatetime;0 | |||
3. Restart browser | |||
4. Open a Private Browsing window | |||
</source> | |||
'''Prefs for: moztestpub list file''' | |||
<source> | |||
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html"); | user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html"); | ||
user_pref("browser.safebrowsing.debug", true); | user_pref("browser.safebrowsing.debug", true); | ||
| Line 440: | Line 430: | ||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("urlclassifier.trackingTable", "test-track-simple, | user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256"); | ||
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple, | user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256"); | ||
</source> | </source> | ||
'''Prefs for: | |||
'''Prefs for: mozstd list file''' | |||
<source lang="bash"> | <source lang="bash"> | ||
| Line 457: | Line 449: | ||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("urlclassifier.trackingTable", "test-track-simple, | user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256"); | ||
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple, | user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256"); | ||
</source> | </source> | ||
'''Prefs for: mozfull list file''' | |||
'''Prefs for: | |||
<source lang="bash"> | <source lang="bash"> | ||
| Line 475: | Line 466: | ||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("urlclassifier.trackingTable", "test-track-simple, | user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256"); | ||
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256"); | user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256"); | ||
</source> | </source> | ||
'''Prefs for: mozstaging list file''' | |||
'''Prefs for: | |||
<source lang="bash"> | <source lang="bash"> | ||
| Line 487: | Line 477: | ||
user_pref("browser.safebrowsing.phishing.enabled", false); | user_pref("browser.safebrowsing.phishing.enabled", false); | ||
user_pref("browser.safebrowsing.malware.enabled", false); | user_pref("browser.safebrowsing.malware.enabled", false); | ||
user_pref("privacy.trackingprotection.enabled", true); | |||
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1); | |||
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256"); | |||
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256"); | |||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstdstaging-track-digest256,mozfullstaging-track-digest256"); | |||
user_pref("urlclassifier.trackingTable", "test-track-simple, | user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256"); | ||
user_pref("urlclassifier. | |||
</source> | </source> | ||
[[#top]] | [[#top]] | ||
| Line 530: | Line 502: | ||
* moztestpub [STAGE ONLY] | * moztestpub [STAGE ONLY] | ||
* mozstd | * mozstd | ||
* mozfull | * mozfull | ||
* mozstaging | * mozstaging | ||
* base | * base | ||
* | * strict | ||
* | * fastblock | ||
| Line 611: | Line 580: | ||
</code> | </code> | ||
==== verify | ==== verify list pages ==== | ||
repeat test 1 x for each of these list groups: | repeat test 1 x for each of these list groups: | ||
* STAGE: mozstd, mozfull, | * STAGE: mozstd, mozfull, moztestpub, base | ||
* PROD: mozstd, mozfull, | * PROD: mozstd, mozfull, moztestpub, base | ||
STEPS: | STEPS: | ||
* | * Set up list | ||
* Verify with matching test page: | |||
https://mozilla.github.io/tracking-test/ads.html | |||
https://mozilla.github.io/tracking-test/analytics.html | |||
https://mozilla.github.io/tracking-test/social.html | |||
https://mozilla.github.io/tracking-test/fastblock.html | |||
https://mozilla.github.io/tracking-test/disconnect.html | |||
https://mozilla.github.io/tracking-test/full.html - for the Strict list | |||
==== verify CNN page ==== | ==== verify CNN page ==== | ||
| Line 676: | Line 646: | ||
* Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists | * Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists | ||
* This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | ||
* Manually trigger an update via the mozilla button on the about:url-classifier page | |||
==== verify disabled ==== | ==== verify disabled ==== | ||
| Line 741: | Line 712: | ||
* Remove <tt>doubleclick.net</tt> and <tt>googlesyndication.com</tt> from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists | * Remove <tt>doubleclick.net</tt> and <tt>googlesyndication.com</tt> from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists | ||
* This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | ||
* Manually trigger an update via the mozilla button on the about:url-classifier page | |||
==== verify disabled ==== | ==== verify disabled ==== | ||