Confirmed users
1,504
edits
| (19 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
<span style="color:#FF0000">This page is deprecated!!!!!! Please see: [https://testrail.stage.mozaws.net/index.php?/suites/view/354&group_by=cases:section_id&group_order=asc TestRail testcase repository]</span> | |||
<span style="color:#FF0000">(for TestRail permissions, file an issue here: https://github.com/mozilla-services/shavar-e2e-tests )</span> | |||
[[Category:Test Plans]] | [[Category:Test Plans]] | ||
= Testplan (Overview) = | = Testplan (Overview) = | ||
| Line 44: | Line 51: | ||
** client-only test! (shavar-server could be completely broken and we wouldn't know) | ** client-only test! (shavar-server could be completely broken and we wouldn't know) | ||
** good test to run first, because if its broken, no reason to proceed | ** good test to run first, because if its broken, no reason to proceed | ||
* | * https://mozilla.github.io/tracking-test/ | ||
* https://mozilla.github.io/tracking-test/ads.html | |||
* https://mozilla.github.io/tracking-test/analytics.html | |||
* https://mozilla.github.io/tracking-test/social.html | |||
** page checks if list delivery/generation is broken | ** page checks if list delivery/generation is broken | ||
** Uses: http://extremetracking.com/ (essentially within an iframe) | ** Uses: http://extremetracking.com/ (essentially within an iframe) | ||
| Line 64: | Line 74: | ||
PRODUCTION: | PRODUCTION: | ||
* shavar.services.mozilla.com - e2e test URL (no loadtesting on PRODUCTION) | * shavar.services.mozilla.com - e2e test URL (no loadtesting on PRODUCTION) | ||
* tracking-protection.cdn.mozilla.net - CDN for lists | * tracking-protection.cdn.mozilla.net - CDN for lists | ||
| Line 101: | Line 110: | ||
* ideally we'd be using the real list | * ideally we'd be using the real list | ||
** we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test | ** we can do that for almost all the end-2-end and validation tests, but we can't do that for the "update" test | ||
[[#top]] | [[#top]] | ||
| Line 119: | Line 121: | ||
=== STAGE === | === STAGE === | ||
1) Verify the download of all lists: | 1) Verify the download of all lists: | ||
$ curl -d" " 'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2' | $ curl -d" " 'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2' | ||
Should return all of these lists: | Should return all of these lists: | ||
ads-track-digest256 | |||
allow-flashallow-digest256 | |||
analytics-track-digest256 | |||
base-track-digest256 | |||
baseeff-track-digest256 | |||
basew3c-track-digest256 | |||
block-flash-digest256 | |||
block-flashsubdoc-digest256 | |||
content-track-digest256 | |||
contenteff-track-digest256 | |||
contentw3c-track-digest256 | |||
except-flash-digest256 | |||
except-flashallow-digest256 | |||
except-flashinfobar-digest256 | |||
except-flashsubdoc-digest256 | |||
except-infobars-digest256 | |||
mozfull-track-digest256 | |||
mozfullstaging-track-digest256 | |||
mozplugin-block-digest256 | |||
mozpub-track-digest256 | |||
mozstd-track-digest256 | |||
mozstd-trackwhite-digest256 | |||
mozstdstaging-track-digest256 | |||
mozstdstaging-trackwhite-digest256 | |||
moztestpub-track-digest256 | |||
moztestpub-trackwhite-digest256 | |||
social-track-digest256 | |||
2) Curl each of the list groups. | 2) Curl each of the list groups. | ||
| Line 149: | Line 157: | ||
Example for MOZSTD: | Example for MOZSTD: | ||
$ curl -d "mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2' | |||
Should return a response similar to this: | Should return a response similar to this: | ||
n:3600 | n:3600 | ||
i:mozstd-track-digest256 | i:mozstd-track-digest256 | ||
u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596 | u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596 | ||
[[#top]] | [[#top]] | ||
=== PRE-PRODUCTION === | === PRE-PRODUCTION === | ||
Stack checks are run on pre-prod to verify that a new app stack returns the same results as the old stack. It verifies the server response. This environment is only up when the tests need to be run, which is only when there is a new stack. | |||
$ curl -k -d" " 'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2' | $ curl -k -d" " 'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2' | ||
Should return all of these lists: | Should return all of these lists: | ||
[* update with next verification] | |||
base-track-digest256 | base-track-digest256 | ||
baseeff-track-digest256 | baseeff-track-digest256 | ||
| Line 186: | Line 191: | ||
mozstdstaging-track-digest256 | mozstdstaging-track-digest256 | ||
mozstdstaging-trackwhite-digest256 | mozstdstaging-trackwhite-digest256 | ||
[[#top]] | [[#top]] | ||
=== PRODUCTION === | === PRODUCTION === | ||
# shavar URL | # shavar URL | ||
$ curl -d" " 'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | $ curl -d" " 'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | ||
# tracking URL (CNAME'd to shavar URL - should produce same output) | # tracking URL (CNAME'd to shavar URL - should produce same output) | ||
$ curl -d" " 'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | $ curl -d" " 'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2' | ||
Should return all of these lists: | Should return all of these lists: | ||
ads-track-digest256 | |||
allow-flashallow-digest256 | |||
analytics-track-digest256 | |||
base-track-digest256 | base-track-digest256 | ||
baseeff-track-digest256 | baseeff-track-digest256 | ||
basew3c-track-digest256 | basew3c-track-digest256 | ||
block-flash-digest256 | |||
block-flashsubdoc-digest256 | |||
content-track-digest256 | content-track-digest256 | ||
contenteff-track-digest256 | contenteff-track-digest256 | ||
contentw3c-track-digest256 | contentw3c-track-digest256 | ||
except-flash-digest256 | |||
except-flashallow-digest256 | |||
except-flashinfobar-digest256 | |||
except-flashsubdoc-digest256 | |||
mozfull-track-digest256 | mozfull-track-digest256 | ||
mozfullstaging-track-digest256 | mozfullstaging-track-digest256 | ||
mozplugin-block-digest256 | mozplugin-block-digest256 | ||
mozplugin2-block-digest256 | mozplugin2-block-digest256 | ||
mozpub-track-digest256 | |||
mozstd-track-digest256 | mozstd-track-digest256 | ||
mozstd-trackwhite-digest256 | mozstd-trackwhite-digest256 | ||
mozstdstaging-track-digest256 | mozstdstaging-track-digest256 | ||
mozstdstaging-trackwhite-digest256 | mozstdstaging-trackwhite-digest256 | ||
social-track-digest256 | |||
Curl for the mozstd list: | Curl for the mozstd list: | ||
# shavar URL | # shavar URL | ||
$ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | ||
# tracking URL (CNAME'd to shavar URL - should produce same output) | # tracking URL (CNAME'd to shavar URL - should produce same output) | ||
$ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2' | ||
Should return this: | Should return this: | ||
n:3600 | n:3600 | ||
i:mozstd-track-digest256 | i:mozstd-track-digest256 | ||
u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596 | u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596 | ||
[[#top]] | [[#top]] | ||
| Line 382: | Line 368: | ||
'''NOTE''' | '''NOTE''' | ||
* Prefs info is permanently located at (https://github.com/mozilla-services/services-test/blob/dev/shavar/e2e-test/prefs.ini) | |||
* Prefs are listed below for stage testing. | * Prefs are listed below for stage testing. | ||
* To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com) | * To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com) | ||
* all list files are tested in both stage and production (except moztestpub - stage only) | * all list files are tested in both stage and production (except moztestpub - stage only) | ||
'''Prefs for: base list file''' | |||
<source lang="bash"> | |||
<source> | |||
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html"); | user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html"); | ||
user_pref("browser.safebrowsing.debug", true); | user_pref("browser.safebrowsing.debug", true); | ||
| Line 396: | Line 382: | ||
user_pref("privacy.trackingprotection.enabled", true); | user_pref("privacy.trackingprotection.enabled", true); | ||
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1); | user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1); | ||
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256"); | |||
user_pref("browser.safebrowsing.provider.mozilla.lists", "block- | user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256"); | ||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | |||
user_pref("urlclassifier.disallow_completions", "block- | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("urlclassifier.trackingTable", "test-track-simple,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256"); | |||
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256"); | |||
</source> | |||
'''Create a Strict list''' | |||
<source lang="bash"> | |||
1. Create a new Nightly profile. | |||
2. Go into about:preferences#privacy and set Tracking Protection to Always. | |||
3. Click on Change Block List and select strict protection. | |||
4. Make these edits in about:config | |||
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1); | |||
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2"); | ||
</source> | |||
'''Create a Cryptomining Fingerprinting list''' | |||
<source lang="bash"> | |||
1. Create a new profile | |||
2. Make these edits in about:config | |||
browser.safebrowsing.provider.mozilla.gethashURL;https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2 | |||
browser.safebrowsing.provider.mozilla.updateURL;https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%MAJOR_VERSION%&pver=2.2 | |||
browser.safebrowsing.provider.mozilla.lists;base-track-digest256,mozstd-trackwhite-digest256,content-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,block-flash-digest256,except-flash-digest256,allow-flashallow-digest256,except-flashallow-digest256,block-flashsubdoc-digest256,except-flashsubdoc-digest256,except-flashinfobar-digest256,ads-track-digest256,social-track-digest256,analytics-track-digest256,fastblock1-track-digest256,fastblock1-trackwhite-digest256,fastblock2-track-digest256,fastblock2-trackwhite-digest256,fastblock3-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256 | |||
urlclassifier.trackingTable;test-track-simple,base-track-digest256,base-cryptomining-track-digest256,base-fingerprinting-track-digest256 | |||
browser.safebrowsing.provider.mozilla.nextupdatetime;0 | |||
3. Restart browser | |||
4. Open a Private Browsing window | |||
</source> | </source> | ||
| Line 475: | Line 486: | ||
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256"); | user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256"); | ||
</source> | </source> | ||
[[#top]] | [[#top]] | ||
| Line 545: | Line 502: | ||
* moztestpub [STAGE ONLY] | * moztestpub [STAGE ONLY] | ||
* mozstd | * mozstd | ||
* mozfull | * mozfull | ||
* mozstaging | * mozstaging | ||
* base | * base | ||
* | * strict | ||
* | * fastblock | ||
| Line 626: | Line 580: | ||
</code> | </code> | ||
==== verify | ==== verify list pages ==== | ||
repeat test 1 x for each of these list groups: | repeat test 1 x for each of these list groups: | ||
* STAGE: mozstd, mozfull, | * STAGE: mozstd, mozfull, moztestpub, base | ||
* PROD: mozstd, mozfull, | * PROD: mozstd, mozfull, moztestpub, base | ||
STEPS: | STEPS: | ||
* | * Set up list | ||
* Verify with matching test page: | |||
https://mozilla.github.io/tracking-test/ads.html | |||
https://mozilla.github.io/tracking-test/analytics.html | |||
https://mozilla.github.io/tracking-test/social.html | |||
https://mozilla.github.io/tracking-test/fastblock.html | |||
https://mozilla.github.io/tracking-test/disconnect.html | |||
https://mozilla.github.io/tracking-test/full.html - for the Strict list | |||
==== verify CNN page ==== | ==== verify CNN page ==== | ||
| Line 679: | Line 634: | ||
* Follow setup instructions for moztestpub file here: [[#Setup|Setup]] | * Follow setup instructions for moztestpub file here: [[#Setup|Setup]] | ||
* Pull up this test page: https:// | * Pull up this test page: https://mozilla.github.io/tracking-test/ | ||
* ensure that the resource is blocked: | * ensure that the resource is blocked: | ||
* You should see: | * You should see: | ||
| Line 691: | Line 646: | ||
* Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists | * Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists | ||
* This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | ||
* Manually trigger an update via the mozilla button on the about:url-classifier page | |||
==== verify disabled ==== | ==== verify disabled ==== | ||
| Line 718: | Line 674: | ||
'''VERIFY''' | '''VERIFY''' | ||
* check that it's no longer blocked by visiting https:// | * check that it's no longer blocked by visiting https://mozilla.github.io/tracking-test/ | ||
* If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat] | * If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: [https://wiki.mozilla.org/File:Tracking_protection_off.png Cat] | ||
| Line 732: | Line 688: | ||
==== verify enabled ==== | ==== verify enabled ==== | ||
* check that it's blocked again by visiting https:// | * check that it's blocked again by visiting https://mozilla.github.io/tracking-test/ | ||
* If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: [https://wiki.mozilla.org/File:Tracking_protection_on.png Fox] | * If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: [https://wiki.mozilla.org/File:Tracking_protection_on.png Fox] | ||
| Line 756: | Line 712: | ||
* Remove <tt>doubleclick.net</tt> and <tt>googlesyndication.com</tt> from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists | * Remove <tt>doubleclick.net</tt> and <tt>googlesyndication.com</tt> from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists | ||
* This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | * This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins) | ||
* Manually trigger an update via the mozilla button on the about:url-classifier page | |||
==== verify disabled ==== | ==== verify disabled ==== | ||