Security/Safe Browsing: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎QA: remove the mozmill test since it has been ported to Marionette)
(→‎Telemetry: add recently added telemetry)
Line 102: Line 102:
* Performance
* Performance
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_CL_CHECK_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_CL_CHECK_TIME]: how long a Safe Browsing lookup took
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_CL_CHECK_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_CL_CHECK_TIME]: how long a Safe Browsing lookup took
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_CL_UPDATE_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_CL_UPDATE_TIME]: how long table updates takes
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=google!google4!other!mozilla&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_CL_KEYED_UPDATE_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=0&trim=1&use_submission_date=0 URLCLASSIFIER_CL_KEYED_UPDATE_TIME]: how long table updates takes
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_LOOKUP_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_LOOKUP_TIME]: time spent in the dbservice while doing a lookup
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_LOOKUP_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_LOOKUP_TIME]: time spent in the dbservice while doing a lookup
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_PS_CONSTRUCT_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PS_CONSTRUCT_TIME]: time spent constructing a PrefixSet
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_PS_CONSTRUCT_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PS_CONSTRUCT_TIME]: time spent constructing a PrefixSet
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_PS_FALLOCATE_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PS_FALLOCATE_TIME]: time spent allocating a PrefixSet
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_PS_FALLOCATE_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PS_FALLOCATE_TIME]: time spent allocating a PrefixSet
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_PS_FILELOAD_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PS_FILELOAD_TIME]: time spent loading PrefixSet from disk
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_PS_FILELOAD_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PS_FILELOAD_TIME]: time spent loading PrefixSet from disk
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_SHUTDOWN_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=0&trim=1&use_submission_date=0 URLCLASSIFIER_SHUTDOWN_TIME]: time spent in the URL Classifier shutdown code
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_VLPS_FALLOCATE_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=0&trim=1&use_submission_date=0 URLCLASSIFIER_VLPS_FALLOCATE_TIME]: time spent allocating a variable-length PrefixSet
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_VLPS_FILELOAD_TIME&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=0&trim=1&use_submission_date=0 URLCLASSIFIER_VLPS_FILELOAD_TIME]: time spent loading a variable-length PrefixSet from disk
* Server-related
* Server-related
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_COMPLETE_TIMEOUT&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_COMPLETE_TIMEOUT]: whether or not a client timed out while contacting the gethash server
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=google!__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_COMPLETE_TIMEOUT2&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_COMPLETE_TIMEOUT2]: whether or not a client timed out while contacting the gethash server
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_COMPLETE_REMOTE_STATUS&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_COMPLETE_REMOTE_STATUS]: HTTP status code returned by the gethash server
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=google!__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_COMPLETE_REMOTE_STATUS2&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_COMPLETE_REMOTE_STATUS2]: HTTP status code returned by the gethash server
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_UPDATE_REMOTE_STATUS&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_UPDATE_REMOTE_STATUS]: HTTP status code returned by the update server
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=google!other!mozilla!other&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_UPDATE_ERROR&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_UPDATE_ERROR]: whether or not an error was encountered while processing an update
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=google!other!mozilla!other&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_UPDATE_REMOTE_STATUS2&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_UPDATE_REMOTE_STATUS2]: HTTP status code returned by the update server
* Database size
* Database size
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_LC_COMPLETIONS&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_LC_COMPLETIONS]: number of entries in the completion cache
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-06&keys=__none__!__none__!__none__&max_channel_version=nightly%252F49&measure=URLCLASSIFIER_LC_COMPLETIONS&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_LC_COMPLETIONS]: number of entries in the completion cache
Line 116: Line 120:
* User interface
* User interface
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-07-01&keys=__none__!__none__!__none__&max_channel_version=nightly%252F50&measure=SECURITY_UI&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-07-01&table=1&trim=1&use_submission_date=0 SECURITY_UI]: number of interstitial pages shown ([https://dxr.mozilla.org/mozilla-central/rev/78dd94ba93c77d0bba45f8e4525947629a305a41/security/manager/ssl/nsISecurityUITelemetry.idl#106-124 malware, phishing], [https://dxr.mozilla.org/mozilla-central/rev/78dd94ba93c77d0bba45f8e4525947629a305a41/security/manager/ssl/nsISecurityUITelemetry.idl#145-153 unwanted]) either in a top-level page or in a frame and the number of times users click on "Ignore this warning", "Get me out of here" or "Why is this blocked?"
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-07-01&keys=__none__!__none__!__none__&max_channel_version=nightly%252F50&measure=SECURITY_UI&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-07-01&table=1&trim=1&use_submission_date=0 SECURITY_UI]: number of interstitial pages shown ([https://dxr.mozilla.org/mozilla-central/rev/78dd94ba93c77d0bba45f8e4525947629a305a41/security/manager/ssl/nsISecurityUITelemetry.idl#106-124 malware, phishing], [https://dxr.mozilla.org/mozilla-central/rev/78dd94ba93c77d0bba45f8e4525947629a305a41/security/manager/ssl/nsISecurityUITelemetry.idl#145-153 unwanted]) either in a top-level page or in a frame and the number of times users click on "Ignore this warning", "Get me out of here" or "Why is this blocked?"
* V4 quality assurance
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_PREFIX_MATCH&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_PREFIX_MATCH]: whether URLs match prefixes in V2, V4 or both
** [https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-01-30&keys=__none__!__none__!__none__&max_channel_version=nightly%252F54&measure=URLCLASSIFIER_VLPS_LOAD_CORRUPT&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-01-23&table=1&trim=1&use_submission_date=0 URLCLASSIFIER_VLPS_LOAD_CORRUPT]: whether or not a variable-length PrefixSet loaded from disk is corrupt


= Links =
= Links =

Revision as of 01:34, 2 February 2017

Note: The Safe Browsing feature in Firefox has been renamed to Phishing Protection, but it's still known as Safe Browsing internally.

Download Protection and Tracking protection have their own separate pages.

History

Google Safe Browsing was an anti-phishing extension released by Google on labs.google.com in December 2005. Google has released this extension to the Mozilla Foundation under MPL 1.1/GPL 2.0/LGPL 2.1 in order that it might be used as part of Firefox if desired. We've landed this change on the trunk as a global extension as of 7 March 2006. You can read the discussion that lead up to to its integration in https://bugzilla.mozilla.org/show_bug.cgi?id=329292

Google started migrating their Safe Browsing to version 4 of the protocol in 2015. We are currently working on integrating V4 in our code base with a incremental approach. That is, we will be landing V4 patches progressively and leave the V2 stack up and running until the V4 is extensively tested and the Shavar server has been upgraded to V4. In other words, V2 and V4 will co-exist for a while to ensure we don't break Safe Browsing. See the V4 implementation plan for the milestones and bugs involved.

Prefs

  • browser.safebrowsing.blockedURIs.enabled: enable the plugin stability blocking (no override or UI)
  • browser.safebrowsing.debug: show debugging info from the JavaScript list update code on the command line
  • browser.safebrowsing.id: what SAFEBROWSING_ID in gethashURL and updateURL maps to
  • browser.safebrowsing.malware.enabled: enable malware protection (includes unwanted as well)
  • browser.safebrowsing.provider.google.gethashURL: server endpoint for completions of malware and phishing lists
  • browser.safebrowsing.provider.google.lists: list of tables coming from the Google Safe Browsing service
  • browser.safebrowsing.provider.google.reportURL: probably unused
  • browser.safebrowsing.provider.google.updateURL: server endpoint for malware and phishing list updates
  • browser.safebrowsing.provider.google.lastupdatetime: timestamp (in ms) of when the last list update happened.
  • browser.safebrowsing.provider.google.nextupdatetime: timestamp (in ms) of when the list should next be downloaded.
  • browser.safebrowsing.reportMalwareMistakeURL: destination for the "This isn't an attack site" button (after ignoring the interstitial warning)
  • browser.safebrowsing.reportPhishMistakeURL: destination for the "This isn't a web forgery" button (after ignoring the interstitial warning)
  • browser.safebrowsing.reportPhishURL: destination for the "Help | Report Web Forgery" menu item
  • urlclassifier.blockedTable: list of tables to use for the plugin stability blocking
  • urlclassifier.disallow_completions: list of tables for which we never call gethash
  • urlclassifier.gethashnoise: the number of fake entries to add to any gethash calls
  • urlclassifier.gethash.timeout_ms: the timeout after which gethash requests should be aborted
  • urlclassifier.malwareTable: list of tables to use when looking for malware (they need to be named *-malware-* or *-unwanted-*)
  • urlclassifier.max-complete-age: the maximum amount of time in seconds that a complete hash will be considered fresh and allowed to match
  • urlclassifier.phishTable: list of tables to use when looking for phishing (they need to be named *-phish-*)

Firefox 49 and earlier:

  • browser.safebrowsing.enabled: enable phishing protection

Firefox 50 and later:

  • browser.safebrowsing.phishing.enabled: enable phishing protection

Documentation

Engineering

Product/Component: Toolkit/Safe Browsing

  • Tracking bug
  • The Firefox implementation is split into a few parts:
    • browser/components/safebrowsing/ (front-end tests)
    • netwerk/base/nsChannelClassifier
    • toolkit/components/url-classifier/ (includes the list manager)
  • Local store is in:
    • ~/.cache/mozilla/firefox/XXXX/safebrowsing/ on Linux
    • ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/ on Mac
    • C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\ on Windows
  • itisatrap.org test pages
  • Telemetry dashboard

Code walkthrough

Both nsBaseChannel::Open() and nsBaseChannel::AsyncOpen() ask for the channel to be "classified" by nsChannelClassifier. There is also a local-only classification that is requested by tracking protection.

The classifier determines the type of URL that it is and then returns the appropriate NS_ERROR code. That causes the channel to be cancelled with that error code.

When the classification state of the page changes, the appropriate UI is shown.

QA

To turn on debugging output, export the following environment variables: 

MOZ_LOG_FILE=/tmp/safebrowsing.log
MOZ_LOG="UrlClassifierDbService:5,nsChannelClassifier:5,UrlClassifierProtocolParser:5,UrlClassifierStreamUpdater:5,UrlClassifierPrefixSet:5"

and also see the browser.safebrowsing.debug pref to see debugging output from the JS pieces of Safe Browsing.

Telemetry

Alerts are sent to safebrowsing-telemetry@mozilla.org.

Links

Retrieved from "https://wiki.mozilla.org/index.php?title=Security/Safe_Browsing&oldid=1161681"