Security/Safe Browsing

Note: The Safe Browsing feature in Firefox has been renamed to Phishing Protection, but it's still known as Safe Browsing internally.

Download Protection and Tracking protection have their own separate pages.

History

Google Safe Browsing was an anti-phishing extension released by Google on labs.google.com in December 2005. Google has released this extension to the Mozilla Foundation under MPL 1.1/GPL 2.0/LGPL 2.1 in order that it might be used as part of Firefox if desired. We've landed this change on the trunk as a global extension as of 7 March 2006. You can read the discussion that lead up to to its integration in https://bugzilla.mozilla.org/show_bug.cgi?id=329292

Google started migrating their Safe Browsing to version 4 of the protocol in 2015. We completed our V4 implementation in late 2017 and shipped it in Firefox 56 via a Shield gradual roll-out.

Prefs

• browser.safebrowsing.blockedURIs.enabled: enable the plugin stability blocking (no override or UI)
• browser.safebrowsing.debug: show debugging info from the JavaScript list update code on the command line as long as browser.dom.window.dump.enabled is also enabled
• browser.safebrowsing.id: what SAFEBROWSING_ID in gethashURL and updateURL maps to
• browser.safebrowsing.malware.enabled: enable malware protection (includes unwanted as well)
• browser.safebrowsing.phishing.enabled: enable phishing protection
• browser.safebrowsing.provider.google.gethashURL: server endpoint for completions of malware and phishing lists
• browser.safebrowsing.provider.google.lists: list of tables coming from the Google Safe Browsing service
• browser.safebrowsing.provider.google.reportURL: probably unused
• browser.safebrowsing.provider.google.updateURL: server endpoint for malware and phishing list updates
• browser.safebrowsing.provider.google.lastupdatetime: timestamp (in ms) of when the last list update happened.
• browser.safebrowsing.provider.google.nextupdatetime: timestamp (in ms) of when the list should next be downloaded.
• browser.safebrowsing.reportMalwareMistakeURL: destination for the "This isn't an attack site" button (after ignoring the interstitial warning)
• browser.safebrowsing.reportPhishMistakeURL: destination for the "This isn't a web forgery" button (after ignoring the interstitial warning)
• browser.safebrowsing.reportPhishURL: destination for the "Help | Report Web Forgery" menu item
• urlclassifier.blockedTable: list of tables to use for the plugin stability blocking
• urlclassifier.disallow_completions: list of tables for which we never call gethash
• urlclassifier.gethashnoise: the number of fake entries to add to any gethash calls. Defaul value: 4. Maximum value: 999 (beyond, the Google request fails with HTTP 400).
• urlclassifier.gethash.timeout_ms: the timeout after which gethash requests should be aborted
• urlclassifier.malwareTable: list of tables to use when looking for malware (they need to be named *-malware-* or *-unwanted-*)
• urlclassifier.max-complete-age: the maximum amount of time in seconds that a complete hash will be considered fresh and allowed to match
• urlclassifier.phishTable: list of tables to use when looking for phishing (they need to be named *-phish-*)
• urlclassifier.skipHostnames: comma-separated list of hostnames to exempt from Safe Browsing checks (hidden, only for temporary hotfix purposes)

Documentation

• Official Google documentation:
  • Safe Browsing protocol: v2.2 and v4
  • User warning requirements
  • Internal documentation available under NDA
  • Android API (requires Google Play Services 9.4)
  • Built-in support in WebView (public in Android O, private in Android N)
  • Public API mailing list
• Design Documentation
  • Server Spec
  • Client Spec
• SUMO
• Overview of how Safe Browsing works in Firefox
• Chromium
  • Design documentation
  • Implementation overview
• Google's advice to site owners:
  • Malware
  • Deceptive content
  • Potentially unwanted or uncommon software

Engineering

Product/Component: Toolkit/Safe Browsing

• Tracking bug (deprecated, do not use)
• The Firefox implementation is split into a few parts:
  • browser/components/safebrowsing/ (front-end tests)
  • netwerk/base/nsChannelClassifier
  • toolkit/components/url-classifier/ (includes the list manager)
• Local store is in:
  • ~/.cache/mozilla/firefox/XXXX/safebrowsing/ on Linux
  • ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/ on Mac
  • C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\ on Windows
• itisatrap.org test pages
• Telemetry dashboard

Code walkthrough

Both nsBaseChannel::Open() and nsBaseChannel::AsyncOpen() ask for the channel to be "classified" by nsChannelClassifier. There is also a local-only classification that is requested by tracking protection.

While we collect information about each of the list matches in nsUrlClassifierClassifyCallback::HandleResult(), which is called for each matched list from nsUrlClassifierLookupCallback::HandleResults(), we pick only the highest priority list match and call OnClassifyComplete() in nsUrlClassifierClassifyCallback::HandleEvent() according to:

• priority of providers
• priority of warning types

Then we return information about the list match. That causes the channel to be cancelled with that error code.

When the classification state of the page changes, the appropriate UI is shown.

Tests

Here are all of the tests which are relevant to Safe Browsing:

./mach gtest UrlClassifier*
./mach test toolkit/components/url-classifier/tests/browser/
./mach test toolkit/components/url-classifier/tests/unit/
./mach test toolkit/components/url-classifier/tests/mochitest/

as well as the ones in testing/firefox-ui/tests/functional/safebrowsing/.

Also relevant are the Tracking Protection tests.

QA

• about:url-classifier provides lots of useful state information
• Test pages
  • Malware, phishing, and unwanted software hard-coded test URLs
  • Phishtank (real phishing sites)
  • Google test pages (we don't implement: Clank Warnings, Client-side phishing detection, Bad IP Warnings)
  • Static test pages for specific bugs
• Meta QA bug
• Info on why certain URLs are blocked
• Script to dump the contents of the local store
• UI tests (Marionette)

To turn on debugging output, export the following environment variables:

MOZ_LOG_FILE=/tmp/safebrowsing.log
MOZ_LOG="UrlClassifierDbService:5,nsChannelClassifier:5,UrlClassifierProtocolParser:5,UrlClassifierStreamUpdater:5,UrlClassifierPrefixSet:5"

and also see these prefs to see debugging output from the JS pieces of Safe Browsing:

 browser.dom.window.dump.enabled = true
 browser.safebrowsing.debug = true

Telemetry

Alerts are sent to safebrowsing-telemetry@mozilla.org.

• Performance
  • URLCLASSIFIER_ASYNC_CLASSIFYLOCAL_TIME: time spent inside AsyncClassifyLocalWithTables()
  • URLCLASSIFIER_CLASSIFYLOCAL_TIME: time spent inside ClassifyLocalWithTables()
  • URLCLASSIFIER_CL_CHECK_TIME: how long a Safe Browsing lookup took
  • URLCLASSIFIER_CL_KEYED_UPDATE_TIME: how long table updates takes
  • URLCLASSIFIER_LOOKUP_TIME_2: time spent in the dbservice while doing a lookup
  • URLCLASSIFIER_PS_CONSTRUCT_TIME: time spent constructing a PrefixSet
  • URLCLASSIFIER_PS_FALLOCATE_TIME: time spent allocating a PrefixSet
  • URLCLASSIFIER_PS_FILELOAD_TIME: time spent loading PrefixSet from disk
  • URLCLASSIFIER_SHUTDOWN_TIME: time spent in the URL Classifier shutdown code
  • URLCLASSIFIER_VLPS_CONSTRUCT_TIME: time spent constructing a variable-length PrefixSet
  • URLCLASSIFIER_VLPS_FALLOCATE_TIME: time spent allocating a variable-length PrefixSet
  • URLCLASSIFIER_VLPS_FILELOAD_TIME: time spent loading a variable-length PrefixSet from disk
• Server-related
  • URLCLASSIFIER_COMPLETE_REMOTE_STATUS2: HTTP status code returned by the gethash server
  • URLCLASSIFIER_COMPLETE_SERVER_RESPONSE_TIME: response time from the completion server
  • URLCLASSIFIER_COMPLETE_TIMEOUT2: whether or not a client timed out while contacting the gethash server
  • URLCLASSIFIER_COMPLETION_ERROR: whether a V4 completion result couldn't be parsed or contained an unknown threat type
  • URLCLASSIFIER_UPDATE_ERROR: whether or not an error was encountered while processing an update
  • URLCLASSIFIER_UPDATE_REMOTE_NETWORK_ERROR: update errors while downloading updates
  • URLCLASSIFIER_UPDATE_REMOTE_STATUS2: HTTP status code returned by the update server
  • URLCLASSIFIER_UPDATE_SERVER_RESPONSE_TIME: response time from the update server
  • URLCLASSIFIER_UPDATE_TIMEOUT: whether or not a client timed out while contacting the update server
• Database size
  • URLCLASSIFIER_LC_COMPLETIONS: number of entries in the completion cache
  • URLCLASSIFIER_LC_PREFIXES: number of entries in the prefix cache
• User interface
  • URLCLASSIFIER_UI_EVENTS: number of interstitial pages shown (malware, phishing, unwanted, harmful) either in a top-level page or in a frame and the number of times users click on "Ignore this warning", "Get me out of here" or "Why is this blocked?"
• V4 quality assurance
  • URLCLASSIFIER_NEGATIVE_CACHE_DURATION: negative cache duration received in fullhash response
  • URLCLASSIFIER_POSITIVE_CACHE_DURATION: positive cache duration received in fullhash response
  • URLCLASSIFIER_VLPS_LOAD_CORRUPT: whether or not a variable-length PrefixSet loaded from disk is corrupt
  • URLCLASSIFIER_VLPS_LONG_PREFIXES (Nightly-only): length of the variable-length prefixes that are sent by Google

Links

• Google reporting forms:
  • Malware
  • Phishing -- Firefox-specific
  • Phishing error (false positive) -- Firefox-specific
  • General (false negatives and false positives)
• StopBadware.org form:
  • Malware error
• API key and account details (internal access only)