Security/Safe Browsing/V4 Implementation: Difference between revisions
No edit summary |
|||
| Line 3: | Line 3: | ||
= Design and Implementation = | = Design and Implementation = | ||
To have v2 and v4 run in parallel, we must carefully refactor some core components like ListManager and HashCompleter. Instead of having a master preference to switch between v2 and v4, we take the "table name driven" approach. The plan is to add a separate provider called "google4", which owns v4 tables: goog-phish-proto, goog-unwanted-proto and goog-malware-proto. The suffix "-proto" indicates that the table should be updated and completed via protobuf. Besides, the new provider google4 has its own updateURL and gethashURL. You can consider "google4" yet another provider like "mozilla". When ListManager and HashCompleter (and any other related components like ProtocolParser) sees table names suffixed by "-proto", they would behave differently. For example, in listmanager.js, while making update request for table "goog-phish-proto", nsIUrlClassifier.makeUpdateRequestV4 will be called to build a v4 specific request. (See {{Bug|1264885}} and {{Bug|1275507}} for more information.) | |||
= Milestones = | = Milestones = | ||
Revision as of 09:59, 18 July 2016
Introduction
Safe Browsing v4 is designed to reduce network bandwidth and disk storage (mainly for mobile devices). The usage of partial URL hashes (aka prefixes) and complete hashes remains the same but the way we get those prefixes and complete hashes is changed. The update and hash completion API will be based on protobuf and content compression (in additional to HTTP compression) is introduced. For further information, see https://developers.google.com/safe-browsing/v4 (The public specification is not complete. For example, the protobuf is not mentioned.)
Design and Implementation
To have v2 and v4 run in parallel, we must carefully refactor some core components like ListManager and HashCompleter. Instead of having a master preference to switch between v2 and v4, we take the "table name driven" approach. The plan is to add a separate provider called "google4", which owns v4 tables: goog-phish-proto, goog-unwanted-proto and goog-malware-proto. The suffix "-proto" indicates that the table should be updated and completed via protobuf. Besides, the new provider google4 has its own updateURL and gethashURL. You can consider "google4" yet another provider like "mozilla". When ListManager and HashCompleter (and any other related components like ProtocolParser) sees table names suffixed by "-proto", they would behave differently. For example, in listmanager.js, while making update request for table "goog-phish-proto", nsIUrlClassifier.makeUpdateRequestV4 will be called to build a v4 specific request. (See bug 1264885 and bug 1275507 for more information.)
Milestones
M0 (2016/7/31)
Deliverables
- Send v4 update request on time
- Parse v4 update response but not store to disk
- Use v4 request backoff settings
- v2 will still be up and running
Notes
- v4 table download/update will be opt-in.
- To test M0 features, modify the following preferences:
- urlclassifier.malwareTable ==> Add goog-malware-proto and goog-unwanted-proto
- urlclassifier.phishTable ==> Add goog-phish-proto
Bugs
13 Total; 0 Open (0%); 12 Resolved (92.31%); 1 Verified (7.69%);
M1 (2016/9/30)
Deliverables
- Store v4 tables to disk (including fixed and variable length prefixes)
- Store table states
- Split v4 tables to different directory per provider
Bugs
41 Total; 1 Open (2.44%); 37 Resolved (90.24%); 3 Verified (7.32%);
M2 (Right before Hawaii Workweek)
Deliverables
- Check v4 prefixes (in addition to v4) but ignore the result
- v2/v4 prefix matching consistency telemetry (e.g. v2/v4 should both 'have' or 'not have' certain URL hash)
- Be careful of the variable length prefixes
Bugs
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 502 [1] => Bad Gateway ) ) 1
M3 (Deadline TBD)
Deliverables
- Do and use v4 complete hashes (in addition to v2) but ignore the result
- v2/v4 URL hash matching consistency telemetry
Bugs
| ID | Summary | Status | Assigned to | Resolution | |
|---|---|---|---|---|---|
| 1296802 | Telemetry pings are run through the URL Classifier | RESOLVED | Thomas Nguyen (:tnguyen) | FIXED | No |
| 1298257 | Implement url matching for variable-length prefix set | RESOLVED | Thomas Nguyen (:tnguyen) | FIXED | No |
| 1311910 | Add telemetry to measure update error rate for V2 and V4 | RESOLVED | Thomas Nguyen (:tnguyen) | FIXED | No |
| 1315893 | Add telemetry to measure update time for V2 and V4 | RESOLVED | Thomas Nguyen (:tnguyen) | FIXED | No |
| 1319286 | Cache v4 table states in memory | RESOLVED | Henry Chang [:hchang] | FIXED | No |
5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);
M4 (Deadline TBD)
Deliverables
- Cache
- Anything else
Bugs
| ID | Summary | Status | Assigned to | Resolution | |
|---|---|---|---|---|---|
| 1305486 | Enable V4 update by default on Nightly only | RESOLVED | Henry Chang [:hchang] | FIXED | No |
| 1313629 | Version-aware (v2/v4) hash completer | RESOLVED | Henry Chang [:hchang] | DUPLICATE | No |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
M5 (Deadline TBD)
Deliverables
- Enable v4 by default!
Bugs
14 Total; 0 Open (0%); 14 Resolved (100%); 0 Verified (0%);