Services/Sync/SimplifiedCrypto - Revision history

Rnewman: Updates.

2011-01-10T21:51:41Z

Updates.

← Older revision		Revision as of 21:51, 10 January 2011
Line 60:		Line 60:
	* Attempt to store, using appropriate race-avoidance technique in case there are multiple clients attempting to upgrade.		* Attempt to store, using appropriate race-avoidance technique in case there are multiple clients attempting to upgrade.
	* Wipe old key data.		* Wipe old key data.
			-- Note that we leave key data in pubkey/privkey, because earlier versions of Sync unfortunately check for these prior to checking storage version. We opted to leave the keys there to get better user experience when an upgrade is needed.

	So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or [[Services/Sync/SyncKey/J-PAKE\|J-PAKE]]-style key distribution.		So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or [[Services/Sync/SyncKey/J-PAKE\|J-PAKE]]-style key distribution.

			Note that each major storage change alters the syncID, and thus PBKDF2 will only work for a single such change -- afterwards, one's passphrase is upgraded into a different sync key, and that won't work.

	=== Bulk keys ===		=== Bulk keys ===
Line 67:		Line 70:
	The server stores one or more bulk keys: one default, and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.		The server stores one or more bulk keys: one default, and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.

	~~TODO: note storage location. Bear~~ in ~~mind that~~ the storage ~~location changed between v4 and v5~~.		Bulk keys are stored in the single WBO storage/crypto/keys.

	Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)		Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)
Line 104:		Line 107:

	This change is incompatible with older clients: not only due to reorganizing the storage namespace, but also because existing clients will be unaware of the simpler encryption mechanism. That means a storage version bump (from 3 to 4).		This change is incompatible with older clients: not only due to reorganizing the storage namespace, but also because existing clients will be unaware of the simpler encryption mechanism. That means a storage version bump (from 3 to 4).

			An error in the argument order of the PBKDF2 function necessitated another version bump, from 4 to 5. Only a small set of nightly builds and add-ons used version 4; these will all prompt to upgrade on launch if a v5 storage version is found on the server.

Rnewman: /* Bulk keys */

2010-12-09T18:58:19Z

Bulk keys

← Older revision		Revision as of 18:58, 9 December 2010
Line 65:		Line 65:
	=== Bulk keys ===		=== Bulk keys ===

	The server stores one or more bulk keys: one default ~~("keys/default")~~, and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.		The server stores one or more bulk keys: one default, and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.

			TODO: note storage location. Bear in mind that the storage location changed between v4 and v5.

	Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)		Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)

	The timestamp on the collections record allows clients to invalidate their key cache when a new key is associated with a collection: the '~~keys~~' collection will appear to have changed.		The timestamp on the collections record allows clients to invalidate their key cache when a new key is associated with a collection: the 'crypto' collection will appear to have changed.

	~~'''Discussion topic''': do we need to preserve (add?) the ability to do per-object encryption? Brian Warner suggested:~~

	:: While not really interesting here, this one-way property is really useful in other situations, like if you derived per-object encryption keys from a parent folder's key. You could then share the whole thing with someone by giving them the parent's key, or share just one object and not give them the ability to get at anything else in that folder. But the sharing discussion is for another day..

	=== HMAC ===		=== HMAC ===

Philikon: /* Deriving encryption and HMAC keys from the Sync Key */

2010-12-09T08:48:59Z

Deriving encryption and HMAC keys from the Sync Key

← Older revision		Revision as of 08:48, 9 December 2010
Line 42:		Line 42:
	First we base32-decode the syncKey from 26 characters to 16 bytes (128 bits). The base32 alphabet is the one specified in [http://tools.ietf.org/html/rfc4648 RFC 4648] except with '''l''' replaced by '''8''' and '''o''' replaced by '''9''':		First we base32-decode the syncKey from 26 characters to 16 bytes (128 bits). The base32 alphabet is the one specified in [http://tools.ietf.org/html/rfc4648 RFC 4648] except with '''l''' replaced by '''8''' and '''o''' replaced by '''9''':

	~~let m~~ = ~~Utils.~~decodeKeyBase32(syncKey);		prk = decodeKeyBase32(syncKey)

	The resulting key is then expanded to an encryption and HMAC key using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]		The resulting key is then expanded to an encryption key T(1) and HMAC key T(2) using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]

	~~// Our extra input to SHA256-HMAC in generateEntry~~
	~~// This includes the full crypto spec; change this when our algo changes.~~
	~~HMAC_INPUT: "Sync-AES_256_CBC-HMAC256",~~

	~~// Reuse the hasher.~~
	~~let h = Utils.makeHMACHasher();~~

	~~// First key.~~
	~~let u = this.username;~~
	~~let k1 = Utils.makeHMACKey("" + HMAC_INPUT + u + "\x01");~~
	~~let enc = Utils.sha256HMACBytes(m, k1, h);~~

	~~// Second key: depends on the output of the first run.~~
	~~let k2 = Utils.makeHMACKey(enc + HMAC_INPUT + u + "\x02");~~
	~~let hmac = Utils.sha256HMACBytes(m, k2, h);~~

	~~enc and hmac are the 256 bit encryption and~~ HMAC ~~keys~~, ~~respectively.~~		info = "Sync-AES_256_CBC-HMAC256" + username
			T(1) = HMAC-SHA256(prk, "" + info + 0x01)
			T(2) = HMAC-SHA256(prk, T(1) + info + 0x02)

	==== Upgrading existing Sync Keys to the new AES key ====		==== Upgrading existing Sync Keys to the new AES key ====

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */ added missing value for HMAC_INPUT

2010-12-04T10:49:12Z

Deriving encryption and HMAC keys from the Sync Key: added missing value for HMAC_INPUT

← Older revision		Revision as of 10:49, 4 December 2010
Line 46:		Line 46:
	The resulting key is then expanded to an encryption and HMAC key using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]		The resulting key is then expanded to an encryption and HMAC key using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]

			// Our extra input to SHA256-HMAC in generateEntry
			// This includes the full crypto spec; change this when our algo changes.
			HMAC_INPUT: "Sync-AES_256_CBC-HMAC256",

	// Reuse the hasher.		// Reuse the hasher.
	let h = Utils.makeHMACHasher();		let h = Utils.makeHMACHasher();

MarcusWolschon: /* Upgrading existing Sync Keys to the new AES key */ Added links to J-PAKE and SyncID in this Wiki for navigating the referenced specifications

2010-12-04T06:52:32Z

Upgrading existing Sync Keys to the new AES key: Added links to J-PAKE and SyncID in this Wiki for navigating the referenced specifications

← Older revision		Revision as of 06:52, 4 December 2010
Line 65:		Line 65:

	* Spot old version		* Spot old version
	* Get a salt (Services.syncID from the meta/global object. The client will be bumping this…)		* Get a salt ([[Labs/Weave/Developer/StorageFormat#Payload:_meta.2Fglobal\|Services.syncID]] from the meta/global object. The client will be bumping this…)
	* Apply PBKDF2 to salt and passphrase to yield our new AES key		* Apply PBKDF2 to salt and passphrase to yield our new AES key
	* Generate bulk keys, encrypt		* Generate bulk keys, encrypt
Line 71:		Line 71:
	* Wipe old key data.		* Wipe old key data.

	So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or J-PAKE-style key distribution.		So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or [[Services/Sync/SyncKey/J-PAKE\|J-PAKE]]-style key distribution.

	=== Bulk keys ===		=== Bulk keys ===

Philikon: /* Passphrase / Sync Key */

2010-11-26T08:29:06Z

Passphrase / Sync Key

← Older revision		Revision as of 08:29, 26 November 2010
Line 36:		Line 36:
	The length of this key is not a big issue: we intend to use J-PAKE for the (infrequent) migration of keys between devices. In any case, 26 is not significantly worse than 20 if typing it does enter the picture, and the use of a nice base32 alphabet makes keyboard entry less error-prone.		The length of this key is not a big issue: we intend to use J-PAKE for the (infrequent) migration of keys between devices. In any case, 26 is not significantly worse than 20 if typing it does enter the picture, and the use of a nice base32 alphabet makes keyboard entry less error-prone.

	As before, the ~~sync key~~ is stored on the client.		As before, the Sync Key is stored on the client. The encryption and hmac keys are derived from it.

	~~Existing users will have their passphrase bootstrapped into an AES key using PBKDF2:~~		==== Deriving encryption and HMAC keys from the Sync Key ====

	* Spot old version
	* Get a salt (Services.syncID from the meta/global object. The client will be bumping this…)
	* Apply PBKDF2 to salt and ~~passphrase to yield our new AES key~~
	* Generate bulk keys~~, encrypt~~
	* Attempt to store, using appropriate race-avoidance technique in case there are multiple clients attempting to upgrade.
	* Wipe old key data.

	~~So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield~~ the ~~new key without any re-entry or J-PAKE-style key distribution.~~

	~~A known fixed string~~ (~~which includes encryption algo details~~) ~~and~~ the ~~username are used as input to SHA256-HMAC~~. ~~The hash operations are chained~~.		First we base32-decode the syncKey from 26 characters to 16 bytes (128 bits). The base32 alphabet is the one specified in [http://tools.ietf.org/html/rfc4648 RFC 4648] except with '''l''' replaced by '''8''' and '''o''' replaced by '''9''':

	let m = Utils.decodeKeyBase32(syncKey);		let m = Utils.decodeKeyBase32(syncKey);

			The resulting key is then expanded to an encryption and HMAC key using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]

	// Reuse the hasher.		// Reuse the hasher.
Line 64:		Line 57:
	let k2 = Utils.makeHMACKey(enc + HMAC_INPUT + u + "\x02");		let k2 = Utils.makeHMACKey(enc + HMAC_INPUT + u + "\x02");
	let hmac = Utils.sha256HMACBytes(m, k2, h);		let hmac = Utils.sha256HMACBytes(m, k2, h);

	~~// Save them.~~
	~~this._encrypt = btoa(enc); // WeaveCrypto expects base64~~
	~~this._hmac = hmac;~~

			enc and hmac are the 256 bit encryption and HMAC keys, respectively.

			==== Upgrading existing Sync Keys to the new AES key ====

			Existing users will have their passphrase bootstrapped into an AES key using PBKDF2:

	The ~~outputs~~ are ~~then used during~~ the key ~~encryption process~~.		* Spot old version
			* Get a salt (Services.syncID from the meta/global object. The client will be bumping this…)
			* Apply PBKDF2 to salt and passphrase to yield our new AES key
			* Generate bulk keys, encrypt
			* Attempt to store, using appropriate race-avoidance technique in case there are multiple clients attempting to upgrade.
			* Wipe old key data.

			So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or J-PAKE-style key distribution.

	=== Bulk keys ===		=== Bulk keys ===

Philikon: /* Passphrase / Sync Key */

2010-11-26T08:20:56Z

Passphrase / Sync Key

← Older revision		Revision as of 08:20, 26 November 2010
Line 52:		Line 52:

	let m = Utils.decodeKeyBase32(syncKey);		let m = Utils.decodeKeyBase32(syncKey);

	// Reuse the hasher.		// Reuse the hasher.
	let h = Utils.makeHMACHasher();		let h = Utils.makeHMACHasher();

Philikon: /* Passphrase / Sync Key */

2010-11-26T08:20:15Z

Passphrase / Sync Key

← Older revision		Revision as of 08:20, 26 November 2010
Line 48:		Line 48:

	So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or J-PAKE-style key distribution.		So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or J-PAKE-style key distribution.

	~~The generated base32 alphanumeric key doesn't actually need to be decoded: it is used as input into hash operations which yield an encryption key and an HMAC key.~~

	~~('''Note:''' however, for NSS convenience in the future, we intend to decode it. Still pending.)~~

	A known fixed string (which includes encryption algo details) and the username are used as input to SHA256-HMAC. The hash operations are chained.		A known fixed string (which includes encryption algo details) and the username are used as input to SHA256-HMAC. The hash operations are chained.

			let m = Utils.decodeKeyBase32(syncKey);

	~~let m = this.keyStr;~~
	~~if (m) {~~
	// Reuse the hasher.		// Reuse the hasher.
	let h = Utils.makeHMACHasher();		let h = Utils.makeHMACHasher();
Line 71:		Line 66:

	// Save them.		// Save them.
	this._encrypt = btoa(enc); // WeaveCrypto expects ~~it.~~		this._encrypt = btoa(enc); // WeaveCrypto expects base64
	this._hmac = hmac;		this._hmac = hmac;
	}

Rnewman: Update.

2010-11-19T18:32:45Z

Update.

← Older revision		Revision as of 18:32, 19 November 2010
Line 24:		Line 24:

	'''tl;dr''': replace the passphrase with an AES key, which will be schlepped around using J-PAKE (so typing it is likely unnecessary). Use this key to indirectly encrypt the 'bulk' symkeys. No RSA involved.		'''tl;dr''': replace the passphrase with an AES key, which will be schlepped around using J-PAKE (so typing it is likely unnecessary). Use this key to indirectly encrypt the 'bulk' symkeys. No RSA involved.

			Existing passphrases will be upgraded to this scheme using PBKDF2.


Line 30:		Line 32:
	Rather than have a user enter a passphrase (which will likely be weak), we have already transitioned to having them generate a "sync key" (which they can replace if they so choose). This is 20 alphanumeric characters.		Rather than have a user enter a passphrase (which will likely be weak), we have already transitioned to having them generate a "sync key" (which they can replace if they so choose). This is 20 alphanumeric characters.

	We propose to expand this to 25 characters, enough for a ~~base36~~-encoded 128-bit AES key ~~(case-insensitive; 23 is enough if case-sensitive, but that's little advantage)~~. This avoids the use of PBKDF2 to routinely bootstrap the sync key into an AES key. Remove the ability for users to enter a key; it's always generated (giving us more confidence in the amount of entropy), and can be regenerated if desired.		We propose to expand this to 26 characters, enough for a base32-encoded 128-bit AES key. This avoids the use of PBKDF2 to routinely bootstrap the sync key into an AES key. Remove the ability for users to enter a key; it's always generated (giving us more confidence in the amount of entropy), and can be regenerated if desired.

	The length of this key is not a big issue: we intend to use J-PAKE for the (infrequent) migration of keys between devices. In any case, 25 is not significantly worse than 20 if typing it does enter the picture.		The length of this key is not a big issue: we intend to use J-PAKE for the (infrequent) migration of keys between devices. In any case, 26 is not significantly worse than 20 if typing it does enter the picture, and the use of a nice base32 alphabet makes keyboard entry less error-prone.

	As before, the sync key is stored on the client.		As before, the sync key is stored on the client.
Line 39:		Line 41:

	* Spot old version		* Spot old version
	* Get a salt (~~'''Proposal''': use the~~ Services.syncID from the meta/global object. ~~Presumably the~~ client will be bumping this…)		* Get a salt (Services.syncID from the meta/global object. The client will be bumping this…)
	* Apply PBKDF2 to salt and passphrase to yield our new AES key		* Apply PBKDF2 to salt and passphrase to yield our new AES key
	* Generate bulk keys, encrypt		* Generate bulk keys, encrypt
Line 47:		Line 49:
	So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or J-PAKE-style key distribution.		So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or J-PAKE-style key distribution.

	The generated ~~base36~~ alphanumeric key doesn't actually need to be decoded: it is used as input into ~~a pair of~~ hash operations which yield an encryption key and an HMAC key.		The generated base32 alphanumeric key doesn't actually need to be decoded: it is used as input into hash operations which yield an encryption key and an HMAC key.

			('''Note:''' however, for NSS convenience in the future, we intend to decode it. Still pending.)

			A known fixed string (which includes encryption algo details) and the username are used as input to SHA256-HMAC. The hash operations are chained.


			let m = this.keyStr;
			if (m) {
			// Reuse the hasher.
			let h = Utils.makeHMACHasher();

			// First key.
			let u = this.username;
			let k1 = Utils.makeHMACKey("" + HMAC_INPUT + u + "\x01");
			let enc = Utils.sha256HMACBytes(m, k1, h);

			// Second key: depends on the output of the first run.
			let k2 = Utils.makeHMACKey(enc + HMAC_INPUT + u + "\x02");
			let hmac = Utils.sha256HMACBytes(m, k2, h);

			// Save them.
			this._encrypt = btoa(enc); // WeaveCrypto expects it.
			this._hmac = hmac;
			}

	~~::key_encr = SHA256("encrypt:" + sync_key)~~
	~~::key_hmac = SHA256("hmac:" + sync_key)~~

	The outputs are then used during the key encryption process.		The outputs are then used during the key encryption process.
Line 58:		Line 82:
	The server stores one or more bulk keys: one default ("keys/default"), and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.		The server stores one or more bulk keys: one default ("keys/default"), and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.

	Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (~~'''TODO''':~~ per-session ~~or persistent caching?)~~		Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)

	~~::philikon: right now~~ they're ~~just cached by per-session. I think we want~~ to ~~store sync_key and data_key persistently though in the password manager (using identity~~.js) ~~and re-fetch any time the keys collection changes~~

	The timestamp on the collections record allows clients to invalidate their key cache when a new key is associated with a collection: the 'keys' collection will appear to have changed.		The timestamp on the collections record allows clients to invalidate their key cache when a new key is associated with a collection: the 'keys' collection will appear to have changed.
Line 70:		Line 92:
	=== HMAC ===		=== HMAC ===

	It's a good practice to use separate keys for HMAC and for encryption. ~~The bulk~~ keys ~~(stored encrypted on the server)~~ are ~~themselves used as input for exactly the same hash operations mentioned above. This yields a pair~~ of keys ~~that can be used for encryption and message authentication:~~		It's a good practice to use separate keys for HMAC and for encryption. Bulk keys are really pairs of keys, each of which is randomly generated.

	~~::data_encr = SHA256("encrypt:" + data_key)~~
	~~::data_hmac = SHA256("hmac:" + data_key)~~

	~~=== Objects ===~~

	~~Now is a great time to partition the "storage" namespace into "keys" and "data", rather than "keys" and everything else (e.g.~~, ~~"tabs"). This makes deletion~~ of ~~just keys or just data much more straightforward~~.

	~~::philikon: this would require~~ a ~~new server API whereas if we stick to~~ the ~~current model we can separate this from a server upgrade~~.		This approach was selected over having a single HMAC key because of the convenience for implementing some sharing-like scenarios.

	== Proposed flows ==		== Proposed flows ==
Line 86:		Line 101:

	* Generate a 128-bit Sync key (25 characters in base36). Store it as an Identity (as we do now.)		* Generate a 128-bit Sync key (25 characters in base36). Store it as an Identity (as we do now.)
	* Generate a random default key. Encrypt it with the sync key, upload it to the server. Store it as an Identity.		* Generate a random default key and HMAC. Encrypt it with the sync key, upload it to the server. Store it as an Identity.
	* Encrypt and upload collections in the obvious way.		* Encrypt and upload collections in the obvious way.

Line 98:		Line 113:
	* (On startup: invalidate/refresh key cache if keys collection has changed. I believe we make this fetch anyway...)		* (On startup: invalidate/refresh key cache if keys collection has changed. I believe we make this fetch anyway...)
	* Retrieve object from collection.		* Retrieve object from collection.
	* Verify HMAC using "keys/hmac". On failure, check for changed keys.
	* Look up key for collection name (defaulting to "keys/default"). Fetch if necessary.		* Look up key for collection name (defaulting to "keys/default"). Fetch if necessary.
			* Verify HMAC using appropriate per-collection or default key. On failure, check for changed keys.
	* Decrypt object.		* Decrypt object.

Philikon: /* Objects */

2010-10-29T22:57:31Z

Objects

← Older revision		Revision as of 22:57, 29 October 2010
Line 78:		Line 78:

	Now is a great time to partition the "storage" namespace into "keys" and "data", rather than "keys" and everything else (e.g., "tabs"). This makes deletion of just keys or just data much more straightforward.		Now is a great time to partition the "storage" namespace into "keys" and "data", rather than "keys" and everything else (e.g., "tabs"). This makes deletion of just keys or just data much more straightforward.

			::philikon: this would require a new server API whereas if we stick to the current model we can separate this from a server upgrade.

	== Proposed flows ==		== Proposed flows ==