Services/Sync/Developer/StorageFormat - Revision history

Rnewman: /* Version 5 */

2011-08-11T00:40:59Z

Version 5

← Older revision		Revision as of 00:40, 11 August 2011
Line 86:		Line 86:
	== Version 5 ==		== Version 5 ==

	Version 5 is similar to Version 2 and 3, except ~~'''encryption''' has been dropped and the HMAC is now calculated with a separate key which is no longer used in its base64 representation but in its raw byte form.~~		Version 5 is similar to Version 2 and 3, except:

	The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle ~~should be~~ used.		* the '''encryption''' field (which specifies the key to use) has been dropped; key selection is now implied by the WBO's collection
			* the HMAC is now calculated with a separate key, used in its raw byte form rather than in base64 encoding.

			The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle is used.

	{\| cellpadding=5		{\| cellpadding=5
Line 110:		Line 113:
	"IV":"GluQHjEH65G0gPk/d/OGmg==",		"IV":"GluQHjEH65G0gPk/d/OGmg==",
	"hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}</pre>		"hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}</pre>


	== Version 3 ==		== Version 3 ==

Philikon: /* Versions 2, 3, and 5 */

2010-12-12T19:44:54Z

Versions 2, 3, and 5

← Older revision		Revision as of 19:44, 12 December 2010
Line 15:		Line 15:
	! id		! id
	\| string		\| string
	\| Record identifier. Starting with version 5 ~~it is highly recommended to use~~ exactly 12 characters from the base64url alphabet ~~for this~~.		\| Record identifier. Starting with version 5 this SHOULD be exactly 12 characters from the base64url alphabet.
	\|-		\|-
	! modified		! modified

Philikon: /* Versions 2, 3, and 5 */

2010-12-12T19:24:28Z

Versions 2, 3, and 5

← Older revision		Revision as of 19:24, 12 December 2010
Line 15:		Line 15:
	! id		! id
	\| string		\| string
	\| ~~Opaque identifying string~~ for ~~the record~~		\| Record identifier. Starting with version 5 it is highly recommended to use exactly 12 characters from the base64url alphabet for this.
	\|-		\|-
	! modified		! modified

Philikon at 19:22, 12 December 2010

2010-12-12T19:22:12Z

← Older revision		Revision as of 19:22, 12 December 2010
Line 7:		Line 7:
	All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the [[Labs/Weave/API\|Sync Server API]], but for a given client storage version, only a subset may be used.		All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the [[Labs/Weave/API\|Sync Server API]], but for a given client storage version, only a subset may be used.

	== Versions 2, 3, and 4 ==		== Versions 2, 3, and 5 ==

	The following describes the JS-object represented by the JSON-string record:		The following describes the JS-object represented by the JSON-string record:
Line 84:		Line 84:
	Individual data engines, e.g., bookmarks, encrypt their [[Labs/Weave/Developer/BrowserObjects\|Browser Objects]] payloads before packing it into the <tt>payload</tt> field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.		Individual data engines, e.g., bookmarks, encrypt their [[Labs/Weave/Developer/BrowserObjects\|Browser Objects]] payloads before packing it into the <tt>payload</tt> field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.

	== Version 4 ==		== Version 5 ==

	Version 4 is similar to Version 2 and 3, except '''encryption''' has been dropped and the HMAC is now calculated with a separate key which is no longer used in its base64 representation but in its raw byte form.		Version 5 is similar to Version 2 and 3, except '''encryption''' has been dropped and the HMAC is now calculated with a separate key which is no longer used in its base64 representation but in its raw byte form.

	The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle should be used.		The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle should be used.
Line 193:		Line 193:
	The payload of the <tt>meta/global</tt> record contains general metadata to describe data like versions and syncID.		The payload of the <tt>meta/global</tt> record contains general metadata to describe data like versions and syncID.

	== Version 2, 3 and 4 ==		== Version 2, 3 and 5 ==

	The following describes the JS-object represented by the JSON-string payload:		The following describes the JS-object represented by the JSON-string payload:
Line 249:		Line 249:
	= Payload: crypto/keys =		= Payload: crypto/keys =

	In storage Version 4, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.		In storage Version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.

	The '''keys''' WBO is encrypted and verified just like any other WBO, except a different key bundle is used. The key bundle for the '''keys''' WBO is derived from the Sync Key using an HKDF with HMAC-SHA256 as the HMAC function (see [http://tools.ietf.org/html/rfc5869 RFC 5869]):		The '''keys''' WBO is encrypted and verified just like any other WBO, except a different key bundle is used. The key bundle for the '''keys''' WBO is derived from the Sync Key using an HKDF with HMAC-SHA256 as the HMAC function (see [http://tools.ietf.org/html/rfc5869 RFC 5869]):
Line 265:		Line 265:


	== Version 4 ==		== Version 5 ==

	{\| cellpadding=5		{\| cellpadding=5

Philikon: /* Payload: crypto/keys */

2010-12-05T02:17:08Z

Payload: crypto/keys

← Older revision		Revision as of 02:17, 5 December 2010
Line 262:		Line 262:

	sync_key = decodeBase32(sync_key_ui.replace('8', 'l').replace('9', 'o'))		sync_key = decodeBase32(sync_key_ui.replace('8', 'l').replace('9', 'o'))
	sync_key_ui = encodeBase32(sync_key).replace('l', '8').replace('o', '8))		sync_key_ui = encodeBase32(sync_key).replace('l', '8').replace('o', '8)

Philikon: /* Payload: crypto/keys */

2010-12-05T02:16:10Z

Payload: crypto/keys

@@ Line 249: / Line 249: @@
 = Payload: crypto/keys =
-In storage format 4, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate and kept in key pairs.
+In storage Version 4, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.
 == Version 4 ==
@@ Line 266: / Line 280: @@
 | Currently defaulting to "crypto"
 |}
 = Payload: keys/pubkey =

Philikon: /* Payload: Encrypted Data Object */

2010-12-05T01:58:10Z

Payload: Encrypted Data Object

← Older revision		Revision as of 01:58, 5 December 2010
Line 86:		Line 86:
	== Version 4 ==		== Version 4 ==

	~~Like~~ Version 2 and 3, except '''encryption''' has been dropped. The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle should be used.		Version 4 is similar to Version 2 and 3, except '''encryption''' has been dropped and the HMAC is now calculated with a separate key which is no longer used in its base64 representation but in its raw byte form.

			The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle should be used.

			{\| cellpadding=5
			! ciphertext
			\| string
			\| Encrypted JSON-stringified Browser Object
			\|-
			! IV
			\| string
			\| Initialization vector used when decrypting the <tt>ciphertext</tt>
			\|-
			! hmac
			\| string
			\| SHA256 HMAC in hex representation, computed on the base64 encoded version of the ciphertext, using the byte-representation of the HMAC key from the same bundle as the encryption key.
			\|}


	=== Example ===		=== Example ===
Line 125:		Line 142:
	! hmac		! hmac
	\| string		\| string
	\| SHA256 HMAC computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm~~. In addition, the hmac is stored in the weave object in HEX format, not base64 like everything else~~.		\| SHA256 HMAC in hex format, computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.
	\|}		\|}

Philikon: /* Payload: crypto/keys */

2010-12-04T18:00:05Z

Payload: crypto/keys

← Older revision		Revision as of 18:00, 4 December 2010
Line 231:		Line 231:

	= Payload: crypto/keys =		= Payload: crypto/keys =

			In storage format 4, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate and kept in key pairs.

	== Version 4 ==		== Version 4 ==

Philikon: /* Version 4 = */

2010-12-04T17:58:05Z

Version 4 =

← Older revision		Revision as of 17:58, 4 December 2010
Line 232:		Line 232:
	= Payload: crypto/keys =		= Payload: crypto/keys =

	== Version 4 ===		== Version 4 ==

	{\| cellpadding=5		{\| cellpadding=5
	! default		! default
	\| array: [encryption key, HMAC key]		\| array
	~~\| Default key pair~~		\| Default key pair: [encryption key, HMAC key]
	\|-		\|-
	! collections		! collections
	\| object		\| object
	\| Mapping of collection name to key ~~pair arrays~~		\| Mapping of collection name to collection-specific key pairs: [encryption key, HMAC key]
	\|-		\|-
	! collection		! collection

Philikon: /* Payload: keys/pubkey */

2010-12-04T17:54:29Z

Payload: keys/pubkey

← Older revision		Revision as of 17:54, 4 December 2010
Line 229:		Line 229:

	----		----

			= Payload: crypto/keys =

			== Version 4 ===

			{\| cellpadding=5
			! default
			\| array: [encryption key, HMAC key]
			\| Default key pair
			\|-
			! collections
			\| object
			\| Mapping of collection name to key pair arrays
			\|-
			! collection
			\| string
			\| Currently defaulting to "crypto"
			\|}

	= Payload: keys/pubkey =		= Payload: keys/pubkey =