Services/Sync/Developer/StorageFormat

From MozillaWiki
Jump to: navigation, search

Under a user's /storage path, there's a set of well-known records and storage formats used by the Weave Sync add-on. This document describes those records and their formats, e.g., JS-object format of the JSON-string payload.

The storage version of the data is stored as part of the meta/global payload.

Weave Basic Object

All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the Sync Server API, but for a given client storage version, only a subset may be used.

Versions 2, 3, and 5

The following describes the JS-object represented by the JSON-string record:

id string Record identifier. Starting with version 5 this SHOULD be exactly 12 characters from the base64url alphabet.
modified number Time when this record was last changed; set by the server
sortindex number Relative importance of this record
payload string String with data; usually a JSON-string
ttl integer (optional) The number of seconds to keep this record. After that time, this item will not be returned.

Example

{"id":"{3ab32a23-822d-424c-a4b8-88da8cf93eb2}0",
 "modified":1278109839.96,
 "sortindex":140,
 "payload":"{\"ciphertext\":\"e2zLWJYX\/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN\/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o\/ax\/3B2cM+6J2Frj6Q83OEW\/QBC8Q6\/XHgtJJlFi6fKWrG+XtFxS2\/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"encryption\":\"..\/crypto\/bookmarks\",\"IV\":\"GluQHjEH65G0gPk\/d\/OGmg==\",\"hmac\":\"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}"}

Changes from v1 -> v2

The client no longer stores and accesses the parentid and predecessorid fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.

Version 1

The following describes the JS-object represented by the JSON-string record:

id string Opaque identifying string for the record
parentid string Record id of a "parent"
predecessorid string Record id of a "predecessor"
modified number Time when this record was last changed; set by the server
sortindex number Relative importance of this record
payload string String with data; usually a JSON-string



Payload: Encrypted Data Object

Individual data engines, e.g., bookmarks, encrypt their Browser Objects payloads before packing it into the payload field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.

Version 5

Version 5 is similar to Version 2 and 3, except:

  • the encryption field (which specifies the key to use) has been dropped; key selection is now implied by the WBO's collection
  • the HMAC is now calculated with a separate key, used in its raw byte form rather than in base64 encoding.

The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle is used.

ciphertext string Encrypted JSON-stringified Browser Object
IV string Initialization vector used when decrypting the ciphertext
hmac string SHA256 HMAC in hex representation, computed on the base64 encoded version of the ciphertext, using the byte-representation of the HMAC key from the same bundle as the encryption key.


Example

{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
 "IV":"GluQHjEH65G0gPk/d/OGmg==",
 "hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}

Version 3

Like Version 2, except encryption is now a relative URL (relative to the Weave Basic Object's URL).

Example

{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
 "encryption":"../crypto/bookmarks",
 "IV":"GluQHjEH65G0gPk/d/OGmg==",
 "hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}

Version 2

The following describes the JS-object represented by the JSON-string payload:

ciphertext string Encrypted JSON-stringified Browser Object
encryption string Url of the crypto record containing the symmetric key to decrypt the ciphertext
IV string Initialization vector used when decrypting the ciphertext
hmac string SHA256 HMAC in hex format, computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.

Once the ciphertext is decrypted to get the Browser Object data, there are additional fields in the JS-object of the decrypted JSON-string:

id string The original id used when encrypting the payload to ensure the encrypted data is of the requested record
deleted boolean Set to true if this record represents a "delete" action

All client Browser Object data like bookmarks, history, and now additionally "clients" are encrypted.

Changes from v1 -> v2

The decrypted ciphertext JSON-string no longer wraps the Browser Object data with an extra [array].

At the payload level, there are two additional fields: IV and hmac. The IV is stored per-record instead of on the /crypto/<engine> so that a different IV can be used per record. The hmac can be used to verify that the encrypted payload has not been tampered with.

Once the ciphertext inside the payload is decrypted to access the Browser Object's payload, there are fields in addition to the usual Browser Object fields for that data type: id and deleted. For all encrypted data objects, there is an id field, which can be used to verify that the encrypted data is for the requested object. Instead of treating empty string payloads as "delete" records, the deleted field will be set to true.

The clients data is no longer a cleartext JSON payload and instead encrypts its Browser Object data like any other encrypted-data engine.

Version 1

The following describes the JS-object represented by the JSON-string payload:

ciphertext string Encrypted JSON-stringified Browser Object, which has been first wrapped with an extra [array], i.e., encrypt(stringify([browserObject]))
encryption string Url of the crypto record containing the symmetric key to decrypt the ciphertext

Empty string "" payloads represent a "delete" action for the record id.



Payload: meta/global

The payload of the meta/global record contains general metadata to describe data like versions and syncID.

Version 2, 3 and 5

The following describes the JS-object represented by the JSON-string payload:

storageVersion number Integer version that can stay the same across multiple client versions so that different versioned clients can sync with each other
syncID string Opaque string that changes when drastic changes happen to the overall data; this causes the client to drop cached keys/data
engines object A hash with fields of engine names and values of objects that contain version and syncID and behave like the version/syncID of this payload but on a per-engine level

Example

{"syncID":"JnvqPEn(6~",
 "storageVersion":3,
 "engines":{"clients":{"version":1,"syncID":"LwjtCQjdsV"},
            "bookmarks":{"version":1,"syncID":"ApPN6v8VY4"},
            "forms":{"version":1,"syncID":"UKeuhB.aOZ"},
            "tabs":{"version":1,"syncID":"G!nU*7H.7j"},
            "history":{"version":1,"syncID":"9Tvy_Vlb44"},
            "passwords":{"version":1,"syncID":"yfBi2v7Pp)"},
            "prefs":{"version":1,"syncID":"*eONx!6GXA"}}}

Changes from v1 -> v2

The storageVersion is now an integer (2) instead of a string, e.g., "1.1", and will only change when the storage format changes. It will no longer automatically changed to the newest client to write the record and instead only change on an incompatible storage format change.

There is an engines object that has engine names as its fields with corresponding value objects with two fields: syncID and version. The syncID for an engine changes when the data for that engine drastically changes, and the client will respond by re-syncing all of its data. The version of an engine acts like the storageVersion and indicates the Browser Object version for the corresponding data.

Version 1

The following describes the JS-object represented by the JSON-string payload:

storageVersion string Version of the newest client that synced
syncID string Opaque string that changes when drastic changes happen to the overall data; this causes the client to drop cached keys/data



Payload: crypto/keys

In storage Version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.

The keys WBO is encrypted and verified just like any other WBO, except a different key bundle is used. The key bundle for the keys WBO is derived from the Sync Key using an HKDF with HMAC-SHA256 as the HMAC function (see RFC 5869):

Pseudo-code:

 HMAC_INPUT = "Sync-AES_256_CBC-HMAC256"
 encryption_key = HMAC-SHA256(sync_key, "" + HMAC_INPUT + username + "\x01")
 hmac_key = HMAC-SHA256(sync_key, encryption_key + HMAC_INPUT + username + "\x02")

Here sync_key is the 16 byte representation of the Sync Key. To translate between the byte and user-readable translation, base32 is used, although with a slightly different alphabet than what RFC 4648 uses. For readability reasons, 'l' has been replaced with '8' and 'o' with '9':

 sync_key = decodeBase32(sync_key_ui.replace('8', 'l').replace('9', 'o'))
 sync_key_ui = encodeBase32(sync_key).replace('l', '8').replace('o', '8)


Version 5

default array Default key pair: [encryption key, HMAC key]
collections object Mapping of collection name to collection-specific key pairs: [encryption key, HMAC key]
collection string Currently defaulting to "crypto"

Example

{"id":"keys",
 "collection":"crypto",
 "collections":{},
 "default:['dGhlc2UtYXJlLWV4YWN0bHktMzItY2hhcmFjdGVycy4=',
           'eWV0LWFub3RoZXItc2V0LW9mLTMyLWNoYXJhY3RlcnM=']}

Payload: keys/pubkey

This is the public key part of the keypair used to encrypt the symmetric keys used to encrypt data.

Version 3

Like Version 2, except that privateKeyUri is now relative to the public key's URL.

Example

{"type":"pubkey",
 "keyData":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArd0DVRP1Ve1ZLZDbdcJvuf/S/zdsy+9qwakS0A2z6D2aUEd0uH61SacUw4StxibSYDcU+4M+zsCZugONoSUCBGcxKAhqStcpxY8SPTwAc3CTMFiTVMHL7rtjq9NYhYT2qFBdrLOmS3nCczJYjTzdig6RhA+GrHzaqmXPXJakQ8uyVYpsWfKj++o+VzNKLXlxRhmw170hzXgtuwVBK5iHCj+ndEQvVOlTQ1g7YFGiMMtOYs4fpc6CFcZm3w4FdmjADluzPk+ZugQkn/Sz0/JZInb0DnSHuRAt86PLfnDiMHQ5GwJynGsE/EYCYZrWrsTp/6y7OBx1Qk4tBuQhbTHtewIDAQAB",
 "privateKeyUri":"privkey"}

Version 1 and 2

The following describes the JS-object represented by the JSON-string payload:

keyData string Value of the public key
privateKeyUri string Url to the corresponding private key
type string Type of the key: "pubkey"



Payload: keys/privkey

This is the private key part of the keypair used to decrypt the symmetric keys used to decrypt data. The private key is encrypted with a key derived from the secret phrase.

Version 3

Like Version 2, except publicKeyUri is now relative to the private key's URL.

Example

{"type":"privkey",
 "salt":"Ag76sNXjS4TVj9r2CeEo/w==",
 "iv":"OX3JxwaP1U5Hh+0qaHRtfg==",
 "keyData":"zOAhty69Qvjl49jjCntvlYK88Hv+KBgrfx6o9HIXdxplSkwOSS6Q2P1ee6KXZor7lBEzuedNMJbog3VrKkNwvTKe5DVuNx8UJVujn1C/A6FR6ml3v/Vimg2tC3MfUHdCBQcN1MX95RM2comwaUdPz+5iR3VtoqE8wHRV+yB7kXD4pbewepQaqCqKrzsZDSGGrBD0LyhQF/YuDkcwadRJs0t2JCEOMjQtqys7nbjOmAOuC4duL/1iWjb6FaTUAfB2CUI0okpjNkLf7XHHiJZOwjYEkpmoGCEXZZvwXtdj3wu30v8EL2tfdGWh7eYhy715fbcMWGBsF6gDpUjwjF+ihfezYZe9FcmA/8neSNJgzoXO4NwNMNBUKCJcWt2y8gZUhLzTvVf103oKqu/bzHQTihcU6RMLU1HVRbr6H3qeck26pDcSqk/ZRJld0WFtx9A+pVD1gzEwCLQ4BCmVMoMMFdyXVH4k0boxjckvIZUUeLpQeGVUj4xb86n48pTZco5AETmRqqVfGK31WjFY9m83+QgwqMCZR9qFMMVJPeyE8tpoqQXhxfJrNyHITgye/EPtVS5s0iLfWq/Qe9jHvIfKa5vlXm5kxyMBHacD07QrZqh4sy8vGPmJfwY8BmhcbeGjqx9yANX5O0KjVJlk2NtJ1yFd4PcMd4VJuVB3hMUPQwP6eZSfe+gj2n2epnL+KesaL+OgD8YRFg9ZyBPqz6VaHfSdLw5e/vmhLzLyiZkYT9K5/7UqFm/7W/VpMselZV7O1sYojIbWeIOcHtJ9wqeOQ20nl6phQJVyKLOY7ApOsLoqYfzkZoG/CrgYKheY20niiwqH+qG6V/rMGaM4RCfpI/2bmWJkhlxyuv+20yMWzRad3H1WpfgxMF0jz4OA3QQKxB+dF/+/50mpveJs7Plgfm3zH/whl+gfOAX6JqGJBFfCA7XUwxDW4EMyGnYj6YMbv3iTHxA30KpA//KmpdF+x6qCP3kbrBHLpNcCk8fqqP6xwojophid/lTieOyEgNL8iGTRpF+XICi+CJNFrIpGdONrmXcGMql2c28V3azX1NGscmXUZstNp0ckw9IuxJovgATgiOTUM+QLDjq2QVCP3Jxol+klK+0mHUGqVCMeQaEb/aG71LbPQNLNBk8SzT59QbBiCOgc6zVx0TtYoLNCbL0hMrocGAHZw1W82y24uAPuDJKMfyVEWJ4HBn6sonzeag1M2WuGJVJJ+o+QuxE1wXaPIYfyzWc6T3741TIc7gl669YjMvdbGxtC57/bjX+heWJyedxlbL9mtoPEApfWsYxrVRGodGXgeYf365HGhU6wjRU4Nq3Tkg/ptEbfPYDv1E83cEwc9UE7yGFXucQZbfggrxrLttPMNABX6ZWnsNOIiuQcolxcebjpBjDQJoE9GD+4XKFXCLcd7gMaS9EOgCig5o9sk6uC2D3jJWVlgCiZoFns2tqW8x2EvJFCPor4hsAgNqMEiyyLwxEiV8EYEnx1uFxxjF4irTY/TzKF0msPNpBuplra26qefrAcqXYgXH2bdx1jnfH7DRMijz41XcDcqWi+VR14SuSdoqT4WYHBJE8f0U9Ng8TPSvk9QLed8eFJmpirDRYTvXkJ9DJP9E39wvAH/W+6cVM//qCENyU=",
 "publicKeyUri":"pubkey"}

Version 1 and 2

The following describes the JS-object represented by the JSON-string payload:

salt string Salt used with the secret phrase
iv string Initialization vector used when decrypting the keyData
keyData string Encrypted private key that needs the secret phrase, salt, and iv to decrypt
publicKeyUri string Url to the corresponding public key
type string Type of the key: "privkey"

Payload: crypto/<engine>

This keyring contains the symmetric key used to encrypt/decrypt Encrypted Data Objects.

Version 3

Like Version 2, except the URLs in the keyring hash are now relative to the symmetric key's URL.

Example

{"keyring":{"../keys/pubkey":{"wrapped":"P3nQWhQtUHpUpdzI8RD0DkW3OSnNcrcOGhWBM6p7a9imIJ3K8RpsEVnhCIBmkprg40lddRgH/o+vDi2jYovxkFKUzM3izLDa016JZud7GWlx50WNpxaWmst76DlXtjnLDK44tJw2UzRJI6jKq/k5rm8anUNrBYWqS7v97+OiHG4viFI9IQCvRrCY0ow+Z37NppoArW/kbna8Dl4UQAkVlQNOgTjvWW6BtiXd3HN7iA4LFgGTJCuQhfMAZel0fRq9vkW6XCZr6OJqGeEXbfmkfwAMQ3S0THCeki6ejAhiGW70OXF+gBqBrYvTiqyVXuRNKRY6TtH+9g8Vb9aj2O6F/w==",
                              "hmac":"682bac23003ceaf660391b8342bb423f2e52200b238eacb9640dc39e69c72a72"}}}

Version 2

The following describes the JS-object represented by the JSON-string payload:

keyring object A hash with fields of key urls used to decrypt the field's value of an object containing the wrapped symmetric key

The following describes the JS-object for keyring:

wrapped string The encrypted symmetric key that is decrypted using the key located at the url that indexed to this entry
hmac string SHA256 HMAC of wrapped and the key derived from the decrypted base64 private key string that would be used to decrypt the wrapped key

Changes from v1 -> v2

There is only one field, keyring, in the payload now that bulkIV is removed.

The keyring object still has field entries of the key to decrypt the wrapped symmetric key, but instead of the wrapped key being the value of the field, it is now an object with two fields: wrapped and hmac. The wrapped symmetric key from v1 is now the value of wrapped. The hmac can be used to verify that the wrapped key has not been tampered with.

Version 1

The following describes the JS-object represented by the JSON-string payload:

bulkIV string Initialization vector used to encrypt/decrypt all data for the corresponding engine
keyring object A hash with fields of key urls used to decrypt the field's value of a wrapped symmetric key