MozillaWiki

Services/Sync/Developer/StorageFormat: Difference between revisions

Page Discussion

Services/Sync/Developer/StorageFormat (view source)

Revision as of 00:40, 11 August 2011

7,199 bytes added ,  11 August 2011
m
→‎Version 5
 
(34 intermediate revisions by 2 users not shown)
Line 7: Line 7:
All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the [[Labs/Weave/API|Sync Server API]], but for a given client storage version, only a subset may be used.
All records from the server come back as a JSON string that represent a JS-object with some attributes. The whole object is described by the [[Labs/Weave/API|Sync Server API]], but for a given client storage version, only a subset may be used.


== Version 2 ==
== Versions 2, 3, and 5  ==


The following describes the JS-object represented by the JSON-string record:
The following describes the JS-object represented by the JSON-string record:  


{| cellpadding=5
{| cellpadding="5"
! id
|-
| string
! id  
| Opaque identifying string for the record
| string  
| Record identifier. Starting with version 5 this SHOULD be exactly 12 characters from the base64url alphabet.
|-
|-
! modified
! modified  
| number
| number  
| Time when this record was last changed; set by the server
| Time when this record was last changed; set by the server
|-
|-
! sortindex
! sortindex  
| number
| number  
| Relative importance of this record
| Relative importance of this record
|-
|-
! payload
! payload  
| string
| string  
| String with data; usually a JSON-string
| String with data; usually a JSON-string
|-
! ttl
| integer (optional)
| The number of seconds to keep this record. After that time, this item will not be returned.
|}
|}


== Changes from v1 -> v2 ==
=== Example  ===
 
<pre>
{"id":"{3ab32a23-822d-424c-a4b8-88da8cf93eb2}0",
"modified":1278109839.96,
"sortindex":140,
"payload":"{\"ciphertext\":\"e2zLWJYX\/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN\/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o\/ax\/3B2cM+6J2Frj6Q83OEW\/QBC8Q6\/XHgtJJlFi6fKWrG+XtFxS2\/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"encryption\":\"..\/crypto\/bookmarks\",\"IV\":\"GluQHjEH65G0gPk\/d\/OGmg==\",\"hmac\":\"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}"}</pre>
 
=== Changes from v1 -&gt; v2 ===


The client no longer stores and accesses the <tt>parentid</tt> and <tt>predecessorid</tt> fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.
The client no longer stores and accesses the <tt>parentid</tt> and <tt>predecessorid</tt> fields in the WBO and instead stores them encrypted inside the Browser Object. Only bookmarks used these fields.
Line 66: Line 79:


----
----
= Payload: Encrypted Data Object =
Individual data engines, e.g., bookmarks, encrypt their [[Labs/Weave/Developer/BrowserObjects|Browser Objects]] payloads before packing it into the <tt>payload</tt> field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.
== Version 5 ==
Version 5 is similar to Version 2 and 3, except:
* the '''encryption''' field (which specifies the key to use) has been dropped; key selection is now implied by the WBO's collection
* the HMAC is now calculated with a separate key, used in its raw byte form rather than in base64 encoding.
The keys with which to verify and decrypt a WBO are now determined based on the collection name. If collection-specific keys do not exist, the default key bundle is used.
{| cellpadding=5
! ciphertext
| string
| Encrypted JSON-stringified Browser Object
|-
! IV
| string
| Initialization vector used when decrypting the <tt>ciphertext</tt>
|-
! hmac
| string
| SHA256 HMAC in hex representation, computed on the base64 encoded version of the ciphertext, using the byte-representation of the HMAC key from the same bundle as the encryption key.
|}




=== Example ===


= Payload: Encrypted Data Object =
<pre>{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
"IV":"GluQHjEH65G0gPk/d/OGmg==",
"hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}</pre>
 
== Version 3 ==
 
Like Version 2, except '''encryption''' is now a relative URL (relative to the Weave Basic Object's URL).
 
=== Example ===


Individual data engines, e.g., bookmarks, encrypt their [[Labs/Weave/Developer/BrowserObjects|Browser Objects]] payloads before packing it into the <tt>payload</tt> field of a Weave Basic Object. There is additional metadata about the encryption to help decrypt the BrowserObject payload.
<pre>{"ciphertext":"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=",
"encryption":"../crypto/bookmarks",
"IV":"GluQHjEH65G0gPk/d/OGmg==",
"hmac":"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289"}</pre>


== Version 2 ==
== Version 2 ==
Line 92: Line 144:
! hmac
! hmac
| string
| string
| SHA256 HMAC computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.  In addition, the hmac is stored in the weave object in HEX format, not base64 like everything else.
| SHA256 HMAC in hex format, computed on the base64 encoded version of the ciphertext, using a base64 encoded version of the key used to encrypt the ciphertext in the first place as the key to the hmac algorithm.
|}
|}


Line 109: Line 161:
All client Browser Object data like bookmarks, history, and now additionally "clients" are encrypted.
All client Browser Object data like bookmarks, history, and now additionally "clients" are encrypted.


== Changes from v1 -> v2 ==
=== Changes from v1 -> v2 ===


The decrypted <tt>ciphertext</tt> JSON-string no longer wraps the Browser Object data with an extra [array].
The decrypted <tt>ciphertext</tt> JSON-string no longer wraps the Browser Object data with an extra [array].
Line 138: Line 190:


----
----


= Payload: meta/global =
= Payload: meta/global =
Line 145: Line 195:
The payload of the <tt>meta/global</tt> record contains general metadata to describe data like versions and syncID.
The payload of the <tt>meta/global</tt> record contains general metadata to describe data like versions and syncID.


== Version 2 ==
== Version 2, 3 and 5 ==


The following describes the JS-object represented by the JSON-string payload:
The following describes the JS-object represented by the JSON-string payload:
Line 163: Line 213:
|}
|}


== Changes from v1 -> v2 ==
=== Example ===
 
<pre>{"syncID":"JnvqPEn(6~",
"storageVersion":3,
"engines":{"clients":{"version":1,"syncID":"LwjtCQjdsV"},
            "bookmarks":{"version":1,"syncID":"ApPN6v8VY4"},
            "forms":{"version":1,"syncID":"UKeuhB.aOZ"},
            "tabs":{"version":1,"syncID":"G!nU*7H.7j"},
            "history":{"version":1,"syncID":"9Tvy_Vlb44"},
            "passwords":{"version":1,"syncID":"yfBi2v7Pp)"},
            "prefs":{"version":1,"syncID":"*eONx!6GXA"}}}</pre>
 
=== Changes from v1 -> v2 ===


The <tt>storageVersion</tt> is now an integer (2) instead of a string, e.g., "1.1", and will only change when the storage format changes. It will no longer automatically changed to the newest client to write the record and instead only change on an incompatible storage format change.
The <tt>storageVersion</tt> is now an integer (2) instead of a string, e.g., "1.1", and will only change when the storage format changes. It will no longer automatically changed to the newest client to write the record and instead only change on an incompatible storage format change.
Line 187: Line 249:
----
----


= Payload: crypto/keys =
In storage Version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO. Encryption and HMAC keys are separate keys and kept in key pairs.


The '''keys''' WBO is encrypted and verified just like any other WBO, except a different key bundle is used. The key bundle for the '''keys''' WBO is derived from the Sync Key using an HKDF with HMAC-SHA256 as the HMAC function (see [http://tools.ietf.org/html/rfc5869 RFC 5869]):


= Payload: keys/pubkey =
Pseudo-code:
 
  HMAC_INPUT = "Sync-AES_256_CBC-HMAC256"
  encryption_key = HMAC-SHA256(sync_key, "" + HMAC_INPUT + username + "\x01")
  hmac_key = HMAC-SHA256(sync_key, encryption_key + HMAC_INPUT + username + "\x02")
 
Here '''sync_key''' is the 16 byte representation of the Sync Key. To translate between the byte and user-readable translation, base32 is used, although with a slightly different alphabet than what [http://tools.ietf.org/html/rfc4648 RFC 4648] uses. For readability reasons, 'l' has been replaced with '8' and 'o' with '9':


This is the public key part of the keypair used to encrypt the symmetric keys used to encrypt data.
  sync_key = decodeBase32(sync_key_ui.replace('8', 'l').replace('9', 'o'))
  sync_key_ui = encodeBase32(sync_key).replace('l', '8').replace('o', '8)


== Version 2 ==


The following describes the JS-object represented by the JSON-string payload:
== Version 5 ==


{| cellpadding=5
{| cellpadding=5
! keyData
! default
| string
| array
| Value of the public key
| Default key pair: [encryption key, HMAC key]
|-
|-
! privateKeyUri
! collections
| string
| object
| Url to the corresponding private key
| Mapping of collection name to collection-specific key pairs: [encryption key, HMAC key]
|-
|-
! type
! collection
| string
| string
| Type of the key: "pubkey"
| Currently defaulting to "crypto"
|}
|}


== Changes from v1 -> v2 ==
=== Example ===
 
{"id":"keys",
  "collection":"crypto",
  "collections":{},
  "default:['dGhlc2UtYXJlLWV4YWN0bHktMzItY2hhcmFjdGVycy4=',
            'eWV0LWFub3RoZXItc2V0LW9mLTMyLWNoYXJhY3RlcnM=']}
 
= Payload: keys/pubkey =


No changes.
This is the public key part of the keypair used to encrypt the symmetric keys used to encrypt data.


== Version 1 ==
== Version 3 ==
 
Like Version 2, except that '''privateKeyUri''' is now relative to the public key's URL.
 
=== Example ===
 
<pre>{"type":"pubkey",
"keyData":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArd0DVRP1Ve1ZLZDbdcJvuf/S/zdsy+9qwakS0A2z6D2aUEd0uH61SacUw4StxibSYDcU+4M+zsCZugONoSUCBGcxKAhqStcpxY8SPTwAc3CTMFiTVMHL7rtjq9NYhYT2qFBdrLOmS3nCczJYjTzdig6RhA+GrHzaqmXPXJakQ8uyVYpsWfKj++o+VzNKLXlxRhmw170hzXgtuwVBK5iHCj+ndEQvVOlTQ1g7YFGiMMtOYs4fpc6CFcZm3w4FdmjADluzPk+ZugQkn/Sz0/JZInb0DnSHuRAt86PLfnDiMHQ5GwJynGsE/EYCYZrWrsTp/6y7OBx1Qk4tBuQhbTHtewIDAQAB",
"privateKeyUri":"privkey"}</pre>
 
== Version 1 and 2 ==


The following describes the JS-object represented by the JSON-string payload:
The following describes the JS-object represented by the JSON-string payload:
Line 236: Line 326:


----
----


= Payload: keys/privkey =
= Payload: keys/privkey =
Line 243: Line 331:
This is the private key part of the keypair used to decrypt the symmetric keys used to decrypt data. The private key is encrypted with a key derived from the secret phrase.
This is the private key part of the keypair used to decrypt the symmetric keys used to decrypt data. The private key is encrypted with a key derived from the secret phrase.


== Version 2 ==
== Version 3 ==
 
The following describes the JS-object represented by the JSON-string payload:


{| cellpadding=5
Like Version 2, except '''publicKeyUri''' is now relative to the private key's URL.
! salt
| string
| Salt used with the secret phrase
|-
! iv
| string
| Initialization vector used when decrypting the <tt>keyData</tt>
|-
! keyData
| string
| Encrypted private key that needs the secret phrase, <tt>salt</tt>, and <tt>iv</tt> to decrypt
|-
! publicKeyUri
| string
| Url to the corresponding public key
|-
! type
| string
| Type of the key: "privkey"
|}


== Changes from v1 -> v2 ==
=== Example ===


No changes.
<pre>{"type":"privkey",
"salt":"Ag76sNXjS4TVj9r2CeEo/w==",
"iv":"OX3JxwaP1U5Hh+0qaHRtfg==",
"keyData":"zOAhty69Qvjl49jjCntvlYK88Hv+KBgrfx6o9HIXdxplSkwOSS6Q2P1ee6KXZor7lBEzuedNMJbog3VrKkNwvTKe5DVuNx8UJVujn1C/A6FR6ml3v/Vimg2tC3MfUHdCBQcN1MX95RM2comwaUdPz+5iR3VtoqE8wHRV+yB7kXD4pbewepQaqCqKrzsZDSGGrBD0LyhQF/YuDkcwadRJs0t2JCEOMjQtqys7nbjOmAOuC4duL/1iWjb6FaTUAfB2CUI0okpjNkLf7XHHiJZOwjYEkpmoGCEXZZvwXtdj3wu30v8EL2tfdGWh7eYhy715fbcMWGBsF6gDpUjwjF+ihfezYZe9FcmA/8neSNJgzoXO4NwNMNBUKCJcWt2y8gZUhLzTvVf103oKqu/bzHQTihcU6RMLU1HVRbr6H3qeck26pDcSqk/ZRJld0WFtx9A+pVD1gzEwCLQ4BCmVMoMMFdyXVH4k0boxjckvIZUUeLpQeGVUj4xb86n48pTZco5AETmRqqVfGK31WjFY9m83+QgwqMCZR9qFMMVJPeyE8tpoqQXhxfJrNyHITgye/EPtVS5s0iLfWq/Qe9jHvIfKa5vlXm5kxyMBHacD07QrZqh4sy8vGPmJfwY8BmhcbeGjqx9yANX5O0KjVJlk2NtJ1yFd4PcMd4VJuVB3hMUPQwP6eZSfe+gj2n2epnL+KesaL+OgD8YRFg9ZyBPqz6VaHfSdLw5e/vmhLzLyiZkYT9K5/7UqFm/7W/VpMselZV7O1sYojIbWeIOcHtJ9wqeOQ20nl6phQJVyKLOY7ApOsLoqYfzkZoG/CrgYKheY20niiwqH+qG6V/rMGaM4RCfpI/2bmWJkhlxyuv+20yMWzRad3H1WpfgxMF0jz4OA3QQKxB+dF/+/50mpveJs7Plgfm3zH/whl+gfOAX6JqGJBFfCA7XUwxDW4EMyGnYj6YMbv3iTHxA30KpA//KmpdF+x6qCP3kbrBHLpNcCk8fqqP6xwojophid/lTieOyEgNL8iGTRpF+XICi+CJNFrIpGdONrmXcGMql2c28V3azX1NGscmXUZstNp0ckw9IuxJovgATgiOTUM+QLDjq2QVCP3Jxol+klK+0mHUGqVCMeQaEb/aG71LbPQNLNBk8SzT59QbBiCOgc6zVx0TtYoLNCbL0hMrocGAHZw1W82y24uAPuDJKMfyVEWJ4HBn6sonzeag1M2WuGJVJJ+o+QuxE1wXaPIYfyzWc6T3741TIc7gl669YjMvdbGxtC57/bjX+heWJyedxlbL9mtoPEApfWsYxrVRGodGXgeYf365HGhU6wjRU4Nq3Tkg/ptEbfPYDv1E83cEwc9UE7yGFXucQZbfggrxrLttPMNABX6ZWnsNOIiuQcolxcebjpBjDQJoE9GD+4XKFXCLcd7gMaS9EOgCig5o9sk6uC2D3jJWVlgCiZoFns2tqW8x2EvJFCPor4hsAgNqMEiyyLwxEiV8EYEnx1uFxxjF4irTY/TzKF0msPNpBuplra26qefrAcqXYgXH2bdx1jnfH7DRMijz41XcDcqWi+VR14SuSdoqT4WYHBJE8f0U9Ng8TPSvk9QLed8eFJmpirDRYTvXkJ9DJP9E39wvAH/W+6cVM//qCENyU=",
"publicKeyUri":"pubkey"}</pre>


== Version 1 ==
== Version 1 and 2 ==


The following describes the JS-object represented by the JSON-string payload:
The following describes the JS-object represented by the JSON-string payload:
Line 299: Line 369:
|}
|}


----


= Payload: crypto/&lt;engine&gt;  =


----
This keyring contains the symmetric key used to encrypt/decrypt Encrypted Data Objects.


== Version 3  ==


Like Version 2, except the URLs in the '''keyring''' hash are now relative to the symmetric key's URL.


= Payload: crypto/<engine> =
=== Example ===


This keyring contains the symmetric key used to encrypt/decrypt Encrypted Data Objects.
<pre>{"keyring":{"../keys/pubkey":{"wrapped":"P3nQWhQtUHpUpdzI8RD0DkW3OSnNcrcOGhWBM6p7a9imIJ3K8RpsEVnhCIBmkprg40lddRgH/o+vDi2jYovxkFKUzM3izLDa016JZud7GWlx50WNpxaWmst76DlXtjnLDK44tJw2UzRJI6jKq/k5rm8anUNrBYWqS7v97+OiHG4viFI9IQCvRrCY0ow+Z37NppoArW/kbna8Dl4UQAkVlQNOgTjvWW6BtiXd3HN7iA4LFgGTJCuQhfMAZel0fRq9vkW6XCZr6OJqGeEXbfmkfwAMQ3S0THCeki6ejAhiGW70OXF+gBqBrYvTiqyVXuRNKRY6TtH+9g8Vb9aj2O6F/w==",
                              "hmac":"682bac23003ceaf660391b8342bb423f2e52200b238eacb9640dc39e69c72a72"}}}</pre>


== Version 2 ==
== Version 2 ==


The following describes the JS-object represented by the JSON-string payload:
The following describes the JS-object represented by the JSON-string payload:  


{| cellpadding=5
{| cellpadding="5"
! keyring
|-
| object
! keyring  
| A hash with fields of key urls used to decrypt the field's value of an object containing the wrapped symmetric key  
| object  
| A hash with fields of key urls used to decrypt the field's value of an object containing the wrapped symmetric key
|}
|}


The following describes the JS-object for <tt>keyring</tt>:
The following describes the JS-object for <tt>keyring</tt>:  


{| cellpadding=5
{| cellpadding="5"
! wrapped
|-
| string
! wrapped  
| string  
| The encrypted symmetric key that is decrypted using the key located at the url that indexed to this entry
| The encrypted symmetric key that is decrypted using the key located at the url that indexed to this entry
|-
|-
! hmac
! hmac  
| string
| string  
| SHA256 HMAC of <tt>wrapped</tt> and the key derived from the decrypted base64 private key string that would be used to decrypt the <tt>wrapped</tt> key
| SHA256 HMAC of <tt>wrapped</tt> and the key derived from the decrypted base64 private key string that would be used to decrypt the <tt>wrapped</tt> key
|}
|}


== Changes from v1 -> v2 ==
=== Changes from v1 -&gt; v2 ===


There is only one field, <tt>keyring</tt>, in the <tt>payload</tt> now that <tt>bulkIV</tt> is removed.
There is only one field, <tt>keyring</tt>, in the <tt>payload</tt> now that <tt>bulkIV</tt> is removed.  


The <tt>keyring</tt> object still has field entries of the key to decrypt the wrapped symmetric key, but instead of the wrapped key being the value of the field, it is now an object with two fields: <tt>wrapped</tt> and <tt>hmac</tt>. The wrapped symmetric key from v1 is now the value of <tt>wrapped</tt>. The <tt>hmac</tt> can be used to verify that the wrapped key has not been tampered with.
The <tt>keyring</tt> object still has field entries of the key to decrypt the wrapped symmetric key, but instead of the wrapped key being the value of the field, it is now an object with two fields: <tt>wrapped</tt> and <tt>hmac</tt>. The wrapped symmetric key from v1 is now the value of <tt>wrapped</tt>. The <tt>hmac</tt> can be used to verify that the wrapped key has not been tampered with.  


== Version 1 ==
== Version 1 ==


The following describes the JS-object represented by the JSON-string payload:
The following describes the JS-object represented by the JSON-string payload:  


{| cellpadding=5
{| cellpadding="5"
! bulkIV
|-
| string
! bulkIV  
| string  
| Initialization vector used to encrypt/decrypt all data for the corresponding engine
| Initialization vector used to encrypt/decrypt all data for the corresponding engine
|-
|-
! keyring
! keyring  
| object
| object  
| A hash with fields of key urls used to decrypt the field's value of a wrapped symmetric key
| A hash with fields of key urls used to decrypt the field's value of a wrapped symmetric key
|}
|}
Rnewman
canmove, Confirmed users
640

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/225682...338575"