MozillaWiki - User contributions [en]

Talk:Service/Sync/FxSync/Developer/BrowserObjects

2011-01-07T07:00:03Z

MarcusWolschon: deleted bookmarks

== deleted bookmarks ==

The "type" value of deleted bookmarks is changed into "item" by at least some Firefox versions.
I did not test if that happens to folders, livemarks,... too. --[[User:MarcusWolschon|MarcusWolschon]] 23:00, 6 January 2011 (PST)

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T10:58:41Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC. 
* Sourcecode of [http://hg.mozilla.org/services/fx-sync/file/12189166cd01/services/sync/modules/util.js#l571 Utils.makeHMACKey()] Sorry, it's a native method.

* What is the value of HMAC_INPUT?
** [http://hg.mozilla.org/services/fx-sync/file/37150bc0bf62/services/sync/modules/constants.js source]
** it is the string "Sync-AES_256_CBC-HMAC256"

* what is enc + HMAC_INPUT + u + "\x02"? (enc is a byte array, the others are strings)
* \x?? = "The Latin-1 character specified by the two hexadecimal digits dd between 00 and FF. ie, copyright symbol is \xA9."
** \x01 = SOH = \u0001 in unicode

== Upgrading existing Sync Keys to the new AES key ==
PBKDF2 iteration count it 4096, key length 128 bit. 
Keep in mind that while everywhere else Base64 is used, this is Base32.

Example (Java):
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T10:54:39Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC. 
* Sourcecode of [http://hg.mozilla.org/services/fx-sync/file/12189166cd01/services/sync/modules/util.js#l571 Utils.makeHMACKey()] Sorry, it's a native method.

* What is the value of HMAC_INPUT?
** [http://hg.mozilla.org/services/fx-sync/file/37150bc0bf62/services/sync/modules/constants.js source]
** it is the string "Sync-AES_256_CBC-HMAC256"
* what is enc + HMAC_INPUT + u + "\x02"? (enc is a byte array, the others are strings)
* \x?? = "The Latin-1 character specified by the two hexadecimal digits dd between 00 and FF. ie, copyright symbol is \xA9."

== Upgrading existing Sync Keys to the new AES key ==
PBKDF2 iteration count it 4096, key length 128 bit. 
Keep in mind that while everywhere else Base64 is used, this is Base32.

Example (Java):
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Services/Sync/SimplifiedCrypto

2010-12-04T10:49:12Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */ added missing value for HMAC_INPUT

== The current state of Sync crypto ==

See [[Labs/Weave/Developer/Crypto]] for thorough details.

In short:

* Generate a RSA key pair
* Generate a symmetric key from the passphrase, using PBKDF2 and a random salt, and encrypt ("wrap") the private key using that symmetric key
* Upload public key, wrapped private key + IV + salt to server
* For each collection, generate a random symkey, encrypt it using the public key, and upload it to the server.
* Each encrypted object contains a relative URI pointing to the key that can decrypt it (in 99.99999% percent of the case this is same, except when the WBO IDs contain slashes and clients get very confused).
* Fetching a decrypted object involves:
** Fetching the encrypted object from the server
** Looking at the key URI in that JSON blob to find the symkey URI
** Fetching (if necessary) from the server and RSA-decrypting the symkey
** Using the symkey to AES-decrypt the object.

== Goal and motivation ==

We want to drop the PKI layer. We don't use it (for the original speculative sharing scenarios), and it costs client computation, server storage, and network bandwidth (~ 16% of our API transactions are key fetches).

== Proposal ==

'''tl;dr''': replace the passphrase with an AES key, which will be schlepped around using J-PAKE (so typing it is likely unnecessary). Use this key to indirectly encrypt the 'bulk' symkeys. No RSA involved.

Existing passphrases will be upgraded to this scheme using PBKDF2.

=== Passphrase / Sync Key ===

Rather than have a user enter a passphrase (which will likely be weak), we have already transitioned to having them generate a "sync key" (which they can replace if they so choose). This is 20 alphanumeric characters.

We propose to expand this to 26 characters, enough for a base32-encoded 128-bit AES key. This avoids the use of PBKDF2 to routinely bootstrap the sync key into an AES key. Remove the ability for users to enter a key; it's always generated (giving us more confidence in the amount of entropy), and can be regenerated if desired.

The length of this key is not a big issue: we intend to use J-PAKE for the (infrequent) migration of keys between devices. In any case, 26 is not significantly worse than 20 if typing it does enter the picture, and the use of a nice base32 alphabet makes keyboard entry less error-prone.

As before, the Sync Key is stored on the client. The encryption and hmac keys are derived from it.

==== Deriving encryption and HMAC keys from the Sync Key ====

First we base32-decode the syncKey from 26 characters to 16 bytes (128 bits). The base32 alphabet is the one specified in [http://tools.ietf.org/html/rfc4648 RFC 4648] except with '''l''' replaced by '''8''' and '''o''' replaced by '''9''':

let m = Utils.decodeKeyBase32(syncKey);

The resulting key is then expanded to an encryption and HMAC key using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]

// Our extra input to SHA256-HMAC in generateEntry
// This includes the full crypto spec; change this when our algo changes.
HMAC_INPUT: "Sync-AES_256_CBC-HMAC256",

// Reuse the hasher.
let h = Utils.makeHMACHasher();

// First key.
let u = this.username;
let k1 = Utils.makeHMACKey("" + HMAC_INPUT + u + "\x01");
let enc = Utils.sha256HMACBytes(m, k1, h);

// Second key: depends on the output of the first run.
let k2 = Utils.makeHMACKey(enc + HMAC_INPUT + u + "\x02");
let hmac = Utils.sha256HMACBytes(m, k2, h);

enc and hmac are the 256 bit encryption and HMAC keys, respectively.

==== Upgrading existing Sync Keys to the new AES key ====

Existing users will have their passphrase bootstrapped into an AES key using PBKDF2:

* Spot old version
* Get a salt ([[Labs/Weave/Developer/StorageFormat#Payload:_meta.2Fglobal|Services.syncID]] from the meta/global object. The client will be bumping this…)
* Apply PBKDF2 to salt and passphrase to yield our new AES key
* Generate bulk keys, encrypt
* Attempt to store, using appropriate race-avoidance technique in case there are multiple clients attempting to upgrade.
* Wipe old key data.

So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or [[Services/Sync/SyncKey/J-PAKE|J-PAKE]]-style key distribution.

=== Bulk keys ===

The server stores one or more bulk keys: one default ("keys/default"), and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.

Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)

The timestamp on the collections record allows clients to invalidate their key cache when a new key is associated with a collection: the 'keys' collection will appear to have changed.

'''Discussion topic''': do we need to preserve (add?) the ability to do per-object encryption? Brian Warner suggested:

:: While not really interesting here, this one-way property is really useful in other situations, like if you derived per-object encryption keys from a parent folder's key. You could then share the whole thing with someone by giving them the parent's key, or share just one object and *not* give them the ability to get at anything else in that folder. But the sharing discussion is for another day..

=== HMAC ===

It's a good practice to use separate keys for HMAC and for encryption. Bulk keys are really pairs of keys, each of which is randomly generated.

This approach was selected over having a single HMAC key because of the convenience for implementing some sharing-like scenarios.

== Proposed flows ==

=== New user ===

* Generate a 128-bit Sync key (25 characters in base36). Store it as an Identity (as we do now.)
* Generate a random default key and HMAC. Encrypt it with the sync key, upload it to the server. Store it as an Identity.
* Encrypt and upload collections in the obvious way.

=== Existing user ===

(See above.)

=== Fetching objects ===

* (On startup: invalidate/refresh key cache if keys collection has changed. I believe we make this fetch anyway...)
* Retrieve object from collection.
* Look up key for collection name (defaulting to "keys/default"). Fetch if necessary.
* Verify HMAC using appropriate per-collection or default key. On failure, check for changed keys.
* Decrypt object.

== Version bump ==

This change is incompatible with older clients: not only due to reorganizing the storage namespace, but also because existing clients will be unaware of the simpler encryption mechanism. That means a storage version bump (from 3 to 4).

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T10:47:29Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC. 
* Sourcecode of [http://hg.mozilla.org/services/fx-sync/file/12189166cd01/services/sync/modules/util.js#l571 Utils.makeHMACKey()] Sorry, it's a native method.

* What is the value of HMAC_INPUT?
** [http://hg.mozilla.org/services/fx-sync/file/37150bc0bf62/services/sync/modules/constants.js source]
** it is the string "Sync-AES_256_CBC-HMAC256"
* what is enc + HMAC_INPUT + u + "\x02"? (enc is a byte array, the others are strings)

== Upgrading existing Sync Keys to the new AES key ==
PBKDF2 iteration count it 4096, key length 128 bit. 
Keep in mind that while everywhere else Base64 is used, this is Base32.

Example (Java):
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T10:32:36Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC. 
* Sourcecode of [http://hg.mozilla.org/services/fx-sync/file/12189166cd01/services/sync/modules/util.js#l571 Utils.makeHMACKey()] Sorry, it's a native method.

* What is the value of HMAC_INPUT?
* what is enc + HMAC_INPUT + u + "\x02"? (enc is a byte array, the others are strings)

== Upgrading existing Sync Keys to the new AES key ==
PBKDF2 iteration count it 4096, key length 128 bit. 
Keep in mind that while everywhere else Base64 is used, this is Base32.

Example (Java):
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T08:29:08Z

MarcusWolschon: /* Upgrading existing Sync Keys to the new AES key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC.

(A deep link to Utils.makeHMACKey() would be helpful here.)

== Upgrading existing Sync Keys to the new AES key ==
PBKDF2 iteration count it 4096, key length 128 bit. 
Keep in mind that while everywhere else Base64 is used, this is Base32.

Example (Java):
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T08:28:34Z

MarcusWolschon: /* Upgrading existing Sync Keys to the new AES key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC.

(A deep link to Utils.makeHMACKey() would be helpful here.)

== Upgrading existing Sync Keys to the new AES key ==
PBKDF2 iteration count it 4096, key length 128 bit.

Example (Java):
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T08:26:49Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */

== sync key representation ==

The sync key is represented to the user as:

X-XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC.

(A deep link to Utils.makeHMACKey() would be helpful here.)

== Upgrading existing Sync Keys to the new AES key ==

example:
<pre>
/**
* See https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto#Upgrading_existing_Sync_Keys_to_the_new_AES_key
* for details on the algorithm.
* @param aV3Passphrase
* @return the v4 syncKey (serves the same purpose as the passphrase before)
* @throws Exception see {@link #passwordToSymmetricKey(char[], byte[])}
* @throws UnsupportedEncodingException should not happen (ASCII)
*/
public String upgradeV3PassphraseToV4SyncKey(final String aV3Passphrase) throws UnsupportedEncodingException, Exception {
String salt = mSyncID;
KeySpec ks = new PBEKeySpec(aV3Passphrase.toCharArray(), salt.getBytes("ASCII"), 4096, 128);
PBKDF2HmacSHA1Factory f = new PBKDF2HmacSHA1Factory();
SecretKey s = f.engineGenerateSecret(ks);

String base32 = biz.wolschon.android.codec.binary.Base32.encode(s.getEncoded()).toLowerCase();
String syncKey = base32.replace('l', '8').replace('o', '9');

syncKey = syncKey.charAt(0)
+ "-" + syncKey.substring(1, 6)
+ "-" + syncKey.substring(6, 11)
+ "-" + syncKey.substring(11, 16)
+ "-" + syncKey.substring(16, 21)
+ "-" + syncKey.substring(21, 26);
Log.d(LOG_TAG, "upgraded passphrase to syncKey \"" + syncKey + "\"");
return syncKey;
}

</pre>

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T07:42:18Z

MarcusWolschon: /* Deriving encryption and HMAC keys from the Sync Key */

Talk:Services/Sync/SimplifiedCrypto

2010-12-04T07:18:47Z

MarcusWolschon: Created page with "== Deriving encryption and HMAC keys from the Sync Key == The hmac used is an SHA-256 HMAC. (A deep link to Utils.makeHMACKey() would be helpful here.)"

== Deriving encryption and HMAC keys from the Sync Key ==

The hmac used is an SHA-256 HMAC.

(A deep link to Utils.makeHMACKey() would be helpful here.)

Services/Sync/SimplifiedCrypto

2010-12-04T06:52:32Z

MarcusWolschon: /* Upgrading existing Sync Keys to the new AES key */ Added links to J-PAKE and SyncID in this Wiki for navigating the referenced specifications

== The current state of Sync crypto ==

See [[Labs/Weave/Developer/Crypto]] for thorough details.

In short:

* Generate a RSA key pair
* Generate a symmetric key from the passphrase, using PBKDF2 and a random salt, and encrypt ("wrap") the private key using that symmetric key
* Upload public key, wrapped private key + IV + salt to server
* For each collection, generate a random symkey, encrypt it using the public key, and upload it to the server.
* Each encrypted object contains a relative URI pointing to the key that can decrypt it (in 99.99999% percent of the case this is same, except when the WBO IDs contain slashes and clients get very confused).
* Fetching a decrypted object involves:
** Fetching the encrypted object from the server
** Looking at the key URI in that JSON blob to find the symkey URI
** Fetching (if necessary) from the server and RSA-decrypting the symkey
** Using the symkey to AES-decrypt the object.

== Goal and motivation ==

We want to drop the PKI layer. We don't use it (for the original speculative sharing scenarios), and it costs client computation, server storage, and network bandwidth (~ 16% of our API transactions are key fetches).

== Proposal ==

'''tl;dr''': replace the passphrase with an AES key, which will be schlepped around using J-PAKE (so typing it is likely unnecessary). Use this key to indirectly encrypt the 'bulk' symkeys. No RSA involved.

Existing passphrases will be upgraded to this scheme using PBKDF2.

=== Passphrase / Sync Key ===

Rather than have a user enter a passphrase (which will likely be weak), we have already transitioned to having them generate a "sync key" (which they can replace if they so choose). This is 20 alphanumeric characters.

We propose to expand this to 26 characters, enough for a base32-encoded 128-bit AES key. This avoids the use of PBKDF2 to routinely bootstrap the sync key into an AES key. Remove the ability for users to enter a key; it's always generated (giving us more confidence in the amount of entropy), and can be regenerated if desired.

The length of this key is not a big issue: we intend to use J-PAKE for the (infrequent) migration of keys between devices. In any case, 26 is not significantly worse than 20 if typing it does enter the picture, and the use of a nice base32 alphabet makes keyboard entry less error-prone.

As before, the Sync Key is stored on the client. The encryption and hmac keys are derived from it.

==== Deriving encryption and HMAC keys from the Sync Key ====

First we base32-decode the syncKey from 26 characters to 16 bytes (128 bits). The base32 alphabet is the one specified in [http://tools.ietf.org/html/rfc4648 RFC 4648] except with '''l''' replaced by '''8''' and '''o''' replaced by '''9''':

let m = Utils.decodeKeyBase32(syncKey);

The resulting key is then expanded to an encryption and HMAC key using the algorithm described in [http://tools.ietf.org/html/rfc5869 RFC 5869]

// Reuse the hasher.
let h = Utils.makeHMACHasher();

// First key.
let u = this.username;
let k1 = Utils.makeHMACKey("" + HMAC_INPUT + u + "\x01");
let enc = Utils.sha256HMACBytes(m, k1, h);

// Second key: depends on the output of the first run.
let k2 = Utils.makeHMACKey(enc + HMAC_INPUT + u + "\x02");
let hmac = Utils.sha256HMACBytes(m, k2, h);

enc and hmac are the 256 bit encryption and HMAC keys, respectively.

==== Upgrading existing Sync Keys to the new AES key ====

Existing users will have their passphrase bootstrapped into an AES key using PBKDF2:

* Spot old version
* Get a salt ([[Labs/Weave/Developer/StorageFormat#Payload:_meta.2Fglobal|Services.syncID]] from the meta/global object. The client will be bumping this…)
* Apply PBKDF2 to salt and passphrase to yield our new AES key
* Generate bulk keys, encrypt
* Attempt to store, using appropriate race-avoidance technique in case there are multiple clients attempting to upgrade.
* Wipe old key data.

So long as the salt is available, other clients can apply PBKDF2 to their stored passphrase and the salt to yield the new key without any re-entry or [[Services/Sync/SyncKey/J-PAKE|J-PAKE]]-style key distribution.

=== Bulk keys ===

The server stores one or more bulk keys: one default ("keys/default"), and an optional set of keys associated with specific collections. This will allow rudimentary sharing scenarios (provide your bookmarks collection key to a web app, and your passwords remain secure). A default key is simpler than having per-engine/collection keys without an obvious need.

Bulk keys are encrypted and HMACed using the sync key outputs, and cached on the client. (Current caching is per-session, but they're stored as identities to make persistence easier to implement.)

The timestamp on the collections record allows clients to invalidate their key cache when a new key is associated with a collection: the 'keys' collection will appear to have changed.

'''Discussion topic''': do we need to preserve (add?) the ability to do per-object encryption? Brian Warner suggested:

:: While not really interesting here, this one-way property is really useful in other situations, like if you derived per-object encryption keys from a parent folder's key. You could then share the whole thing with someone by giving them the parent's key, or share just one object and *not* give them the ability to get at anything else in that folder. But the sharing discussion is for another day..

=== HMAC ===

It's a good practice to use separate keys for HMAC and for encryption. Bulk keys are really pairs of keys, each of which is randomly generated.

This approach was selected over having a single HMAC key because of the convenience for implementing some sharing-like scenarios.

== Proposed flows ==

=== New user ===

* Generate a 128-bit Sync key (25 characters in base36). Store it as an Identity (as we do now.)
* Generate a random default key and HMAC. Encrypt it with the sync key, upload it to the server. Store it as an Identity.
* Encrypt and upload collections in the obvious way.

=== Existing user ===

(See above.)

=== Fetching objects ===

* (On startup: invalidate/refresh key cache if keys collection has changed. I believe we make this fetch anyway...)
* Retrieve object from collection.
* Look up key for collection name (defaulting to "keys/default"). Fetch if necessary.
* Verify HMAC using appropriate per-collection or default key. On failure, check for changed keys.
* Decrypt object.

== Version bump ==

This change is incompatible with older clients: not only due to reorganizing the storage namespace, but also because existing clients will be unaware of the simpler encryption mechanism. That means a storage version bump (from 3 to 4).

Talk:CloudServices/Sync

2010-12-04T06:32:45Z

MarcusWolschon: /* more links */

== more links ==
[[Services/Sync/Experimental Clients|Experimental Clients]] 
[[Services/Sync/SimplifiedCrypto|Simplified Crypto]] (used with storage format 4) 
[https://wiki.mozilla.org/Labs/Weave/Developer/Crypto Crypto]] (used with storage format 1-3)

Talk:CloudServices/Sync

2010-12-04T06:31:44Z

MarcusWolschon: /* more links */

== more links ==
[[Services/Sync/Experimental Clients|Experimental Clients]] 
[[Services/Sync/SimplifiedCrypto|Simplified Crypto]] (used with storage format 4) 
[[/Labs/Weave/Developer/Crypto|Crypto]] (used with storage format 1-3)

Talk:CloudServices/Sync

2010-12-04T06:30:38Z

MarcusWolschon: /* more links */

== more links ==
[[Services/Sync/Experimental Clients|Experimental Clients]]
[[Services/Sync/SimplifiedCrypto|Simplified Crypto]] (used with storage format 4)
[[Services/Sync/Crypto|Crypto]] (used with storage format 1-3)

Talk:CloudServices/Sync

2010-12-04T06:30:04Z

MarcusWolschon: /* more links */

== more links ==
[[Services/Sync/Experimental Clients|Experimental Clients]]
[[SimplifiedCrypto]] (used with storage format 4)
[[Crypto]] (used with storage format 1-3)

Talk:Services/Sync/Experimental Clients/iPhone

2010-12-04T06:29:02Z

MarcusWolschon: Created page with "== 404 == "Not found: weaveclient-iphone ""

== 404 ==
"Not found: weaveclient-iphone "

Talk:CloudServices/Sync

2010-12-04T06:27:31Z

MarcusWolschon: Created page with "== more links == Experimental Clients"

== more links ==
[[Services/Sync/Experimental Clients|Experimental Clients]]

Services/Sync/Experimental Clients

2010-12-04T06:26:16Z

MarcusWolschon: /* Experimental Clients */

<onlyinclude>
= Experimental Clients =

* [[Weave/Experimental_Clients/Web|Web based data viewer]]
* [[Weave/Experimental_Clients/iPhone|iPhone data viewer]]
* [[Weave/Experimental_Clients/WebOS|WebOS data viewer]]
* [[Weave/Experimental_Clients/Python|Python based command line data viewer]]
* [http://sourceforge.net/userapps/mediawiki/marcuswolschon/index.php?title=Projects/AndroidDolphinHDFirefoxSyncPlugin Weave for DolphinHD on Android]
</onlyinclude>

Talk:Services/Sync/Server/API/Storage/1.1

2010-12-04T06:22:40Z

MarcusWolschon: Created page with "== WBOs== See [https://wiki.mozilla.org/Labs/Weave/Developer/StorageFormat StorageFormat] for the actual payloads stored in WBOs and their meaning. == current version == See [..."

== WBOs==

See [https://wiki.mozilla.org/Labs/Weave/Developer/StorageFormat StorageFormat] for the actual payloads stored in WBOs and their meaning.

== current version ==

See [https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API 1.0] for the version of this specification that is currently in use.

Talk:Services/Sync/Server/API/Storage/1.0

2010-12-04T06:22:08Z

MarcusWolschon: /* WBOs */

== WBOs==

See [https://wiki.mozilla.org/Labs/Weave/Developer/StorageFormat StorageFormat] for the actual payloads stored in WBOs and their meaning.

== new version ==

See [https://wiki.mozilla.org/Labs/Weave/Sync/1.1/API 1.1] for the next version of this specification.

Talk:Services/Sync/Server/API/Storage/1.0

2010-12-04T06:21:03Z

MarcusWolschon: Created page with "== WBOs== See [https://wiki.mozilla.org/Labs/Weave/Developer/StorageFormat StorageFormat] for the actual payloads stored in WBOs and their meaning."

== WBOs==

See [https://wiki.mozilla.org/Labs/Weave/Developer/StorageFormat StorageFormat] for the actual payloads stored in WBOs and their meaning.

Talk:Services/Sync/Developer/StorageFormat

2010-12-04T06:19:56Z

MarcusWolschon: /* WBO */

== WBO ==
The elements described here are stored as [https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API#Weave_Basic_Object_.28WBO.29 WBO]=Weave Basic Objects.

Regarding the stored cryptographic keys, see [https://wiki.mozilla.org/Labs/Weave/Developer/Crypto here(V1-V3)] and [https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto here(V4)] for how they are ultimately derived from the user's password.

== Version 4 ==
The latest desktop-mozilla uses vesion 4.
The changes include at least a very different login and crypto (see [https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto here]).

Talk:Services/Sync/Developer/StorageFormat

2010-12-04T06:17:18Z

MarcusWolschon: Created page with "== WBO == The elements described here are stored as WBO=Weave Basic Objects. See [https://wiki.mozilla.org/Labs/Weave/Developer/BrowserObjects BrowserObjects] for more details. ..."

== WBO ==
The elements described here are stored as WBO=Weave Basic Objects.
See [https://wiki.mozilla.org/Labs/Weave/Developer/BrowserObjects BrowserObjects] for more details.

Regarding the stored cryptographic keys, see [https://wiki.mozilla.org/Labs/Weave/Developer/Crypto here(V1-V3)] and [https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto here(V4)] for how they are ultimately derived from the user's password.

== Version 4 ==
The latest desktop-mozilla uses vesion 4.
The changes include at least a very different login and crypto (see [https://wiki.mozilla.org/Services/Sync/SimplifiedCrypto here]).