CloudServices/Sagrada/TokenServer

Goals

So here's the challenge we face. Current login for sync looks like this:

1. provide username and password
2. we log into ldap with that username and password and grab your sync node
3. we check the sync node against the url you've accessed, and use that to configure where your data is stored.

This solution works great for centralized login. It's fast, has a minimum number of steps, and caches the data centrally. The system that does node-assignment is lightweight, since the client and server both cache the result, and has support for multiple applications with the /node/<app> API protocol.

However, this breaks horribly when we don't have centralized login. And adding support for browserid to the SyncStorage protocol means that we're now there. We're going to get valid requests from users who don't have an account in LDAP. We won't even know, when they make a first request, if the node-assignment server has ever heard of them.

So, we have a bunch of requirements for the system. Not all of them are must-haves, but they're all things we need to think about trading off in whatever system gets designed:

• need to support multiple services (not necessarily centrally)
• need to be able to assign users to different machines as a service scales out, or somehow distribute them
• need to consistently send a user back to the same server once they've been assigned
• need to give operations some level of control over how users are allocated
• need to provide some recourse if a particular node dies
• need to handle exhaustion attacks. For example, I could set up an primary that just auto-approved any username, then loop through users until all nodes were full.
• need support for future developments like bucketed assignment
• Needs to be a system that scales infinitely.

Proposed Design

This solution proposes to use a token-based authentication system. A user that wants to connect to one of our service asks to a central server an access token.

The central server, a.k.a. the Login Server checks the authenticity of the user with a supported authentication method, and attributes to the user a server he needs to use with that token.

The server, a.k.a. the Service Node, that gets called controls the validity of the token included in the request. Token have a limited lifespan.

Definitions and assumptions

First, a few definitions:

• Service: a service Mozilla provides, like Sync or Easy Setup.
• Login Server: used to authenticate user, returns tokens that can be used to authenticate to our services.
• Node: an URL that identifies a service, like http://phx345
• Service Node: a server that contains the service, and can be mapped to several Nodes (URLs)
• Node Assignment Server: a service that can attribute to a user a node.
• User DB: a database that keeps the user/node relation
• Cluster: Group of webheads and storage devices that make up a set of Service Nodes.
• Colo: physical datacenter, may contain multiple clusters
• Master Secret: the secret shared between Login Server and Service Node. Never used directly, only for deriving other secrets.
• Signing Secret: derived from the master secret, used to sign auth and metadata tokens. For example: sig-secret = HKDF_Expand(master-secret, "SIGN")
• Encryption Secret: derived from the master secret, used to encrypt the metadata token. For example: enc-secret = HKDF_Expand(master-secret, "ENCRYPT")
• Token Secret: derived from the master secret and auth token, used as oauth_consumer_secret. This is the only secret shared with the client and is different for each token. For example: token-secret = HKDF_Expand(master-secret, auth-token)

Some assumptions:

• A Login Server detains the secret for all the Service Nodes for a given Service.
• Any given webhead in a cluster can receive calls to all service nodes in the cluster.
• The Login Server will support only BrowserID at first, but could support any authentication protocol in the future, as long as it can be done with a single call.
• All servers are time-synced

Flow

Here's the proposed two-step flow (with Browser ID):

1. the client trades a browser id assertion for a session token
2. the client turns the session token into a oauth token

Getting a session token:

Client                       Login Server                  BID         User DB          Node Assignment Server   
===========================================================================================================
                                 |                          |             |                    |                
request token ---- [1] --------->|------> verify --- [2] -->|             |                    |                
                                 |       get node -- [3] ---|------------>|--> lookup          |                
                                 |                          |             |<-- return node     |                   
                                 |  attribute node --[4]----|-------------|------------------->|--> set node    
                                 |                          |             |                    |<-- node        
                                 |<--- build token   [5]    |             |                    |               
keep token <-------- [6] --------|                          |             |                    |                 
create signed auth header [7]    |                          |             |                    |                

Calling the service:

Client                                          Service Node
============================================================
call node --------------- [8] ---|---------------->|--> verify token 
                                 |                 |<-- process request [9]
get response <-------------------|-----------------|

• the client request a token, giving its browser id assertion [1]

    POST /request_token HTTP/1.1
    Host: token.services.mozilla.com
    Content-Type: application/json
    X-Authentication-Method: Browser-ID   (optional header since Browser-ID is the default)
    
    {"audience":XXX,"assertion":XXX}     

• the Login Server checks the browser id assertion [2] this step will be done locally without calling an external browserid server -- but this could potentially happen (we can use pyvep + use the BID.org certificate)
• the Login Server asks the Users DB if the user is already allocated to a node. [3]
• if the user is not allocated to a node, the Login Server asks a new one to the Node Assignment Server [4]
• the Login Server creates a response with a session token [5], using the user id, a time stamp and a secret string only known by the selected node and itself, and sends it back to the user along with a secret derived from the shared secret, using HKDF (https://tools.ietf.org/html/rfc5869). It also adds the node url in the response, and optionaly a metadata token. [6]

  HTTP/1.1 200 OK
  Content-Type: application/json
  
  {'oauth_consumer_key': <token>,
   'oauth_consumer_secret': <derived-secret>,
   'service_entry': <node>,
   'metadata': <metadata-token>
   }

• the client calculates with the information received, an OAuth authorization header, and saves the node location. [6]
• the client calls the right node, using the special Authorization header. It tramsmits as-is the token and the metadata token if provided [7]

    POST /request HTTP/1.1
    Host: some.node.services.mozilla.com
    Authorization: OAuth realm="Example",
                   oauth_consumer_key=<token>  
                   metadata=<metadata-token>,
                   oauth_token="kkk9d7dh3k39sjv7",
                   oauth_signature_method="HMAC-SHA1",
                   oauth_timestamp="137131201",   (client timestamp)
                   oauth_nonce="7d8f3e4a",
                   oauth_signature="bYT5CMsGcbgUdFHObYMEfcx6bsw%3D"

• the node is able with its shared secret to validate that the request is valid, by calculating the derived key given the salt and the shared secret [9]. If it's an invalid or expired token, the node returns a 401

Tokens

A token is a json encoded mapping. They are two tokens:

• the authorization token: contains the user application id and the expiration date.
• the metadata token: contains app-specific data (optional)

Authorization Token

The keys of the Authorization Token are:

• expires: an expire timestamp (UTC) defaults to current time + 30 mn
• uid: the app-specific user id (the user id integer in the case of sync)

Example:

 auth_token = {'uid': '123', 'expires': 1324654308.907832}  

The token is signed using the salted derived secret and base64-ed. The signature is HMAC-SHA1:

 auth_token, signature = HMAC-SHA1(auth_token, secret_key)
 auth_token = b64encode(auth_token, salt, signature)

The authorization token is not encrypted

Metadata token (optional)

The keys of the Metadata token are free-form. This token can include anything needed by the application to function.

It's passed as-is by the client to the Service Node

Example:

 app_token = {'email': 'my@email.com', 'someparam': 1324654308.907832}  

To avoid information leakage, the token is encrypted and signed using the shared secret and base64-ed. The encryption is AES-CBC and signature is HMAC-SHA1:

 app_token, signature = AES-CBC+HMAC-SHA1(app_token, secret_key)
 app_token = b64encode(app_token, signature)

The metadata token is crypted

Shared Secrets File

Each Service Node has a unique secret per Node it serves, it shares with the Login Server. A secret is an hex string of 256 chars from [a-f0-9]

Example of generating such string:

 >>> import binascii, os
 >>> print binascii.b2a_hex(os.urandom(256))[:256]
 21c100e75c02af215e2bf523b0...0505ff951

Ops create secrets for each Node, and maintain for each cluster a file containing all secrets. The file is deployed on the Login Server and on each Service Node. The Login Server has all clusters files.

Each file is a CSV file called /var/moz/shared_secrets/CLUSTER, where CLUSTER is the name of the cluster,

Example:

   phx1,secret
   phx2,secret
   ...

Secret Update Process

When an existing secret needs to be changed for whatever reason, the current secret becomes the old secret. The reason is to avoid existing tokens to be rejected when the secret is changed.

The new secret is inserted to the Node's line on each file :

   phx1,new secret,oldsecret
   phx2,secret
   ...

The Service Nodes are the first ones to be updated, then the Login Server is updated in turn, so the new tokens are immediatly recognized by the Nodes. In the interim, the Service Node fallbacks to the old secret when a token verification fails and there's an old secret in the file.

The Login Server only works with a single secret, so ignores the old secret when it creates tokens.

The old secret is pruned eventually.

Backward Compatibility

XXX TBU

Older versions of the system will use completely different API entrypoints - the old /user api, and the 1.1 /sync api. Those will need to be maintained during the transition, though new clusters should spin up with only 2.0 support.

We should watch logs to study 1.1 falloff and consolidate those users through migration as they diminish.

However, There are a couple of points that need to be synced up:

• The database that assigns nodes needs to be shared between the two. We should add a column for "1.0 acceptable" and update the old system to only look at those columns. Alternately, could work with ops to just have an "all old assignments go to this one cluster", in which case, the db doesn't need to be shared.

• There will be a migration that moves all the user node data from LDAP to the tokenserver. However, we need to make sure that any subsequent migrations update this data. This ensures that a user with a pre-2 client and post-2 client point at the same place, and that people moving to the new systems will have the right node. We can't punt this, because if a node goes down post-migration, a user who switches over afterwards is stuck on it. (at the very least, we need to purge these nodes from the 2.0 db).

• will need to migrate all user login data over to the browserid servers, but that's not relevant to tokenserver.

Infra/Scaling

On the Login Server

The flow is:

1. the user ask for a token, with a browser id assertion
2. the server verifies locally the assertion [CPU bound]
3. the server calls the User DB [I/O Bound]
4. the server calls the Node Assignment Server [I/O Bound] (optional)
5. the server builds the token and sends it back [CPU bound]
6. the user uses the node for the time of the ttl (30mn)

So, for 100k users it means we'll do 200k requests on the Login Server per hour, so 50 RPS. For 1M users, 500 RPS. For 10M users, 5000 RPS. For 100M users, 50000 RPS.

Deployment

• A Login Server is stateless, so we can deploy as many as we want and have Zeus load balance over them
• A Login Server sees all secrets, so it can be cross-cluster / cross-datacenter
• The shared secrets files can stay in memory -- updating the files should ping the app so we reload them
• The User DB is the current LDAP, and may evolve into a more specialised metadata DB later

On each Service Node

Flow :

1. the server checks the token [CPU Bound]
2. the server process the request [Sync = I/O Bound]

APIS v1.0

Unless stated otherwise, all APIs are using application/json for the requests and responses content types.

POST /1.0/request_token

Asks for new token given some credentials. By default, the authentication mechanism is Browser ID but the X-Authentication-Protocol can be used to explicitly pick a protocol. If the server does not support the authentication protocol provided, a 400 is returned.

When the authentication protocol requires something else than an Authorization header, the data is provided in the request body.

Example for Browser-Id:

POST /request_token
Host: token.services.mozilla.com
Content-Type: application/json

{'audience': XXX,
 'assertion': XXX}

This API returns several values in a json mapping:

• oauth_consumer_key - a signed authorization token, containing the user's id and expiration
• oauth_consumer_secret - a secret containing a secret derived from the shared secret
• service_entry: a node url
• metadata - a signed an encrypted token, containing app-specific metadata - optional

Example:

HTTP/1.1 200 OK
Content-Type: application/json

{'oauth_consumer_key': <token>,
 'oauth_consumer_secret': <derived-secret>,
 'service_entry': <node>,
 'metadata': <metadata-token>,
}

Phase 1

[End of January? Need to check with ally]

End to end prototype with low-level scaling

• Fully defined API, including headers and errors
• Assigns Nodes
• Maintains Node state for a user (in the existing LDAP)
• Issues valid tokens
• Downs nodes if needed

Phase 2

[End of Q1?]

Scalable implementation of the above in place.

• Migration
• Operational support scripts (TBD)
• Logging and Metrics