Identity/AttachedServices/StorageProtocolZero

Summary

This is a working proposal for the PICL Storage API to store and retrieve simple objects called Basic Storage Objects, which are organized into named collections. It is based on the Mozilla SyncStorge 2.0 API with some simplifications.

This will not be the final protocol used by PICL. It is an attempt to re-use existing protocol work to enable rapid early iteration.

Basic Storage Objects

Basic Storage Objects have the following fields:

Parameter	Default	Type/Max	Description
id	required	string, 64	An identifying string. For a user, the id must be unique for a BSO within a collection, though objects in different collections may have the same ID. BSO ids may only contain characters from the urlsafe-base64 alphabet (i.e. alphanumerics, underscore and hyphen).
version	none	int > 0, 16	The version number at which this object was last modified. This is a server-assigned integer that is set automatically on each write; any client-supplied value for this field is ignored.
timestamp	none	int > 0, 16	The timestamp at which this object was last modified, in milliseconds since UNIX epoch (1970-01-01 00:00:00 UTC). This is set automatically by the server according to its own clock; any client-supplied value for this field is ignored.
payload	empty string	string, 256k	A string containing the data of the record. The structure of this string is defined separately for each BSO type. This spec makes no requirements for its format. In practice, JSONObjects are common.
deleted	false	bool	Whether this object is deleted.

Collections

Each BSO is assigned to a collection with other related BSOs. Collection names may only contain characters from the urlsafe-base64 alphabet (i.e. alphanumeric characters, underscore and hyphen).

Collections are created implicitly when a BSO is stored in them for the first time. They continue to exist indefinitely since the BSO records they contain cannot be deleted, only marked for deletion.

Version Numbers

In order to allow multiple clients to coordinate their changes, the server associates a version number with the data stored for each user. This is a server-assigned integer value that increases with every modification made to the user’s data.

The version number is tracked at three levels of nesting:

• The current version number is increased whenever any change is made to the user’s data. It applies to the store as a whole.
• Each collection has a last-modified version giving the version number at which that collection was last modified. It will always be less than or equal to the current version number.
• Each BSO has a last-modified version giving the version number at which that item was last modified. It will always be less than or equal to the last-modified version of the containing collection.

The version number is guaranteed to be monotonically increasing and can be used for coordination and conflict management. It is not guaranteed to increment sequentially, and clients should treat it as an opaque integer value. As a special case, resources that do not exist are considered to have a last-modified version number of zero.

API Endpoints

Each user is identified by an opaque, alphanumeric userid, and their data is accessed via URLs of the form:

   https://<server-url>/<userid>/<api-instruction>

The server currently uses the PiCL NULL authentication model, and the userid is the same as the user’s authentication token; obviously this will change as the project moves forward.

All data transfer between server and client is utf8-encoded JSON bodies, with content-type of application/json.

GET https://<server-url>/<userid>/info/collections

Returns a mapping of collection names associated with the account to the last-modified version number for each collection.

Example response:

   { “version”: 42,     // the current version number for the whole store
     “collections”:   { // the last-modified version number for each collection
       “tabs”: 42,
       “bookmarks”:  38
   }}

Possible HTTP status codes:

• 200 OK: everything went fine.
• 304 Not Modified: the current version number is less than or equal to the value specified in the X-If-Modified-Since-Version header.

GET https://<server-url>/<userid>/storage/<collection>

Returns a list of all BSOs in the collection, or all those matching the specified query parameters.

Query parameters:

• ids: a comma-separated list of ids. Only objects whose id is in this list will be returned. A maximum of 100 ids may be provided.
• newer: a version number. Only objects whose last-modified version number is strictly greater than this value will be returned.

Example response:

   { “version”:  42,    // the last-modified version number for this collection
     “items”:  [        // the BSOs that match this query
       { “id”: “mybookmark”, “version”: 17, … },
       { “id”: “abookmark”, “version”: 42, … },
   ]}

Possible HTTP status codes:

• 200 OK: success was had
• 304 Not Modified: the last-modified version number of the collection is less than or equal to the value in the X-If-Modified-Since-Version header.
• 400 Bad Request: something was wrong with the request, e.g. too many ids were included in the query parameter.
• 404 Not Found: the user has no such collection.

POST https://<server-url>/<userid>/storage/<collection>

Creates or updates BSOs within a collection, and responds with the new version number for that collection. A maximum of 100 BSOs may be included in the request.

The request body must be a JSON array of BSO objects, which will be iterated over. Any BSOs that already exist will have their data updated with the provided fields; BSOs that do not exist will be created. Deletion is achieved by setting the “deleted” field to true.

Example response:

   { “version”: 42 }

Possible HTTP status codes:

• 200 OK: the request completed successfully, creating the collection if necessary.
• 400 Bad Request: something was wrong with the request, e.g. too many BSOs were included in the request.
• 412 Precondition Failed: the last-modified version number of the collection is greater than the value in the X-If-Unmodified-Since-Version header.

Request Headers

X-If-Modified-Since-Version

This header may be added to any GET request, set to a version number. If the last-modified version of the target resource is less than or equal to the version number given, then a 304 Not Modified response will be returned and re-transmission of the unchanged data will be avoided.

It is similar to the standard HTTP If-Modified-Since header, but the value is an opaque version number rather than a timestamp.

If the value of this header is not a valid positive integer, or if the X-If-Unmodified-Since-Version header is also present, then a 400 Bad Request response will be returned.

X-If-Unmodified-Since-Version

This header may be added to any request to a collection or item, set to a version number. If the last-modified version of the target resource is greater than the version number given, the request will fail with a 412 Precondition Failed response.

It is similar to the standard HTTP If-Unmodified-Since header, but the value is an opaque version number rather than a timestamp.

If the value of this header is not a valid positive integer, or if the X-If-Modified-Since-Version header is also present, then a 400 Bad Request response will be returned.

Response Headers

X-Last-Modified-Version

This header gives the last-modified version number of the target resource as seen during processing of the request, and will be included in all success responses (200, 201, 204).

When given in response to a write request, this will be equal to the new current version number and the new last-modified version number of any BSOs created or changed by the request.

It is similar to the standard HTTP Last-Modified header, but the value is an opaque version number rather than a timestamp.