CloudServices/At a glance
TOKEN SERVER
What does it do?
This solution uses a token-based authentication system. When a user wants to connect to one of our service , they get an access token by asking a central server.
System Requirements
- The system needs to support multiple services (not necessarily centrally)
- The system needs to be able to assign users to different machines as a service scales out, or somehow distribute them
- The system should consistently send a user back to the same server once they’ve been assigned
- The system needs to give operations some level of control over how users are allocated
- The system should provide some recourse if a particular node dies
- The system that can handle exhaustion attacks. For example, I could set up an primary
- The system should auto-approve any username, then loop through users until all nodes were full.
- The system needs to support future developments like bucketed assignment
- system that scales infinitely.
Assumptions
- A Login Server detains the secret for all the Service Nodes for a given Service.
- Any given webhead in a cluster can receive calls to all service nodes in the cluster.
- The Login Server will support only BrowserID at first, but could support any authentication protocol in the future, as long as it can be done with a single call
- All servers are time-synced
- The expires value for a token is a fixed value per application. For example it could be 30 minutes for Sync and 2 hours for bipostal.
- The Login Server keeps a white list of domains for BID verifications
Resources
Specs and Docs:
LOCATION
What does it do?
The Mozilla Location Service is an open service which lets devices determine their location based on network infrastructure like WiFi access points and cell towers. Project Overview
Requirements
- Javascript
- HTML
Get Involved
- As an individual you can join our active community and start mapping the wireless networks in your neighborhood or contribute to one of the client applications.
- As a developer you can directly use the service or contribute to the development of the service itself.
- As an organization you can contact us to find out more about collaboration opportunities.
- As the owner of a WiFi access point you can learn more about opting out of this service and the privacy protections we built into the service.
Instructions on how one can contribute to the project
MESSAGE QUEUE
What does it do?
Message Queue provides a service for applications to queue messages for its clients. Project Overview
Requirements
Notifications
The Notifications project needs Queuey for storing messages on behalf of users that want to receive them. Each user gets their own queue.
If a user needs to get notifications for general topics, the Notifications application will create a queue and clients will poll multiple queues.
The first version could be considered a Message Store rather than a queue as it supports a much richer set of query semantics and does not let public consumers remove messages. Messages can be removed via the entire queue being deleted by the App or by expiring.
Requirements:
- Service App can create queues
- Service App can add messages to queue
- Messages on queues expire
- Clients may read any queue they are aware of
Socorro
The second version allows authenticated applications to queue messages and for clients to consume them. This model allows for a worker model where jobs are added to a queue that multiple clients may be watching, and each message will be given to an individual client.
Requirements:
- Service App can create queues, which can be partitioned
- Service App can add messages to queue, and specify partition to retain ordering
- Clients can ask for information about how many partitions a queue has
- Clients may read any queue, and its partitions that they are aware of
Architecture
Queue Web Application
queuey (Python web application providing RESTful API)
Queue Storage Backend
Cassandra
Coordination Service
(Only used when consuming from the queue)
Apache Zookeeper
Queue Consumption
* Consumers coordinate with coordination service
* Consumers split queues/partitions amongst themselves
* Consumers record in coordination service the farthest they've processed
in a queue/partition
* Consumers rebalance queue/partition allocation when consumers are added or
removed using coordination service
* Consumption and rebalancing is done entirely client-side
Queuey is composed of two core parts:
- A web application (handles the RESTful API)
- Storage back-end (used by the web application to store queues/messages)
The storage back-end is pluggable, and the current default storage back-end is Cassandra.
Unlike other MQ products, Queuey does not hand out messages amongst all clients reading from the same queue. Every client is responsible for tracking how far it has read into the queue, if only a single client should see a message, then the queue should be partitioned, and clients should decide who reads which partition of the queue.
Different performance and message guarantee characteristics can be configured by changing the deployment strategy and Cassandra replication and read / write options. Therefore multiple Queuey deployments will be necessary, and Apps should use the deployment with the operational characteristics desired.
Since messages are stored and read by timestamp from Cassandra, and Cassandra only has eventual consistency, there is an enforced delay in how soon a message is available for reading to ensure a consistent picture of the queue. This helps ensure that written messages will show up when reading the queue to avoid 'losing' messages by reading past where they appeared in the queue.
For more on what kind of probabilities are involved in various replication factors and differing read/write CL's, see:
In the event that the clients are deleting messages in their queue as they've been read, the delay is unimportant. Enforcing a proper delay is only required when clients read but never delete messages (and thus track how far into the queue they've read based on time-stamp).
When using queue's that are to be consumed, they must be declared up-front as a partitioned queue. The amount of partitions should also be specified, and new messages will be randomly partitioned. If messages should be processed in order, they can be inserted into a single partition to enforce ordering. All messages that are randomly partitioned should be considered loosely ordered.
Queue Workers
A worker library called Qdo handles coordinating and processing messages off queues in a job processing setup. The Qdo library utilizes Apache Zookeeper for worker and queue coordination.
Workers coordinate to divide up the queue's and partitions in each queue so that no queue/partition has multiple readers. This avoids the need for read locking a queue, and how far into each host+queue+partition is stored in Zookeeper.
This model is based exactly on how Apache Kafka workers divide up queues to work on.
API
API can be found on the queuey API docs page.
Engineers
- Ben Bangert
- Hanno Schlichting