Confirmed users
192
edits
Services/MessageQueuing: Difference between revisions
Jump to navigation
Jump to search
adding architecture bit
(adding architecture bit) |
|||
| Line 13: | Line 13: | ||
=== Phase 1 === | === Phase 1 === | ||
'''NOTE: This phase is being somewhat skipped at the moment to Phase 2''' | |||
The first version of the MQ will focus on providing a useful service for the | The first version of the MQ will focus on providing a useful service for the | ||
| Line 33: | Line 35: | ||
* Messages on queues expire | * Messages on queues expire | ||
* Clients may read any queue they are aware of | * Clients may read any queue they are aware of | ||
=== Phase 2 === | === Phase 2 === | ||
| Line 49: | Line 52: | ||
* Service App can create queues | * Service App can create queues | ||
* Service App can add messages to queue | * Service App can add messages to queue | ||
* Authenticated clients may consume | * Authenticated clients may consume messages from a queue. | ||
* Authenticated clients may mark a message for consumption with a reservation TTL | * Authenticated clients may mark a message for consumption with a reservation TTL. | ||
== Architecture == | |||
For scalability purposes, since some messages may be retained for periods of time | |||
per Phase 1 requirements, the initial choice of a backend is Cassandra. However, | |||
for Phase 2 requirements, Cassandra is missing the ability to manage and coordinate | |||
queue consumers so Zookeeper is being used for distributed synchronization. | |||
When used for Notifications, each user will have a single queue per Notification | |||
Application, this helps ensure that even for an individual user receiving many | |||
messages, they are partitioned by the Notification Application to ensure | |||
a more level distribution rather than a single deep queue. | |||
Under the Phase 2 use-case, it is most likely desired that massive amounts of | |||
messages may be intended for the same queue, which would normally result in a | |||
single extremely deep queue. Deep queue's do not scale well horizontally, | |||
especially when using Cassandra which maximizes through-put across many | |||
row-keys. | |||
The other issue is that to consume messages, there are only two effective ways | |||
of marking a message as consumed: | |||
1. The message may be deleted after it has been sent. The only problem in | |||
this case is that there is still no guarantee it has been processed, and | |||
acquiring a lock to consume a message, and the resulting write-back to | |||
delete it is expensive. | |||
2. One consumer per queue/partition. This requires some guess-work up-front | |||
about how many consumers will be around. Since consumers can consume from | |||
multiple queue/partition's at once, there needs to be at least as many | |||
partitions for a queue, as desired consumers. The advantage with this | |||
approach is that locking is only necessary when adding/removing consumers | |||
to ensure one consumer per partition. | |||
This MessageQueue project goes with the second option, and only incurs the | |||
lock during consumer addition/removal. The MessageQueue also does not track | |||
the state or last message read in the queue/partition's, it is the consumers | |||
responsibility to track how far it has read and processed successfully. There | |||
is an API call available to record with the MessageQueue how far a consumer | |||
has successfully processed in a queue/partition. | |||
When reading messages for a processing workload, they should be read in batches | |||
for performance to avoid network latency overhead. | |||
Since messages are stored and read by timestamp from Cassandra, and Cassandra | |||
only has eventual consistency, there is an enforced delay in how soon a message | |||
is availalable for reading to ensure a consistent picture of the queue. This | |||
is no less than 5 seconds after insertion, and at most about 15 seconds. | |||
When using queue's that are to be consumed, they must be declared up-front as | |||
a ''partioned'' queue. The amount of partitions should also be specified, and | |||
new messages will be randomly partioned. If messages should be processed in | |||
order, they can be inserted into a single partition to enforce ordering. All | |||
messages that are randomly partitioned should be considered loosely ordered. | |||
== Proposed API (Phase 1) == | == Proposed API (Phase 1) == | ||