Confirmed users
192
edits
Services/MessageQueuing: Difference between revisions
Jump to navigation
Jump to search
no edit summary
mNo edit summary |
No edit summary |
||
| Line 50: | Line 50: | ||
Requirements: | Requirements: | ||
* Service App can create queues | * Service App can create queues, which can be partitioned | ||
* Service App can add messages to queue | * Service App can add messages to queue, and specify partition to retain ordering | ||
* | * Clients can ask for information about how many partitions a queue has | ||
* | * Clients may read any queue, and its partitions that they are aware of | ||
== | == Design Background == | ||
For scalability purposes, since some messages may be retained for periods of time | For scalability purposes, since some messages may be retained for periods of time | ||
| Line 109: | Line 109: | ||
messages that are randomly partitioned should be considered loosely ordered. | messages that are randomly partitioned should be considered loosely ordered. | ||
== | == Architecture == | ||
'''Queue Storage Backend''' | |||
Cassandra | |||
'''Coordination Service''' | |||
(Only used when consuming from the queue) | |||
Apache Zookeeper | |||
'''Queue Consumption''' | |||
* Consumers coordinate with coordination service | |||
* Consumers split queues/partitions amongst themselves | |||
* Consumers record in coordination service the farthest they've processed | |||
in a queue/partition | |||
* Consumers rebalance queue/partition allocation when consumers are added or | |||
removed using coordination service | |||
* Consumption and rebalancing is done entirely client-side | |||
== Proposed API == | |||
Applications allowed to use the '''Message Queue''' | |||
will be given an application key that must be sent with every request, and | will be given an application key that must be sent with every request, and | ||
their IP must be on an accepted IP list for the given application key. | their IP must be on an accepted IP list for the given application key. | ||
All methods must include the header 'X-Application-Name' which indicates the name of the | |||
application using the queue. Queue's are unique within an application. | |||
For methods requiring authentication (internal apps), the application key must | |||
be sent as a HTTP header named 'ApplicationKey'. | |||
=== Internal Apps === | === Internal Apps === | ||
These methods are authenticated by IP, and are intended for use by Services Applications. | |||
'''POST /queue''' | '''POST /queue''' | ||
Creates a new queue | Creates a new queue | ||
Params: | |||
''queue_name'' (Optional) - Name of the queue to create | |||
''partitions'' (Optional) - How many partitions the queue should have (defaults to 1) | |||
'''DELETE /queue/{queue_name}''' | '''DELETE /queue/{queue_name}''' | ||
Deletes a given queue created by the App. | Deletes a given queue created by the App. | ||
Params: | |||
''delete'' (Optional) - If set to ''false'', then the queue contents will be deleted, but the queue will remain registered. | |||
'''POST /queue/{queue_name}''' | '''POST /queue/{queue_name}''' | ||
| Line 139: | Line 176: | ||
'''GET /queue/{queue_name}''' | '''GET /queue/{queue_name}''' | ||
Returns | Returns messages for the queue. | ||
Params: | |||
''since_timestamp'' - All messages newer than this timestamp, should be formatted as seconds since epoch in GMT | ''since_timestamp'' (Optional) - All messages newer than this timestamp, should be formatted as seconds since epoch in GMT | ||
''limit'' - Only return N amount of messages | ''limit'' (Optional) - Only return N amount of messages, defaults to 100 | ||
''order'' (Optional) - 'descending' or 'ascending', defaults to descending | |||
Messages are returned in order of newest to oldest. | Messages are returned in order of newest to oldest. | ||
'''GET /queue/{queue_name}/info''' | |||
Returns information about the queue. | |||
Example response: | |||
{ | |||
'partitions': 4, | |||
'created': 1322521547 | |||
} | |||
== Use Cases == | == Use Cases == | ||
| Line 161: | Line 211: | ||
* Devices can read messages since XXX | * Devices can read messages since XXX | ||
* Messages are encoded as JSON | * Messages are encoded as JSON | ||
=== Socorro === | |||
Socorro is Mozilla's crash reporting service, receiving in excess of 8 thousand | |||
crashes per second during peaks. The MQ will be utilized to hold crash reports as | |||
they come in, and used by consumers to process the crash reports. | |||
* Application Key and Application Queue's will be created during setup. | |||
* Webheads will be adding messages to the queue by queue name, with enough parititons | |||
setup for the various consumers. | |||
* Consumers will process crashes off the queues | |||