|
|
| Line 1: |
Line 1: |
| == Planning Questions == | | = Overview = |
|
| |
|
| What is the simplest MQ system and features to add from there?
| | The '''Message Queue''' project is part of [[Services/Roadmap/Sagrada|Project Sagrada]] |
| | , providing a service for applications to queue messages for readers. |
|
| |
|
| *Put a single message on a queue for later retrieval in a FIFO manner, non-persistent, via a RESTful service to push and poll for messages (send XX many at once per client instruction?)
| | The first version of the MQ will focus on providing a useful service for the |
| *Transient messages (Semi-durable, higher throughput at the possible loss of some recent messages in a crash)
| | [[Services/Notifications|Notifications project]] to enqueue messages for many |
| *Persistent messages (Accessible until consumed)
| | queues and for subscribers to poll for messages. |
| *Prioritizing of messages? Or allow for multiple queues and have the client prioritize where it pulls messages from?
| |
| *Allow messages to go unacknowledged to allow a client to 'time-out' on handling it?
| |
| *Message push (persistent MQ connection), reduces latency but increases complexity. Pub/Sub or load-balanced on multiple connected clients?
| |
| *Chat and 'live' apps using the MQ for comet?
| |
|
| |
|
| Trade-offs of implementation / back-ends
| | = Project = |
|
| |
|
| *Non-persistent allows use of memcached for higher throughput
| | == Engineers == |
| *Semi-durable allows backends that do occasional fsync's (Mongo / Redis)
| |
| *Fully persistent (i.e., backed by a db, RabbitMQ persist) would be much slower
| |
|
| |
|
| Or should an App creator get to choose which characteristic they want when creating a queue?
| | * Ben Bangert |
| i.e. someone making a real-time chat/broadcast system could choose non-persistent for lots of messages of little importance if they're missed...
| |
| someone sending verification emails would choose fully persistent to ensure it actually gets sent eventually
| |
|
| |
|
| Crypto
| | == Use Cases == |
|
| |
|
| *Payload could be encrypted at the client layer
| | === Notifications === |
| *If routing keys are used (RabbitMQ style brokering), those could be encrypted as well
| |
| *Crypto less useful unless other tasks needed to hide payloads, which should likely be done on client-side
| |
|
| |
|
| Apps should be able to create both durable and temporary queues on-demand, as well as assign a TTL to a queue for temporary use.
| | The new version of [[Services/Notifications/Push|Notifications]] will use the |
| | MQ to store messages for users from various websites and allow users to poll |
| | the MQ directly for new notifications they have allowed. |
|
| |
|
| Authentication
| | * Internal app add's messages to a queue by queue name |
| | | * Devices can read a queue by name |
| *maybe [[Services/AppKeys|AppKeys]] would work, depending on if only the App could send messages... might need something else if end-user clients can directly push messages via Javascript | | * Devices can read the X most recent messages |
| *If used for message passing between end-user's vs. a separate Service app, clients would need permission to be on the queue, likely based on whether they're 'logged in' to the App using the queue
| | * Devices can read messages since XXX |
| | |
| Message Handling
| |
| | |
| *Should the MQ Service do anything beyond merely holding messages and allowing connections for their retrieval?
| |
| *Simplest has no built-in handling, app would need to poll MQ service for new messages
| |
| *Allow a message payload to contain a callback to a RESTful service (URL + POST payload)?
| |
| *Message TTL?
| |
| | |
| Queues
| |
| | |
| *Single default queue per App seems reasonable
| |
| *Multiple queues per App? Routing of messages to multiple queues based on route key like in RabbitMQ? | |
| *Temporary queues for return-paths? (i.e., using a queue to farm out something then waiting for it to complete and return data)
| |
| | |
| Queue Workers?
| |
| | |
| Simplest: No queue workers, service calls back into special App URL
| |
| Drawback: Can't determine as easily if the Apps are busy processing 'task' jobs or end-user requests
| |
| | |
| *Introduces additional challenge of requiring Apps to have the code the worker should call for a message accessible to the MQ Worker Service | |
| *Feels like a separate service from the MQ Service (though highly related) | |
| *Requires worker pools on machines, worker watchers, resource monitoring, worker killers for excessive resource usage, deploying additional machines as App tasks require more task processing power under load
| |
| *Provide 'built-in' queue workers for delayed common tasks? i.e. sending email, update caches.
| |
| *Handling 'excessive resource usage' requires consistent way to determine CPU usage per task regardless of existing machine utilization, kill workers that use more than XXX RSS memory, etc.
| |
| | |
| Introspection / Administration
| |
| | |
| *How much should one be able to see about current queue activity? Total queues per app? Total messages in each queue? Ability to examine payload of individual message?
| |
| *Most basic feature-set should include some way for an App to clear/reset one of its queues
| |