Confirmed users
192
edits
Services/MessageQueuing: Difference between revisions
Jump to navigation
Jump to search
Added more questions and thoughts on workers and use cases
(more refinement of concepts) |
(Added more questions and thoughts on workers and use cases) |
||
| Line 3: | Line 3: | ||
What is the simplest MQ system and features to add from there? | What is the simplest MQ system and features to add from there? | ||
*Put a single message on a queue for later retrieval in a FIFO manner, non-persistent, | *Put a single message on a queue for later retrieval in a FIFO manner, non-persistent, via a RESTful service to push and poll for messages (send XX many at once per client instruction?) | ||
*Transient messages (Semi-durable, higher throughput at the possible loss of some recent messages in a crash) | *Transient messages (Semi-durable, higher throughput at the possible loss of some recent messages in a crash) | ||
*Persistent messages (Accessible until consumed) | *Persistent messages (Accessible until consumed) | ||
*Prioritizing of messages? Or allow for multiple queues and have the client prioritize where it pulls messages from? | *Prioritizing of messages? Or allow for multiple queues and have the client prioritize where it pulls messages from? | ||
*Allow messages to go | *Allow messages to go unacknowledged to allow a client to 'time-out' on handling it? | ||
*Message push (persistent MQ connection), reduces latency but increases complexity. Pub/Sub or load-balanced on multiple connected clients? | *Message push (persistent MQ connection), reduces latency but increases complexity. Pub/Sub or load-balanced on multiple connected clients? | ||
Trade-offs of | Trade-offs of implementation | ||
* | *Non-persistent allows use of memcached for higher throughput | ||
* | *Fully persistent (i.e., backed by a db) would be much slower | ||
Crypto | Crypto | ||
| Line 22: | Line 21: | ||
Apps should be able to create both durable and temporary queues on-demand, as well as assign a TTL to a queue for temporary use. | Apps should be able to create both durable and temporary queues on-demand, as well as assign a TTL to a queue for temporary use. | ||
Authentication | |||
*maybe [[Services/AppKeys|AppKeys]] would work, depending on if only the App could send messages... might need something else if end-user clients can directly push messages via Javascript | |||
Message Handling | |||
*Should the MQ Service do anything beyond merely holding messages and allowing connections for their retrieval? | |||
*Simplest has no built-in handling, app would need to poll MQ service for new messages | |||
*Allow a message payload to contain a callback to a RESTful service (URL + POST payload)? | |||
Queue Workers? | |||
*Introduces additional challenge of requiring Apps to have the code the worker should call for a message accessible to the MQ Worker Service | |||
*Feels like a separate service from the MQ Service (though highly related) | |||
*Requires worker pools on machines, worker watchers, resource monitoring, worker killers for excessive resource usage | |||
*Provide 'built-in' queue workers for delayed common tasks? i.e. sending email, update caches. | |||