Confirmed users
192
edits
Services/MessageQueuing: Difference between revisions
m
→Planning Questions
(Added more questions and thoughts on workers and use cases) |
|||
| Line 10: | Line 10: | ||
*Message push (persistent MQ connection), reduces latency but increases complexity. Pub/Sub or load-balanced on multiple connected clients? | *Message push (persistent MQ connection), reduces latency but increases complexity. Pub/Sub or load-balanced on multiple connected clients? | ||
Trade-offs of implementation | Trade-offs of implementation / back-ends | ||
*Non-persistent allows use of memcached for higher throughput | *Non-persistent allows use of memcached for higher throughput | ||
*Semi-durable allows backends that do occasional fsync's (Mongo / Redis) | |||
*Fully persistent (i.e., backed by a db) would be much slower | *Fully persistent (i.e., backed by a db) would be much slower | ||
Or should an App creator get to choose which characteristic they want when creating a queue? | |||
i.e. someone making a real-time chat/broadcast system could choose non-persistent for lots of messages of little importance if they're missed... | |||
someone sending verification emails would choose fully persistent to ensure it actually gets sent eventually | |||
Crypto | Crypto | ||
| Line 31: | Line 36: | ||
*Simplest has no built-in handling, app would need to poll MQ service for new messages | *Simplest has no built-in handling, app would need to poll MQ service for new messages | ||
*Allow a message payload to contain a callback to a RESTful service (URL + POST payload)? | *Allow a message payload to contain a callback to a RESTful service (URL + POST payload)? | ||
*Message TTL? | |||
Queues | |||
*Single default queue per App seems reasonable | |||
*Multiple queues per App? Routing of messages to multiple queues based on route key like in RabbitMQ? | |||
*Temporary queues for return-paths? (i.e., using a queue to farm out something then waiting for it to complete and return data) | |||
Queue Workers? | Queue Workers? | ||
| Line 36: | Line 47: | ||
*Introduces additional challenge of requiring Apps to have the code the worker should call for a message accessible to the MQ Worker Service | *Introduces additional challenge of requiring Apps to have the code the worker should call for a message accessible to the MQ Worker Service | ||
*Feels like a separate service from the MQ Service (though highly related) | *Feels like a separate service from the MQ Service (though highly related) | ||
*Requires worker pools on machines, worker watchers, resource monitoring, worker killers for excessive resource usage | *Requires worker pools on machines, worker watchers, resource monitoring, worker killers for excessive resource usage, deploying additional machines as App tasks require more task processing power under load | ||
*Provide 'built-in' queue workers for delayed common tasks? i.e. sending email, update caches. | *Provide 'built-in' queue workers for delayed common tasks? i.e. sending email, update caches. | ||
*Handling 'excessive resource usage' requires consistent way to determine CPU usage per task regardless of existing machine utilization, kill workers that use more than XXX RSS memory, etc. | |||
Introspection / Administration | |||
*How much should one be able to see about current queue activity? Total queues per app? Total messages in each queue? Ability to examine payload of individual message? | |||
*Most basic feature-set should include some way for an App to clear/reset one of its queues | |||