I have 20000 devices which send the messages that I push to RabbitMQ, on the other side I have 1 or more worker services that process the message and reply to a custom response queue on RabbitMQ. I need to ensure strict message ordering in RabbitMQ while also achieving high throughput.
I've searched the web and these are the options that I came up with:
- Single queue, single consumer
Guarantees order but limits throughput since only one consumer can process messages at a time. - Single queue, single consumer with
ConcurrentMessageLimit
> 1
Risks breaking order due to concurrent processing within the consumer. - Queue per device, single consumer per queue
Could work, but would need to create a lot of queues. Get's complicated if user adds a new device or removes one. - RabbitMQ Super Streams with Single Active Consumer
This option looks promising, however in the end it really looks like option 3 with added throughput benefits. Or am I missing something? As far as I understand I still need to have 20000 streams and handle the cases when device is added or remove to do the cleanup of individual streams.
For example, if I have 3 queues as in the official example this would still mean that I have 3 active consumers but with concurrency limit = 1. Which means I can achieve high throughput by adding as many streams as necessary to suffice for my workload. Did I get it right? What if workload increases - does this mean I need to be able to add streams dynamically, is this even supported by super streams?