"justin.adam.miller at gmail.com" <justin.adam.miller at gmail.com> writes:
I have an app that will have, at any one time, < 500 connections to a
single broker, across multiple machines, all of whom I want as
consumers of one queue so that rabbit can load balance amongst them.
I have an app that will have, at any one time, < 500 connections to a
single broker, across multiple machines, all of whom I want as
consumers of one queue so that rabbit can load balance amongst them.
relatively modest hardware.
As for 500 consumers of a single queue, it doesn't seem likely to be a
problem. But you could prototype it and see what you get. Try to match
your expected message throughput, message size distribution, and
consumption intervals (how long between a message being delivered and
being acked). And try with hardware similar or less capable than what
you expect to use in production.
Is handling so few connections/consumers a joke for rabbit? Should I
consider clustering, and if so, what does clustering buy me in this
case? The one thing I don't love about clustering is that, without
active/passive failover, the node that declares the queue is a single
point of failure.
Currently clustering won't buy you anything in this context.consider clustering, and if so, what does clustering buy me in this
case? The one thing I don't love about clustering is that, without
active/passive failover, the node that declares the queue is a single
point of failure.
As for active/active failover, there will be developments in that area
soon. But even then, a single queue will remain a potential bottleneck.
The idea of clustering along with multiple queues bound to one
exchange is appealing (since exchange and binding information is
replicated across the cluster), but rabbit will not round robin
messages to all the bound queues -- there's no exchange type (that I
know of) that has that behavior.
If I knew the bound queues ahead of time, then I could pick one at
random, but it seems like determining that queue would be cumbersome,
inefficient and/or costly at best.
If you were to go down this route, there may be advantages to usingexchange is appealing (since exchange and binding information is
replicated across the cluster), but rabbit will not round robin
messages to all the bound queues -- there's no exchange type (that I
know of) that has that behavior.
If I knew the bound queues ahead of time, then I could pick one at
random, but it seems like determining that queue would be cumbersome,
inefficient and/or costly at best.
multiple brokers rather than clustering. E.g., have one queue in each
broker, with each consumer coonsuming from one of them. Then each
producer connects to each broker and publishes each message to one of
them, chosen at random. This would allow you to do things like rolling
upgrades of the brokers. Clustering won't, as we don't support mixed
version clusters.
--
David Wragg
Staff Engineer, RabbitMQ
VMware, Inc.
