On Sun, Jan 22, 2012 at 4:46 AM, Simone Busoli wrote:
On Sat, Jan 21, 2012 at 23:58, Elias Levy wrote:
We are thinking of exposing a portion of our message bus to customers.
To do we we will be creating a new cluster that can be exposed and will be
linking it to our existing cluster.
I've looked a the Federation plug-in, but it has a feature disqualifies
it for our purposes. We'd like to keep communications from our existing
cluster to the new exposed cluster unidirectional. That means now allowing
incoming connections into the existing cluster, in case there is ever a
RabbitMQ vulnerability (or say, an Erlang or OpenSSL vulnerability) that
would allow the external cluster to be compromised. Since both cluster are
running RabbitMQ, we don't want an attacker to be able to jump from the
external cluster to the internal one. From what I've gathered, the AMQP
client that the Federation plug-in uses executed within the downstream
broker, and thus it has to be able to make an AMQP connection to the
Is that correct? Any plans to allow for the opposite behavior (upstream
connects to the downstream broker)?
I don't know the internals of the federation plugin well enough to be able
to answer this question. I know that it uses exchange-exchange bindings and
I'm not sure if it's even feasible to have it work in the way you describe.
I think you are wrong. The Federation plug-in does not use
exchange-to-exchange bindings. Such bindings are not supported across
virtual hosts, never mind across clusters. The Federation plug-in merely
provides a nice illusion of such binding by using an internal AMQP client
as I described.
Any other takers?
That leaves us with the Shovel plug-in, which can be configured to run on
either the upstream or downstream broker.
One problem with the Shovel plug-in is that it appears you can only
manage the shovels within the configuration file. Is there no way to
manage shovels online, either through AMQP or a management plug-in?
No, there is no way to do that except in the configuration file.
That's too bad.
If neither of those are possible, can RabbitMQ reload its configuration
No, you cannot do that. Nonetheless be aware that shovel is not hard to
set up manually as an external component from the broker, although with
some gotchas. We've been doing that for some time before the federation
plugin was published.
Interesting. Do you have any documentation on how one may go about doing
so? Or are you merely standing up a new RabbitMQ instance with the shovel
configured, and not using the broker in those new instances?
One disadvantage of running the shovel appart from the downstream and
upstream brokers is that now the messages must cross the network twice,
once from the upstream to the shovel, and then from the shovel to the
downstream. With the shovel within the upstream or downstream broker only
one network transfer is needed.
Finally, its unclear whether you should only define a shovel in a single
node in a cluster or in multiple ones? If you define it in multiple ones,
does it create multiple shovel clients competing for messages? If not,
will a shovel fail over if the node with the existing client fails?
Shovel is a low level mechanis, so whatever you configure it, it will do
what you told it to, i.e. pick messages from one queue and publish them to
some exchange. How you set it up is then up to you according to what you
want to accomplish.
I understand that. My question is whether the shovel is cluster aware,
like the federation plug-in.
The federation plug-in, when configured on all nodes, will only make a
single connection to the upstream broker, and if the nodes with the
connection dies, another node will create a new one.
My understanding is that the shovel is not cluster aware. If you configure
the shovel on each node in your cluster, you'll end up with as many shovels
and connections as there are nodes. That's not particularly bad, as it
will give you load balancing across nodes, but I want to confirm my