We are thinking of exposing a portion of our message bus to customers. To
do we we will be creating a new cluster that can be exposed and will be
linking it to our existing cluster.

I've looked a the Federation plug-in, but it has a feature disqualifies it
for our purposes. We'd like to keep communications from our existing
cluster to the new exposed cluster unidirectional. That means now allowing
incoming connections into the existing cluster, in case there is ever a
RabbitMQ vulnerability (or say, an Erlang or OpenSSL vulnerability) that
would allow the external cluster to be compromised. Since both cluster are
running RabbitMQ, we don't want an attacker to be able to jump from the
external cluster to the internal one. From what I've gathered, the AMQP
client that the Federation plug-in uses executed within the downstream
broker, and thus it has to be able to make an AMQP connection to the
upstream broker.

Is that correct? Any plans to allow for the opposite behavior (upstream
connects to the downstream broker)?

That leaves us with the Shovel plug-in, which can be configured to run on
either the upstream or downstream broker.

One problem with the Shovel plug-in is that it appears you can only manage
the shovels within the configuration file. Is there no way to manage
shovels online, either through AMQP or a management plug-in?

If neither of those are possible, can RabbitMQ reload its configuration
without restarting?

Finally, its unclear whether you should only define a shovel in a single
node in a cluster or in multiple ones? If you define it in multiple ones,
does it create multiple shovel clients competing for messages? If not,
will a shovel fail over if the node with the existing client fails?

Elias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120121/c5fcd934/attachment.htm>

Search Discussions

  • Simone Busoli at Jan 22, 2012 at 12:46 pm

    On Sat, Jan 21, 2012 at 23:58, Elias Levy wrote:

    We are thinking of exposing a portion of our message bus to customers. To
    do we we will be creating a new cluster that can be exposed and will be
    linking it to our existing cluster.

    I've looked a the Federation plug-in, but it has a feature disqualifies it
    for our purposes. We'd like to keep communications from our existing
    cluster to the new exposed cluster unidirectional. That means now allowing
    incoming connections into the existing cluster, in case there is ever a
    RabbitMQ vulnerability (or say, an Erlang or OpenSSL vulnerability) that
    would allow the external cluster to be compromised. Since both cluster are
    running RabbitMQ, we don't want an attacker to be able to jump from the
    external cluster to the internal one. From what I've gathered, the AMQP
    client that the Federation plug-in uses executed within the downstream
    broker, and thus it has to be able to make an AMQP connection to the
    upstream broker.

    Is that correct? Any plans to allow for the opposite behavior (upstream
    connects to the downstream broker)?
    I don't know the internals of the federation plugin well enough to be able
    to answer this question. I know that it uses exchange-exchange bindings and
    I'm not sure if it's even feasible to have it work in the way you describe.

    That leaves us with the Shovel plug-in, which can be configured to run on
    either the upstream or downstream broker.

    One problem with the Shovel plug-in is that it appears you can only manage
    the shovels within the configuration file. Is there no way to manage
    shovels online, either through AMQP or a management plug-in?
    No, there is no way to do that except in the configuration file.

    If neither of those are possible, can RabbitMQ reload its configuration
    without restarting?
    No, you cannot do that. Nonetheless be aware that shovel is not hard to set
    up manually as an external component from the broker, although with some
    gotchas. We've been doing that for some time before the federation plugin
    was published.

    Finally, its unclear whether you should only define a shovel in a single
    node in a cluster or in multiple ones? If you define it in multiple ones,
    does it create multiple shovel clients competing for messages? If not,
    will a shovel fail over if the node with the existing client fails?
    Shovel is a low level mechanis, so whatever you configure it, it will do
    what you told it to, i.e. pick messages from one queue and publish them to
    some exchange. How you set it up is then up to you according to what you
    want to accomplish.

    Elias




    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120122/7b6da4b9/attachment.htm>
  • Elias Levy at Jan 23, 2012 at 9:44 pm

    On Sun, Jan 22, 2012 at 4:46 AM, Simone Busoli wrote:
    On Sat, Jan 21, 2012 at 23:58, Elias Levy wrote:

    We are thinking of exposing a portion of our message bus to customers.
    To do we we will be creating a new cluster that can be exposed and will be
    linking it to our existing cluster.

    I've looked a the Federation plug-in, but it has a feature disqualifies
    it for our purposes. We'd like to keep communications from our existing
    cluster to the new exposed cluster unidirectional. That means now allowing
    incoming connections into the existing cluster, in case there is ever a
    RabbitMQ vulnerability (or say, an Erlang or OpenSSL vulnerability) that
    would allow the external cluster to be compromised. Since both cluster are
    running RabbitMQ, we don't want an attacker to be able to jump from the
    external cluster to the internal one. From what I've gathered, the AMQP
    client that the Federation plug-in uses executed within the downstream
    broker, and thus it has to be able to make an AMQP connection to the
    upstream broker.

    Is that correct? Any plans to allow for the opposite behavior (upstream
    connects to the downstream broker)?
    I don't know the internals of the federation plugin well enough to be able
    to answer this question. I know that it uses exchange-exchange bindings and
    I'm not sure if it's even feasible to have it work in the way you describe.
    I think you are wrong. The Federation plug-in does not use
    exchange-to-exchange bindings. Such bindings are not supported across
    virtual hosts, never mind across clusters. The Federation plug-in merely
    provides a nice illusion of such binding by using an internal AMQP client
    as I described.

    Any other takers?

    That leaves us with the Shovel plug-in, which can be configured to run on
    either the upstream or downstream broker.

    One problem with the Shovel plug-in is that it appears you can only
    manage the shovels within the configuration file. Is there no way to
    manage shovels online, either through AMQP or a management plug-in?
    No, there is no way to do that except in the configuration file.
    That's too bad.

    If neither of those are possible, can RabbitMQ reload its configuration
    without restarting?
    No, you cannot do that. Nonetheless be aware that shovel is not hard to
    set up manually as an external component from the broker, although with
    some gotchas. We've been doing that for some time before the federation
    plugin was published.
    Interesting. Do you have any documentation on how one may go about doing
    so? Or are you merely standing up a new RabbitMQ instance with the shovel
    configured, and not using the broker in those new instances?

    One disadvantage of running the shovel appart from the downstream and
    upstream brokers is that now the messages must cross the network twice,
    once from the upstream to the shovel, and then from the shovel to the
    downstream. With the shovel within the upstream or downstream broker only
    one network transfer is needed.

    Finally, its unclear whether you should only define a shovel in a single
    node in a cluster or in multiple ones? If you define it in multiple ones,
    does it create multiple shovel clients competing for messages? If not,
    will a shovel fail over if the node with the existing client fails?
    Shovel is a low level mechanis, so whatever you configure it, it will do
    what you told it to, i.e. pick messages from one queue and publish them to
    some exchange. How you set it up is then up to you according to what you
    want to accomplish.
    I understand that. My question is whether the shovel is cluster aware,
    like the federation plug-in.

    The federation plug-in, when configured on all nodes, will only make a
    single connection to the upstream broker, and if the nodes with the
    connection dies, another node will create a new one.

    My understanding is that the shovel is not cluster aware. If you configure
    the shovel on each node in your cluster, you'll end up with as many shovels
    and connections as there are nodes. That's not particularly bad, as it
    will give you load balancing across nodes, but I want to confirm my
    understanding.

    Elias
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120123/212f988e/attachment.htm>
  • Simone Busoli at Jan 23, 2012 at 9:48 pm
    What I was trying to say is that it uses ex-ex bindings internally, not
    across brokers
    On Mon, Jan 23, 2012 at 22:44, Elias Levy wrote:

    I think you are wrong. The Federation plug-in does not use
    exchange-to-exchange bindings
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120123/3d513b32/attachment.htm>
  • Elias Levy at Jan 23, 2012 at 10:15 pm

    On Mon, Jan 23, 2012 at 1:48 PM, Simone Busoli wrote:

    What I was trying to say is that it uses ex-ex bindings internally, not
    across brokers
    In what context does it do that? Are you referring to an ex-ex binding
    between the x-federated exchange and its backing exchange (the exchange of
    the selected type)?

    Elias
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120123/c119555d/attachment.htm>
  • Simone Busoli at Jan 23, 2012 at 10:33 pm
    exactly, but as I said I don't know much more about the internals of the
    mechanism
    On Mon, Jan 23, 2012 at 23:15, Elias Levy wrote:
    On Mon, Jan 23, 2012 at 1:48 PM, Simone Busoli wrote:

    What I was trying to say is that it uses ex-ex bindings internally, not
    across brokers
    In what context does it do that? Are you referring to an ex-ex binding
    between the x-federated exchange and its backing exchange (the exchange of
    the selected type)?

    Elias
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120123/5482388c/attachment.htm>
  • Simon MacMullen at Jan 24, 2012 at 11:55 am

    On 23/01/12 22:15, Elias Levy wrote:
    On Mon, Jan 23, 2012 at 1:48 PM, Simone Busoli <simone.busoli at gmail.com
    wrote:

    What I was trying to say is that it uses ex-ex bindings internally,
    not across brokers


    In what context does it do that? Are you referring to an ex-ex binding
    between the x-federated exchange and its backing exchange (the exchange
    of the selected type)?
    Federation requires e-e bindings in the upstream, it creates an internal
    exchange (that it uses to be able to create a load of bindings atomically).

    The connection between the federation exchange and the backing exchange
    is not really a binding.

    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, VMware
  • Simon MacMullen at Jan 24, 2012 at 11:55 am

    On 21/01/12 22:58, Elias Levy wrote:
    We are thinking of exposing a portion of our message bus to customers.
    To do we we will be creating a new cluster that can be exposed and
    will be linking it to our existing cluster.

    I've looked a the Federation plug-in, but it has a feature disqualifies
    it for our purposes. We'd like to keep communications from our
    existing cluster to the new exposed cluster unidirectional. That means
    now allowing incoming connections into the existing cluster, in case
    there is ever a RabbitMQ vulnerability (or say, an Erlang or OpenSSL
    vulnerability) that would allow the external cluster to be compromised.
    Since both cluster are running RabbitMQ, we don't want an attacker to
    be able to jump from the external cluster to the internal one. From
    what I've gathered, the AMQP client that the Federation plug-in uses
    executed within the downstream broker, and thus it has to be able to
    make an AMQP connection to the upstream broker.

    Is that correct? Any plans to allow for the opposite behavior (upstream
    connects to the downstream broker)?
    Not really. Although that could be useful in some situations it does
    mean the plugin wouldn't be able to be smart about which messages to
    send over gap, so it would start to look a lot more like the shovel.
    That leaves us with the Shovel plug-in, which can be configured to run
    on either the upstream or downstream broker.

    One problem with the Shovel plug-in is that it appears you can only
    manage the shovels within the configuration file. Is there no way to
    manage shovels online, either through AMQP or a management plug-in?
    Not now, we'd like to do that in the future.
    If neither of those are possible, can RabbitMQ reload its configuration
    without restarting?
    Not now, we'd like to do that in the future.
    Finally, its unclear whether you should only define a shovel in a single
    node in a cluster or in multiple ones? If you define it in multiple
    ones, does it create multiple shovel clients competing for messages? Yes.
    If
    not, will a shovel fail over if the node with the existing client fails?
    No.

    (Maybe we should look at that...)

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Elias Levy at Jan 24, 2012 at 6:05 pm
    From: Simon MacMullen <simon at rabbitmq.com>
    Federation requires e-e bindings in the upstream, it creates an internal
    exchange (that it uses to be able to create a load of bindings
    atomically).


    Simon,

    Interesting. I suppose you create a new exchange, create a number of
    bindings against the exchange, then create an exchange-to-exchange binding
    against the new exchange to apply all those bindings atomically, yes?

    That's a good trick to know, since AMQP does not give you a way to create
    multiple bindings atomically.

    I am curious as to whether this helps at all with binding churn. I know
    its recommended to long term bindings to avoid binding churn, but this is
    often not possible, as a client may need to change its bindings
    dynamically. Would making use private client exchange, creating the
    dynamic bindings against it, and the creating the single e2e binding help
    with binding churn?

    I imagine it would not, since the additional bindings still will cause lots
    of changes to the distributed Mnesia DB.

    Elias
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120124/f2e47629/attachment.htm>
  • Simon MacMullen at Jan 25, 2012 at 11:13 am

    On 24/01/12 18:05, Elias Levy wrote:
    I am curious as to whether this helps at all with binding churn. I know
    its recommended to long term bindings to avoid binding churn, but this
    is often not possible, as a client may need to change its bindings
    dynamically. Would making use private client exchange, creating the
    dynamic bindings against it, and the creating the single e2e binding
    help with binding churn?
    No, it's still the same underlying operation. Of course "avoid binding
    churn" is just a special case of "avoid unnecessary work" - if the work
    is necessary you can't avoid it...

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJan 21, '12 at 10:58p
activeJan 25, '12 at 11:13a
posts10
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase