Hi All,

During my investigations for designing a completely fail safe messaging
scenario the ideal option was to have local brokers that shovel to a central
broker. That way if the central brokers ever went down messages would just
be queued locally for delivery later.

I finally got around to using the shovel plugin and while it had everything
I was looking for, there were some drawbacks (for me at least). It seems
that whenever I want to create a queue that gets pushed to an exchange on
another server I have to edit the rabbitmq.config for this and then restart
the service. I'm just curious if there is a softer way to handle this that
allows for more dynamic forwarding (one feature I always liked about AMQP is
that client apps can define exchanges and queues rather than having an admin
set them up).


I'm also open to other possibilities for HA that handle when the main broker
cluster goes offline or is unreachable. :)

Thanks,
James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111011/e1ff03fc/attachment.htm>

Search Discussions

  • Simon MacMullen at Oct 13, 2011 at 1:58 pm

    On 11/10/11 17:51, James Carr wrote:
    I'm just curious
    if there is a softer way to handle this that allows for more dynamic
    forwarding (one feature I always liked about AMQP is that client apps
    can define exchanges and queues rather than having an admin set them up).
    You could check out the federation plugin:

    http://www.rabbitmq.com/plugins.html#rabbitmq_federation
    http://hg.rabbitmq.com/rabbitmq-federation/file/default/README
    http://www.rabbitmq.com/blog/2011/06/22/federation-plugin-preview-release/

    That's intended to be a slightly higher-level thing than the shovel.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • James Carr at Oct 19, 2011 at 7:53 pm
    Hi,

    Reading the documentation, I noticed this statement:

    "There are of course limitations, since this is a preview release. The
    worst is that federation is not compatible with clustering. You
    shouldn?t use the federation plugin in a cluster. This is the first
    thing we?re going to fix."

    Just so I can be sure I parsed that statement correctly, it's not
    compatible when used as part of a cluster, not if the upstream is a
    cluster. Is that correct?

    Thanks,
    James

    On Thu, Oct 13, 2011 at 8:58 AM, Simon MacMullen wrote:
    On 11/10/11 17:51, James Carr wrote:

    ?I'm just curious
    if there is a softer way to handle this that allows for more dynamic
    forwarding (one feature I always liked about AMQP is that client apps
    can define exchanges and queues rather than having an admin set them up).
    You could check out the federation plugin:

    http://www.rabbitmq.com/plugins.html#rabbitmq_federation
    http://hg.rabbitmq.com/rabbitmq-federation/file/default/README
    http://www.rabbitmq.com/blog/2011/06/22/federation-plugin-preview-release/

    That's intended to be a slightly higher-level thing than the shovel.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Oct 20, 2011 at 8:25 am

    On 19/10/2011 8:53PM, James Carr wrote:
    Reading the documentation, I noticed this statement:

    "There are of course limitations, since this is a preview release. The
    worst is that federation is not compatible with clustering. You
    shouldn?t use the federation plugin in a cluster. This is the first
    thing we?re going to fix."
    Hmm. That's actually part of the blog post about the 2.5.0 preview
    release. This is already fixed in 2.6.x. I wonder how I could make that
    clearer.

    (But for the record it was the downstream that couldn't be a cluster.)

    Cheers, Simon
  • James Carr at Oct 20, 2011 at 2:24 pm
    Thanks, I tried it and it works advertised, but I feel like I may have
    misunderstood it.

    I've set it up and have messages published to a local exchange being
    routed to an exchange on a central broker. however when the central
    broker is down there's no local queuing of messages until it comes
    back up.

    Of course, maybe I am just designing my topology wrong. Maybe instead
    of separate exchanges per application I should just have one exchange,
    shoveled out to the remote exchange and then multiple queues bound to
    that exchange. The exchange per app was going to make things
    heavyweight with the shovel plugin since I'd have to restart local
    brokers for each new exchange, but with a single exchange, single
    local queue and multiple remote queues might make things more
    effecient.

    Maybe the federation plugin includes a backing queue that can be
    switched on or am I missing something?

    Thanks,
    James

    On Thu, Oct 20, 2011 at 3:25 AM, Simon MacMullen wrote:
    On 19/10/2011 8:53PM, James Carr wrote:

    Reading the documentation, I noticed this statement:

    "There are of course limitations, since this is a preview release. The
    worst is that federation is not compatible with clustering. You
    shouldn?t use the federation plugin in a cluster. This is the first
    thing we?re going to fix."
    Hmm. That's actually part of the blog post about the 2.5.0 preview release.
    This is already fixed in 2.6.x. I wonder how I could make that clearer.

    (But for the record it was the downstream that couldn't be a cluster.)

    Cheers, Simon
  • Simon MacMullen at Oct 20, 2011 at 4:26 pm

    On 20/10/11 15:24, James Carr wrote:
    Thanks, I tried it and it works advertised, but I feel like I may have
    misunderstood it.
    Just to be clear, we are still talking about the federation exchange, right?
    I've set it up and have messages published to a local exchange being
    routed to an exchange on a central broker. however when the central
    broker is down there's no local queuing of messages until it comes
    back up.
    That should really be happening. I am assuming your local exchange is
    the upstream and the central exchange is the downstream. When the
    downstream connects to the upstream it should create a queue (with a
    name beginning "federation: " on the upstream. Can you see this in mgmt
    / rabbitmqctl? Of course, if you use the queue_expires parameter the
    upstream queue will expire eventually.
    Of course, maybe I am just designing my topology wrong. Maybe instead
    of separate exchanges per application I should just have one exchange,
    shoveled out to the remote exchange and then multiple queues bound to
    that exchange. The exchange per app was going to make things
    heavyweight with the shovel plugin since I'd have to restart local
    brokers for each new exchange, but with a single exchange, single
    local queue and multiple remote queues might make things more
    effecient.

    Maybe the federation plugin includes a backing queue that can be
    switched on or am I missing something?
    It is always on - even if the downstream does not disconnect then the
    upstream might publish messages faster than they can get sent across the
    link.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • James Carr at Oct 20, 2011 at 6:37 pm
    Maybe I misunderstand? :)

    Given that I have a cluster of brokers used for a single geographical
    location and each machine that publishes messages has a local broker
    that pushes messages out to the central cluster:

    The cluster is the upstream
    The local brokers are the downstream


    Or is it the other way around?


    Thanks,
    James
    On Thu, Oct 20, 2011 at 11:26 AM, Simon MacMullen wrote:
    On 20/10/11 15:24, James Carr wrote:

    Thanks, I tried it and it works advertised, but I feel like I may have
    misunderstood it.
    Just to be clear, we are still talking about the federation exchange, right?
    I've set it up and have messages published to a local exchange being
    routed to an exchange on a central broker. however when the central
    broker is down there's no local queuing of messages until it comes
    back up.
    That should really be happening. I am assuming your local exchange is the
    upstream and the central exchange is the downstream. When the downstream
    connects to the upstream it should create a queue (with a name beginning
    "federation: " on the upstream. Can you see this in mgmt / rabbitmqctl? Of
    course, if you use the queue_expires parameter the upstream queue will
    expire eventually.
    Of course, maybe I am just designing my topology wrong. Maybe instead
    of separate exchanges per application I should just have one exchange,
    shoveled out to the remote exchange and then multiple queues bound to
    that exchange. The exchange per app was going to make things
    heavyweight with the shovel plugin since I'd have to restart local
    brokers for each new exchange, but with a single exchange, single
    local queue and multiple remote queues might make things more
    effecient.

    Maybe the federation plugin includes a backing queue that can be
    switched on or am I ?missing something?
    It is always on - even if the downstream does not disconnect then the
    upstream might publish messages faster than they can get sent across the
    link.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Simon MacMullen at Oct 21, 2011 at 9:26 am

    On 20/10/11 19:37, James Carr wrote:
    Maybe I misunderstand? :)

    Given that I have a cluster of brokers used for a single geographical
    location and each machine that publishes messages has a local broker
    that pushes messages out to the central cluster:

    The cluster is the upstream
    The local brokers are the downstream


    Or is it the other way around?
    It is. Upstream and downstream refer to message flow.

    (Of course you can set it up in both directions too, but it sounds like
    that's not what you want.)

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedOct 11, '11 at 4:51p
activeOct 21, '11 at 9:26a
posts8
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Simon MacMullen: 4 posts James Carr: 4 posts

People

Translate

site design / logo © 2021 Grokbase