I was trying to understand how the clustering support works in RabbitMQ

1) From the documentation the message queue data is not clustered, so if you
have queue created on one machine and the queue is not persisted then that
box going down will cause the queued messages to be destroyed. Is this a
correct interpretation?

I have a system that is creating perishable queue messages, i.e. they need
to be acted on in 100-200 ms or it's to late a timeout will occur further up
the chain. I would like to setup a cluster an protect as many messages as
possible. It would not be worth creating a persistent queue because it
would slow down the processing and as I mentioned these are perishable
messages.

It seems like I should create different queues on each machine in the
cluster and try to load balance across those queues, either by having the
producers write too the local queue, round robin or some other mechanism.
The workers in turn would listen on all queues. Does this sound
reasonable? It requires more configuration than I was hoping for because at
least the workers have to be aware on another queue being created if another
node is added.

Thanks,
Hugh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090605/aa8d1698/attachment.htm

Search Discussions

  • Jason J. W. Williams at Jun 5, 2009 at 4:11 pm
    Hi Hugh,
    Queue data is not replicated to other members of the cluster (whether it's a
    persistent or non-persistent queue), so if a cluster member dies, all queue
    data goes poof if it's non-persistent. However, if it is persistent, while
    queue data doesn't go poof (still in the log) when you restart the node the
    queue data won't auto replay into the cluster. You'll have to do that
    manually.

    -J
    On Fri, Jun 5, 2009 at 8:07 AM, Hugh Watkins wrote:


    I was trying to understand how the clustering support works in RabbitMQ

    1) From the documentation the message queue data is not clustered, so if
    you have queue created on one machine and the queue is not persisted then
    that box going down will cause the queued messages to be destroyed. Is this
    a correct interpretation?

    I have a system that is creating perishable queue messages, i.e. they need
    to be acted on in 100-200 ms or it's to late a timeout will occur further up
    the chain. I would like to setup a cluster an protect as many messages as
    possible. It would not be worth creating a persistent queue because it
    would slow down the processing and as I mentioned these are perishable
    messages.

    It seems like I should create different queues on each machine in the
    cluster and try to load balance across those queues, either by having the
    producers write too the local queue, round robin or some other mechanism.
    The workers in turn would listen on all queues. Does this sound
    reasonable? It requires more configuration than I was hoping for because at
    least the workers have to be aware on another queue being created if another
    node is added.

    Thanks,
    Hugh

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090605/1e236f9e/attachment.htm
  • Keith Irwin at Jun 5, 2009 at 4:25 pm

    On Jun 5, 2009, at 9:11 AM, Jason J. W. Williams wrote:

    Hi Hugh,

    Queue data is not replicated to other members of the cluster
    (whether it's a persistent or non-persistent queue), so if a cluster
    member dies, all queue data goes poof if it's non-persistent.
    However, if it is persistent, while queue data doesn't go poof
    (still in the log) when you restart the node the queue data won't
    auto replay into the cluster. You'll have to do that manually.
    Are you saying that if an instance of RabbitMQ does down and is
    brought back up, its persistent messages won't be delivered when a
    consumer re-attaches? (I'm not sure what you mean by "manually".)

    If that's the case, that seems quite broken and not fault-tolerant at
    all.

    Surely, I'm misunderstanding.....

    I so need to get out from under my workload and write some tests.

    Keith
    -J

    On Fri, Jun 5, 2009 at 8:07 AM, Hugh Watkins wrote:

    I was trying to understand how the clustering support works in
    RabbitMQ

    1) From the documentation the message queue data is not clustered,
    so if you have queue created on one machine and the queue is not
    persisted then that box going down will cause the queued messages to
    be destroyed. Is this a correct interpretation?

    I have a system that is creating perishable queue messages, i.e.
    they need to be acted on in 100-200 ms or it's to late a timeout
    will occur further up the chain. I would like to setup a cluster an
    protect as many messages as possible. It would not be worth
    creating a persistent queue because it would slow down the
    processing and as I mentioned these are perishable messages.

    It seems like I should create different queues on each machine in
    the cluster and try to load balance across those queues, either by
    having the producers write too the local queue, round robin or some
    other mechanism. The workers in turn would listen on all queues.
    Does this sound reasonable? It requires more configuration than I
    was hoping for because at least the workers have to be aware on
    another queue being created if another node is added.

    Thanks,
    Hugh

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090605/8b8caa9d/attachment.htm
  • Matthias Radestock at Jun 8, 2009 at 7:11 pm
    Keith, Jason,

    Keith Irwin wrote:
    On Jun 5, 2009, at 9:11 AM, Jason J. W. Williams wrote:
    when you restart the node the queue data won't auto replay into the
    cluster. You'll have to do that manually.
    Are you saying that if an instance of RabbitMQ does down and is brought
    back up, its persistent messages won't be delivered when a consumer
    re-attaches? (I'm not sure what you mean by "manually".)

    If that's the case, that seems quite broken and not fault-tolerant at all.

    Surely, I'm misunderstanding.....
    Persisted messages *do* get re-queued automatically on the appropriate
    durable queues when a node restarts.

    Jason, did that not work for you? If so, a test case would be appreciated.


    Regards,

    Matthias.
  • Keith Irwin at Jun 8, 2009 at 7:31 pm

    On Mon, Jun 8, 2009 at 12:11 PM, Matthias Radestock wrote:

    Keith, Jason,

    Keith Irwin wrote:
    On Jun 5, 2009, at 9:11 AM, Jason J. W. Williams wrote:

    when you restart the node the queue data won't auto replay into the
    cluster. You'll have to do that manually.
    Are you saying that if an instance of RabbitMQ does down and is brought
    back up, its persistent messages won't be delivered when a consumer
    re-attaches? (I'm not sure what you mean by "manually".)

    If that's the case, that seems quite broken and not fault-tolerant at all.

    Surely, I'm misunderstanding.....
    Persisted messages *do* get re-queued automatically on the appropriate
    durable queues when a node restarts.

    That's good to know! In my research (thus far), about the only AMQP
    implementation I know of that does full replication across a cluster (or
    claims to), is Apache QPID. I'm trying to convince myself that trusting
    anyone's guaranteed messaging strategy is probably not that wise.

    Keith


    Jason, did that not work for you? If so, a test case would be appreciated.


    Regards,

    Matthias.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090608/e22f7950/attachment.htm
  • Alexis Richardson at Jun 8, 2009 at 7:41 pm
    Keith

    As a matter of practicality we have found that 'eventual delivery'
    (ie. what RabbitMQ currently does) combined with various 'lightweight'
    forms of replicated flows, makes sense for many HA cases. If you want
    a combination of HA, reliability *and* timely delivery *and*
    idempotency, then things get a little more 'fun'.

    alexis


    On Mon, Jun 8, 2009 at 8:31 PM, Keith Irwinwrote:
    On Mon, Jun 8, 2009 at 12:11 PM, Matthias Radestock wrote:

    Keith, Jason,

    Keith Irwin wrote:
    On Jun 5, 2009, at 9:11 AM, Jason J. W. Williams wrote:

    when you restart the node the queue data won't auto replay into the
    cluster. ?You'll have to do that manually.
    Are you saying that if an instance of RabbitMQ does down and is brought
    back up, its persistent messages won't be delivered when a consumer
    re-attaches? ?(I'm not sure what you mean by "manually".)

    If that's the case, that seems quite broken and not fault-tolerant at
    all.

    Surely, I'm misunderstanding.....
    Persisted messages *do* get re-queued automatically on the appropriate
    durable queues when a node restarts.
    That's good to know! In my research (thus far), about the only AMQP
    implementation I know of that does full replication across a cluster (or
    claims to), is Apache QPID. I'm trying to convince myself that trusting
    anyone's guaranteed messaging strategy is probably not that wise.
    Keith
    Jason, did that not work for you? If so, a test case would be appreciated.


    Regards,

    Matthias.

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Jason J. W. Williams at Jun 8, 2009 at 8:36 pm
    Hey Matthias,

    In a clustered scenario no that did not work. But I haven't tried it
    since 1.5.1.

    -J

    On Mon, Jun 8, 2009 at 1:11 PM, Matthias Radestockwrote:
    Keith, Jason,

    Keith Irwin wrote:
    On Jun 5, 2009, at 9:11 AM, Jason J. W. Williams wrote:

    when you restart the node the queue data won't auto replay into the
    cluster. ?You'll have to do that manually.
    Are you saying that if an instance of RabbitMQ does down and is brought
    back up, its persistent messages won't be delivered when a consumer
    re-attaches? ?(I'm not sure what you mean by "manually".)

    If that's the case, that seems quite broken and not fault-tolerant at all.

    Surely, I'm misunderstanding.....
    Persisted messages *do* get re-queued automatically on the appropriate
    durable queues when a node restarts.

    Jason, did that not work for you? If so, a test case would be appreciated.


    Regards,

    Matthias.
  • Matthias Radestock at Jun 8, 2009 at 8:48 pm
    Jason,

    Jason J. W. Williams wrote:
    In a clustered scenario no that did not work. But I haven't tried it
    since 1.5.1.
    There was a problem with the recovery of *bindings* in a clustered setup
    that we fixed in 1.5.5. But I have not seen any reports of problems with
    the recovery of *messages* in the 1.5.x releases. So please re-run your
    tests and let us know the results. If there is a bug in the message
    recovery we'll look into it straight away.


    Regards,

    Matthias.
  • Jason J. W. Williams at Jun 8, 2009 at 8:50 pm
    Hi Matthias,

    I'll recheck. The issue we ran into was:

    Node A & Node B

    Node A contains queue 1.
    Node A dies. Persistent unserviced messages remain in queue 1.
    Queue 1 moves to Node B.
    Node A restarts and rejoins cluster.
    Unserviced messages from before failure, do not replay into queue 1 as
    located now on Node B.

    -J


    On Mon, Jun 8, 2009 at 2:48 PM, Matthias Radestockwrote:
    Jason,

    Jason J. W. Williams wrote:
    In a clustered scenario no that did not work. But I haven't tried it
    since 1.5.1.
    There was a problem with the recovery of *bindings* in a clustered setup
    that we fixed in 1.5.5. But I have not seen any reports of problems with the
    recovery of *messages* in the 1.5.x releases. So please re-run your tests
    and let us know the results. If there is a bug in the message recovery we'll
    look into it straight away.


    Regards,

    Matthias.
  • Matthias Radestock at Jun 8, 2009 at 9:19 pm
    Jason,

    Jason J. W. Williams wrote:
    I'll recheck. The issue we ran into was:

    Node A & Node B

    Node A contains queue 1.
    Node A dies. Persistent unserviced messages remain in queue 1.
    Queue 1 moves to Node B.
    Node A restarts and rejoins cluster.
    Unserviced messages from before failure, do not replay into queue 1 as
    located now on Node B.
    Interesting!

    How did you "move" the queue between nodes? That is not a feature that
    is supported in RabbitMQ (yet).

    I suppose that while node A is down a client may connect to node B and
    declare a queue with the same name. I am actually not sure what happens
    in that case, and how it affects re-queueing of persisted messages when
    node A subsequently recovers. So some test scenarios along these lines
    and observations of current behaviour would be very much appreciated.


    Regards,

    Matthias.
  • Jason J. W. Williams at Jun 8, 2009 at 9:30 pm
    Hey Matthias,

    My wording is a bit sloppy. :-) My consumer recreated the queue on
    reconnect. Then when Node A came up, it saw the queue already existed
    and didn't replay into it.

    -J

    On Mon, Jun 8, 2009 at 3:19 PM, Matthias Radestockwrote:
    Jason,

    Jason J. W. Williams wrote:
    I'll recheck. The issue we ran into was:

    Node A & Node B

    Node A contains queue 1.
    Node A dies. Persistent unserviced messages remain in queue 1.
    Queue 1 moves to Node B.
    Node A restarts and rejoins cluster.
    Unserviced messages from before failure, do not replay into queue 1 as
    located now on Node B.
    Interesting!

    How did you "move" the queue between nodes? That is not a feature that is
    supported in RabbitMQ (yet).

    I suppose that while node A is down a client may connect to node B and
    declare a queue with the same name. I am actually not sure what happens in
    that case, and how it affects re-queueing of persisted messages when node A
    subsequently recovers. So some test scenarios along these lines and
    observations of current behaviour would be very much appreciated.


    Regards,

    Matthias.
  • Matthias Radestock at Jun 9, 2009 at 11:02 am
    Jason,


    Jason J. W. Williams wrote:
    My wording is a bit sloppy. :-) My consumer recreated the queue on
    reconnect. Then when Node A came up, it saw the queue already existed
    and didn't replay into it.
    I have just tested this (on 1.5.5 and default, but I am pretty sure it
    would work the same on any 1.5.x version):

    1) create a two-node cluster - with nodes 'rabbit' and 'hare'
    2) connect to 'hare', declare a durable queue 'foo', and send a
    persistent message to it
    3) shut down 'hare'
    4) connect to 'rabbit', declare a durable queue 'foo', and send a
    persistent message to it
    5) restart 'hare'
    6) run 'rabbitmqctl list_queues'

    The last step shows a queue 'foo' with two messages, indicating that the
    message which was sent to the queue on 'hare' got requeued in the new
    queue on 'rabbit' when 'hare' recovered.

    Whether or not that is the right thing to do is debatable, but it's
    certainly not obviously broken.


    Matthias.
  • Arvind Jayaprakash at Jun 5, 2009 at 6:55 pm

    On Jun 05, Hugh Watkins wrote:
    I was trying to understand how the clustering support works in RabbitMQ

    1) From the documentation the message queue data is not clustered, so if you
    have queue created on one machine and the queue is not persisted then that
    box going down will cause the queued messages to be destroyed. Is this a
    correct interpretation?

    I have a system that is creating perishable queue messages, i.e. they need
    to be acted on in 100-200 ms or it's to late a timeout will occur further up
    the chain. I would like to setup a cluster an protect as many messages as
    possible. It would not be worth creating a persistent queue because it
    would slow down the processing and as I mentioned these are perishable
    messages.

    It seems like I should create different queues on each machine in the
    cluster and try to load balance across those queues, either by having the
    producers write too the local queue, round robin or some other mechanism.
    The workers in turn would listen on all queues. Does this sound
    reasonable? It requires more configuration than I was hoping for because at
    least the workers have to be aware on another queue being created if another
    node is added.
    Clustering is not going to help in increasing the queue reliability in
    the current implementation. If you intend to replicate queues on
    multiple nodes, they might as well be standalone nodes. Clustering
    increases the availability of only bindings and exchanges.
  • Alexis Richardson at Jun 5, 2009 at 7:29 pm
    Guys

    I recommend Tony's recent london geek night talk on this subject. I
    think I posted the URL in the last day or two.

    alexis


    On Fri, Jun 5, 2009 at 7:55 PM, Arvind Jayaprakashwrote:
    On Jun 05, Hugh Watkins wrote:
    I was trying to understand how the clustering support works in RabbitMQ

    1) From the documentation the message queue data is not clustered, so if you
    have queue created on one machine and the queue is not persisted then that
    box going down will cause the queued messages to be destroyed. Is this a
    correct interpretation?

    I have a system that is creating perishable queue messages, i.e. they need
    to be acted on in 100-200 ms or it's to late a timeout will occur further up
    the chain. ?I would like to setup a cluster an protect as many messages as
    possible. ?It would not be worth creating a persistent queue ?because it
    would slow down the processing and as I mentioned these are perishable
    messages.

    It seems like I should create different queues on each machine in the
    cluster and try to load balance across those queues, either by having the
    producers write too the local queue, round robin or some other mechanism.
    The workers in turn would listen on all queues. ?Does this sound
    reasonable? ?It requires more configuration than I was hoping for because at
    least the workers have to be aware on another queue being created if another
    node is added.
    Clustering is not going to help in increasing the queue reliability in
    the current implementation. If you intend to replicate queues on
    multiple nodes, they might as well be standalone nodes. Clustering
    increases the availability of only bindings and exchanges.

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 5, '09 at 2:07p
activeJun 9, '09 at 11:02a
posts14
users6
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase