Hi,

Is it possible to use RabbitMQ as a reliable message queue? What I
mean is to make it preserve the order of messages in queues even for
non ack'ed durable messages.
Putting non acked messages into the end of the queue just doesn't make sence.

http://www.rabbitmq.com/faq.html#message-ordering

Thank you!

Search Discussions

  • Jason J. W. Williams at Jun 14, 2011 at 4:08 pm

    Is it possible to use RabbitMQ as a reliable message queue? What I
    mean is to make it preserve the order of messages in queues even for
    non ack'ed durable messages.
    Putting non acked messages into the end of the queue just doesn't make sence.
    Actually, it would be difficult to preserve the order in the case of
    non-ack. Let's say you've got M1, M2, M3, M4 in the queue (like the
    FAQ describes). If you have two consumers, and C1 is consuming M1 and
    C2 is consuming M2. C1 finishes, acks M1 and starts consuming M3.
    However, C2 fails and disconnects without acking M2, where should M2
    go in the order? M3 is already being consumed. The only sane approach
    is to treat M2 as a new submission and append it to the end of the
    queue.

    -J
  • Emile Joubert at Jun 15, 2011 at 9:54 am

    On 14/06/11 17:08, Jason J. W. Williams wrote:
    Is it possible to use RabbitMQ as a reliable message queue? What I
    mean is to make it preserve the order of messages in queues even for
    non ack'ed durable messages.
    Putting non acked messages into the end of the queue just doesn't make sence.
    Actually, it would be difficult to preserve the order in the case of
    non-ack. Let's say you've got M1, M2, M3, M4 in the queue (like the
    FAQ describes). If you have two consumers, and C1 is consuming M1 and
    C2 is consuming M2. C1 finishes, acks M1 and starts consuming M3.
    However, C2 fails and disconnects without acking M2, where should M2
    go in the order? M3 is already being consumed. The only sane approach
    is to treat M2 as a new submission and append it to the end of the
    queue.
    Yes, it is not obvious how to requeue in the presence of multiple
    consumers. But it is possible to do better in the case of a single
    consumer where the inconsistency can't arise. At present rabbit always
    requeues at the back of the queue, i.e. treat it as a new message. This
    is consistent with the spec which only guarantees ordering along the
    same path from a single producer to a single consumer.


    -Emile
  • Toby Corkindale at Jun 16, 2011 at 1:14 am

    On 15/06/11 19:54, Emile Joubert wrote:
    On 14/06/11 17:08, Jason J. W. Williams wrote:
    Is it possible to use RabbitMQ as a reliable message queue? What I
    mean is to make it preserve the order of messages in queues even for
    non ack'ed durable messages.
    Putting non acked messages into the end of the queue just doesn't make sence.
    Actually, it would be difficult to preserve the order in the case of
    non-ack. Let's say you've got M1, M2, M3, M4 in the queue (like the
    FAQ describes). If you have two consumers, and C1 is consuming M1 and
    C2 is consuming M2. C1 finishes, acks M1 and starts consuming M3.
    However, C2 fails and disconnects without acking M2, where should M2
    go in the order? M3 is already being consumed. The only sane approach
    is to treat M2 as a new submission and append it to the end of the
    queue.
    Yes, it is not obvious how to requeue in the presence of multiple
    consumers. But it is possible to do better in the case of a single
    consumer where the inconsistency can't arise. At present rabbit always
    requeues at the back of the queue, i.e. treat it as a new message. This
    is consistent with the spec which only guarantees ordering along the
    same path from a single producer to a single consumer.
    I quite like the "send to the back of the queue" behaviour.
    In the event of a bad message that is always failing/crashing on the
    consumer, it'll get pushed to the back of the queue to allow other good
    messages to be processed.

    In a system where the messages are always delivered in-order, that bad
    message would effectively lock the system up.

    Toby
  • T-zex at Jun 16, 2011 at 8:14 am
    Unexpected shut down of a process (power failure) will result in
    "good" message being put to the end of the queue - a very unexpected
    behaviour. In case of a "bad" message application should decide that
    it is bad and what should be done with it - requeue (unlikely) or get
    rid of it.

    On Thu, Jun 16, 2011 at 2:14 AM, Toby Corkindale
    wrote:
    On 15/06/11 19:54, Emile Joubert wrote:
    On 14/06/11 17:08, Jason J. W. Williams wrote:

    Is it possible to use RabbitMQ as a reliable message queue? What I
    mean is to make it preserve the order of messages in queues even for
    non ack'ed durable messages.
    Putting non acked messages into the end of the queue just doesn't make
    sence.
    Actually, it would be difficult to preserve the order in the case of
    non-ack. Let's say you've got M1, M2, M3, M4 in the queue (like the
    FAQ describes). If you have two consumers, and C1 is consuming M1 and
    C2 is consuming M2. C1 finishes, acks M1 and starts consuming M3.
    However, C2 fails and disconnects without acking M2, where should M2
    go in the order? M3 is already being consumed. The only sane approach
    is to treat M2 as a new submission and append it to the end of the
    queue.
    Yes, it is not obvious how to requeue in the presence of multiple
    consumers. But it is possible to do better in the case of a single
    consumer where the inconsistency can't arise. At present rabbit always
    requeues at the back of the queue, i.e. treat it as a new message. This
    is consistent with the spec which only guarantees ordering along the
    same path from a single producer to a single consumer.
    I quite like the "send to the back of the queue" behaviour.
    In the event of a bad message that is always failing/crashing on the
    consumer, it'll get pushed to the back of the queue to allow other good
    messages to be processed.

    In a system where the messages are always delivered in-order, that bad
    message would effectively lock the system up.

    Toby
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Toby Corkindale at Jun 16, 2011 at 8:17 am

    On 16/06/11 18:14, T-zex wrote:
    Unexpected shut down of a process (power failure) will result in
    "good" message being put to the end of the queue - a very unexpected
    behaviour. In case of a "bad" message application should decide that
    it is bad and what should be done with it - requeue (unlikely) or get
    rid of it.
    In an ideal world, yes, but in practice it's much better to assume the
    worst.
    On Thu, Jun 16, 2011 at 2:14 AM, Toby Corkindale
    wrote:
    On 15/06/11 19:54, Emile Joubert wrote:
    On 14/06/11 17:08, Jason J. W. Williams wrote:

    Is it possible to use RabbitMQ as a reliable message queue? What I
    mean is to make it preserve the order of messages in queues even for
    non ack'ed durable messages.
    Putting non acked messages into the end of the queue just doesn't make
    sence.
    Actually, it would be difficult to preserve the order in the case of
    non-ack. Let's say you've got M1, M2, M3, M4 in the queue (like the
    FAQ describes). If you have two consumers, and C1 is consuming M1 and
    C2 is consuming M2. C1 finishes, acks M1 and starts consuming M3.
    However, C2 fails and disconnects without acking M2, where should M2
    go in the order? M3 is already being consumed. The only sane approach
    is to treat M2 as a new submission and append it to the end of the
    queue.
    Yes, it is not obvious how to requeue in the presence of multiple
    consumers. But it is possible to do better in the case of a single
    consumer where the inconsistency can't arise. At present rabbit always
    requeues at the back of the queue, i.e. treat it as a new message. This
    is consistent with the spec which only guarantees ordering along the
    same path from a single producer to a single consumer.
    I quite like the "send to the back of the queue" behaviour.
    In the event of a bad message that is always failing/crashing on the
    consumer, it'll get pushed to the back of the queue to allow other good
    messages to be processed.

    In a system where the messages are always delivered in-order, that bad
    message would effectively lock the system up.

    Toby
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

    --
    .signature
  • Jason J. W. Williams at Jun 16, 2011 at 5:47 pm

    On Thu, Jun 16, 2011 at 2:14 AM, T-zex wrote:
    Unexpected shut down of a process (power failure) will result in
    "good" message being put to the end of the queue - a very unexpected
    behaviour. In case of a "bad" message application should decide that
    it is bad and what should be done with it - requeue (unlikely) or get
    rid of it.
    Well technically you can do that already. If you don't want it
    requeued just consume it and toss it, if you want it requeued just
    recreate the channel. I'm not sure that a message in the process of
    being consumed when the broker dies is requeued to the back. My
    experience is the queue comes up in the state it was in when the
    broker died.

    -J
  • Matthias Radestock at Jun 16, 2011 at 5:53 pm

    On 16/06/11 18:47, Jason J. W. Williams wrote:
    On Thu, Jun 16, 2011 at 2:14 AM, T-zexwrote:
    Unexpected shut down of a process (power failure) will result in
    "good" message being put to the end of the queue - a very
    unexpected behaviour. In case of a "bad" message application should
    decide that it is bad and what should be done with it - requeue
    (unlikely) or get rid of it.
    Well technically you can do that already. If you don't want it
    requeued just consume it and toss it,
    ...or reject the message with basic.reject(requeue=false).
    if you want it requeued just recreate the channel.
    ...or reject the message with basic.reject(requeue=true).
    I'm not sure that a message in the process of being consumed when the
    broker dies is requeued to the back.
    They are. But we have some work underway to change that.


    Matthias.
  • Jason J. W. Williams at Jun 16, 2011 at 5:59 pm

    ...or reject the message with basic.reject(requeue=false).
    if you want it requeued just recreate the channel.
    ...or reject the message with basic.reject(requeue=true).
    Couldn't remember if reject had been supported yet. :)

    -J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 14, '11 at 4:00p
activeJun 16, '11 at 5:59p
posts9
users5
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase