FAQ
we have scenarios where we pick up a message at a time, perform a bunch of
operations, publish a new message and then ack the message we just picked
up. Recently we have started to publish the new message inside a transaction
since this guarantees that the message is persisted (it's a persistent
message), since we want to minimize the odds of message loss. However, we
have started to notice that the original message we pick up is marked as
unacknowledged after we complete our process. Is there something
fundamentally wrong in what we're doing, or is our approach correct?

thanks
Vishnu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100518/011b2c08/attachment.htm

Search Discussions

  • Michael Bridgen at May 18, 2010 at 11:04 am

    vishnu wrote:
    we have scenarios where we pick up a message at a time, perform a bunch
    of operations, publish a new message and then ack the message we just
    picked up. Recently we have started to publish the new message inside a
    transaction since this guarantees that the message is persisted (it's a
    persistent message), since we want to minimize the odds of message loss.
    However, we have started to notice that the original message we pick up
    is marked as unacknowledged after we complete our process. Is there
    something fundamentally wrong in what we're doing, or is our approach
    correct?
    The approach is correct. What order are you doing things in?
    Transactions are implicitly started immediately after a commit or
    rollback, and acknowledgements are transactional, so if you're
    committing after the publish but before the ack --

    basic.publish(new_message)
    tx.commit()
    basic.ack(old_message)

    the state at this point is a published message, and an uncommitted
    transaction with the ack. If you do this:

    basic.publish(new_message)
    basic.ack(old_message)
    tx.commit()

    then you'll have atomically acked the old message and published the new
    one; which is, I believe, what you want.


    Michael.
  • Vishnu at May 19, 2010 at 6:00 am
    resending this to the list :)
    On Tue, May 18, 2010 at 7:59 PM, vishnu wrote:

    well actually we're doing
    basic.consume

    some stuff
    tx.select
    basic.publish
    tx.commit

    some stuff
    basic.ack

    currently, we're trying to make sure the publish is inside a transaction.
    On Tue, May 18, 2010 at 4:34 PM, Michael Bridgen wrote:

    vishnu wrote:
    we have scenarios where we pick up a message at a time, perform a bunch
    of operations, publish a new message and then ack the message we just
    picked up. Recently we have started to publish the new message inside a
    transaction since this guarantees that the message is persisted (it's a
    persistent message), since we want to minimize the odds of message loss.
    However, we have started to notice that the original message we pick up
    is marked as unacknowledged after we complete our process. Is there
    something fundamentally wrong in what we're doing, or is our approach
    correct?
    The approach is correct. What order are you doing things in?
    Transactions are implicitly started immediately after a commit or
    rollback, and acknowledgements are transactional, so if you're
    committing after the publish but before the ack --

    basic.publish(new_message)
    tx.commit()
    basic.ack(old_message)

    the state at this point is a published message, and an uncommitted
    transaction with the ack. If you do this:

    basic.publish(new_message)
    basic.ack(old_message)
    tx.commit()

    then you'll have atomically acked the old message and published the new
    one; which is, I believe, what you want.


    Michael.


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100519/8017f715/attachment.htm
  • Matthias Radestock at May 19, 2010 at 6:22 am

    vishnu wrote:
    well actually we're doing
    basic.consume

    some stuff
    tx.select
    basic.publish
    tx.commit

    some stuff
    basic.ack

    currently, we're trying to make sure the publish is inside a
    transaction.
    As Michael pointed out ...
    <quote>
    Transactions are implicitly started immediately after a commit or
    rollback, and acknowledgements are transactional, so if you're
    committing after the publish but before the ack [which is exactly what
    you are doing] then the state at this point is a published message, and
    an uncommitted transaction with the ack.
    </quote>

    Unless the code does a subsequent commit the ack will be lost.

    You either need to
    a) issue a second commit, after the ack,
    b) move the ack to before the first commit, or
    c) use two channels - one for consuming and acking non-transactionally,
    and the other for publishing transactionally.


    Regards,

    Matthias.
  • Michael Bridgen at May 19, 2010 at 9:20 am

    On Tue, May 18, 2010 at 7:59 PM, vishnu <pathsny at gmail.com
    wrote:

    well actually we're doing
    basic.consume

    some stuff
    tx.select
    basic.publish
    tx.commit

    some stuff
    basic.ack

    currently, we're trying to make sure the publish is inside a
    transaction.
    Right, that fits the first pattern. You will want to move the ack
    inside the transaction. Then the ack of the old incoming message will
    happen atomically with publishing the outgoing message, and you won't
    see a published outgoing message along with an unacked incoming message.

    basic.consume

    some stuff
    tx.select
    basic.publish

    some stuff
    basic.ack
    tx.commit


    Michael.
    On Tue, May 18, 2010 at 4:34 PM, Michael Bridgen <mikeb at rabbitmq.com
    wrote:

    vishnu wrote:
    we have scenarios where we pick up a message at a time,
    perform a bunch
    of operations, publish a new message and then ack the message we just
    picked up. Recently we have started to publish the new
    message inside a
    transaction since this guarantees that the message is
    persisted (it's a
    persistent message), since we want to minimize the odds of
    message loss.
    However, we have started to notice that the original message
    we pick up
    is marked as unacknowledged after we complete our process. Is there
    something fundamentally wrong in what we're doing, or is our approach
    correct?
    The approach is correct. What order are you doing things in?
    Transactions are implicitly started immediately after a commit or
    rollback, and acknowledgements are transactional, so if you're
    committing after the publish but before the ack --

    basic.publish(new_message)
    tx.commit()
    basic.ack(old_message)

    the state at this point is a published message, and an uncommitted
    transaction with the ack. If you do this:

    basic.publish(new_message)
    basic.ack(old_message)
    tx.commit()

    then you'll have atomically acked the old message and published
    the new
    one; which is, I believe, what you want.


    Michael.


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    <mailto:rabbitmq-discuss at lists.rabbitmq.com>
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss




    ------------------------------------------------------------------------

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Vishnu at May 19, 2010 at 10:30 am
    aha thanks. I sorta get it now

    why does rabbitmq implicitly start a transaction after a commit. Does this
    mean after a commit every single operation now has to be transactional?
    thanks
    On Wed, May 19, 2010 at 2:50 PM, Michael Bridgen wrote:

    On Tue, May 18, 2010 at 7:59 PM, vishnu <pathsny at gmail.com <mailto:
    pathsny at gmail.com>> wrote:

    well actually we're doing
    basic.consume

    some stuff
    tx.select
    basic.publish
    tx.commit

    some stuff
    basic.ack

    currently, we're trying to make sure the publish is inside a
    transaction.
    Right, that fits the first pattern. You will want to move the ack inside
    the transaction. Then the ack of the old incoming message will happen
    atomically with publishing the outgoing message, and you won't see a
    published outgoing message along with an unacked incoming message.


    basic.consume

    some stuff
    tx.select
    basic.publish

    some stuff
    basic.ack
    tx.commit


    Michael.

    On Tue, May 18, 2010 at 4:34 PM, Michael Bridgen <mikeb at rabbitmq.com
    wrote:

    vishnu wrote:
    we have scenarios where we pick up a message at a time,
    perform a bunch
    of operations, publish a new message and then ack the message we just
    picked up. Recently we have started to publish the new
    message inside a
    transaction since this guarantees that the message is
    persisted (it's a
    persistent message), since we want to minimize the odds of
    message loss.
    However, we have started to notice that the original message
    we pick up
    is marked as unacknowledged after we complete our process. Is there
    something fundamentally wrong in what we're doing, or is our approach
    correct?
    The approach is correct. What order are you doing things in?
    Transactions are implicitly started immediately after a commit or
    rollback, and acknowledgements are transactional, so if you're
    committing after the publish but before the ack --

    basic.publish(new_message)
    tx.commit()
    basic.ack(old_message)

    the state at this point is a published message, and an uncommitted
    transaction with the ack. If you do this:

    basic.publish(new_message)
    basic.ack(old_message)
    tx.commit()

    then you'll have atomically acked the old message and published
    the new
    one; which is, I believe, what you want.


    Michael.


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    <mailto:rabbitmq-discuss at lists.rabbitmq.com>


    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss




    ------------------------------------------------------------------------


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100519/612554cb/attachment.htm
  • Matthew Sackman at May 19, 2010 at 10:33 am

    On Wed, May 19, 2010 at 04:00:54PM +0530, vishnu wrote:
    why does rabbitmq implicitly start a transaction after a commit. Does this
    mean after a commit every single operation now has to be transactional?
    Yup. Once a channel has had a tx.select, it is transactional until it is
    closed. Feel free to use several channels.

    Matthew
  • Matthias Radestock at May 19, 2010 at 10:39 am

    Matthew Sackman wrote:
    On Wed, May 19, 2010 at 04:00:54PM +0530, vishnu wrote:
    why does rabbitmq implicitly start a transaction after a commit. Does this
    mean after a commit every single operation now has to be transactional?
    Yup. Once a channel has had a tx.select, it is transactional until it is
    closed.
    ...and this behaviour is mandated by the AMQP spec, so rabbit has no
    choice in the matter.

    Matthias.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedMay 18, '10 at 10:42a
activeMay 19, '10 at 10:39a
posts8
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase