I apologize if this is an ill-informed question: In a nutshell, the
thing I am trying to accomplish is blocking a producer until it has
received acks (or even nacks) from all consumers on temporary queues
bound to a given (fanout) exchange for a single message. From the
docs, my sense is that using confirms and waitForConfirms would be
preferable to full on transactions, but as each instance of the
producer will be on its own thread, I am open to using transactions.
The main point of confusion for me here is whether I can get multiple
acks (from all consumers) for one message without having any prior
information about the number of queues attached to the exchange. That
is, I don't just want to know that one queue has received the message,
but that all queues bound to the exchange have done so. It seems clear
that if I send one message per attached queue, I could create a set of
seqNos and check against that to verify acks for all of the messages,
but without a list of bound queues (which I cannot see how to get),
that does not seem like an option. In any case, it seems likely that
there is something that I am missing here. That said, I would
appreciate any thoughts (or even clarifying questions) on this.

Thanks,
Jonathan

Search Discussions

  • Alexandru Scvorţov at Sep 23, 2011 at 9:58 am
    Hi Jonathan,
    I apologize if this is an ill-informed question: In a nutshell, the
    thing I am trying to accomplish is blocking a producer until it has
    received acks (or even nacks) from all consumers on temporary queues
    bound to a given (fanout) exchange for a single message. From the
    docs, my sense is that using confirms and waitForConfirms would be
    preferable to full on transactions,
    Confirms are usually the way to go.

    With confirms enabled, calling waitForConfirms would block the producer
    until all published messages on that channel are acknowledged *by the
    broker*. The broker, in turn, will only acknowledge the message if 1)
    it has been persisted to disk; *or* if 2) it has been delivered to a
    consumer on each of the queues it had reached.

    You can't really specify that you only want the acknowledgement if the
    messages are delivered and not if they're persisted.

    Also, you said "temporary queues". Confirms will only work as described
    above if the message is persistent and if the queues it lands on are
    durable. A message that lands on a non-durable queue is considered
    acknowledged on that queue immediately (of course, the broker will still
    wait for it to be persisted or acknowledged on all the other queues it
    reached, before acknowledging it to the producer).
    but as each instance of the
    producer will be on its own thread,
    That's fine. Just make sure to use a separate channel for each
    producer (calling waitForConfirms on the same channel from multiple
    threads will almost certainly not do what you want it to).
    The main point of confusion for me here is whether I can get multiple
    acks (from all consumers) for one message without having any prior
    information about the number of queues attached to the exchange. That
    is, I don't just want to know that one queue has received the message,
    but that all queues bound to the exchange have done so. It seems clear
    Not quite sure what you mean. A published message, even if it gets sent
    to multiple queues, is still *one* message with a certain sequence
    number. A published message will be acknowledged by the broker only
    once, when it has been written to disk or when it has been successfully
    sent to a consumer from each queue.

    Consumer acks and publisher acks are completely different things. A
    consumer acks a message to tell *the broker* (not the publisher) that it
    has taken responsibility for the message (by using it up, by forwarding
    it, by doing something with it). The broker acks a message to tell *the
    publisher* that it has taken responsibility for the message (by saving
    it to disk, or by handing it over to consumers).

    When a message reaches a queue, and is then delivered to a consumer, the
    broker will consider the message delivered only when the consumer
    acknowledges it. So, if it weren't for the "writing the message to
    disk" part, it would work exactly the way you're expecting it to.

    What sort of system are you designing?

    Hope this helps.

    Cheers,
    Alex

    On Thu, Sep 22, 2011 at 12:22:50PM -0700, gneeri wrote:
    I apologize if this is an ill-informed question: In a nutshell, the
    thing I am trying to accomplish is blocking a producer until it has
    received acks (or even nacks) from all consumers on temporary queues
    bound to a given (fanout) exchange for a single message. From the
    docs, my sense is that using confirms and waitForConfirms would be
    preferable to full on transactions, but as each instance of the
    producer will be on its own thread, I am open to using transactions.
    The main point of confusion for me here is whether I can get multiple
    acks (from all consumers) for one message without having any prior
    information about the number of queues attached to the exchange. That
    is, I don't just want to know that one queue has received the message,
    but that all queues bound to the exchange have done so. It seems clear
    that if I send one message per attached queue, I could create a set of
    seqNos and check against that to verify acks for all of the messages,
    but without a list of bound queues (which I cannot see how to get),
    that does not seem like an option. In any case, it seems likely that
    there is something that I am missing here. That said, I would
    appreciate any thoughts (or even clarifying questions) on this.

    Thanks,
    Jonathan
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Gneeri at Sep 23, 2011 at 4:19 pm
    Hi Alex,

    Thank you very much for your detailed reply. It is quite helpful and I
    do have one specific question with respect to your notes, but think it
    might be most useful if I briefly respond to your question about the
    intended system before inquiring further:

    The system is composed of a dynamic array of servers, each of which
    support user logins. In order to prevent data corruption (i.e.,
    concurrency problems), my intent is that, upon login to any of the
    servers, the server logged into broadcasts a message out to the other
    servers, telling them to revoke control over the logging in user's
    data. Then, and only once all of the other servers (subscribed to the
    login exchange at startup) have ack'd the request to relinquish
    control, do I want the login server to proceed and take control of the
    user's data.
  • Gneeri at Sep 23, 2011 at 8:26 pm
    Hi Alex,

    So to follow up, I have run some tests with durable queues and
    waitForConfirms, which are not giving me the results I would expect.

    The test is set up as follows:

    1. I have a producer publishing with confirmSelect and waitForConfirms
    to a fanout exchange

    2. I have 2 consumers, attached to durable queues bound to the fanout
    exchange

    3. both consumers have autoAck set to false and use manual basicAck
    after processing

    4. I have put a delay (via Thread.sleep()) into one of the consumers

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack to queue
    which would in turn send an ack to the fanout exchange, followed by an
    ack to the producer. This, however, does not appear to be the case and
    the producer appears to be executing code below the waitForConfirms()
    call, well before the delayed consumer sends back its ack. For good
    measure, I have also tried placing the sleep() call before the
    consumer calls nextDelivery() in case the simple acceptance of the
    payload from the queue would be enough to pass acks back up the chain
    to the producer. This too, did not appear to work.

    Ultimately, then, my sense is that I have misconstrued your earlier
    remarks. Earlier (below), I passed along the intended functionality.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required. In any case, if you could let me know what I am
    missing in this test (i.e., if there is a correctable way of making
    this work) or if I need to attempt an implementation based on one
    message per server, I would very much appreciate it.

    Thanks in advance,
    Jonathan
    On Sep 23, 12:19?pm, gneeri wrote:
    Hi Alex,

    Thank you very much for your detailed reply. It is quite helpful and I
    do have one specific question with respect to your notes, but think it
    might be most useful if I briefly respond to your question about the
    intended system before inquiring further:

    The system is composed of a dynamic array of servers, each of which
    support user logins. In order to prevent data corruption (i.e.,
    concurrency problems), my intent is that, upon login to any of the
    servers, the server logged into broadcasts a message out to the other
    servers, telling them to revoke control over the logging in user's
    data. Then, and only once all of the other servers (subscribed to the
    login exchange at startup) have ack'd the request to relinquish
    control, do I want the login server to proceed and take control of the
    user's data.

    From there, I expect the blocking requirement is pretty clear. Also,
    insofar as the array of servers is dynamic, I'm hoping my desire to
    use a single message over a "fanout" exchange will make some sense (as
    opposed to having to send a single message to each server via a named
    queue (or perhaps routing key) that would require me to have a list of
    all of the servers in advance of publishing).

    Which brings me to my question(s): when you say: "The broker, in turn,
    will only acknowledge the message if . . . 2) it has been delivered to
    a consumer on each of the queues it had reached"--does this mean that
    the broker (in this case, fanout exchange) will only send an ack to
    the publisher after a consumer on each of the attached queues has been
    reached? After some very rough testing with multiple consumers
    attached to the same fanout exchange, it has appeared to me that the
    publisher may get an ack prior to one of the consumers sending its ack
    back to the broker. This may just be the result of bad test code (on
    my part) *or* it may relate the fact that I am using temporary queues
    (e.g., channel.queueDeclare().getQueue()) rather than durable queues,
    as you suggest below. If that latter is the case, are you suggesting
    that using durable queues would make the broker wait on all consumers
    for all of the queues before returning an ack to the publisher? If so,
    that sounds like it would do the trick.

    Sorry again, for being so verbose and thanks very much for your help.

    Cheers,
    Jonathan

    On Sep 23, 5:58?am, Alexandru Scvor?ov wrote:






    Hi Jonathan,
    I apologize if this is an ill-informed question: In a nutshell, the
    thing I am trying to accomplish is blocking a producer until it has
    received acks (or even nacks) from all consumers on temporary queues
    bound to a given (fanout) exchange for a single message. From the
    docs, my sense is that using confirms and waitForConfirms would be
    preferable to full on transactions,
    Confirms are usually the way to go.
    With confirms enabled, calling waitForConfirms would block the producer
    until all published messages on that channel are acknowledged *by the
    broker*. ?The broker, in turn, will only acknowledge the message if 1)
    it has been persisted to disk; *or* if 2) it has been delivered to a
    consumer on each of the queues it had reached.
    You can't really specify that you only want the acknowledgement if the
    messages are delivered and not if they're persisted.
    Also, you said "temporary queues". ?Confirms will only work as described
    above if the message is persistent and if the queues it lands on are
    durable. ?A message that lands on a non-durable queue is considered
    acknowledged on that queue immediately (of course, the broker will still
    wait for it to be persisted or acknowledged on all the other queues it
    reached, before acknowledging it to the producer).
    but as each instance of the
    producer will be on its own thread,
    That's fine. ?Just make sure to use a separate channel for each
    producer (calling waitForConfirms on the same channel from multiple
    threads will almost certainly not do what you want it to).
    The main point of confusion for me here is whether I can get multiple
    acks (from all consumers) for one message without having any prior
    information about the number of queues attached to the exchange. That
    is, I don't just want to know that one queue has received the message,
    but that all queues bound to the exchange have done so. It seems clear
    Not quite sure what you mean. ?A published message, even if it gets sent
    to multiple queues, is still *one* message with a certain sequence
    number. ?A published message will be acknowledged by the broker only
    once, when it has been written to disk or when it has been successfully
    sent to a consumer from each queue.
    Consumer acks and publisher acks are completely different things. ?A
    consumer acks a message to tell *the broker* (not the publisher) that it
    has taken responsibility for the message (by using it up, by forwarding
    it, by doing something with it). ?The broker acks a message to tell *the
    publisher* that it has taken responsibility for the message (by saving
    it to disk, or by handing it over to consumers).
    When a message reaches a queue, and is then delivered to a consumer, the
    broker will consider the message delivered only when the consumer
    acknowledges it. ?So, if it weren't for the "writing the message to
    disk" part, it would work exactly the way you're expecting it to.
    What sort of system are you designing?
    Hope this helps.
    Cheers,
    Alex
    On Thu, Sep 22, 2011 at 12:22:50PM -0700, gneeri wrote:
    I apologize if this is an ill-informed question: In a nutshell, the
    thing I am trying to accomplish is blocking a producer until it has
    received acks (or even nacks) from all consumers on temporary queues
    bound to a given (fanout) exchange for a single message. From the
    docs, my sense is that using confirms and waitForConfirms would be
    preferable to full on transactions, but as each instance of the
    producer will be on its own thread, I am open to using transactions.
    The main point of confusion for me here is whether I can get multiple
    acks (from all consumers) for one message without having any prior
    information about the number of queues attached to the exchange. That
    is, I don't just want to know that one queue has received the message,
    but that all queues bound to the exchange have done so. It seems clear
    that if I send one message per attached queue, I could create a set of
    seqNos and check against that to verify acks for all of the messages,
    but without a list of bound queues (which I cannot see how to get),
    that does not seem like an option. In any case, it seems likely that
    there is something that I am missing here. That said, I would
    appreciate any thoughts (or even clarifying questions) on this.
    Thanks,
    Jonathan
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Matthias Radestock at Sep 23, 2011 at 10:55 pm

    On 23/09/11 21:26, gneeri wrote:
    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.

    They concern the interaction between producers and the broker, and have
    nothing to do with consumer acks.

    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..


    Regards,

    Matthias.
  • Gneeri at Sep 23, 2011 at 11:44 pm
    Hi Matthias,

    Thanks for clarifying that. From the explication of the rules for
    confirms, it had appeared to me that the relationships to queues and
    consumers had more weight in the confirmation process. That not being
    the case, I have 2 brief follow up questions:

    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?

    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.

    Thanks in advance,
    Jonathan
    On Sep 23, 6:55?pm, Matthias Radestock wrote:
    On 23/09/11 21:26, gneeri wrote:

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.

    They concern the interaction between producers and the broker, and ?have
    nothing to do with consumer acks.

    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..

    Regards,

    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Alexandru Scvorţov at Sep 26, 2011 at 9:54 am
    Hi Jonathan,
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    Not really, no. I don't think there's anyway to get what you want
    automatically.
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    There's no AMQP way of finding how many queues are bound to an exchange.
    As you say, there's the management API, but using that is likely to lead
    to races (what happens if you query the number of queues, then one of
    the machines dies before it sends its acknowledgement?).

    You could kind of get what you want with AMQP. Use a fanout exchange
    like you want for broadcast messages, but use a direct exchange for
    direct communication between machines. You could use the fanout
    exchange to make sure that all the machines know about each other (in
    particular, that each knows the queues the others bound to the direct
    exchange, so that they can communicate directly). And you could use the
    direct exchange to publish persistent, confirm, mandatory, immediate messages
    directly to other machines. With these messages, the sender will be
    notified if the other machine received the message (only a confirm is
    received for the publish), or if the other machine went offline in the past
    (a basic.return and a confirm is received). This way, the sending
    machine would know which other machines received its message and, so,
    will know to wait for the correct number of responses. The queue bound
    to the direct exchange would have to be durable, auto-delete for this to
    work (durable for confirms to work and auto-delete so that once the sole
    consumer dies, the queue is deleted and publishing to it will result in
    a basic.return). In addition, when processing "please log out this
    user" messages, a machine should *not* acknowledge that it has
    received the message until after it publishes its response (this way, if
    the machine dies while processing, the sender will get a basic.return for
    it).

    Hope this helps.

    Cheers,
    Alex
    On Fri, Sep 23, 2011 at 04:44:14PM -0700, gneeri wrote:
    Hi Matthias,

    Thanks for clarifying that. From the explication of the rules for
    confirms, it had appeared to me that the relationships to queues and
    consumers had more weight in the confirmation process. That not being
    the case, I have 2 brief follow up questions:

    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?

    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.

    Thanks in advance,
    Jonathan
    On Sep 23, 6:55?pm, Matthias Radestock wrote:
    On 23/09/11 21:26, gneeri wrote:

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.

    They concern the interaction between producers and the broker, and ?have
    nothing to do with consumer acks.

    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..

    Regards,

    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Gneeri at Sep 26, 2011 at 3:04 pm
    Hi Alex,

    Thanks so much for the detailed reply. I had begun thinking along
    these lines, but really appreciate the level of detail regarding the
    specific types of publish requests, queues and exchanges. This is
    really helpful and will likely save me many hours of trial and error.
    I do have just one quick question though regarding the following:

    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."

    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code? If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?

    Thanks again, really, I very much appreciate your help.

    Cheers,
    Jonathan

    On Sep 26, 5:54?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    Not really, no. ?I don't think there's anyway to get what you want
    automatically.
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    There's no AMQP way of finding how many queues are bound to an exchange.
    As you say, there's the management API, but using that is likely to lead
    to races (what happens if you query the number of queues, then one of
    the machines dies before it sends its acknowledgement?).

    You could kind of get what you want with AMQP. ?Use a fanout exchange
    like you want for broadcast messages, but use a direct exchange for
    direct communication between machines. ?You could use the fanout
    exchange to make sure that all the machines know about each other (in
    particular, that each knows the queues the others bound to the direct
    exchange, so that they can communicate directly). ?And you could use the
    direct exchange to publish persistent, confirm, mandatory, immediate messages
    directly to other machines. ?With these messages, the sender will be
    notified if the other machine received the message (only a confirm is
    received for the publish), or if the other machine went offline in the past
    (a basic.return and a confirm is received). ?This way, the sending
    machine would know which other machines received its message and, so,
    will know to wait for the correct number of responses. ?The queue bound
    to the direct exchange would have to be durable, auto-delete for this to
    work (durable for confirms to work and auto-delete so that once the sole
    consumer dies, the queue is deleted and publishing to it will result in
    a basic.return). ?In addition, when processing "please log out this
    user" messages, a machine should *not* acknowledge that it has
    received the message until after it publishes its response (this way, if
    the machine dies while processing, the sender will get a basic.return for
    it).

    Hope this helps.

    Cheers,
    Alex








    On Fri, Sep 23, 2011 at 04:44:14PM -0700, gneeri wrote:
    Hi Matthias,
    Thanks for clarifying that. From the explication of the rules for
    confirms, it had appeared to me that the relationships to queues and
    consumers had more weight in the confirmation process. That not being
    the case, I have 2 brief follow up questions:
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    Thanks in advance,
    Jonathan
    On Sep 23, 6:55?pm, Matthias Radestock wrote:
    On 23/09/11 21:26, gneeri wrote:

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.
    They concern the interaction between producers and the broker, and ?have
    nothing to do with consumer acks.
    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..
    Regards,
    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Alexandru Scvorţov at Sep 26, 2011 at 3:34 pm
    Hi Jonathan,
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."

    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code?
    The exchange type has nothing to do with it. When you consume from a
    queue, you specify the autoAck parameter. If this is set to true
    (which it is, by default), then the server will not wait for the
    consumer to acknowledge receipt of the message (so, it's effectively
    auto-ack'ing all messages). If it's set to false, the broker will not
    consider the message delivered to the consumer until it explicitly ack's
    the message. See the API guide for an example (I assume you're using
    the Java client, but the other clients have very similar APIs):
    http://www.rabbitmq.com/api-guide.html#consuming
    If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Oh. I missed that. Yes, I think that would work.

    Cheers,
    Alex
    On Mon, Sep 26, 2011 at 08:04:28AM -0700, gneeri wrote:
    Hi Alex,

    Thanks so much for the detailed reply. I had begun thinking along
    these lines, but really appreciate the level of detail regarding the
    specific types of publish requests, queues and exchanges. This is
    really helpful and will likely save me many hours of trial and error.
    I do have just one quick question though regarding the following:

    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."

    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code? If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?

    Thanks again, really, I very much appreciate your help.

    Cheers,
    Jonathan

    On Sep 26, 5:54?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    Not really, no. ?I don't think there's anyway to get what you want
    automatically.
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    There's no AMQP way of finding how many queues are bound to an exchange.
    As you say, there's the management API, but using that is likely to lead
    to races (what happens if you query the number of queues, then one of
    the machines dies before it sends its acknowledgement?).

    You could kind of get what you want with AMQP. ?Use a fanout exchange
    like you want for broadcast messages, but use a direct exchange for
    direct communication between machines. ?You could use the fanout
    exchange to make sure that all the machines know about each other (in
    particular, that each knows the queues the others bound to the direct
    exchange, so that they can communicate directly). ?And you could use the
    direct exchange to publish persistent, confirm, mandatory, immediate messages
    directly to other machines. ?With these messages, the sender will be
    notified if the other machine received the message (only a confirm is
    received for the publish), or if the other machine went offline in the past
    (a basic.return and a confirm is received). ?This way, the sending
    machine would know which other machines received its message and, so,
    will know to wait for the correct number of responses. ?The queue bound
    to the direct exchange would have to be durable, auto-delete for this to
    work (durable for confirms to work and auto-delete so that once the sole
    consumer dies, the queue is deleted and publishing to it will result in
    a basic.return). ?In addition, when processing "please log out this
    user" messages, a machine should *not* acknowledge that it has
    received the message until after it publishes its response (this way, if
    the machine dies while processing, the sender will get a basic.return for
    it).

    Hope this helps.

    Cheers,
    Alex








    On Fri, Sep 23, 2011 at 04:44:14PM -0700, gneeri wrote:
    Hi Matthias,
    Thanks for clarifying that. From the explication of the rules for
    confirms, it had appeared to me that the relationships to queues and
    consumers had more weight in the confirmation process. That not being
    the case, I have 2 brief follow up questions:
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    Thanks in advance,
    Jonathan
    On Sep 23, 6:55?pm, Matthias Radestock wrote:
    On 23/09/11 21:26, gneeri wrote:

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.
    They concern the interaction between producers and the broker, and ?have
    nothing to do with consumer acks.
    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..
    Regards,
    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Gneeri at Sep 26, 2011 at 4:04 pm
    Hi Alex,

    This is a point of confusion for me then. In a previous note where I
    mentioned setting autoAck to false and manually using basicAck,
    Matthias replied as follows:

    "Confirms mean one thing and one thing only: that the broker has
    accepted
    responsibility for the message.

    "They concern the interaction between producers and the broker, and
    have
    nothing to do with consumer acks."

    If in the fanout context I could have used consumer acks to hold off
    broker acks to the publisher, I would have expected that I could
    effectively block confirms until after I sent basicAcks from the
    consumer to the broker and then on up to the publisher. From
    Matthias's note (and my tests), however, it appears that the broker
    sends an ack back to the publisher as soon as it takes control of the
    message and so issues a confirm without waiting on anything from the
    consumer. In the direct exchange context, however, I took your notes
    to be indicating that the broker there will wait for an ack from the
    consumer, such that if the consumer died while in process, but before
    sending the ack, the broker would return a basic.return along with the
    confirm. If this is not the case, then, my sense is that I will have
    to have the consumers publish a response directly back to original
    publisher as Matthias notes--while using the fanout broadcast to keep
    track of the number of expected responses (as you indicate).

    Does that make sense? Basically, it still orbits around a question of
    how and/or if consumer acks can be made to effect a producer. I am
    guessing that my implementation will have to require the return
    publications (responses), but any clarification you could add to the
    consumer ack, confirm chain would be greatly appreciated.

    Thanks,
    Jonathan

    On Sep 26, 11:34?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."
    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code?
    The exchange type has nothing to do with it. ?When you consume from a
    queue, you specify the autoAck parameter. ?If this is set to true
    (which it is, by default), then the server will not wait for the
    consumer to acknowledge receipt of the message (so, it's effectively
    auto-ack'ing all messages). ?If it's set to false, the broker will not
    consider the message delivered to the consumer until it explicitly ack's
    the message. ?See the API guide for an example (I assume you're using
    the Java client, but the other clients have very similar APIs):
    ?http://www.rabbitmq.com/api-guide.html#consuming
    If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Oh. ?I missed that. ?Yes, I think that would work.

    Cheers,
    Alex








    On Mon, Sep 26, 2011 at 08:04:28AM -0700, gneeri wrote:
    Hi Alex,
    Thanks so much for the detailed reply. I had begun thinking along
    these lines, but really appreciate the level of detail regarding the
    specific types of publish requests, queues and exchanges. This is
    really helpful and will likely save me many hours of trial and error.
    I do have just one quick question though regarding the following:
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."
    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code? If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Thanks again, really, I very much appreciate your help.
    Cheers,
    Jonathan
    On Sep 26, 5:54?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    Not really, no. ?I don't think there's anyway to get what you want
    automatically.
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    There's no AMQP way of finding how many queues are bound to an exchange.
    As you say, there's the management API, but using that is likely to lead
    to races (what happens if you query the number of queues, then one of
    the machines dies before it sends its acknowledgement?).
    You could kind of get what you want with AMQP. ?Use a fanout exchange
    like you want for broadcast messages, but use a direct exchange for
    direct communication between machines. ?You could use the fanout
    exchange to make sure that all the machines know about each other (in
    particular, that each knows the queues the others bound to the direct
    exchange, so that they can communicate directly). ?And you could use the
    direct exchange to publish persistent, confirm, mandatory, immediate messages
    directly to other machines. ?With these messages, the sender will be
    notified if the other machine received the message (only a confirm is
    received for the publish), or if the other machine went offline in the past
    (a basic.return and a confirm is received). ?This way, the sending
    machine would know which other machines received its message and, so,
    will know to wait for the correct number of responses. ?The queue bound
    to the direct exchange would have to be durable, auto-delete for this to
    work (durable for confirms to work and auto-delete so that once the sole
    consumer dies, the queue is deleted and publishing to it will result in
    a basic.return). ?In addition, when processing "please log out this
    user" messages, a machine should *not* acknowledge that it has
    received the message until after it publishes its response (this way, if
    the machine dies while processing, the sender will get a basic.return for
    it).
    Hope this helps.
    Cheers,
    Alex
    On Fri, Sep 23, 2011 at 04:44:14PM -0700, gneeri wrote:
    Hi Matthias,
    Thanks for clarifying that. From the explication of the rules for
    confirms, it had appeared to me that the relationships to queues and
    consumers had more weight in the confirmation process. That not being
    the case, I have 2 brief follow up questions:
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    Thanks in advance,
    Jonathan
    On Sep 23, 6:55?pm, Matthias Radestock wrote:
    On 23/09/11 21:26, gneeri wrote:

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.
    They concern the interaction between producers and the broker, and ?have
    nothing to do with consumer acks.
    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..
    Regards,
    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Alexandru Scvorţov at Sep 26, 2011 at 4:28 pm
    Hi Jonathan,
    This is a point of confusion for me then. In a previous note where I
    mentioned setting autoAck to false and manually using basicAck,
    Matthias replied as follows:
    Uf, I forgot about the persister in the last message. Sorry.
    If in the fanout context I could have used consumer acks to hold off
    broker acks to the publisher, I would have expected that I could
    effectively block confirms until after I sent basicAcks from the
    consumer to the broker and then on up to the publisher. From
    Matthias's note (and my tests), however, it appears that the broker
    sends an ack back to the publisher as soon as it takes control of the
    message and so issues a confirm without waiting on anything from the
    consumer.
    He's right. I was hoping the immediate flag will hold off the confirm,
    but, of course, it won't.
    In the direct exchange context, however, I took your notes
    to be indicating that the broker there will wait for an ack from the
    consumer, such that if the consumer died while in process, but before
    sending the ack, the broker would return a basic.return along with the
    confirm. If this is not the case, then, my sense is that I will have
    to have the consumers publish a response directly back to original
    publisher as Matthias notes--while using the fanout broadcast to keep
    track of the number of expected responses (as you indicate).
    You will have to have consumers send something to the original
    publisher and a fanout to keep track of responses. You'll only need a
    timeout of some sort, so that if any of the consumers takes to long to
    respond, you react somehow (that probably means the consumer died
    processing the message).

    You'll still need the mandatory, immediate flags to at least see that
    the consumer was alive and was consuming from the queue at the time the
    message was sent.
    Does that make sense? Basically, it still orbits around a question of
    how and/or if consumer acks can be made to effect a producer. I am
    guessing that my implementation will have to require the return
    publications (responses), but any clarification you could add to the
    consumer ack, confirm chain would be greatly appreciated.
    You'll need to publish back. The only things confirms will give you in
    this case is the guarantee that basic.returns happen before confirms.
    So, once you receive the confirm, you know the message has reached the
    consumer.

    There's also no need to delay acknowledgement on the consumer side.

    Sorry for the confusion.

    Alex

    On Mon, Sep 26, 2011 at 09:04:06AM -0700, gneeri wrote:
    Hi Alex,

    This is a point of confusion for me then. In a previous note where I
    mentioned setting autoAck to false and manually using basicAck,
    Matthias replied as follows:

    "Confirms mean one thing and one thing only: that the broker has
    accepted
    responsibility for the message.

    "They concern the interaction between producers and the broker, and
    have
    nothing to do with consumer acks."

    If in the fanout context I could have used consumer acks to hold off
    broker acks to the publisher, I would have expected that I could
    effectively block confirms until after I sent basicAcks from the
    consumer to the broker and then on up to the publisher. From
    Matthias's note (and my tests), however, it appears that the broker
    sends an ack back to the publisher as soon as it takes control of the
    message and so issues a confirm without waiting on anything from the
    consumer. In the direct exchange context, however, I took your notes
    to be indicating that the broker there will wait for an ack from the
    consumer, such that if the consumer died while in process, but before
    sending the ack, the broker would return a basic.return along with the
    confirm. If this is not the case, then, my sense is that I will have
    to have the consumers publish a response directly back to original
    publisher as Matthias notes--while using the fanout broadcast to keep
    track of the number of expected responses (as you indicate).

    Does that make sense? Basically, it still orbits around a question of
    how and/or if consumer acks can be made to effect a producer. I am
    guessing that my implementation will have to require the return
    publications (responses), but any clarification you could add to the
    consumer ack, confirm chain would be greatly appreciated.

    Thanks,
    Jonathan

    On Sep 26, 11:34?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."
    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code?
    The exchange type has nothing to do with it. ?When you consume from a
    queue, you specify the autoAck parameter. ?If this is set to true
    (which it is, by default), then the server will not wait for the
    consumer to acknowledge receipt of the message (so, it's effectively
    auto-ack'ing all messages). ?If it's set to false, the broker will not
    consider the message delivered to the consumer until it explicitly ack's
    the message. ?See the API guide for an example (I assume you're using
    the Java client, but the other clients have very similar APIs):
    ?http://www.rabbitmq.com/api-guide.html#consuming
    If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Oh. ?I missed that. ?Yes, I think that would work.

    Cheers,
    Alex








    On Mon, Sep 26, 2011 at 08:04:28AM -0700, gneeri wrote:
    Hi Alex,
    Thanks so much for the detailed reply. I had begun thinking along
    these lines, but really appreciate the level of detail regarding the
    specific types of publish requests, queues and exchanges. This is
    really helpful and will likely save me many hours of trial and error.
    I do have just one quick question though regarding the following:
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."
    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code? If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Thanks again, really, I very much appreciate your help.
    Cheers,
    Jonathan
    On Sep 26, 5:54?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    Not really, no. ?I don't think there's anyway to get what you want
    automatically.
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    There's no AMQP way of finding how many queues are bound to an exchange.
    As you say, there's the management API, but using that is likely to lead
    to races (what happens if you query the number of queues, then one of
    the machines dies before it sends its acknowledgement?).
    You could kind of get what you want with AMQP. ?Use a fanout exchange
    like you want for broadcast messages, but use a direct exchange for
    direct communication between machines. ?You could use the fanout
    exchange to make sure that all the machines know about each other (in
    particular, that each knows the queues the others bound to the direct
    exchange, so that they can communicate directly). ?And you could use the
    direct exchange to publish persistent, confirm, mandatory, immediate messages
    directly to other machines. ?With these messages, the sender will be
    notified if the other machine received the message (only a confirm is
    received for the publish), or if the other machine went offline in the past
    (a basic.return and a confirm is received). ?This way, the sending
    machine would know which other machines received its message and, so,
    will know to wait for the correct number of responses. ?The queue bound
    to the direct exchange would have to be durable, auto-delete for this to
    work (durable for confirms to work and auto-delete so that once the sole
    consumer dies, the queue is deleted and publishing to it will result in
    a basic.return). ?In addition, when processing "please log out this
    user" messages, a machine should *not* acknowledge that it has
    received the message until after it publishes its response (this way, if
    the machine dies while processing, the sender will get a basic.return for
    it).
    Hope this helps.
    Cheers,
    Alex
    On Fri, Sep 23, 2011 at 04:44:14PM -0700, gneeri wrote:
    Hi Matthias,
    Thanks for clarifying that. From the explication of the rules for
    confirms, it had appeared to me that the relationships to queues and
    consumers had more weight in the confirmation process. That not being
    the case, I have 2 brief follow up questions:
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    Thanks in advance,
    Jonathan
    On Sep 23, 6:55?pm, Matthias Radestock wrote:
    On 23/09/11 21:26, gneeri wrote:

    My expectation, then, is that producer would publish its message and
    then be blocked until the delayed consumer returns an ack
    Confirms mean one thing and one thing only: that the broker has accepted
    responsibility for the message.
    They concern the interaction between producers and the broker, and ?have
    nothing to do with consumer acks.
    Yes, there are some conditions in which a consumer ack can trigger a
    confirm in RabbitMQ, but that is an implementation detail and not
    something you'd want to design an application around.
    My hope is that I will not have to aggregate a list of servers and
    pass along a single message to each server, but perhaps that is what
    will be required.
    You can still send a single message via a fanout exchange. But the
    consumers then in turn need to send a message back to the sender. Your
    sender will need to know how many consumers there are so it knows when
    it has gotten all responses..
    Regards,
    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Gneeri at Sep 26, 2011 at 4:42 pm
    Hi Alex,

    No problem at all, and again, thanks for the clarification. I really
    do appreciate the help. That said, I think I now have a decent sense
    of how to proceed. I'll let you know if I run into any perplexities,
    but will try not to take much more of your time.

    Thanks again,
    Jonathan
    On Sep 26, 12:28?pm, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    This is a point of confusion for me then. In a previous note where I
    mentioned setting autoAck to false and manually using basicAck,
    Matthias replied as follows:
    Uf, I forgot about the persister in the last message. ?Sorry.
    If in the fanout context I could have used consumer acks to hold off
    broker acks to the publisher, I would have expected that I could
    effectively block confirms until after I sent basicAcks from the
    consumer to the broker and then on up to the publisher. From
    Matthias's note (and my tests), however, it appears that the broker
    sends an ack back to the publisher as soon as it takes control of the
    message and so issues a confirm without waiting on anything from the
    consumer.
    He's right. ?I was hoping the immediate flag will hold off the confirm,
    but, of course, it won't.
    In the direct exchange context, however, I took your notes
    to be indicating that the broker there will wait for an ack from the
    consumer, such that if the consumer died while in process, but before
    sending the ack, the broker would return a basic.return along with the
    confirm. If this is not the case, then, my sense is that I will have
    to have the consumers publish a response directly back to original
    publisher as Matthias notes--while using the fanout broadcast to keep
    track of the number of expected responses (as you indicate).
    You will have to have consumers send something to the original
    publisher and a fanout to keep track of responses. ?You'll only need a
    timeout of some sort, so that if any of the consumers takes to long to
    respond, you react somehow (that probably means the consumer died
    processing the message).

    You'll still need the mandatory, immediate flags to at least see that
    the consumer was alive and was consuming from the queue at the time the
    message was sent.
    Does that make sense? Basically, it still orbits around a question of
    how and/or if consumer acks can be made to effect a producer. I am
    guessing that my implementation will have to require the return
    publications (responses), but any clarification you could add to the
    consumer ack, confirm chain would be greatly appreciated.
    You'll need to publish back. ?The only things confirms will give you in
    this case is the guarantee that basic.returns happen before confirms.
    So, once you receive the confirm, you know the message has reached the
    consumer.

    There's also no need to delay acknowledgement on the consumer side.

    Sorry for the confusion.

    Alex






    On Mon, Sep 26, 2011 at 09:04:06AM -0700, gneeri wrote:
    Hi Alex,
    This is a point of confusion for me then. In a previous note where I
    mentioned setting autoAck to false and manually using basicAck,
    Matthias replied as follows:
    "Confirms mean one thing and one thing only: that the broker has
    accepted
    responsibility for the message.
    "They concern the interaction between producers and the broker, and
    have
    nothing to do with consumer acks."
    If in the fanout context I could have used consumer acks to hold off
    broker acks to the publisher, I would have expected that I could
    effectively block confirms until after I sent basicAcks from the
    consumer to the broker and then on up to the publisher. From
    Matthias's note (and my tests), however, it appears that the broker
    sends an ack back to the publisher as soon as it takes control of the
    message and so issues a confirm without waiting on anything from the
    consumer. In the direct exchange context, however, I took your notes
    to be indicating that the broker there will wait for an ack from the
    consumer, such that if the consumer died while in process, but before
    sending the ack, the broker would return a basic.return along with the
    confirm. If this is not the case, then, my sense is that I will have
    to have the consumers publish a response directly back to original
    publisher as Matthias notes--while using the fanout broadcast to keep
    track of the number of expected responses (as you indicate).
    Does that make sense? Basically, it still orbits around a question of
    how and/or if consumer acks can be made to effect a producer. I am
    guessing that my implementation will have to require the return
    publications (responses), but any clarification you could add to the
    consumer ack, confirm chain would be greatly appreciated.
    Thanks,
    Jonathan
    On Sep 26, 11:34?am, Alexandru Scvor?ov <alexan... at rabbitmq.com>
    wrote:
    Hi Jonathan,
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."
    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code?
    The exchange type has nothing to do with it. ?When you consume from a
    queue, you specify the autoAck parameter. ?If this is set to true
    (which it is, by default), then the server will not wait for the
    consumer to acknowledge receipt of the message (so, it's effectively
    auto-ack'ing all messages). ?If it's set to false, the broker will not
    consider the message delivered to the consumer until it explicitly ack's
    the message. ?See the API guide for an example (I assume you're using
    the Java client, but the other clients have very similar APIs):
    ?http://www.rabbitmq.com/api-guide.html#consuming
    If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Oh. ?I missed that. ?Yes, I think that would work.
    Cheers,
    Alex
    On Mon, Sep 26, 2011 at 08:04:28AM -0700, gneeri wrote:
    Hi Alex,
    Thanks so much for the detailed reply. I had begun thinking along
    these lines, but really appreciate the level of detail regarding the
    specific types of publish requests, queues and exchanges. This is
    really helpful and will likely save me many hours of trial and error.
    I do have just one quick question though regarding the following:
    "In addition, when processing "please log out this user" messages, a
    machine should *not* acknowledge that it has received the message
    until after it publishes its response (this way, if the machine dies
    while processing, the sender will get a basic.return for it)."
    Am I correct in understanding from this that, in the direct exchange
    context, I can control when the consumer machine sends back its ack,
    by calling something like basicAck at some point in the consumption/
    delivery code? If so, I would expect that I could skip the return
    publishing (i.e., the response) from the consumer to the producer and
    simply check confirms and confirms + basic.returns for each message
    sent directly to each machine. Moreover, if, prior to publishing, I
    associate the machine (or direct queue corresponding to it) to the
    message to be sent, I should be able to find out which machines
    present which form of response (i.e., confirm or confirm +
    basic.return). Does this make sense or am I still reading too much
    into the consumer ack (such that, for example, the confirmation
    received by the publisher just represents the handoff of the message
    to the queue and not its uptake by the consumer)?
    Thanks again, really, I very much appreciate your help.
    Cheers,
    Jonathan
    On Sep 26, 5:54?am, Alexandru Scvor?ov wrote:
    Hi Jonathan,
    1. Do consumer acks have more significance in transactions, such that
    I could use those to effect the functionality I'm looking for?
    Not really, no. ?I don't think there's anyway to get what you want
    automatically.
    2. If not, is there any way (apart from the admin HTTP API) for
    determining the number of consumers (e.g., bound queues) for a given
    exchange? This way I could test the number of reply messages as you
    suggest; otherwise, with needing to register and account for a dynamic
    array of servers, I would lose a good deal of what I had hoped would
    be the benefit of a fanout broadcast.
    There's no AMQP way of finding how many queues are bound to an exchange.
    As you say, there's the management API, but using that is likely to lead
    to races (what happens if you query the number of queues, then one of
    the machines dies before it sends its acknowledgement?).
    You could kind of get what you want with AMQP. ?Use a fanout exchange
    like you want for broadcast messages, but use a direct exchange for
    direct communication between machines. ?You could use the fanout
    exchange to make sure that all the machines know about each other (in
    particular, that each knows the queues the others bound to the direct
    exchange, so that they can communicate directly). ?And you could
    ...

    read more ?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedSep 22, '11 at 7:22p
activeSep 26, '11 at 4:42p
posts12
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase