Hi,

I've made some effort to answer these questions on my own, but I'm
still a bit unsure. I'm designing an application for a webserver that
needs to publish events to a broker as part of the work to render a
request. The information is important, but in the event of a broker
failure or broker throttling, the most important thing is to get the
web request rendered. If that means messages get dropped, that's fine
- the consumers are idempotent and we can replay the messages later if
necessary.

I'd like this information to be published with as little latency to
the rendering thread as possible, so large delays while publishing
messages (20ms+) aren't acceptable, even if the broker is down or
throttling. Essentially, I want a situation where publishing gets
done, but the publishing process and/or the state of the broker don't
impact the process of rendering the request. I'm using the Java
client library.

My first inclination was to put all messages in a blocking queue and
have a single thread dedicated to publishing the contents of that
queue to the broker. If the broker goes down that's ok - a bound on
the queue will make sure we don't exceed the server's heap capacity.
The problem there is that such queues synchronize all writers on a
single lock, and given the high concurrency of the server there could
be a lot of contention. I want to avoid a single point of
synchronization.

There are lock-free queue implementations, but they don't offer an
easy way to implement a size bound.

Rather than queuing locally, isn't RabbitMQ for queueing? So I
figured I could keep a single connection around for all publishers in
the JVM, with one thread responsible for maintaining the connection.
Since the server uses a thread pool to render requests, every thread
can have their own channel in a threadlocal, only creating a new
channel if the underlying connection breaks and is reestablished or
they haven't created one before. If the connection is down when the
rendering thread tries to publish, we'll just drop the message.
Otherwise each rendering thread can publish straight to RabbitMQ.

A couple of questions about doing it this way. First, does publishing
a message require a roundtrip to the broker? I dug around on the
topic and came to some part of the AMQP spec that indicates publishing
is a "cast", not a "call", so it seems like it shouldn't. ACKs come
back asynchronously, so I'm assuming publishing is essentially a 0ms
operation, but I want to be sure. Second, if the broker is throttling
via "TCP backpressure" (which I'm having a hard time finding a good
definition for) will that increase publisher latency? I see that I
can set a socket timeout, but I think that only applies if the broker
is actually down.

Thanks!

- Eric

Search Discussions

  • Simon MacMullen at Feb 21, 2012 at 5:58 pm

    On 21/02/12 17:51, Eric wrote:
    First, does publishing
    a message require a roundtrip to the broker?
    No. Well, there are various ways you can force it to, but by default
    it's async.
    so I'm assuming publishing is essentially a 0ms
    operation
    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?
    but I want to be sure. Second, if the broker is throttling
    via "TCP backpressure" (which I'm having a hard time finding a good
    definition for) will that increase publisher latency?
    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.

    Would you rather drop the message at this point?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Eric at Feb 21, 2012 at 6:11 pm
    Thanks for the quick reply.
    No. Well, there are various ways you can force it to, but by default
    it's async.
    Awesome! Great that publishing is async.
    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?
    Right, sorry. I mean ~0ms from the point of view of the client. I
    don't care too much about message latency from publisher to the
    eventual consumer, just so long as the publisher doesn't experience
    any delay.
    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.

    Would you rather drop the message at this point?
    "TCP flow control" got me to a helpful Wikipedia article much faster,
    so maybe that's better. :) I would rather drop the message at that
    point. Ideally there'd be some timeout to specify there... if I've
    been blocked for a few ms, it's time to give up. But publisher
    throttling is supposed to be transparent to the publisher, right?
    On Feb 21, 9:58?am, Simon MacMullen wrote:
    On 21/02/12 17:51, Eric wrote:
    ?> First, does publishing
    ?> a message require a roundtrip to the broker?

    No. Well, there are various ways you can force it to, but by default
    it's async.

    ?> so I'm assuming publishing is essentially a 0ms
    ?> operation

    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?

    ?> but I want to be sure. ?Second, if the broker is throttling
    ?> via "TCP backpressure" (which I'm having a hard time finding a good
    ?> definition for) will that increase publisher latency?

    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.

    Would you rather drop the message at this point?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Eric at Feb 22, 2012 at 5:44 pm
    I think I found my answer in another thread:
    http://groups.google.com/group/rabbitmq-discuss/browse_thread/thread/dda41e9e8e089eaf/2c5995dc93a7a3df?lnk=gst&q=Memory+Management+Concerns+%2F+Questions#2c5995dc93a7a3df

    In summary: there's no way to know you're being throttled and no way
    to timeout during a throttled publish. That means that publishing
    should be really fast! ...unless the broker's out of memory, and then
    you'll be stuck waiting for an undefined amount of time. That's a
    really disappointing situation.

    The solution proposed in that thread is to use a blocking queue with a
    single publisher, which I threw out already due to worries about lock
    contention.
    On Feb 21, 10:11?am, Eric wrote:
    Thanks for the quick reply.
    No. Well, there are various ways you can force it to, but by default
    it's async.
    Awesome! ?Great that publishing is async.
    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?
    Right, sorry. ?I mean ~0ms from the point of view of the client. ?I
    don't care too much about message latency from publisher to the
    eventual consumer, just so long as the publisher doesn't experience
    any delay.
    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.
    Would you rather drop the message at this point?
    "TCP flow control" got me to a helpful Wikipedia article much faster,
    so maybe that's better. ?:) ?I would rather drop the message at that
    point. ?Ideally there'd be some timeout to specify there... if I've
    been blocked for a few ms, it's time to give up. ?But publisher
    throttling is supposed to be transparent to the publisher, right?

    On Feb 21, 9:58?am, Simon MacMullen wrote:






    On 21/02/12 17:51, Eric wrote:
    ?> First, does publishing
    ?> a message require a roundtrip to the broker?
    No. Well, there are various ways you can force it to, but by default
    it's async.
    ?> so I'm assuming publishing is essentially a 0ms
    ?> operation
    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?
    ?> but I want to be sure. ?Second, if the broker is throttling
    ?> via "TCP backpressure" (which I'm having a hard time finding a good
    ?> definition for) will that increase publisher latency?
    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.
    Would you rather drop the message at this point?
    Cheers, Simon
    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Eric at Feb 22, 2012 at 5:48 pm
    Is it possible to configure the broker to never page to disk so that
    it never throttles? Basically, when it hits its high watermark for
    memory can it just drop inbound messages?
    On Feb 22, 9:44?am, Eric wrote:
    I think I found my answer in another thread:http://groups.google.com/group/rabbitmq-discuss/browse_thread/thread/...

    In summary: ?there's no way to know you're being throttled and no way
    to timeout during a throttled publish. ?That means that publishing
    should be really fast! ...unless the broker's out of memory, and then
    you'll be stuck waiting for an undefined amount of time. ?That's a
    really disappointing situation.

    The solution proposed in that thread is to use a blocking queue with a
    single publisher, which I threw out already due to worries about lock
    contention.

    On Feb 21, 10:11?am, Eric wrote:






    Thanks for the quick reply.
    No. Well, there are various ways you can force it to, but by default
    it's async.
    Awesome! ?Great that publishing is async.
    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?
    Right, sorry. ?I mean ~0ms from the point of view of the client. ?I
    don't care too much about message latency from publisher to the
    eventual consumer, just so long as the publisher doesn't experience
    any delay.
    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.
    Would you rather drop the message at this point?
    "TCP flow control" got me to a helpful Wikipedia article much faster,
    so maybe that's better. ?:) ?I would rather drop the message at that
    point. ?Ideally there'd be some timeout to specify there... if I've
    been blocked for a few ms, it's time to give up. ?But publisher
    throttling is supposed to be transparent to the publisher, right?
    On Feb 21, 9:58?am, Simon MacMullen wrote:

    On 21/02/12 17:51, Eric wrote:
    ?> First, does publishing
    ?> a message require a roundtrip to the broker?
    No. Well, there are various ways you can force it to, but by default
    it's async.
    ?> so I'm assuming publishing is essentially a 0ms
    ?> operation
    Well, you still have the one way network latency - or do you mean from
    the point of view of the client?
    ?> but I want to be sure. ?Second, if the broker is throttling
    ?> via "TCP backpressure" (which I'm having a hard time finding a good
    ?> definition for) will that increase publisher latency?
    Maybe we should call it TCP flow control? But yes, if your broker is out
    of memory, publishing will block until the broker is ready to read again.
    Would you rather drop the message at this point?
    Cheers, Simon
    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Matthew Sackman at Feb 22, 2012 at 5:52 pm

    On Wed, Feb 22, 2012 at 09:48:58AM -0800, Eric wrote:
    Is it possible to configure the broker to never page to disk so that
    it never throttles? Basically, when it hits its high watermark for
    memory can it just drop inbound messages?
    Nope.

    Matthew
  • Simon MacMullen at Feb 23, 2012 at 12:09 pm

    On 22/02/12 17:52, Matthew Sackman wrote:
    Is it possible to configure the broker to never page to disk so that
    it never throttles? Basically, when it hits its high watermark for
    memory can it just drop inbound messages?
    Nope.
    I find myself wondering whether some such feature may be useful though.
    At the moment Rabbit is concerned with never throwing anything away, how
    many people would find it useful to drop messages when the server is busy?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Michael Bridgen at Feb 23, 2012 at 12:15 pm

    On 02/23/2012 12:09 PM, Simon MacMullen wrote:
    On 22/02/12 17:52, Matthew Sackman wrote:
    Is it possible to configure the broker to never page to disk so that
    it never throttles? Basically, when it hits its high watermark for
    memory can it just drop inbound messages?
    Nope.
    I find myself wondering whether some such feature may be useful though.
    At the moment Rabbit is concerned with never throwing anything away, how
    many people would find it useful to drop messages when the server is busy?
    If data needs to be timely in preference to being reliable -- better to
    throw away data if it looks like connections are blocked, and send fresh
    data when things are clear again.

    -Michael
  • Eric at Feb 23, 2012 at 8:27 pm
    Well... I would find it useful. And the person in the thread I linked
    to would have found it useful as well.

    It's pretty tough to go into a production environment with the
    statement "Publishing should be fast unless the broker is throttling.
    If the broker is throttling we won't be able to tell, we don't have a
    good way of knowing beforehand, and we don't know how long the threads
    will be blocked". It's useful in any case where an application would
    rather not publish the message rather than waiting to publish it
    indefinitely.

    The net result is that a struggling broker will cause a chain
    reaction; publishers will block indefinitely, which may bring the
    publishing application to a halt. In that case I'd like to abandon
    the publishing process, but it's quite impossible at that point - it's
    too late, the broker is already throttling, and there's no timeout.
    The thread could block for seconds, or more. I'm concerned about not
    throwing the message away, but I'm more concerned about being able to
    isolate services from RabbitMQ broker events.

    I know that this is solved by decoupling the publishing process, but
    then I'm back to the data structure conundrum I mentioned in my first
    post. It's very elegant if a thread can simply publish a message and
    move on with life, and that's only problematic if there's no guarantee
    as to how long that publish can take.

    One more question along this line. The flow control document says "By
    default, when the RabbitMQ server uses above 40% of the installed RAM,
    it raises a memory alarm... Connection heartbeat monitoring gets
    disabled too." If I set up a connection heartbeat on the shared
    connection, will the heartbeat detect a broker alarm that exceeds the
    interval and disconnect? A disconnect would terminate the publisher
    channels and free things up.

    - Eric
    On Feb 23, 4:15?am, Michael Bridgen wrote:
    On 02/23/2012 12:09 PM, Simon MacMullen wrote:
    On 22/02/12 17:52, Matthew Sackman wrote:
    Is it possible to configure the broker to never page to disk so that
    it never throttles? Basically, when it hits its high watermark for
    memory can it just drop inbound messages?
    Nope.
    I find myself wondering whether some such feature may be useful though.
    At the moment Rabbit is concerned with never throwing anything away, how
    many people would find it useful to drop messages when the server is busy?
    If data needs to be timely in preference to being reliable -- better to
    throw away data if it looks like connections are blocked, and send fresh
    data when things are clear again.

    -Michael
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Feb 24, 2012 at 11:20 am

    On 23/02/12 20:27, Eric wrote:
    Well... I would find it useful. And the person in the thread I linked
    to would have found it useful as well.
    Sure, I'm trying to gauge how widespread such a demand is. It looks like
    we could plausibly do something in this direction.
    It's pretty tough to go into a production environment with the
    statement "Publishing should be fast unless the broker is throttling.
    If the broker is throttling we won't be able to tell, we don't have a
    good way of knowing beforehand, and we don't know how long the threads
    will be blocked". It's useful in any case where an application would
    rather not publish the message rather than waiting to publish it
    indefinitely.
    Yeah, the problem is that we've historically optimised for Never
    Throwing Away Messages Ever Ever...
    The net result is that a struggling broker will cause a chain
    reaction; publishers will block indefinitely, which may bring the
    publishing application to a halt. In that case I'd like to abandon
    the publishing process, but it's quite impossible at that point - it's
    too late, the broker is already throttling, and there's no timeout.
    The thread could block for seconds, or more. I'm concerned about not
    throwing the message away, but I'm more concerned about being able to
    isolate services from RabbitMQ broker events.
    If it's any consolation the flow control stuff that's coming in 2.8.0
    will make it much harder to make the memory alarm go off, so blocking
    for seconds should be much less likely.
    One more question along this line. The flow control document says "By
    default, when the RabbitMQ server uses above 40% of the installed RAM,
    it raises a memory alarm... Connection heartbeat monitoring gets
    disabled too." If I set up a connection heartbeat on the shared
    connection, will the heartbeat detect a broker alarm that exceeds the
    interval and disconnect? A disconnect would terminate the publisher
    channels and free things up.
    No - the whole point of disabling heartbeats is to make sure connections
    stay up when this happens...

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Eric at Feb 24, 2012 at 6:07 pm

    Sure, I'm trying to gauge how widespread such a demand is. It looks like
    we could plausibly do something in this direction.
    Seems like it's not very widespread. That's fine - I'm probably
    pushing towards a less common use case. We can deal with somewhat
    unreliable messaging but we'd really like a broker. 0MQ is great for
    unreliable messaging but it's brokerless, and we'd rather not write
    our own broker. RabbitMQ has a great broker, it's just sort of...
    narcissistic. :)
    If it's any consolation the flow control stuff that's coming in 2.8.0
    will make it much harder to make the memory alarm go off, so blocking
    for seconds should be much less likely.
    In reality, the throttling case is extremely unlikely anyway, right?
    You basically have to create a case where publishers are outpacing
    consumers and/or the speed of disk writes on the broker. So you'd
    have to publish something like 100mb/s with little or no consumption,
    and our application is roughly 5mb/minute. So we'd probably never
    throttle. It's just that having some kind of guarantee is nice from
    an operational standpoint.

    In conclusion, I finally did what I should have done in the first
    place. I mocked up the threading scenario we're likely to see in
    production and benchmarked it against Java's LinkedBlockingQueue and
    ArrayBlockingQueue. Compared to each producer thread having its own
    linked list (the completely uncontended scenario) the blocking queues
    introduced virtually no additional waiting - a microsecond or two at
    most. That's the case even at rates 50x or more than we'd see in
    production. So blocking queue it is, and that's a comfortingly simple
    solution.

    Thanks for all the help and apologies for under-researching what I was
    doing.

    - Eric
    On Feb 24, 3:20?am, Simon MacMullen wrote:
    On 23/02/12 20:27, Eric wrote:

    Well... I would find it useful. ?And the person in the thread I linked
    to would have found it useful as well.
    Sure, I'm trying to gauge how widespread such a demand is. It looks like
    we could plausibly do something in this direction.
    It's pretty tough to go into a production environment with the
    statement "Publishing should be fast unless the broker is throttling.
    If the broker is throttling we won't be able to tell, we don't have a
    good way of knowing beforehand, and we don't know how long the threads
    will be blocked". ?It's useful in any case where an application would
    rather not publish the message rather than waiting to publish it
    indefinitely.
    Yeah, the problem is that we've historically optimised for Never
    Throwing Away Messages Ever Ever...
    The net result is that a struggling broker will cause a chain
    reaction; publishers will block indefinitely, which may bring the
    publishing application to a halt. ?In that case I'd like to abandon
    the publishing process, but it's quite impossible at that point - it's
    too late, the broker is already throttling, and there's no timeout.
    The thread could block for seconds, or more. ?I'm concerned about not
    throwing the message away, but I'm more concerned about being able to
    isolate services from RabbitMQ broker events.
    If it's any consolation the flow control stuff that's coming in 2.8.0
    will make it much harder to make the memory alarm go off, so blocking
    for seconds should be much less likely.
    One more question along this line. ?The flow control document says "By
    default, when the RabbitMQ server uses above 40% of the installed RAM,
    it raises a memory alarm... Connection heartbeat monitoring gets
    disabled too." ?If I set up a connection heartbeat on the shared
    connection, will the heartbeat detect a broker alarm that exceeds the
    interval and disconnect? ?A disconnect would terminate the publisher
    channels and free things up.
    No - the whole point of disabling heartbeats is to make sure connections
    stay up when this happens...

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Feb 27, 2012 at 10:42 am

    On 24/02/12 18:07, Eric wrote:
    In reality, the throttling case is extremely unlikely anyway, right?
    You basically have to create a case where publishers are outpacing
    consumers and/or the speed of disk writes on the broker. So you'd
    have to publish something like 100mb/s with little or no consumption,
    and our application is roughly 5mb/minute.
    With small messages in fact the first thing you run into is being
    CPU-bound making routing decisions. But the general principle holds, yes.
    In conclusion, I finally did what I should have done in the first
    place. I mocked up the threading scenario we're likely to see in
    production and benchmarked it against Java's LinkedBlockingQueue and
    ArrayBlockingQueue. Compared to each producer thread having its own
    linked list (the completely uncontended scenario) the blocking queues
    introduced virtually no additional waiting - a microsecond or two at
    most. That's the case even at rates 50x or more than we'd see in
    production. So blocking queue it is, and that's a comfortingly simple
    solution.

    Thanks for all the help and apologies for under-researching what I was
    doing.
    Ah, good.

    I had a quick look at implementing non-blocking publish on Java on
    Friday. Unfortunately it's not possible to do this without NIO on Java,
    so a prerequisite becomes "port the Java client to NIO", which... makes
    it less likely to happen I'm afraid.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Anton Lebedevich at Feb 28, 2012 at 5:41 am
    Hello.
    I had the same situation. Web application written in java that
    publishes on each request and needs to respond fast (even with 50x
    error). Rabbitmq throttling once halted all connected web frontends
    causing whole cluster failure.
    Then I used a separate publishing thread with blocking queue and
    single lock contention wasn't a problem for me. But publishing rate on
    a single machine was quite low about hundred/s.
    On Fri, Feb 24, 2012 at 3:20 PM, Simon MacMullen wrote:
    On 23/02/12 20:27, Eric wrote:

    Well... I would find it useful. ?And the person in the thread I linked
    to would have found it useful as well.

    Sure, I'm trying to gauge how widespread such a demand is. It looks like we
    could plausibly do something in this direction.

    It's pretty tough to go into a production environment with the
    statement "Publishing should be fast unless the broker is throttling.
    If the broker is throttling we won't be able to tell, we don't have a
    good way of knowing beforehand, and we don't know how long the threads
    will be blocked". ?It's useful in any case where an application would
    rather not publish the message rather than waiting to publish it
    indefinitely.

    Yeah, the problem is that we've historically optimised for Never Throwing
    Away Messages Ever Ever...

    The net result is that a struggling broker will cause a chain
    reaction; publishers will block indefinitely, which may bring the
    publishing application to a halt. ?In that case I'd like to abandon
    the publishing process, but it's quite impossible at that point - it's
    too late, the broker is already throttling, and there's no timeout.
    The thread could block for seconds, or more. ?I'm concerned about not
    throwing the message away, but I'm more concerned about being able to
    isolate services from RabbitMQ broker events.

    If it's any consolation the flow control stuff that's coming in 2.8.0 will
    make it much harder to make the memory alarm go off, so blocking for seconds
    should be much less likely.

    One more question along this line. ?The flow control document says "By
    default, when the RabbitMQ server uses above 40% of the installed RAM,
    it raises a memory alarm... Connection heartbeat monitoring gets
    disabled too." ?If I set up a connection heartbeat on the shared
    connection, will the heartbeat detect a broker alarm that exceeds the
    interval and disconnect? ?A disconnect would terminate the publisher
    channels and free things up.

    No - the whole point of disabling heartbeats is to make sure connections
    stay up when this happens...


    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 21, '12 at 5:51p
activeFeb 28, '12 at 5:41a
posts13
users5
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase