We're regularly able to break rabbits tracking of consumer counts on a
queue, such that it says "X consumers" whereas the reality is "X - N
consumers" where N>0. This seems to be triggered by disagreement as to
the state of sockets, likely caused by an unclean disconnect by a
client. This could be caused by any number of things, such as
rebooting on of our client VMs or maybe just a connection drop inside
our providers network (a common and unwelcome event).

We've seen this problem on 2.5.1 + R14B01, and are continuing to see
it on 2.7.1 + R15B. I can try to provide additional details on
request.

My concerns on consumer counts are two-fold. First, we have no way to
validate that num_procs==num_consumers, an important metric when
validating that our platform is running correctly. Second, we don't
know if rabbit is pushing messages to these ghost consumers, and if
so, what's happening to them.

-Aaron


--
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com

Search Discussions

  • Simon MacMullen at Feb 15, 2012 at 3:22 pm
    Do you see these connections in rabbitmqctl / mgmt?

    Have you enabled heartbeats?

    Cheers, Simon
    On 15/02/12 15:17, Aaron Westendorf wrote:
    We're regularly able to break rabbits tracking of consumer counts on a
    queue, such that it says "X consumers" whereas the reality is "X - N
    consumers" where N>0. This seems to be triggered by disagreement as to
    the state of sockets, likely caused by an unclean disconnect by a
    client. This could be caused by any number of things, such as
    rebooting on of our client VMs or maybe just a connection drop inside
    our providers network (a common and unwelcome event).

    We've seen this problem on 2.5.1 + R14B01, and are continuing to see
    it on 2.7.1 + R15B. I can try to provide additional details on
    request.

    My concerns on consumer counts are two-fold. First, we have no way to
    validate that num_procs==num_consumers, an important metric when
    validating that our platform is running correctly. Second, we don't
    know if rabbit is pushing messages to these ghost consumers, and if
    so, what's happening to them.

    -Aaron

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Aaron Westendorf at Feb 15, 2012 at 3:58 pm
    I need to run reports to figure out which ports are diff'd between
    client and server, then mine rabbitmqctl to see what it thinks. That's
    on deck for the day but may take awhile or get pushed off in favor of
    other needs.

    We haven't enabled heartbeats because until recently, we weren't
    guaranteed that we could timely acknowledge a heartbeat while also
    handling a message. If heartbeats will rid us of any question, I'll
    gladly turn them back on.

    -Aaron
    On Wed, Feb 15, 2012 at 10:22 AM, Simon MacMullen wrote:
    Do you see these connections in rabbitmqctl / mgmt?

    Have you enabled heartbeats?

    Cheers, Simon

    On 15/02/12 15:17, Aaron Westendorf wrote:

    We're regularly able to break rabbits tracking of consumer counts on a
    queue, such that it says "X consumers" whereas the reality is "X - N
    consumers" where N>0. This seems to be triggered by disagreement as to
    the state of sockets, likely caused by an unclean disconnect by a
    client. This could be caused by any number of things, such as
    rebooting on of our client VMs or maybe just a connection drop inside
    our providers network (a common and unwelcome event).

    We've seen this problem on 2.5.1 + R14B01, and are continuing to see
    it on 2.7.1 + R15B. ?I can try to provide additional details on
    request.

    My concerns on consumer counts are two-fold. First, we have no way to
    validate that num_procs==num_consumers, an important metric when
    validating that our platform is running correctly. Second, we don't
    know if rabbit is pushing messages to these ghost consumers, and if
    so, what's happening to them.

    -Aaron

    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


    --
    Aaron Westendorf
    Senior Software Engineer
    Agora Games
    359 Broadway
    Troy, NY 12180
    Phone: 518.268.1000
    aaron at agoragames.com
    www.agoragames.com
  • Simon MacMullen at Feb 15, 2012 at 4:00 pm

    On 15/02/12 15:58, Aaron Westendorf wrote:
    We haven't enabled heartbeats because until recently, we weren't
    guaranteed that we could timely acknowledge a heartbeat while also
    handling a message. If heartbeats will rid us of any question, I'll
    gladly turn them back on.
    Well, heartbeats will definitely detect connections that have died but
    where this has not been noticed by the underlying OS, which sounds like
    what you're seeing.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Aaron Westendorf at Feb 15, 2012 at 4:13 pm
    That is sensible and we now have the capacity to do so. Thank you.

    cheers,
    Aaron


    On Wed, Feb 15, 2012 at 11:00 AM, Simon MacMullen wrote:
    On 15/02/12 15:58, Aaron Westendorf wrote:

    We haven't enabled heartbeats because until recently, we weren't
    guaranteed that we could timely acknowledge a heartbeat while also
    handling a message. If heartbeats will rid us of any question, I'll
    gladly turn them back on.

    Well, heartbeats will definitely detect connections that have died but where
    this has not been noticed by the underlying OS, which sounds like what
    you're seeing.

    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, VMware


    --
    Aaron Westendorf
    Senior Software Engineer
    Agora Games
    359 Broadway
    Troy, NY 12180
    Phone: 518.268.1000
    aaron at agoragames.com
    www.agoragames.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 15, '12 at 3:17p
activeFeb 15, '12 at 4:13p
posts5
users2
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2021 Grokbase