Hi all,

I'm using 3 rabbit boxes in active-active configuration and sometimes I
experience "strange" behavior, ranging from abnormally high CPU usage, to
service crash.
I noticed that Connections count went to >800. Channels count is ~1600.
There were around 30 connections I expected from my app servers and
everything else looked like this:

{"name":"<'rabbit at rabbitmq-stg-1'.2.10055.0>","node":"rabbit at rabbitmq-stg-1","protocol":"Direct
0-9-1","user":"xxx","vhost":"/","client_properties":{},"type":"direct"}
{"name":"<'rabbit at rabbitmq-stg-1'.2.10072.0>","node":"rabbit at rabbitmq-stg-1","protocol":"Direct
0-9-1","user":"xxx","vhost":"/","client_properties":{},"type":"direct"}
...

Note that "rabbitmq-stg-1" node is connected to itself hundreds of times
via "Direct 0-9-1" protocol .
I'm not sure either those enormous amount of connections are the cause of
trouble, but seems suspicious.
Anybody knows where to look at?

Vadim.

--

Search Discussions

  • Simon MacMullen at Feb 8, 2012 at 10:34 am

    On 07/02/12 23:53, Vadim Chekan wrote:
    Note that "rabbitmq-stg-1" node is connected to itself hundreds of times
    via "Direct 0-9-1" protocol .
    I'm not sure either those enormous amount of connections are the cause
    of trouble, but seems suspicious.
    Anybody knows where to look at?
    So these are connections made with the Erlang client in direct mode. HA
    doesn't do that. Assuming you are not using the direct Erlang client in
    your own apps, it must be a plugin. The mgmt plugin uses short lived
    direct connections, and shovel and federation use long lived ones - are
    you using either of these?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Vadim Chekan at Feb 8, 2012 at 5:22 pm
    Hi Simon,
    Thanks for your reply. I think you are right about plugin. It started when
    we added "ping" to load balancer. We use http's "/api/" interface to test
    for aliveness. Is it a kosher approach?
    Meanwhile we will turn aliveness test off and will try to reproduce the
    problem with a script.

    Vadim.
    On Wed, Feb 8, 2012 at 2:34 AM, Simon MacMullen wrote:
    On 07/02/12 23:53, Vadim Chekan wrote:

    Note that "rabbitmq-stg-1" node is connected to itself hundreds of times
    via "Direct 0-9-1" protocol .
    I'm not sure either those enormous amount of connections are the cause
    of trouble, but seems suspicious.
    Anybody knows where to look at?
    So these are connections made with the Erlang client in direct mode. HA
    doesn't do that. Assuming you are not using the direct Erlang client in
    your own apps, it must be a plugin. The mgmt plugin uses short lived direct
    connections, and shovel and federation use long lived ones - are you using
    either of these?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
    ______________________________**_________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
    https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>


    --
  • Simon MacMullen at Feb 8, 2012 at 5:35 pm

    On 08/02/12 17:22, Vadim Chekan wrote:
    We use http's "/api/" interface to test for aliveness. Is it a kosher
    approach?
    It certainly should be. I recall there was a bug in the aliveness test
    in some old version - what are you using?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Vadim Chekan at Feb 8, 2012 at 8:50 pm
    Rabbit: 2.7.1, erlang: R14B03, CentOS, 64bits.
    We've just ran a test with 5 threads querying the API and rabbit is rock
    solid. So we are looking suspiciously at our load balancer right now. Will
    report the findings...

    Thanks for your help Simon,
    Vadim.
    On Wed, Feb 8, 2012 at 9:35 AM, Simon MacMullen wrote:
    On 08/02/12 17:22, Vadim Chekan wrote:

    We use http's "/api/" interface to test for aliveness. Is it a kosher
    approach?
    It certainly should be. I recall there was a bug in the aliveness test in
    some old version - what are you using?


    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware


    --
  • Vadim Chekan at Feb 9, 2012 at 2:44 am
    Hmm, we hit it very hard with scripts which simulate aliveness probe, but
    no result. I hope it is related to another problem I've described in
    another mail.

    Vadim.
    On Wed, Feb 8, 2012 at 12:50 PM, Vadim Chekan wrote:

    Rabbit: 2.7.1, erlang: R14B03, CentOS, 64bits.
    We've just ran a test with 5 threads querying the API and rabbit is rock
    solid. So we are looking suspiciously at our load balancer right now. Will
    report the findings...

    Thanks for your help Simon,
    Vadim.

    On Wed, Feb 8, 2012 at 9:35 AM, Simon MacMullen wrote:
    On 08/02/12 17:22, Vadim Chekan wrote:

    We use http's "/api/" interface to test for aliveness. Is it a kosher
    approach?
    It certainly should be. I recall there was a bug in the aliveness test in
    some old version - what are you using?


    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware


    --
    From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is
    explicitly specified


    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 7, '12 at 11:53p
activeFeb 9, '12 at 2:44a
posts6
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Vadim Chekan: 4 posts Simon MacMullen: 2 posts

People

Translate

site design / logo © 2017 Grokbase