Hello,
we have a cluster of three RabbitMQs (3.0.1). After having too many unacked messages (because of an error on client side) we ran into a memory error ("Cannot allocate 1167696400 bytes of memory (of type "old_heap")".
Now the management plugin shows "blocked" and "blocking" connections. Additional one of these connections has a channel with 1.6m unacked messages. These connections and channels are not displayed when calling "rabbitmqctl list_connections" or "rabbitmqctl list_channels"!
These connections/channels survived the restart of the client server and the RabbitMQ server.
Closing the connection with the management plugin results in this error:


{throw,{error,{not_a_connection_pid,<0.449.0>}}, [{rabbit_networking,close_connection,2}, {rabbit_mgmt_wm_connection,delete_resource,2}, {webmachine_resource,resource_call,3}, {webmachine_resource,do,3}, {webmachine_decision_core,resource_call,1}, {webmachine_decision_core,decision,1}, {webmachine_decision_core,handle_request,2}, {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}


This is the ouput of the rest client for one of these connections (ip addresses censored):


{
"recv_oct": 636365,
"recv_cnt": 949,
"send_oct": 1696,
"send_cnt": 164,
"send_pend": 0,
"state": "blocked",
"last_blocked_by": "resource",
"last_blocked_age": 1.251632,
"channels": 5,
"recv_oct_details":
{
"rate": 0,
"interval": 32663327,
"last_event": 1365522007990
},
"send_oct_details":
{
"rate": 0,
"interval": 32663327,
"last_event": 1365522007990
},
"type": "network",
"node": "m12n-production at rabbitmq1",
"name": "###.##.1.102:13353 -> ###.##.1.70:5672",
"port": 5672,
"peer_port": 13353,
"host": "###.##.1.70",
"peer_host": "###.##.1.102",
"ssl": false,
"peer_cert_subject": "",
"peer_cert_issuer": "",
"peer_cert_validity": "",
"auth_mechanism": "PLAIN",
"ssl_protocol": "",
"ssl_key_exchange": "",
"ssl_cipher": "",
"ssl_hash": "",
"protocol": "AMQP 0-9-1",
"user": "m12n",
"vhost": "prod",
"timeout": 30,
"frame_max": 131072,
"client_properties":
{
"product": "RabbitMQ",
"information": "Licensed under the MPL. See http://www.rabbitmq.com/",
"platform": "Java",
"capabilities":
{
"exchange_exchange_bindings": true,
"consumer_cancel_notify": true,
"basic.nack": true,
"publisher_confirms": true
},
"copyright": "Copyright (C) 2007-2012 VMware, Inc.",
"version": "3.0.1"
}
}


How can we kill these connections/channels?


Greetings, Dirk

Search Discussions

  • Simon MacMullen at Apr 17, 2013 at 9:33 am
    Hi.


    These connections / channels do not exist. You're seeing a bug in the
    management plugin where it will retain information about connections and
    channels that were alive on a cluster node when it crashed.


    This bug was fixed in RabbitMQ 3.0.3.


    Cheers, Simon

    On 17/04/13 09:57, Dirk Plate wrote:
    Hello,
    we have a cluster of three RabbitMQs (3.0.1). After having too many unacked messages (because of an error on client side) we ran into a memory error ("Cannot allocate 1167696400 bytes of memory (of type "old_heap")".
    Now the management plugin shows "blocked" and "blocking" connections. Additional one of these connections has a channel with 1.6m unacked messages. These connections and channels are not displayed when calling "rabbitmqctl list_connections" or "rabbitmqctl list_channels"!
    These connections/channels survived the restart of the client server and the RabbitMQ server.
    Closing the connection with the management plugin results in this error:

    {throw,{error,{not_a_connection_pid,<0.449.0>}}, [{rabbit_networking,close_connection,2}, {rabbit_mgmt_wm_connection,delete_resource,2}, {webmachine_resource,resource_call,3}, {webmachine_resource,do,3}, {webmachine_decision_core,resource_call,1}, {webmachine_decision_core,decision,1}, {webmachine_decision_core,handle_request,2}, {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}

    This is the ouput of the rest client for one of these connections (ip addresses censored):

    {
    "recv_oct": 636365,
    "recv_cnt": 949,
    "send_oct": 1696,
    "send_cnt": 164,
    "send_pend": 0,
    "state": "blocked",
    "last_blocked_by": "resource",
    "last_blocked_age": 1.251632,
    "channels": 5,
    "recv_oct_details":
    {
    "rate": 0,
    "interval": 32663327,
    "last_event": 1365522007990
    },
    "send_oct_details":
    {
    "rate": 0,
    "interval": 32663327,
    "last_event": 1365522007990
    },
    "type": "network",
    "node": "m12n-production at rabbitmq1",
    "name": "###.##.1.102:13353 -> ###.##.1.70:5672",
    "port": 5672,
    "peer_port": 13353,
    "host": "###.##.1.70",
    "peer_host": "###.##.1.102",
    "ssl": false,
    "peer_cert_subject": "",
    "peer_cert_issuer": "",
    "peer_cert_validity": "",
    "auth_mechanism": "PLAIN",
    "ssl_protocol": "",
    "ssl_key_exchange": "",
    "ssl_cipher": "",
    "ssl_hash": "",
    "protocol": "AMQP 0-9-1",
    "user": "m12n",
    "vhost": "prod",
    "timeout": 30,
    "frame_max": 131072,
    "client_properties":
    {
    "product": "RabbitMQ",
    "information": "Licensed under the MPL. See http://www.rabbitmq.com/",
    "platform": "Java",
    "capabilities":
    {
    "exchange_exchange_bindings": true,
    "consumer_cancel_notify": true,
    "basic.nack": true,
    "publisher_confirms": true
    },
    "copyright": "Copyright (C) 2007-2012 VMware, Inc.",
    "version": "3.0.1"
    }
    }

    How can we kill these connections/channels?

    Greetings, Dirk

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



    --
    Simon MacMullen
    RabbitMQ, VMware
  • Simon MacMullen at Apr 17, 2013 at 10:04 am

    On 17/04/13 10:48, Dirk Plate wrote:
    Hi,
    thanks for this information.
    Is it possible to reset the management plugin (where is this information stored?).

    Hi. This information lives in the management statistics database. You
    can kill that process with e.g.:


    $ rabbitmqctl eval 'exit(global:whereis_name(rabbit_mgmt_db), bang).'


    and it will start up again and reconstruct all its data without the
    phantom connections / channels.


    Cheers, Simon




    --
    Simon MacMullen
    RabbitMQ, VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedApr 17, '13 at 8:57a
activeApr 17, '13 at 10:04a
posts3
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Simon MacMullen: 2 posts Dirk Plate: 1 post

People

Translate

site design / logo © 2017 Grokbase