Had a node crash in a cluster (due to memory exhaustion). When clients then
reconnected to a healthy node, bind commands failed to return. And when the
crashed node was restarted the logs was flooded with:


=ERROR REPORT==== 5-Aug-2013::23:45:50 ===
Discarding message
{'$gen_call',{<0.7975.0>,#Ref<0.0.0.244595>},{deliver,{delivery,true,<5068.31152.5031>,{basic_message,{resource,<<8
bytes>>,exchange,<<16 bytes>>},[<<7 bytes>>],{content,60,{'P_basic',<<24
bytes>>,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},<<27
bytes>>,rabbit_framing_amqp_0_9_1,[<<299 bytes>>]},<<16
bytes>>,false},396},true}} from <0.7975.0> to <0.26389.5794> in an old
incarnation (3) of this node (1)


The only solution was to restart all nodes in the cluster.








--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Node-crash-tp28595.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

Search Discussions

  • Emile Joubert at Aug 6, 2013 at 10:37 am
    Hi Carl,

    On 06/08/13 08:15, carlhoerberg wrote:
    Had a node crash in a cluster (due to memory exhaustion). When clients then
    reconnected to a healthy node, bind commands failed to return. And when the
    crashed node was restarted the logs was flooded with:

    =ERROR REPORT==== 5-Aug-2013::23:45:50 ===
    Discarding message
    {'$gen_call',{<0.7975.0>,#Ref<0.0.0.244595>},{deliver,{delivery,true,<5068.31152.5031>,{basic_message,{resource,<<8
    bytes>>,exchange,<<16 bytes>>},[<<7 bytes>>],{content,60,{'P_basic',<<24
    bytes>>,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},<<27
    bytes>>,rabbit_framing_amqp_0_9_1,[<<299 bytes>>]},<<16
    bytes>>,false},396},true}} from <0.7975.0> to <0.26389.5794> in an old
    incarnation (3) of this node (1)

    The only solution was to restart all nodes in the cluster.

    One or two of these messages (Discarding message ... in an old
    incarnation) does not indicate an error condition, and can safely be
    ignored.


    If you see large numbers of these messages then that does indicate a
    problem. A cause has been identified and fixed since 3.1.3. The release
    notes containing the fix will contain the tag "25670". You should
    upgrade when this release becomes available.


    (You can obtain the fix sooner by building from source, making sure to
    use the "stable" branch: http://www.rabbitmq.com/build-server.html)










    -Emile
  • Carlhoerberg at Aug 6, 2013 at 10:44 am
    Excellent! Thanks!






    On Tuesday 6 August 2013 at 12:39, Emile Joubert [via RabbitMQ] wrote:

    Hi Carl,
    On 06/08/13 08:15, carlhoerberg wrote:

    Had a node crash in a cluster (due to memory exhaustion). When clients then
    reconnected to a healthy node, bind commands failed to return. And when the
    crashed node was restarted the logs was flooded with:

    =ERROR REPORT==== 5-Aug-2013::23:45:50 ==> > Discarding message
    {'$gen_call',{<0.7975.0>,#Ref<0.0.0.244595>},{deliver,{delivery,true,<5068.31152.5031>,{basic_message,{resource,<<8
    bytes>>,exchange,<<16 bytes>>},[<<7 bytes>>],{content,60,{'P_basic',<<24
    bytes>>,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},<<27
    bytes>>,rabbit_framing_amqp_0_9_1,[<<299 bytes>>]},<<16
    bytes>>,false},396},true}} from <0.7975.0> to <0.26389.5794> in an old
    incarnation (3) of this node (1)

    The only solution was to restart all nodes in the cluster.
    One or two of these messages (Discarding message ... in an old
    incarnation) does not indicate an error condition, and can safely be
    ignored.

    If you see large numbers of these messages then that does indicate a
    problem. A cause has been identified and fixed since 3.1.3. The release
    notes containing the fix will contain the tag "25670". You should
    upgrade when this release becomes available.

    (You can obtain the fix sooner by building from source, making sure to
    use the "stable" branch: http://www.rabbitmq.com/build-server.html)





    -Emile





    _______________________________________________
    rabbitmq-discuss mailing list
    [hidden email] (/user/SendEmail.jtp?type=node&node(605&i=0)
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


    If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/Node-crash-tp28595p28605.html
    To unsubscribe from Node crash, click here (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node(595&code=Y2FybC5ob2VyYmVyZ0BnbWFpbC5jb218Mjg1OTV8LTEyNDcxMDc4NjM=).
    NAML (http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)













    --
    View this message in context: http://rabbitmq.1065348.n5.nabble.com/Node-crash-tp28595p28606.html
    Sent from the RabbitMQ mailing list archive at Nabble.com.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130806/fcf83796/attachment.htm>
  • Simon MacMullen at Aug 6, 2013 at 11:00 am

    On 06/08/13 11:37, Emile Joubert wrote:
    If you see large numbers of these messages then that does indicate a
    problem. A cause has been identified and fixed since 3.1.3. The release
    notes containing the fix will contain the tag "25670". You should
    upgrade when this release becomes available.

    Just for clarification, this will be in 3.1.4.


    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, Pivotal

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 6, '13 at 7:15a
activeAug 6, '13 at 11:00a
posts4
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase