Hi all,

Rabbitmq crashed and in the log file I can find three different type of errors.
I attached here a snip of the log file, these issues are due to a problem in the server?
We are running Rabbimq 2.5.1.

Thanks
--
Andrea
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111110/51f264b9/attachment.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Crash1.txt
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111110/51f264b9/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Crash2.txt
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111110/51f264b9/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Crash3.txt
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111110/51f264b9/attachment-0002.txt>

Search Discussions

  • Emile Joubert at Nov 10, 2011 at 1:01 pm
    Hi Andrea,
    On 10/11/11 10:17, Rosa, Andrea wrote:
    Rabbitmq crashed and in the log file I can find three different type
    of errors.
    Is it possible that some .beam files in your RabbitMQ installation are
    missing? It is unlikely that these errors appeared spontaneously - what
    event preceded their onset?


    -Emile
  • Emile Joubert at Nov 10, 2011 at 1:14 pm

    On 10/11/11 13:01, Emile Joubert wrote:
    Hi Andrea,
    On 10/11/11 10:17, Rosa, Andrea wrote:
    Rabbitmq crashed and in the log file I can find three different type
    of errors.
    Of course we'll be able to make better guesses with access to more of
    the logfile. Could you send the complete logfile as well as the sasl
    logfile (compressed) to support at rabbitmq.com ?


    Emile
  • Rosa, Andrea at Nov 10, 2011 at 1:36 pm
    Hi

    Thank for your time, I'll do all my best to give you the complete logfile
    thanks
    --
    Andrea
    -----Original Message-----
    From: Emile Joubert [mailto:emile at rabbitmq.com]
    Sent: 10 November 2011 13:14
    To: Rosa, Andrea
    Cc: rabbitmq-discuss at lists.rabbitmq.com
    Subject: Re: [rabbitmq-discuss] Rabbitmq crash
    On 10/11/11 13:01, Emile Joubert wrote:
    Hi Andrea,
    On 10/11/11 10:17, Rosa, Andrea wrote:
    Rabbitmq crashed and in the log file I can find three different
    type
    of errors.
    Of course we'll be able to make better guesses with access to more of
    the logfile. Could you send the complete logfile as well as the sasl
    logfile (compressed) to support at rabbitmq.com ?


    Emile
  • Dave Greggory at Nov 14, 2011 at 3:17 pm
    We're running RabbitMQ 2.5.1 in production and it crashed last week. Crash = Dropped all connections and process died (i.e. rabbitmqctl status and ps -ef | grep rabbitmq both showed that the process itself was not running).?

    Setup -
    2 non-clustered RabbitMQ nodes behind a load balancer with only 1 node being active in the load balancer (2nd one is there for failover in case situations like this).?
    No special config (All defaults)
    Only Management plugin used.?

    CentOS Linux?2.6.18-164.el5 x86_64
    Erlang R13B04

    Clients mostly Java-library 2.5.1 (some older apps with 2.3.1 and 2.0.0 clients).?

    I attached the logs (errors starting at 8-Nov-2011::18:13:02 mark). I also backed up mnesia in its error state and can provide if needed. There's an erl_crash.dump which I can provide if needed as well.?No CPU or memory spikes.

    I was able to start it up again no problem. Just saw the following in the logs at startup (which seemed unusual).

    ? ?=WARNING REPORT==== 8-Nov-2011::18:42:03 ===

    ? ?msg_store_persistent: recovery terms differ from present
    ? ?rebuilding indices from scratch


    We did not upgraded to 2.6.1 because it seemed to be a buggy release (purely based on this mailing list)... i.e. HA nodes config didn't work, etc. I will upgrade our QA environment to 2.7.0 shortly but probably will not go live with 2.7.0 for at least a month (to verify stability).?In the mean time, we'd like to understand what happened.

    We've been pretty happy with RabbitMQ so far, and it has been quite stable on the server side (the java client library is another issue, but that's for another topic). I'm excited to start using HA functionality in 2.7.0.?
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: incident-logs.tar.gz
    Type: application/x-gzip
    Size: 111956 bytes
    Desc: not available
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111114/c0d57e3f/attachment.bin>
    -------------- next part --------------
    An embedded and charset-unspecified text was scrubbed...
    Name: report-minimal-post-restart.txt
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111114/c0d57e3f/attachment.txt>
  • Matthias Radestock at Nov 14, 2011 at 3:37 pm

    On 14/11/11 15:17, Dave Greggory wrote:
    We're running RabbitMQ 2.5.1 in production and it crashed last week.
    The error in the logs is the same as reported in
    http://old.nabble.com/Can%27t-Determine-Cause-of-541---INTERNAL_ERROR-ts32825393.html

    The advice, as there, is to upgrade.

    Matthias.
  • Dave Greggory at Nov 14, 2011 at 3:47 pm
    Thanks for the quick response. But that thread is not very clear on what exactly the problem is.

    1) What went wrong? Why did it happen all of a sudden after running perfectly for months? What triggered it? It must be serious bug to cause RabbitMQ process itself to die.
    2) Have you identified the bug that caused it?

    We'd be happy to upgrade, but we'd like to know with a bit more confidence that upgrading is going to fix the problem.?

    Thanks again
    Dave


    ________________________________
    From: Matthias Radestock <matthias at rabbitmq.com>
    To: Dave Greggory <davegreggory at yahoo.com>
    Cc: "rabbitmq-discuss at lists.rabbitmq.com" <rabbitmq-discuss at lists.rabbitmq.com>
    Sent: Monday, November 14, 2011 10:37 AM
    Subject: Re: [rabbitmq-discuss] RabbitMQ Crash
    On 14/11/11 15:17, Dave Greggory wrote:
    We're running RabbitMQ 2.5.1 in production and it crashed last week.
    The error in the logs is the same as reported in http://old.nabble.com/Can%27t-Determine-Cause-of-541---INTERNAL_ERROR-ts32825393.html

    The advice, as there, is to upgrade.

    Matthias.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111114/30cdf8fd/attachment.htm>
  • Matthias Radestock at Nov 14, 2011 at 3:58 pm
    Dave,
    On 14/11/11 15:47, Dave Greggory wrote:
    Thanks for the quick response. But that thread is not very clear on what
    exactly the problem is.

    1) What went wrong? Why did it happen all of a sudden after running
    perfectly for months? What triggered it?
    We have no idea.
    It must be serious bug to cause RabbitMQ process itself to die.
    Correct. But it clearly is also very rare.
    2) Have you identified the bug that caused it? No.
    We'd be happy to upgrade, but we'd like to know with a bit more
    confidence that upgrading is going to fix the problem.
    As I said in the other thread, the code around the area in which the
    crash occurred has changed substantially in the last few releases. So
    while upgrading may not fix the problem, there is a good chance that it
    will and if it doesn't then we'll get better data points to help us
    track down the problem.

    Regards,

    Matthias.
  • Dave Greggory at Nov 14, 2011 at 4:14 pm
    :-(

    Since we don't know what the problem is and it happens rarely, there's no way to confirm even after upgrading that the problem has been fixed. Is there any more information that we can provide so you can troubleshoot the problem? Should we open a ticket using our support contract with SpringSource/RabbitMQ support?


    ________________________________
    From: Matthias Radestock <matthias at rabbitmq.com>
    To: Dave Greggory <davegreggory at yahoo.com>
    Cc: "rabbitmq-discuss at lists.rabbitmq.com" <rabbitmq-discuss at lists.rabbitmq.com>
    Sent: Monday, November 14, 2011 10:58 AM
    Subject: Re: [rabbitmq-discuss] RabbitMQ Crash

    Dave,
    On 14/11/11 15:47, Dave Greggory wrote:
    Thanks for the quick response. But that thread is not very clear on what
    exactly the problem is.

    1) What went wrong? Why did it happen all of a sudden after running
    perfectly for months? What triggered it?
    We have no idea.
    It must be serious bug to cause RabbitMQ process itself to die.
    Correct. But it clearly is also very rare.
    2) Have you identified the bug that caused it? No.
    We'd be happy to upgrade, but we'd like to know with a bit more
    confidence that upgrading is going to fix the problem.
    As I said in the other thread, the code around the area in which the crash occurred has changed substantially in the last few releases. So while upgrading may not fix the problem, there is a good chance that it will and if it doesn't then we'll get better data points to help us track down the problem.

    Regards,

    Matthias.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111114/804376cf/attachment.htm>
  • Matthias Radestock at Nov 15, 2011 at 9:14 am
    Dave,
    On 14/11/11 16:14, Dave Greggory wrote:
    Since we don't know what the problem is and it happens rarely, there's
    no way to confirm even after upgrading that the problem has been fixed.
    Is there any more information that we can provide so you can
    troubleshoot the problem?
    I do now have a theory for why the problem may occur.

    How long was the rabbit running before it crashed? Could you send us the
    *complete* rabbit log, going all the way from the startup to the crash?
    It's probably too long for the list, so please send it to
    info at rabbitmq.com. Thanks.

    Regards,

    Matthias.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedNov 10, '11 at 10:17a
activeNov 15, '11 at 9:14a
posts10
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase