Seemingly at random, our rabbitmq application seems to crash. The node
seems to stay up, but I will get a notice that the application is down
an I have to run the start_app command.

Here is the rabbit-sasl.log from the period of time where the crash
seems to occur:

=SUPERVISOR REPORT==== 28-Feb-2009::01:37:29 ===
Supervisor: {local,mnesia_kernel_sup}
Context: child_terminated
Reason: shutdown
Offender: [{pid,<0.61.0>},
{name,mnesia_monitor},
{mfa,{mnesia_monitor,start,[]}},
{restart_type,permanent},
{shutdown,3000},
{child_type,worker}]


=SUPERVISOR REPORT==== 28-Feb-2009::01:37:29 ===
Supervisor: {local,mnesia_kernel_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.61.0>},
{name,mnesia_monitor},
{mfa,{mnesia_monitor,start,[]}},
{restart_type,permanent},
{shutdown,3000},
{child_type,worker}]


=SUPERVISOR REPORT==== 28-Feb-2009::01:37:29 ===
Supervisor: {local,mnesia_sup}
Context: child_terminated
Reason: shutdown
Offender: [{pid,<0.60.0>},
{name,mnesia_kernel_sup},
{mfa,{mnesia_kernel_sup,start,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]


=SUPERVISOR REPORT==== 28-Feb-2009::01:37:29 ===
Supervisor: {local,mnesia_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.60.0>},
{name,mnesia_kernel_sup},
{mfa,{mnesia_kernel_sup,start,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]


=CRASH REPORT==== 28-Feb-2009::01:42:17 ===
crasher:
pid: <0.25316.9>
registered_name: []
error_info: {aborted,{no_exists,[rabbit_config,profiling_enabled]}}
initial_call: {rabbit_reader,init,[<0.140.0>]}
ancestors: [rabbit_tcp_client_sup,rabbit_sup,<0.106.0>]
messages: []
links: [<0.140.0>,#Port<0.1545808>]
dictionary: []
trap_exit: true
status: running
heap_size: 233
stack_size: 21
reductions: 102
neighbours:

=SUPERVISOR REPORT==== 28-Feb-2009::01:42:17 ===
Supervisor: {local,rabbit_tcp_client_sup}
Context: child_terminated
Reason: {aborted,{no_exists,
[rabbit_config,profiling_enabled]}}
Offender: [{pid,<0.25316.9>},
{name,tcp_client},
{mfa,{rabbit_reader,start_link,[]}},
{restart_type,temporary},
{shutdown,brutal_kill},
{child_type,worker}]


=CRASH REPORT==== 28-Feb-2009::15:23:27 ===
crasher:
pid: <0.9179.11>
registered_name: []
error_info: {aborted,{no_exists,[rabbit_config,profiling_enabled]}}
initial_call: {rabbit_reader,init,[<0.140.0>]}
ancestors: [rabbit_tcp_client_sup,rabbit_sup,<0.106.0>]
messages: []
links: [<0.140.0>,#Port<0.1595616>]
dictionary: []
trap_exit: true
status: running
heap_size: 233
stack_size: 21
reductions: 102
neighbours:

=SUPERVISOR REPORT==== 28-Feb-2009::15:23:27 ===
Supervisor: {local,rabbit_tcp_client_sup}
Context: child_terminated
Reason: {aborted,{no_exists,
[rabbit_config,profiling_enabled]}}
Offender: [{pid,<0.9179.11>},
{name,tcp_client},
{mfa,{rabbit_reader,start_link,[]}},
{restart_type,temporary},
{shutdown,brutal_kill},
{child_type,worker}]

Matt George
Software Developer | Emma?
matt.george at myemma.com
800.595.4401
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090302/3bf7c111/attachment.htm

Search Discussions

  • Ben Hood at Mar 2, 2009 at 4:56 pm
    George,
    On Mon, Mar 2, 2009 at 3:20 PM, Matt George wrote:
    =CRASH REPORT==== 28-Feb-2009::01:42:17 ===
    ??crasher:
    ?? ?pid: <0.25316.9>
    ?? ?registered_name: []
    ?? ?error_info: {aborted,{no_exists,[rabbit_config,profiling_enabled]}}
    ?? ?initial_call: {rabbit_reader,init,[<0.140.0>]}
    ?? ?ancestors: [rabbit_tcp_client_sup,rabbit_sup,<0.106.0>]
    ?? ?messages: []
    This looks like you are missing a table called rabbit_config. To
    verify this, use the mnesia:info/0 command from the shell. After
    having checked this, it might be an idea to find out how this could
    come about, seeing as ensuring that the correct tables exist is part
    of the boot sequence.

    HTH,

    Ben
  • Ben Hood at Mar 2, 2009 at 5:10 pm

    ---------- Forwarded message ----------
    From: Ben Hood <0x6e6562 at gmail.com>
    Date: Mon, Mar 2, 2009 at 5:09 PM
    Subject: Re: [rabbitmq-discuss] application keeps crashing
    To: Matt George <matt.george at myemma.com>

    Matt - Sorry I called you George beforehand :-(
    On Mon, Mar 2, 2009 at 5:04 PM, Matt George wrote:
    What is the best way to run that command? I have just a little bit of
    experience with the erlang shell, I am slowly getting more into using it.
    Probably the easiest way (without using any of the Rabbit machinery)
    is to just execute the following:

    $ erl -mnesia dir PATH_TO_YOUR_MNESIA DIRECTORY -s mnesia

    Erlang (BEAM) emulator version 5.6.5 [source] [smp:2]
    [async-threads:0] [hipe] [kernel-poll:false]

    Eshell V5.6.5 ?(abort with ^G)
    1> mnesia:info().

    .....Output follows here

    Ben
  • Ben Hood at Mar 2, 2009 at 10:54 pm
    Matt,
    On Mon, Mar 2, 2009 at 5:13 PM, Matt George wrote:
    ---> Active tables <---
    schema ? ? ? ? : with 1 ? ? ? ?records occupying 388 ? ? ?words of mem
    ===> System info in version "4.3.5", debug level = none <===
    opt_disc. Directory "/Users/mgeorge/Mnesia.nonode at nohost" is NOT used.
    use fallback at restart = false
    running db nodes ? = [nonode at nohost]
    stopped db nodes ? = []
    master node tables = []
    remote ? ? ? ? ? ? = []
    ram_copies ? ? ? ? = [schema]
    disc_copies ? ? ? ?= []
    disc_only_copies ? = []
    Well this means that there was no mnesia database in that directory,
    which is a slight problem :-(

    On face value, one may conclude that there was no Rabbit installed in
    the standard location on that particular node, but this contradicts
    the fact that Rabbit was actually running.

    Can you maybe tell us about the details of your installation
    (including any clustering or virtualization you may have used)?

    Ben
  • Ben Hood at Mar 4, 2009 at 12:42 pm
    Matt,
    On Tue, Mar 3, 2009 at 2:57 PM, Matt George wrote:
    I talked to the guys who setup the box and they told me that it's a standard
    vanilla ubuntu 8.04 install, on a vmware virtual server.
    The rabbitmq install was from the debian package via apt-get, and was
    upgraded from 1.5.1 to 1.5.3 the same way. After we installed
    rabbitmq, nothing else was done in terms of configuration.
    Let me know if you need anymore info
    OK, so you're saying that you can reproduce this behaviour by using
    apt-get to upgrade from 1.5.1 to 1.5.3?

    Ben
  • Dmitriy Samovskiy at Mar 4, 2009 at 3:31 pm
    Hi all,
    I talked to the guys who setup the box and they told me that it's a standard
    vanilla ubuntu 8.04 install, on a vmware virtual server.
    The rabbitmq install was from the debian package via apt-get, and was
    upgraded from 1.5.1 to 1.5.3 the same way. After we installed
    rabbitmq, nothing else was done in terms of configuration.
    Let me know if you need anymore info
    OK, so you're saying that you can reproduce this behaviour by using
    apt-get to upgrade from 1.5.1 to 1.5.3?
    I just did an upgrade from 1.5.1 to 1.5.3 on my vm with "apt-get install rabbitmq-server"
    and it worked well, didn't break anything. It's Debian Etch however, not Ubuntu.

    Matt, do you have any config in /etc/default/rabbitmq? 1.5.3 had some filename changes in
    this area (file got renamed to rabbitmq.conf), please see Tony's announcement:

    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-February/003479.html

    - Dmitriy
  • Matt George at Mar 4, 2009 at 3:37 pm
    I do not have a /etc/default/rabbitmq present. Our install should have
    been just a vanilla install.

    thanks,
    Matt

    Matt George
    Software Developer | Emma?
    matt.george at myemma.com
    800.595.4401
    615.292.0777 (fax)

    Emma helps organizations everywhere communicate & market in style.
    Visit us online at http://www.myemma.com
    On Mar 4, 2009, at 9:31 AM, Dmitriy Samovskiy wrote:


    Hi all,
    I talked to the guys who setup the box and they told me that it's
    a standard
    vanilla ubuntu 8.04 install, on a vmware virtual server.
    The rabbitmq install was from the debian package via apt-get, and
    was
    upgraded from 1.5.1 to 1.5.3 the same way. After we installed
    rabbitmq, nothing else was done in terms of configuration.
    Let me know if you need anymore info
    OK, so you're saying that you can reproduce this behaviour by using
    apt-get to upgrade from 1.5.1 to 1.5.3?
    I just did an upgrade from 1.5.1 to 1.5.3 on my vm with "apt-get
    install rabbitmq-server" and it worked well, didn't break anything.
    It's Debian Etch however, not Ubuntu.

    Matt, do you have any config in /etc/default/rabbitmq? 1.5.3 had
    some filename changes in this area (file got renamed to
    rabbitmq.conf), please see Tony's announcement:

    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-February/003479.html

    - Dmitriy
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090304/da807878/attachment.htm
  • Ben Hood at Mar 4, 2009 at 5:42 pm
    Matt,
    On Wed, Mar 4, 2009 at 3:37 PM, Matt George wrote:
    I do not have a?/etc/default/rabbitmq present. Our install should have been
    just a vanilla install.
    OK.

    To be able to help you, I'm going to need to be able to reproduce your
    error somehow. Considering that you've virtualized your environment,
    are you able to provide an image with your setup that will recreate
    the symptom?

    Ben
  • Ben Hood at Mar 7, 2009 at 1:27 pm
    Matt,
    On Fri, Mar 6, 2009 at 8:58 PM, Matt George wrote:

    Unfortunately, I cannot get a copy of that virtual machine right now.
    Earlier you had said that there might be something wrong with the mnesia db.
    Yes - the error message you were getting was due to a case clause that
    didn't check for the non-existence of the rabbit_config table in
    mnesia, because the non-existence of the table is not something that
    would normally occur.
    Where would I generally look for that location to see if it's messed up?
    Well you would do exactly what you did beforehand, i.e. look at the
    directory where you tell Rabbit to put the mnesia files, which is
    /var/lib/rabbitmq/mnesia in your case.
    ask because we have a dev server, with the same basic setup as our prod
    server,
    that seems to be working completely fine. Which leads me to think that it
    might be a setup or data issue.
    thanks,
    Possibly. This is why is important to able to capture the state of the
    VM when the problem happens again, so that we can look into what is
    missing and maybe try to find out why.

    Ben
  • Ben Hood at Mar 9, 2009 at 3:14 pm
    Matt,
    On Mon, Mar 9, 2009 at 2:29 PM, Matt George wrote:
    We had another crash over the weekend and I saw some stuff in the logs this
    time that I thought might help so I am attaching them with this email.
    OK, the sasl log shows the original symptom where the rabbit_config
    table doesn't exist. Is this on the same environment as the previous
    occurences? You said that you were unable to reproduce this on your
    development environment. I am wondering if the underlying mnesia
    tables are somehow being deleted. If you could do a fresh installation
    of 1.5.3 on the particular environment that is giving you grief, maybe
    you will be able to reproduce this issue.
    One
    of the things that I noticed was that it looks like somehow the connections
    seem to be opening but not closing for some time.
    This is not an issue. As AMQP is a connection orientated protocol,
    connections can last a long time.
    I am using the pyamqplib
    library to make my connections. From the rabbitmq standpoint, is it better
    to keep a connection open, or to get a new connection for every message?
    The former, as that is what the protocol is designed to do. However,
    we have been able to maintain 20000 concurrent connections to a single
    instance (before getting bored), so it would be pretty hard to
    overload it.

    Looking at the normal log, you have a few thousand standard entries
    that document clients connecting and disconnecting.

    Towards the back end of the log file (at about 21:19:56 on 8/3/09) you
    have a lot of connection_closed_abruptly errors in the log. This is
    usually the result of the client resetting the socket without
    observing the AMQP shutdown protocol (in plain english, it looks as if
    a number of clients have just died). Are you sure that the Rabbit
    instance has become unavailable at this point? Do you have another
    client to test this with?

    Ben
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: rabbit.log.1
    Type: application/octet-stream
    Size: 890779 bytes
    Desc: not available
    Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090309/32efd88d/attachment.obj
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: rabbit-sasl.log.1
    Type: application/octet-stream
    Size: 6074 bytes
    Desc: not available
    Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090309/32efd88d/attachment-0001.obj
  • Ben Hood at Mar 9, 2009 at 3:27 pm
    Matt,
    On Mon, Mar 9, 2009 at 3:19 PM, Matt George wrote:
    I think we might be able to do a fresh install on the prod box later this
    week. Is that the one you were talking about?
    Would it be best to try and uninstall the current Rabbitmq instance and then
    install again?
    Yes. Ideally you should start from fresh and install the latest stable
    of Rabbit, which is 1.5.3.

    BTW - If you keep this discussion on list, then you might get some
    help from other members of the community :-)

    Ben
  • Ben Hood at Mar 4, 2009 at 5:36 pm
    Just cc'ing the list :-)


    ---------- Forwarded message ----------
    From: Matt George <matt.george at myemma.com>
    Date: Wed, Mar 4, 2009 at 3:16 PM
    Subject: Re: [rabbitmq-discuss] application keeps crashing
    To: Ben Hood <0x6e6562 at gmail.com>


    No I'm sorry for the confusion. I was saying that we had upgraded the
    to 1.5.3 using apt-get.
    thanks,
    Matt
    Matt George
    Software Developer?|?Emma?
    matt.george at myemma.com
    800.595.4401
    615.292.0777 (fax)
    Emma helps organizations everywhere communicate & market in style.
    Visit us online at?http://www.myemma.com
    - Show quoted text -
    On Mar 4, 2009, at 6:42 AM, Ben Hood wrote:

    Matt,

    On Tue, Mar 3, 2009 at 2:57 PM, Matt George wrote:

    I talked to the guys who setup the box and they told me that it's a standard

    vanilla ubuntu 8.04 install, on a vmware virtual server.

    The rabbitmq install was from the debian package via apt-get, and was

    upgraded from 1.5.1 to 1.5.3 the same way. After we installed

    rabbitmq, nothing else was done in terms of configuration.

    Let me know if you need anymore info

    OK, so you're saying that you can reproduce this behaviour by using
    apt-get to upgrade from 1.5.1 to 1.5.3?

    Ben

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedMar 2, '09 at 3:20p
activeMar 9, '09 at 3:27p
posts12
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase