Hi,


RabbitMQ 2.8.6 terminated leaving the following in logs:


=ERROR REPORT==== 25-Sep-2012::23:36:07 ===
** Generic server mnesia_sync terminating
** Last message in was timeout
** When Server state == {state,[{<0.31634.9>,#Ref<0.0.8.19505>}],true}
** Reason for termination ==
**
{{badmatch,{error,{file_error,"/var/lib/rabbitmq/mnesia/rabbit at stgback/LATEST.LOG",
enospc}}},
[{mnesia_sync,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}


=ERROR REPORT==== 25-Sep-2012::23:36:07 ===
** Generic server <0.31634.9> terminating
** Last message in was {basic_cancel,<0.31633.9>,<<"633">>,
{'basic.cancel_ok',<<"633">>}}
** When Server state == {q,
{amqqueue,
{resource,<<"/">>,queue,
<<"704b40896d5947df8a0838616ba18462">>},
false,true,none,
[{<<"x-expires">>,signedint,300000}],
<0.31634.9>,[],undefined},
none,true,rabbit_variable_queue,
{vqstate,
{0,{[],[]}},
{0,{[],[]}},
{delta,undefined,0,undefined},
{0,{[],[]}},
{0,{[],[]}},
0,
{0,nil},
undefined,
{0,nil},
{qistate,


"/var/lib/rabbitmq/mnesia/rabbit at stgback/queues/D297LZX7459XHE45UP1X6ZYM1",
{{dict,0,16,16,8,80,48,


{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[]


What could be the reason?
What additional info is necessary?


I looked through /var/lib/rabbitmq/mnesia/rabbit at stgback/LATEST.LOG. Some
binary format, no obvious things found.


Thank you!


Eugene












--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Rabbit-Crash-tp22219.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

Search Discussions

  • Simon MacMullen at Sep 26, 2012 at 3:44 pm

    On 26/09/12 16:14, eugene wrote:
    RabbitMQ 2.8.6 terminated leaving the following in logs:

    <snip>

    {{badmatch,{error,{file_error,"/var/lib/rabbitmq/mnesia/rabbit at stgback/LATEST.LOG",
    enospc}}},

    You ran out of disc space, which RabbitMQ does not handle well.


    Which is why we added disc space monitoring in 2.8.2 - when you get to <
    1GB free disc space RabbitMQ should start blocking publishes.


    But in your case it (presumably) didn't. Or else you managed to get
    RabbitMQ to fill up the disc without accepting publishes.


    In either case I am curious as to how you managed to do that :-) Have
    you changed the disc space limit? Were you declaring vast numbers of
    queues / exchanges / etc? Is
    /var/lib/rabbitmq/mnesia/rabbit at stgback/LATEST.LOG on a tiny partition?
    Can you post the contents of /var/log/rabbitmq somewhere?


    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, VMware
  • Eugene at Sep 26, 2012 at 4:13 pm
    Hi Simon,


    Thank you for replying!


    I think you're right. The node Rabbit is running on is a development one
    with limited disk space.


    Knowing that, I put the following in rabbit config:


    {rabbit, [{disk_free_limit, 2097152}
    ]},


    It might have influenced.
    I just did not expect rabbit to crash, rather to raise some alarms.


    Regards,


    Eugene












    --
    View this message in context: http://rabbitmq.1065348.n5.nabble.com/Rabbit-Crash-tp22219p22226.html
    Sent from the RabbitMQ mailing list archive at Nabble.com.
  • Matthias Radestock at Sep 26, 2012 at 10:45 pm
    Eugene,

    On 26/09/12 17:13, eugene wrote:
    I think you're right. The node Rabbit is running on is a development one
    with limited disk space.

    Knowing that, I put the following in rabbit config:

    {rabbit, [{disk_free_limit, 2097152}
    ]},

    It might have influenced.
    I just did not expect rabbit to crash, rather to raise some alarms.

    I suspect the rabbit logs will show that an alarm was raised.


    Rabbit's response to such an alarm is to block all publishers, since
    messages are generally the main source of memory and disk space growth
    in rabbit, by some margin.


    However, other activities such as declaring durable exchanges/queues
    require disk space too. So does writing the log file (which, if it's on
    the same partition as the db dir will eat into the latter's space).
    Rabbit makes no attempt to curtail these activities.


    Regards,


    Matthias.
  • Simon MacMullen at Sep 27, 2012 at 9:14 am

    On 26/09/12 23:45, Matthias Radestock wrote:
    I think you're right. The node Rabbit is running on is a development one
    with limited disk space.

    Knowing that, I put the following in rabbit config:

    {rabbit, [{disk_free_limit, 2097152}
    ]},

    It might have influenced.
    I just did not expect rabbit to crash, rather to raise some alarms.
    I suspect the rabbit logs will show that an alarm was raised.

    Rabbit's response to such an alarm is to block all publishers, since
    messages are generally the main source of memory and disk space growth
    in rabbit, by some margin.

    In addition, if the alarm goes off when there is only 2MB of disc space
    left, you're essentially stuck anyway - even blocking publishers is very
    unlikely to happen fast enough to stop you from consuming disc space
    before you run out of space.


    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedSep 26, '12 at 3:14p
activeSep 27, '12 at 9:14a
posts5
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase