Hi All,

I need some advice on moving /var/lib/rabbitmq to a new server with
more RAM. What has happened is that on our 8GB RAM server we've had a
surge in activity and one of our queues has grown much faster than it
can be consumed by our "workers".

This has resulted in RabbitMQ crashing, with an erl_crash.dump file
appearing.

At present when I attempt to start the server up again, it tries to
roll the persister.LOG file and recover what was in the queue at the
time of the crash. However even with rabbitmq.config set up as below,
the vm_memory_high_watermark alarm is triggered and erl_crash.dump is
created all over again.

[
{rabbit, [{vm_memory_high_watermark, 0.9}]}
].

I've got another server with 24GB of RAM, so I set it's HOSTNAME to
the same as the 8GB server and installed the same version of RabbitMQ.
I then copied over the contents of /var/lib/rabbitmq and attempted to
start RabbitMQ. I no longer hit the vm_memory_high_watermark however
Erlang seems to close/shutdown while attempting to recover
persister.LOG.

Is there a step I am missing?

Thank You.
Chris.

Search Discussions

  • Matthew Sackman at Aug 23, 2011 at 11:16 am
    Hi Chris,
    On Sun, Aug 21, 2011 at 05:08:39PM -0700, ChrisStoyles wrote:
    I need some advice on moving /var/lib/rabbitmq to a new server with
    more RAM. What has happened is that on our 8GB RAM server we've had a
    surge in activity and one of our queues has grown much faster than it
    can be consumed by our "workers".

    This has resulted in RabbitMQ crashing, with an erl_crash.dump file
    appearing.

    At present when I attempt to start the server up again, it tries to
    roll the persister.LOG file and recover what was in the queue at the
    time of the crash. However even with rabbitmq.config set up as below,
    the vm_memory_high_watermark alarm is triggered and erl_crash.dump is
    created all over again.
    Erm, persister.LOG suggests a RabbitMQ pre 2.0.0. Is that true? We would
    very strongly recommend you upgrade asap.
    [
    {rabbit, [{vm_memory_high_watermark, 0.9}]}
    ].
    Yeah, that's not going to make any difference to pre 2.0 Rabbits. Pre
    2.0, all msgs are always held in RAM.
    I've got another server with 24GB of RAM, so I set it's HOSTNAME to
    the same as the 8GB server and installed the same version of RabbitMQ.
    I then copied over the contents of /var/lib/rabbitmq and attempted to
    start RabbitMQ. I no longer hit the vm_memory_high_watermark however
    Erlang seems to close/shutdown while attempting to recover
    persister.LOG.
    That's odd - if the hostname and the nodename of Rabbit is the same, it
    should start up I think. Is there anything in the logs?

    Matthew
  • ChrisStoyles at Aug 23, 2011 at 9:51 pm
    Hi Matthew,

    Yes we were running 1.7.2 :) we are planning to upgrade just as soon
    as things settle down after this event.

    Thanks for posting, I actually didn't get an email alert from Google
    Groups that a new message had been posted hence the delay in getting
    back to you.

    It turns out our problem was to do with a bug in Erlang which was
    noted and patched here:
    http://erlang.2086793.n4.nabble.com/Patch-for-18-exabyte-memory-allocation-failure-td3342133.html

    We downloaded the latest stable release of Erlang and compiled from
    source. After this RabbitMQ came up without any problems whatsoever,
    turns out this was an Erlang bug not RabbitMQ (because Ubuntu 10.04LTS
    packages a very old version of both RabbitMQ and Erlang).

    Thanks for your help.

    Chris.
    On Aug 23, 7:16?pm, Matthew Sackman wrote:
    Hi Chris,
    On Sun, Aug 21, 2011 at 05:08:39PM -0700, ChrisStoyles wrote:
    I need some advice on moving /var/lib/rabbitmq to a new server with
    more RAM. What has happened is that on our 8GB RAM server we've had a
    surge in activity and one of our queues has grown much faster than it
    can be consumed by our "workers".
    This has resulted in RabbitMQ crashing, with an erl_crash.dump file
    appearing.
    At present when I attempt to start the server up again, it tries to
    roll the persister.LOG file and recover what was in the queue at the
    time of the crash. However even with rabbitmq.config set up as below,
    the vm_memory_high_watermark alarm is triggered and erl_crash.dump is
    created all over again.
    Erm, persister.LOG suggests a RabbitMQ pre 2.0.0. Is that true? We would
    very strongly recommend you upgrade asap.
    [
    ? {rabbit, [{vm_memory_high_watermark, 0.9}]}
    ].
    Yeah, that's not going to make any difference to pre 2.0 Rabbits. Pre
    2.0, all msgs are always held in RAM.
    I've got another server with 24GB of RAM, so I set it's HOSTNAME to
    the same as the 8GB server and installed the same version of RabbitMQ.
    I then copied over the contents of /var/lib/rabbitmq and attempted to
    start RabbitMQ. I no longer hit the vm_memory_high_watermark however
    Erlang seems to close/shutdown while attempting to recover
    persister.LOG.
    That's odd - if the hostname and the nodename of Rabbit is the same, it
    should start up I think. Is there anything in the logs?

    Matthew
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 22, '11 at 12:08a
activeAug 23, '11 at 9:51p
posts3
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

ChrisStoyles: 2 posts Matthew Sackman: 1 post

People

Translate

site design / logo © 2022 Grokbase