Hi,

We have been running a single RabbitMQ instance for 6 months now (RabbitMQ
2.2).
Off late we observed RabbitMQ was consuming 11G of Memory when the entire
queue-size was about 7MB.
We decided to do a restart of RabbitMQ (thinking this was a memory leak).
On doing a restart of the APP we kept getting restart TIME-OUT without any
messages in the startup_{err,log).

We kept on retrying the restart and after 10minutes RabbitMQ restarted on
its own, without any message or log of what went wrong.

What was an expected down time of <1min ended up being a down time of 10
min.
Also it forced our team to frantically set up RabbittMQ on a new machine
with all the queues and Exchanges (the original Rabbit came back up in the
meanwhile).

We are trying to analyze why this happened to us. Specifically answer the
following questions

- Why was rabbitmq consuming high amounts of memory? Was it a memory
leak? Is the memory leak fixed in the later version.
- Why did rabbitMQ not restart immediately? Why was no message present in
the log.
- In situations like these how do we start a new rabbitmq instance
automatically with all the queues and preferably data.
- How do we upgrade to latest Rabbit without any data loss?

Any opinions and thoughts on this are welcome.

-Abhishek Kona
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110802/05857590/attachment.htm>

Search Discussions

  • Allan Kamau at Aug 3, 2011 at 7:09 am

    On Tue, Aug 2, 2011 at 5:00 PM, Abhishek K wrote:
    Hi,
    We have been running a single RabbitMQ instance for 6 months now (RabbitMQ
    2.2).
    Off late we observed RabbitMQ was consuming 11G of Memory when the entire
    queue-size was about 7MB.
    We decided to do a restart of RabbitMQ (thinking this was a memory leak).
    On doing a restart of the APP we kept getting restart TIME-OUT without any
    messages in the startup_{err,log).
    We kept on retrying the restart and after 10minutes RabbitMQ restarted on
    its own, without any message or log of what went wrong.
    What was an expected down time of <1min ended up being a down time of 10
    min.
    Also it forced our team to frantically set up RabbittMQ on a new machine
    with all the queues and Exchanges (the original Rabbit came back up in the
    meanwhile).
    We are trying to analyze why this happened to us. Specifically answer the
    following questions

    Why was rabbitmq consuming high amounts of memory? Was it a memory leak? Is
    the memory leak fixed in the later version.
    Why did rabbitMQ not restart immediately? Why was no message present in the
    log.
    In situations like these how do we start a new rabbitmq instance
    automatically with all the queues and preferably data.
    How do we upgrade to latest Rabbit without any data loss?

    Any opinions and thoughts on this are welcome.
    -Abhishek Kona
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    You may want to include specify the OS and version you are running and
    perhaps detail changes you may have made recently that may shed some
    light to the cause of behaviour you are observing.

    Allan.
  • Abhishek K at Aug 3, 2011 at 10:15 am
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage rate
    increased by 20%.

    -Abhishek Kona
    On Wed, Aug 3, 2011 at 12:39 PM, Allan Kamau wrote:
    On Tue, Aug 2, 2011 at 5:00 PM, Abhishek K wrote:
    Hi,
    We have been running a single RabbitMQ instance for 6 months now (RabbitMQ
    2.2).
    Off late we observed RabbitMQ was consuming 11G of Memory when the entire
    queue-size was about 7MB.
    We decided to do a restart of RabbitMQ (thinking this was a memory leak).
    On doing a restart of the APP we kept getting restart TIME-OUT without any
    messages in the startup_{err,log).
    We kept on retrying the restart and after 10minutes RabbitMQ restarted on
    its own, without any message or log of what went wrong.
    What was an expected down time of <1min ended up being a down time of 10
    min.
    Also it forced our team to frantically set up RabbittMQ on a new machine
    with all the queues and Exchanges (the original Rabbit came back up in the
    meanwhile).
    We are trying to analyze why this happened to us. Specifically answer the
    following questions

    Why was rabbitmq consuming high amounts of memory? Was it a memory leak? Is
    the memory leak fixed in the later version.
    Why did rabbitMQ not restart immediately? Why was no message present in the
    log.
    In situations like these how do we start a new rabbitmq instance
    automatically with all the queues and preferably data.
    How do we upgrade to latest Rabbit without any data loss?

    Any opinions and thoughts on this are welcome.
    -Abhishek Kona
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    You may want to include specify the OS and version you are running and
    perhaps detail changes you may have made recently that may shed some
    light to the cause of behaviour you are observing.

    Allan.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110803/1154887e/attachment.htm>
  • Emile Joubert at Aug 3, 2011 at 11:51 am
    Hi Abhishek,
    On 03/08/11 11:15, Abhishek K wrote:
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage rate
    increased by 20%.
    You don't mention the version of Erlang you use. It is worth bearing in
    mind that later versions of Erlang do better garbage collection. Also,
    how much RAM is installed on the server? You should generally expect
    that Erlang / RabbitMQ will use as much RAM as it can.

    You said that the broker uses 11GB RAM and that the queue size was 7MB -
    how were these figures determined? If that 7MB is distributed across a
    large number of queues then that is a possible explanation for the high
    memory usage. If the startup after a clean shutdown takes an inordinate
    amount of time then you might have more than 7MB worth of messages on
    the broker.

    You should also inspect the results from running all the "rabbitmqctl
    list_*" commands to make sure your application isn't using up too many
    resources (e.g. queues or exchanges) on the broker.

    RabbitMQ version 2.5.1 adds more memory reporting features to the
    "rabbitmqctl status" command, with a breakdown of where memory is being
    used. This is not available on version 2.2.0, but it is another possible
    reason to upgrade.



    Regards

    Emile
  • Abhishek K at Aug 3, 2011 at 5:17 pm
    I am using Erlang R12B. The server had around 20GB of Ram.

    There were around 30 queues on the server with the total memory consumption
    calculated with
    $ rabbitmqctl list_queues name memory

    -Abhishek Kona

    Abhishek Kona
    Department of Computer Engineering
    National Institute of Technology
    Karnataka, India

    On Wed, Aug 3, 2011 at 5:21 PM, Emile Joubert wrote:

    Hi Abhishek,
    On 03/08/11 11:15, Abhishek K wrote:
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage rate
    increased by 20%.
    You don't mention the version of Erlang you use. It is worth bearing in
    mind that later versions of Erlang do better garbage collection. Also,
    how much RAM is installed on the server? You should generally expect
    that Erlang / RabbitMQ will use as much RAM as it can.

    You said that the broker uses 11GB RAM and that the queue size was 7MB -
    how were these figures determined? If that 7MB is distributed across a
    large number of queues then that is a possible explanation for the high
    memory usage. If the startup after a clean shutdown takes an inordinate
    amount of time then you might have more than 7MB worth of messages on
    the broker.

    You should also inspect the results from running all the "rabbitmqctl
    list_*" commands to make sure your application isn't using up too many
    resources (e.g. queues or exchanges) on the broker.

    RabbitMQ version 2.5.1 adds more memory reporting features to the
    "rabbitmqctl status" command, with a breakdown of where memory is being
    used. This is not available on version 2.2.0, but it is another possible
    reason to upgrade.



    Regards

    Emile
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110803/1611660b/attachment.htm>
  • Jason J. W. Williams at Aug 3, 2011 at 5:25 pm
    What was the configuration of the exchanges (type), queues (type and
    flags) and bindings? Also were the queues long-lived or are they being
    created and destroyed constantly?

    -J
    On Wed, Aug 3, 2011 at 11:17 AM, Abhishek K wrote:
    I am using Erlang R12B. The server had around 20GB of Ram.
    There were around 30 queues on the server with the total memory consumption
    calculated with
    $ rabbitmqctl list_queues name memory
    -Abhishek Kona

    Abhishek Kona
    Department of Computer Engineering
    National Institute of Technology
    Karnataka, India

    On Wed, Aug 3, 2011 at 5:21 PM, Emile Joubert wrote:

    Hi Abhishek,
    On 03/08/11 11:15, Abhishek K wrote:
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage rate
    increased by 20%.
    You don't mention the version of Erlang you use. It is worth bearing in
    mind that later versions of Erlang do better garbage collection. Also,
    how much RAM is installed on the server? You should generally expect
    that Erlang / RabbitMQ will use as much RAM as it can.

    You said that the broker uses 11GB RAM and that the queue size was 7MB -
    how were these figures determined? If that 7MB is distributed across a
    large number of queues then that is a possible explanation for the high
    memory usage. If the startup after a clean ?shutdown takes an inordinate
    amount of time then you might have more than 7MB worth of messages on
    the broker.

    You should also inspect the results from running all the "rabbitmqctl
    list_*" commands to make sure your application isn't using up too many
    resources (e.g. queues or exchanges) on the broker.

    RabbitMQ version 2.5.1 adds more memory reporting features to the
    "rabbitmqctl status" command, with a breakdown of where memory is being
    used. This is not available on version 2.2.0, but it is another possible
    reason to upgrade.



    Regards

    Emile

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Abhishek K at Aug 4, 2011 at 2:30 am
    Hi,

    All the queues were long lived.
    There was only a single exchange all queues bound to that exchange, all
    queues were durable.

    The routing key was an individual string for each queue.

    -Abhishek Kona
    On Wed, Aug 3, 2011 at 10:55 PM, Jason J. W. Williams wrote:

    What was the configuration of the exchanges (type), queues (type and
    flags) and bindings? Also were the queues long-lived or are they being
    created and destroyed constantly?

    -J
    On Wed, Aug 3, 2011 at 11:17 AM, Abhishek K wrote:
    I am using Erlang R12B. The server had around 20GB of Ram.
    There were around 30 queues on the server with the total memory
    consumption
    calculated with
    $ rabbitmqctl list_queues name memory
    -Abhishek Kona

    Abhishek Kona
    Department of Computer Engineering
    National Institute of Technology
    Karnataka, India

    On Wed, Aug 3, 2011 at 5:21 PM, Emile Joubert wrote:

    Hi Abhishek,
    On 03/08/11 11:15, Abhishek K wrote:
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage rate
    increased by 20%.
    You don't mention the version of Erlang you use. It is worth bearing in
    mind that later versions of Erlang do better garbage collection. Also,
    how much RAM is installed on the server? You should generally expect
    that Erlang / RabbitMQ will use as much RAM as it can.

    You said that the broker uses 11GB RAM and that the queue size was 7MB -
    how were these figures determined? If that 7MB is distributed across a
    large number of queues then that is a possible explanation for the high
    memory usage. If the startup after a clean shutdown takes an inordinate
    amount of time then you might have more than 7MB worth of messages on
    the broker.

    You should also inspect the results from running all the "rabbitmqctl
    list_*" commands to make sure your application isn't using up too many
    resources (e.g. queues or exchanges) on the broker.

    RabbitMQ version 2.5.1 adds more memory reporting features to the
    "rabbitmqctl status" command, with a breakdown of where memory is being
    used. This is not available on version 2.2.0, but it is another possible
    reason to upgrade.



    Regards

    Emile

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110804/d6f9e4ea/attachment.htm>
  • Alex Lovell-Troy at Aug 4, 2011 at 7:58 am
    This sounds like the behavior I've seen when you consume messages
    without acking them.

    Does that ring a bell for anyone else?

    -alex
    On Thu, Aug 4, 2011 at 3:30 AM, Abhishek K wrote:
    Hi,
    All the queues were long lived.
    There was only a single exchange all queues bound to that exchange, all
    queues were durable.
    The routing key was an individual string for each queue.
    -Abhishek Kona
    On Wed, Aug 3, 2011 at 10:55 PM, Jason J. W. Williams
    wrote:
    What was the configuration of the exchanges (type), queues (type and
    flags) and bindings? Also were the queues long-lived or are they being
    created and destroyed constantly?

    -J

    On Wed, Aug 3, 2011 at 11:17 AM, Abhishek K <abhishek.kona at gmail.com>
    wrote:
    I am using Erlang R12B. The server had around 20GB of Ram.
    There were around 30 queues on the server with the total memory
    consumption
    calculated with
    $ rabbitmqctl list_queues name memory
    -Abhishek Kona

    Abhishek Kona
    Department of Computer Engineering
    National Institute of Technology
    Karnataka, India


    On Wed, Aug 3, 2011 at 5:21 PM, Emile Joubert <emile at rabbitmq.com>
    wrote:
    Hi Abhishek,
    On 03/08/11 11:15, Abhishek K wrote:
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage rate
    increased by 20%.
    You don't mention the version of Erlang you use. It is worth bearing in
    mind that later versions of Erlang do better garbage collection. Also,
    how much RAM is installed on the server? You should generally expect
    that Erlang / RabbitMQ will use as much RAM as it can.

    You said that the broker uses 11GB RAM and that the queue size was 7MB
    -
    how were these figures determined? If that 7MB is distributed across a
    large number of queues then that is a possible explanation for the high
    memory usage. If the startup after a clean ?shutdown takes an
    inordinate
    amount of time then you might have more than 7MB worth of messages on
    the broker.

    You should also inspect the results from running all the "rabbitmqctl
    list_*" commands to make sure your application isn't using up too many
    resources (e.g. queues or exchanges) on the broker.

    RabbitMQ version 2.5.1 adds more memory reporting features to the
    "rabbitmqctl status" command, with a breakdown of where memory is being
    used. This is not available on version 2.2.0, but it is another
    possible
    reason to upgrade.



    Regards

    Emile

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Abhishek K at Aug 4, 2011 at 12:13 pm
    -Messages are regularly acked by the application using RabbitMQ.
    The application is Thrift Server and before returning the Thrift RPC call,
    the RabbitMQ ack is sent back.

    -Abhishek Kona


    On Thu, Aug 4, 2011 at 1:28 PM, Alex Lovell-Troy
    wrote:
    This sounds like the behavior I've seen when you consume messages
    without acking them.

    Does that ring a bell for anyone else?

    -alex
    On Thu, Aug 4, 2011 at 3:30 AM, Abhishek K wrote:
    Hi,
    All the queues were long lived.
    There was only a single exchange all queues bound to that exchange, all
    queues were durable.
    The routing key was an individual string for each queue.
    -Abhishek Kona
    On Wed, Aug 3, 2011 at 10:55 PM, Jason J. W. Williams
    wrote:
    What was the configuration of the exchanges (type), queues (type and
    flags) and bindings? Also were the queues long-lived or are they being
    created and destroyed constantly?

    -J

    On Wed, Aug 3, 2011 at 11:17 AM, Abhishek K <abhishek.kona at gmail.com>
    wrote:
    I am using Erlang R12B. The server had around 20GB of Ram.
    There were around 30 queues on the server with the total memory
    consumption
    calculated with
    $ rabbitmqctl list_queues name memory
    -Abhishek Kona

    Abhishek Kona
    Department of Computer Engineering
    National Institute of Technology
    Karnataka, India


    On Wed, Aug 3, 2011 at 5:21 PM, Emile Joubert <emile at rabbitmq.com>
    wrote:
    Hi Abhishek,
    On 03/08/11 11:15, Abhishek K wrote:
    I am running RabbitMQ 2.20 on Debian Lenny (2.6.26).
    There was no changes in the code using RabbitMQ, only the usage
    rate
    increased by 20%.
    You don't mention the version of Erlang you use. It is worth bearing
    in
    mind that later versions of Erlang do better garbage collection.
    Also,
    how much RAM is installed on the server? You should generally expect
    that Erlang / RabbitMQ will use as much RAM as it can.

    You said that the broker uses 11GB RAM and that the queue size was
    7MB
    -
    how were these figures determined? If that 7MB is distributed across
    a
    large number of queues then that is a possible explanation for the
    high
    memory usage. If the startup after a clean shutdown takes an
    inordinate
    amount of time then you might have more than 7MB worth of messages on
    the broker.

    You should also inspect the results from running all the "rabbitmqctl
    list_*" commands to make sure your application isn't using up too
    many
    resources (e.g. queues or exchanges) on the broker.

    RabbitMQ version 2.5.1 adds more memory reporting features to the
    "rabbitmqctl status" command, with a breakdown of where memory is
    being
    used. This is not available on version 2.2.0, but it is another
    possible
    reason to upgrade.



    Regards

    Emile

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110804/bb1e2c9f/attachment.htm>
  • Matthias Radestock at Aug 4, 2011 at 12:20 pm
    Abhishek,
    On 04/08/11 13:13, Abhishek K wrote:
    -Messages are regularly acked by the application using RabbitMQ.
    As Emile suggested, please upgrade Erlang and RabbitMQ. I reckon the
    problem is more likely in the application than in either of these, but
    upgrading will make diagnosis easier. If after the upgrade the problem
    arises again, please send us the output of 'rabbitmqctl report'.

    Regards,

    Matthias.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 2, '11 at 2:00p
activeAug 4, '11 at 12:20p
posts10
users6
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase