Hi all,

Following Rabbitmq discuss "unexplained broker shutdown", what are the risks to configure the vm_memory_high_watermark to a higher value in order to not start paging messages at 8% of installed RAM but at 30% or more?

Thanks in advance

Sylvain
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles ou ayant un caract?re priv?. S'ils ne vous sont pas destin?s, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque mani?re que ce soit le contenu. Si ce message vous a ?t? transmis par erreur, merci d'en informer l'exp?diteur et de supprimer imm?diatement de votre syst?me informatique ce courriel ainsi que tous les documents qui y sont attach?s."
******
" This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110811/d39ef45e/attachment.htm>

Search Discussions

  • Matthew Sackman at Aug 11, 2011 at 4:23 pm

    On Thu, Aug 11, 2011 at 06:15:45PM +0200, CORDIER-AKKA Sylvain (MORPHO) wrote:
    Following Rabbitmq discuss "unexplained broker shutdown", what are the risks to configure the vm_memory_high_watermark to a higher value in order to not start paging messages at 8% of installed RAM but at 30% or more?
    In extremis, it could crash due to the Erlang VM being unable to
    allocate sufficient RAM for things like garbage collection. Note this is
    really quite unlikely, but not impossible.

    Also, Rabbit can't actually stop Erlang from using more RAM. There are
    scenarios in which Rabbit will eat up vastly more than the 40% it's
    notionally meant to throttle publishers at, so a good amount of buffer
    space is well advised. Finally, a good amount of space for the OS to
    cache disk in is also helpful.

    More likely is that you run out of RAM, which means you then have no
    disk cache, so performance will suffer, and then Rabbit will start being
    swapped out to disk, which will utterly cripple performance. Once
    performance starts to drop, as it's likely Rabbit is overloaded anyway
    (otherwise it wouldn't have used up so much RAM), performance will fall
    off very rapidly, and this can go from bad to worse.

    Matthew
  • CORDIER-AKKA Sylvain (MORPHO) at Aug 16, 2011 at 8:27 am
    Thank you for this explanation, but I search again a solution about the memory :
    Our RabbitMQ host has got 64GB installed RAM and RabbitMQ server process uses in normal use about 5GB of RAM.
    The normal use point is just above the 8% of installed RAM (8% of 64GB is 5.12GB) and a small variation of the use point makes the RabbitMQ starts paging messages and causes a final system slowdown.

    If the vm_memory_high_watermark is not the good solution, what are the available solutions?
    Increase the amount of installed RAM to 96GB for the system starts paging at 7.68GB?
    Isn't there another way to delay RabbitMQ to start swapping?

    Best regards

    Sylvain

    -----Original Message-----
    From: rabbitmq-discuss-bounces at lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] On Behalf Of Matthew Sackman
    Sent: Thursday, August 11, 2011 6:23 PM
    To: rabbitmq-discuss at lists.rabbitmq.com
    Subject: Re: [rabbitmq-discuss] Configure VM memory high watermark
    On Thu, Aug 11, 2011 at 06:15:45PM +0200, CORDIER-AKKA Sylvain (MORPHO) wrote:
    Following Rabbitmq discuss "unexplained broker shutdown", what are the risks to configure the vm_memory_high_watermark to a higher value in order to not start paging messages at 8% of installed RAM but at 30% or more?
    In extremis, it could crash due to the Erlang VM being unable to
    allocate sufficient RAM for things like garbage collection. Note this is
    really quite unlikely, but not impossible.

    Also, Rabbit can't actually stop Erlang from using more RAM. There are
    scenarios in which Rabbit will eat up vastly more than the 40% it's
    notionally meant to throttle publishers at, so a good amount of buffer
    space is well advised. Finally, a good amount of space for the OS to
    cache disk in is also helpful.

    More likely is that you run out of RAM, which means you then have no
    disk cache, so performance will suffer, and then Rabbit will start being
    swapped out to disk, which will utterly cripple performance. Once
    performance starts to drop, as it's likely Rabbit is overloaded anyway
    (otherwise it wouldn't have used up so much RAM), performance will fall
    off very rapidly, and this can go from bad to worse.

    Matthew
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    #
    " Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles ou ayant un caract?re priv?. S'ils ne vous sont pas destin?s, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque mani?re que ce soit le contenu. Si ce message vous a ?t? transmis par erreur, merci d'en informer l'exp?diteur et de supprimer imm?diatement de votre syst?me informatique ce courriel ainsi que tous les documents qui y sont attach?s."
    ******
    " This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
    #
  • Matthew Sackman at Aug 19, 2011 at 2:08 pm
    Hi Sylvain,
    On Tue, Aug 16, 2011 at 10:27:20AM +0200, CORDIER-AKKA Sylvain (MORPHO) wrote:
    Our RabbitMQ host has got 64GB installed RAM and RabbitMQ server
    process uses in normal use about 5GB of RAM. The normal use point is
    just above the 8% of installed RAM (8% of 64GB is 5.12GB) and a small
    variation of the use point makes the RabbitMQ starts paging messages
    and causes a final system slowdown.

    If the vm_memory_high_watermark is not the good solution, what are the
    available solutions?
    Increase the amount of installed RAM to 96GB for the system starts
    paging at 7.68GB?
    Isn't there another way to delay RabbitMQ to start swapping?
    Well. The thing that Rabbit is trying to ensure is that should RAM
    continue to be eaten up, by the time it gets to around 26GB used,
    everything should be safely on disk and that the transition to putting
    things onto disk is a smooth one. Obviously, unless you have insanely
    fast disks, writing out 26GB of messages to disk takes some time, which
    is why Rabbit starts writing out quite early on.

    The use of 0.4 as the default highwater mark is based on the fear that
    GC by the Erlang VM could cause a tempory spike of near double the
    amount of RAM in the machine. As the amount of RAM you have installed
    gets greater and greater, I'd have thought the odds of there being such
    a spike reduces. Thus maybe in such machines, a much higher highwater
    mark would be reasonable.

    If you have the time, what I'd suggest is that you try with the
    watermark much higher (say, 0.8) and then stress test it and see if you
    can get Rabbit/Erlang to fail (most likely it'll be a a "cannot allocate
    X bytes" malloc-style failure). Reduce the watermark and repeat until
    you have confidence that for your use case, there will be no fatal
    spikes and you still have utilisation of RAM that you're happy with.

    Matthew

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 11, '11 at 4:15p
activeAug 19, '11 at 2:08p
posts4
users2
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase