Hi all

First I need to send kudos to all of you, I've been using rabbit for a year
now and has given me a lot of satisfactions. It reminded me the day I use a
Database instead of a bunch of files.. the same but with data flows! Like a
marriage, satisfactions and problems come in the same pack! Just some
background info:

* Im running rabbitmq 1.7.0-3 on a debian testing distribution with
python-amqplib 0.6.1-1
* It's a VPS with 2GB of RAM, *BUT* when the used memory it's aboout 80%,
"tasks may be killed off pre-maturely, we do this to prevent a total OOM
scenario where the VPS goes dead" (literal words from the provider)
* I was using ~15 queues with 1kb messages and a rate of ~20
messages/minute

Like a week ago I've made the following changes:
* I'm using 50 queues with a rate of ~60 messages/minute
* One of them (I call it "the fat queue" ) has 50kb messages
* Processing at the rear end is rate fixed ... so I need rabbit to act as
a memory/disk buffer

A: High speed producer : 50000 1K messages in a second or so.
B: Translator: reads from A_q and generates a 50kb message from a single 1kb
message and publishes it on the fat_queue.
C: (Slow) Fixed speed consumer: grabs the 50kb message and does it's magic.

A process -> A_q queue -> B process -> fat_queue -> many C processes (even
that I have many Cs, A is much faster)


Here is the problem when everythig is almost empty:

rabbitmqctl list_queues memory | awk '{T += $1;}END{print T;}'
5703040

this means that the total memory used for queues is arround 5Mb but RES
memory for beam.smp process is 358Mb !!!

And when I start using it, things get worse. beam.smp just jumps from 400m
to1.6G (in jumps of 100-200m per second) and well... my VPS provider just
decides to kill almost everything on that machine. Right now I'm killing the
B process after a bunch of messages, but actually i don't know how to
estimate how much memory rabbit will be using per message.


I've heard the "new persister" (what a name, eh!) is "much better" with
"memory issues":
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-May/007251.html

taken from:
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-February/006397.html
=INFO REPORT==== 16-Feb-2010::04:42:05 ===
Memory limit set to 3196MB.
I'd love that functionality! when was implemented? or what parameters I need
to pass to rabbit?

Greetings!

Nico C?sar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100603/7cbbbfee/attachment.htm>

Search Discussions

  • Marek Majkowski at Jul 14, 2010 at 1:02 pm
    Nicol?s

    First, sorry for timing, I must have missed your message somehow.

    2010/6/3 Nicol?s C?sar <nico at nicocesar.com>:
    A process -> A_q queue -> B process -> fat_queue -> many C processes (even
    that I have many Cs,?? A is much faster)

    Here is the problem when everythig is almost empty:

    rabbitmqctl list_queues memory | awk '{T += $1;}END{print T;}'
    5703040

    this means that the total memory used for queues is arround 5Mb but RES
    memory for beam.smp process is 358Mb !!!
    You're right. rabbitmqctl memory report isn't anything useful.

    Actually, it's impossible to give any decent memory statistics. Consider
    a fanout exchange, where a single message gets distributed into multiple queues.
    RabbitMQ actually doesn't copy the message, it shares it between
    queues - what should memory usage return in that case?

    And when I start using it, things get worse. beam.smp just jumps from 400m
    to1.6G (in jumps of 100-200m per second) and well... my VPS provider just
    decides to kill almost everything on that machine. Right now I'm killing the
    B process after a bunch of messages, but actually i don't know how to
    estimate how much memory rabbit will be using per message.
    RabbitMQ stores messages body as erlang binaries. They are handled
    by a bit different garbage collection algorithms than normal erlang runtime
    (that's another reason why we don't really know how much memory
    messages are using).

    Generally, garbage collector once in a while decides that it needs to
    defragment the memory. To do so it allocates a big memory block
    and copies used data there. After that's done, it frees the old memory block.

    But for a while erlang will use _twice_ as much memory as it really needs.

    That can be one of the reasons why you see the huge memory consumption.
    Another reason may be badly tune garbage collector. In one
    of my installations I noticed that setting some magic erlang gc option made
    rabbit use way less memory. But tuning gc internals is not a thing we want
    to recommend.
    I've heard the "new persister" (what a name, eh!) is "much better" with
    "memory issues":
    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-May/007251.html
    It definitely is. If you can afford compiling rabbit from source, please give
    it a try. Just do "hg up bug21673; make run". More info here:
    http://www.rabbitmq.com/build-server.html#obtaining

    Using erlang R13B03 or later may also help (it introduces better
    garbage collection for binaries)
    taken from:
    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-February/006397.html
    =INFO REPORT==== 16-Feb-2010::04:42:05 ===
    Memory limit set to 3196MB.
    I'd love that functionality! when was implemented? or what parameters I need
    to pass to rabbit?
    The idea behind it, is to tell the producers that they should stop
    sending messages
    when rabbit is using too much memory. To make it useful your client
    must support
    channel.flow amqp command, which tells the client to stop sending.
    I'm afraid python-amqplib doesn't support that.

    Some server side documentation:
    http://www.rabbitmq.com/extensions.html#memsup

    This functionality works a bit differently in the new persister
    branch. It gives a hint
    to rabbit about how much memory rabbit may use. If it's using more - data gets
    saved to the disk. So theoretically you should never see channel.flow
    (if you have fast enough disk).

    Cheers,
    Marek Majkowski
  • Marek Majkowski at Jul 14, 2010 at 1:33 pm
    (resent, looks like previously it haven't gone to the mailing list)

    Nicol?s

    First, sorry for timing, I must have missed your message somehow.

    2010/6/3 Nicol?s C?sar <nico at nicocesar.com>:
    A process -> A_q queue -> B process -> fat_queue -> many C processes (even
    that I have many Cs,?? A is much faster)

    Here is the problem when everythig is almost empty:

    rabbitmqctl list_queues memory | awk '{T += $1;}END{print T;}'
    5703040

    this means that the total memory used for queues is arround 5Mb but RES
    memory for beam.smp process is 358Mb !!!
    You're right. rabbitmqctl memory report isn't anything useful.

    Actually, it's impossible to give any decent memory statistics. Consider
    a fanout exchange, where a single message gets distributed into multiple queues.
    RabbitMQ actually doesn't copy the message, it shares it between
    queues - what should memory usage return in that case?

    And when I start using it, things get worse. beam.smp just jumps from 400m
    to1.6G (in jumps of 100-200m per second) and well... my VPS provider just
    decides to kill almost everything on that machine. Right now I'm killing the
    B process after a bunch of messages, but actually i don't know how to
    estimate how much memory rabbit will be using per message.
    RabbitMQ stores messages body as erlang binaries. They are handled
    by a bit different garbage collection algorithms than normal erlang runtime
    (that's another reason why we don't really know how much memory
    messages are using).

    Generally, garbage collector once in a while decides that it needs to
    defragment the memory. To do so it allocates a big memory block
    and copies used data there. After that's done, it frees the old memory block.

    But for a while erlang will use _twice_ as much memory as it really needs.

    That can be one of the reasons why you see the huge memory consumption.
    Another reason may be badly tune garbage collector. In one
    of my installations I noticed that setting some magic erlang gc option made
    rabbit use way less memory. But tuning gc internals is not a thing we want
    to recommend.
    I've heard the "new persister" (what a name, eh!) is "much better" with
    "memory issues":
    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-May/007251.html
    It definitely is. If you can afford compiling rabbit from source, please give
    it a try. Just do "hg up bug21673; make run". More info here:
    ?http://www.rabbitmq.com/build-server.html#obtaining

    Using erlang R13B03 or later may also help (it introduces better
    garbage collection for binaries)
    taken from:
    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-February/006397.html
    =INFO REPORT==== 16-Feb-2010::04:42:05 ===
    Memory limit set to 3196MB.
    I'd love that functionality! when was implemented? or what parameters I need
    to pass to rabbit?
    The idea behind it, is to tell the producers that they should stop
    sending messages
    when rabbit is using too much memory. To make it useful your client
    must support
    channel.flow amqp command, which tells the client to stop sending.
    I'm afraid python-amqplib doesn't support that.

    Some server side documentation:
    ?http://www.rabbitmq.com/extensions.html#memsup

    This functionality works a bit differently in the new persister
    branch. It gives a hint
    to rabbit about how much memory rabbit may use. If it's using more - data gets
    saved to the disk. So theoretically you should never see channel.flow
    (if you have fast enough disk).

    Cheers,
    ?Marek Majkowski

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 3, '10 at 12:38p
activeJul 14, '10 at 1:33p
posts3
users2
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase