Hi.

I'm a having issues with a huge memory consumption with rabbit.

I'm uting rabbit 1.7.2 with Debian Linux


----- /etc/rabbitmq/rabbitmq.config ----
[{rabbit, [{vm_memory_high_watermark, 0.2}]}].
------------------------------------------------------

I hit the maximum:
------ tail -f /var/log/rabbitmq/rabbit.log | grep -i memory ------
vm_memory_high_watermark set. Memory used:434801384 allowed:429496729
alarm_handler: {set,{vm_memory_high_watermark,[]}}
------------------------------------------------------


I'm sopposed to use channel.flow to stop the producer. My python code is:

-------------------------------------------
while not self.chan.flow(True):
sys.stdout.write("W")
sys.stdout.flush()
time.sleep(1)

self.chan.basic_publish(msg,
exchange=self.config_amqp["exchange"],
routing_key=self.config_amqp['routing_key'])
-------------------------------------------

What i'm doing wrong?

BTW, Im using delivery_mode=2 and a durable exchange and queue

PS: thanks tonyg for your irclog:
http://dev.rabbitmq.com/irclog/index.php?date 10-05-13

Nico C?sar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100608/39ea8726/attachment.htm>

Search Discussions

  • Matthew Sackman at Jun 8, 2010 at 2:05 pm
    Hi Nicol?s,
    On Tue, Jun 08, 2010 at 08:33:24AM -0300, Nicol?s C?sar wrote:
    ----- /etc/rabbitmq/rabbitmq.config ----
    [{rabbit, [{vm_memory_high_watermark, 0.2}]}].
    ------------------------------------------------------

    I hit the maximum:
    ------ tail -f /var/log/rabbitmq/rabbit.log | grep -i memory ------
    vm_memory_high_watermark set. Memory used:434801384 allowed:429496729
    alarm_handler: {set,{vm_memory_high_watermark,[]}}
    ------------------------------------------------------


    I'm sopposed to use channel.flow to stop the producer. My python code is:

    -------------------------------------------
    while not self.chan.flow(True):
    sys.stdout.write("W")
    sys.stdout.flush()
    time.sleep(1)

    self.chan.basic_publish(msg,
    exchange=self.config_amqp["exchange"],
    routing_key=self.config_amqp['routing_key'])
    -------------------------------------------

    What i'm doing wrong?
    When you hit the memory watermark, the *server* sends to *you* the
    channel.flow{active=false} which you're meant to acknowledge by sending
    back a channel.flow_ok{active=false}. However, depending on which python
    client you're using, this may not be implemented - I'm aware Tony's
    produced an experimental patch for pyamqplib, and I'm not sure about the
    status of flow control in Pika.

    I think what you're doing is sending a channel.flow to the server which
    is telling the server whether or not to send message to you, not the
    other way around.

    Matthew
  • Nicolás César at Jun 8, 2010 at 8:04 pm
    2010/6/8 Matthew Sackman <matthew at rabbitmq.com>
    Hi Nicol?s,

    On Tue, Jun 08, 2010 at 08:33:24AM -0300, Nicol?s C?sar wrote:
    (..)
    What i'm doing wrong?
    When you hit the memory watermark, the *server* sends to *you* the
    channel.flow{activeúlse} which you're meant to acknowledge by sending
    back a channel.flow_ok{activeúlse}. However, depending on which python
    client you're using, this may not be implemented - I'm aware Tony's
    produced an experimental patch for pyamqplib, and I'm not sure about the
    status of flow control in Pika.

    Thanks Matthew for the excelent response. You probably are talking about
    this:
    http://code.google.com/p/py-amqplib/issues/detail?id


    I think what you're doing is sending a channel.flow to the server which
    is telling the server whether or not to send message to you, not the
    other way around.
    Thanks for the explanation. I'll see that patch and get my things going
    smoothly
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100608/813cad2b/attachment.htm>
  • Nicolás César at Jun 9, 2010 at 2:50 pm

    El 8 de junio de 2010 17:04, Nicol?s C?sar <nico at nicocesar.com> escribi?:

    2010/6/8 Matthew Sackman <matthew at rabbitmq.com>
    Hi Nicol?s,

    On Tue, Jun 08, 2010 at 08:33:24AM -0300, Nicol?s C?sar wrote:
    (..)
    What i'm doing wrong?
    When you hit the memory watermark, the *server* sends to *you* the
    channel.flow{activeúlse} which you're meant to acknowledge by sending
    back a channel.flow_ok{activeúlse}. However, depending on which python
    client you're using, this may not be implemented - I'm aware Tony's
    produced an experimental patch for pyamqplib, and I'm not sure about the
    status of flow control in Pika.

    Thanks Matthew for the excelent response. You probably are talking about
    this:
    http://code.google.com/p/py-amqplib/issues/detail?id


    I think what you're doing is sending a channel.flow to the server which
    is telling the server whether or not to send message to you, not the
    other way around.
    Thanks for the explanation. I'll see that patch and get my things going
    smoothly

    I've applied the patch... works just fine. If I use chan.publish( ...,
    block_on_flow_control=True ) I get the expected result and (if I set it to
    false I get a convienent exception)

    But now I've hitted the "vm_memory_high_watermark set." ; every producer is
    blocked but the memory stills allocated. (using >400M for 50 empty queues)
    for more than an hour.

    When vm_memory_high_watermark will be cleared? Can I force something to get
    that memory back to normal. (I refuse to shut down rabbit, since I'm
    debugging to get this into production)

    Greetings


    Nico C?sar
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100609/6192a0ca/attachment.htm>
  • Simon MacMullen at Jun 9, 2010 at 3:03 pm

    On 09/06/10 15:50, Nicol?s C?sar wrote:
    I've applied the patch... works just fine. If I use chan.publish( ...,
    block_on_flow_control=True ) I get the expected result and (if I set it
    to false I get a convienent exception)

    But now I've hitted the "vm_memory_high_watermark set." ; every producer
    is blocked but the memory stills allocated. (using >400M for 50 empty
    queues) for more than an hour.

    When vm_memory_high_watermark will be cleared? Can I force something to
    get that memory back to normal. (I refuse to shut down rabbit, since I'm
    debugging to get this into production)
    Are you acking the messages after consuming them?

    Cheers, Simon
  • Nicolás César at Jun 9, 2010 at 3:12 pm
    2010/6/9 Simon MacMullen <simon at rabbitmq.com>
    On 09/06/10 15:50, Nicol?s C?sar wrote:

    When vm_memory_high_watermark will be cleared? Can I force something to
    get that memory back to normal. (I refuse to shut down rabbit, since I'm
    debugging to get this into production)
    Are you acking the messages after consuming them?
    yes. using:

    self.chan_in.basic_ack(msg.delivery_info['delivery_tag'])

    I've checked using
    /usr/sbin/rabbitmqctl list_queues name messages_unacknowledged
    /usr/sbin/rabbitmqctl list_channels pid messages_unacknowledged

    and yep... everything is in 0...

    Is there any other way to checkit? Is there a way to get a memory dump from
    the vm to see whats going on?

    Nico C?sar
    http://blog.nicocesar.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100609/5b73309b/attachment.htm>
  • Simon MacMullen at Jun 9, 2010 at 4:15 pm

    On 09/06/10 16:12, Nicol?s C?sar wrote:
    Is there any other way to checkit? Is there a way to get a memory dump
    from the vm to see whats going on?
    Well, there's erlang:memory/0.

    If you're using the Debian packages, type:

    $ sudo -u rabbitmq -H erl -remsh rabbit@`hostname` -sname foo

    to establish a connection to the RabbitMQ Erlang process then at the
    Erlang prompt:

    memory().

    to give some high level statistics about what's using memory.

    Other things to think about:

    * Are you using the most recent Erlang? (R13B04) GC has improved recently.

    * Are you stopping the queues from hibernating? If you've written
    something like a shell script to repeatedly invoke rabbitmqctl
    list_queues this will keep the queue processes from hibernating which in
    turn stops them from GCing on hibernate. If queues are busy this is not
    a problem as they also GC every (n) reductions, but if they're
    almost-but-not-quite idle it can be a problem. It doesn't have to be a
    shell script, anything that means the queues never get 10 seconds or so
    idle.

    Cheers, Simon
  • Nicolás César at Jun 9, 2010 at 9:27 pm
    2010/6/9 Simon MacMullen <simon at rabbitmq.com>
    On 09/06/10 16:12, Nicol?s C?sar wrote:

    Is there any other way to checkit? Is there a way to get a memory dump
    from the vm to see whats going on?
    Well, there's erlang:memory/0.

    If you're using the Debian packages, type:

    $ sudo -u rabbitmq -H erl -remsh rabbit@`hostname` -sname foo

    to establish a connection to the RabbitMQ Erlang process then at the Erlang
    prompt:

    memory().

    to give some high level statistics about what's using memory.
    Cool!!

    (rabbit at sendsorium)1> memory().
    [{total,27420688},
    {processes,8840004},
    {processes_used,8793876},
    {system,18580684},
    {atom,683329},
    {atom_used,650462},
    {binary,11081456},
    {code,5263067},
    {ets,452988}]

    but in top I see beam.smp using:
    VIRT1m RES�m

    none of those numbers get close....

    How can I use this information to get better debugging?


    Other things to think about:

    * Are you using the most recent Erlang? (R13B04) GC has improved recently.
    I've been using:
    Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0]
    [kernel-poll:false]

    Is that old? I'm trying other resources before changing erlang version (and
    taking my Debian to 'testing' instead of 'stable') . Thanks for the advice.
    Now I'm aware of that... if everything fails, I'll do it!

    * Are you stopping the queues from hibernating? If you've written something
    like a shell script to repeatedly invoke rabbitmqctl list_queues this will
    keep the queue processes from hibernating which in turn stops them from
    GCing on hibernate. If queues are busy this is not a problem as they also GC
    every (n) reductions, but if they're almost-but-not-quite idle it can be a
    problem. It doesn't have to be a shell script, anything that means the
    queues never get 10 seconds or so idle.


    This is really nice data!! Yes. I have scripts but they're cron'ed every
    minute.... I'm changing that and testing again.

    One small detail. I'm using:

    -----------------------------------------------------
    while True:
    msg = self.chan.basic_get(self.config['queue'])
    if msg:
    ....
    time.sleep(2.0)
    ----------------------------------------------------

    because I need a fixed-rate consumer . ... could this be a similar case of
    the "almost-but-not-quite idle queue"?
    is there any way to tell the broker to consume at a fixed rate with
    chan.basic_consume(...callback = f)?

    Greetings

    Nico C?sar
    http://blog.nicocesar.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100609/c250c728/attachment.htm>
  • Simon MacMullen at Jun 10, 2010 at 11:26 am

    On 09/06/10 22:27, Nicol?s C?sar wrote:
    (rabbit at sendsorium)1> memory().
    [{total,27420688}, <snip>
    but in top I see beam.smp using:
    VIRT1m RES�m
    So there is some disparity here but not as much as when you were seeing
    400M used. I take it this was a different run?
    I've been using:
    Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0]
    [kernel-poll:false]

    Is that old? I'm trying other resources before changing erlang version
    (and taking my Debian to 'testing' instead of 'stable') . Thanks for the
    advice. Now I'm aware of that... if everything fails, I'll do it!
    It's old enough to make a difference. If you're in general concerned
    about memory use then R13B04 should improve things. But see below for
    something more important...
    * Are you stopping the queues from hibernating? If you've written
    something like a shell script to repeatedly invoke rabbitmqctl
    list_queues this will keep the queue processes from hibernating
    which in turn stops them from GCing on hibernate. If queues are busy
    this is not a problem as they also GC every (n) reductions, but if
    they're almost-but-not-quite idle it can be a problem. It doesn't
    have to be a shell script, anything that means the queues never get
    10 seconds or so idle.


    This is really nice data!! Yes. I have scripts but they're cron'ed every
    minute.... I'm changing that and testing again.
    Every minute *should* be OK, it's more like every couple of seconds
    which can prevent the hibernation.
    One small detail. I'm using:

    -----------------------------------------------------
    while True:
    msg = self.chan.basic_get(self.config['queue'])
    if msg:
    ....
    time.sleep(2.0)
    ----------------------------------------------------

    because I need a fixed-rate consumer . ... could this be a similar case
    of the "almost-but-not-quite idle queue"?
    Yes, as you'll still be waking the queue up every 2 seconds even if it's
    empty. I suspect this is your real problem.
    is there any way to tell the broker to consume at a fixed rate with
    chan.basic_consume(...callback = f)?
    Yes, basic.qos. This not-very-well-named method lets you specify how
    many unacked messages the server will send to you at any time. So set
    qos to a low value, then use basic_consume with a sleep before acking.

    This is an interesting case though, I'll file a bug on this behaviour.

    Cheers, Simon
  • Nicolás César at Jun 10, 2010 at 1:02 pm

    El 10 de junio de 2010 08:26, Simon MacMullen <simon at rabbitmq.com> escribi?:
    On 09/06/10 22:27, Nicol?s C?sar wrote:

    (rabbit at sendsorium)1> memory().
    [{total,27420688},
    <snip>

    but in top I see beam.smp using:
    VIRT1m RES�m
    So there is some disparity here but not as much as when you were seeing
    400M used. I take it this was a different run?


    Yess... the OOM assassin was with his gun killing my processes! Rabbit went
    down fist! Thanks to you I have the memory(). trick for the next time.

    * Are you stopping the queues from hibernating? If you've written
    something like a shell script to repeatedly invoke rabbitmqctl
    list_queues this will keep the queue processes from hibernating
    which in turn stops them from GCing on hibernate. If queues are busy
    this is not a problem as they also GC every (n) reductions, but if
    they're almost-but-not-quite idle it can be a problem. It doesn't
    have to be a shell script, anything that means the queues never get
    10 seconds or so idle.


    This is really nice data!! Yes. I have scripts but they're cron'ed every
    minute.... I'm changing that and testing again.
    Every minute *should* be OK, it's more like every couple of seconds which
    can prevent the hibernation.

    I've changed it to 10 minutes and there were no significant behavior
    changes.

    One small detail. I'm using:

    -----------------------------------------------------
    while True:
    msg = self.chan.basic_get(self.config['queue'])
    if msg:
    ....
    time.sleep(2.0)
    ----------------------------------------------------

    because I need a fixed-rate consumer . ... could this be a similar case
    of the "almost-but-not-quite idle queue"?
    Yes, as you'll still be waking the queue up every 2 seconds even if it's
    empty. I suspect this is your real problem.

    Bingo! thanks Simon for your help in this issue!! now it's on the mail
    archive, I hope this mail helps next googling rabbiter. :)

    is there any way to tell the broker to consume at a fixed rate with
    chan.basic_consume(...callback = f)?
    Yes, basic.qos. This not-very-well-named method lets you specify how many
    unacked messages the server will send to you at any time. So set qos to a
    low value, then use basic_consume with a sleep before acking.
    Nice to hear a yes here!! I'll be changing my code to get this working!
    Thanks!

    Nico Cesar
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100610/8178669e/attachment.htm>
  • Nicolás César at Jun 10, 2010 at 3:00 pm
    Ops! this mail went to Simon only, I'm posting here if any has an extra tip
    for me. Thanks

    Nico C?sar


    ---------- Mensaje reenviado ----------
    De: Nicol?s C?sar <nico at nicocesar.com>
    Fecha: 10 de junio de 2010 11:18
    Asunto: Re: [rabbitmq-discuss] chan.flow and vm_memory_high_watermark
    Para: Simon MacMullen <simon at rabbitmq.com>


    El 10 de junio de 2010 10:02, Nicol?s C?sar <nico at nicocesar.com> escribi?:
    (..)

    is there any way to tell the broker to consume at a fixed rate with
    chan.basic_consume(...callback = f)?
    Yes, basic.qos. This not-very-well-named method lets you specify how many
    unacked messages the server will send to you at any time. So set qos to a
    low value, then use basic_consume with a sleep before acking.
    Nice to hear a yes here!! I'll be changing my code to get this working!
    Thanks!
    Simon,

    I'm sorry but I just don't get the use of basic_qos and fixed rate consuming
    toghether!. My investigation so far leads that the parameters I have for a
    QoS are:

    basic_qos(prefetch_size, prefetch_count, a_global)

    soppose that I use

    basic_qos(prefetch_size=0, prefetch_count=1, a_global=False)


    How can I limit "consume 1 message every 2 seconds" (that's my fixed rete)
    using chan.basic_consume(...callback = f)?

    If you could give me any help here would be very appreciated.

    Greetings,

    Nico Cesar
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100610/bde1d719/attachment.htm>
  • Scott Brooks at Jun 10, 2010 at 3:30 pm
    You would consume with acks turned on.

    Then once you ack a message, rabbitmq will send you another.

    You can use all the extra parameters to rabbitmqctl list_queues to see
    how many messages are sent but not acknowledged, etc.

    Scott

    2010/6/10 Nicol?s C?sar <nico at nicocesar.com>:
    Ops! this mail went to Simon only, I'm posting here if any has an extra tip
    for me. Thanks

    Nico C?sar


    ---------- Mensaje reenviado ----------
    De: Nicol?s C?sar <nico at nicocesar.com>
    Fecha: 10 de junio de 2010 11:18
    Asunto: Re: [rabbitmq-discuss] chan.flow and vm_memory_high_watermark
    Para: Simon MacMullen <simon at rabbitmq.com>


    El 10 de junio de 2010 10:02, Nicol?s C?sar <nico at nicocesar.com> escribi?:
    (..)
    is there any way to tell the broker to consume at a fixed rate with
    chan.basic_consume(...callback = f)?
    Yes, basic.qos. This not-very-well-named method lets you specify how many
    unacked messages the server will send to you at any time. So set qos to a
    low value, then use basic_consume with a sleep before acking.
    Nice to hear a yes here!! I'll be changing my code to get this working!
    Thanks!
    Simon,

    I'm sorry but I just don't get the use of basic_qos and fixed rate consuming
    toghether!. My investigation so far leads that the parameters I have for a
    QoS are:

    basic_qos(prefetch_size, prefetch_count, a_global)

    soppose that I use

    basic_qos(prefetch_size=0, prefetch_count=1, a_global=False)

    How can I limit "consume 1 message every 2 seconds" (that's my fixed rete)
    using chan.basic_consume(...callback = f)?

    If you could give me any help here would be very appreciated.

    Greetings,

    Nico Cesar


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Jun 10, 2010 at 3:53 pm

    On 10/06/10 16:00, Nicol?s C?sar wrote:
    I'm sorry but I just don't get the use of basic_qos and fixed rate
    consuming toghether!. My investigation so far leads that the parameters
    I have for a QoS are:

    basic_qos(prefetch_size, prefetch_count, a_global)

    soppose that I use

    basic_qos(prefetch_size=0, prefetch_count=1, a_global=False)


    How can I limit "consume 1 message every 2 seconds" (that's my fixed
    rete) using chan.basic_consume(...callback = f)?
    Those qos methods look reasonable. You would need to then delay the ack
    you send in the consume callback. Be careful not to just block there,
    that will block all the frames flowing down the connection on other
    channels. You need to use some sort of timer.

    Are you using Pika? Marek has an example of this sort of thing at:

    http://github.com/majek/pika/blob/master/examples/demo_channel_flow_asyncore.py

    Cheers, Simon

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 8, '10 at 11:33a
activeJun 10, '10 at 3:53p
posts13
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase