One of our systems overfed the rabbit, and it's now a bit angry. The
strange problem that we're having, though, is that it seems that the
commit after the thousandth ack'd message is triggering the persister
log rollover, which is then causing beam to run out of memory. The
only thing we're doing is basic_get'ing messages, acking them, and
then committing (the channel is transactional). We've also tried
committing every thousand messages; either way, the first commit after
the thousandth ack'd message is causing the persister log rollover,
followed immediately by beam crashing due to being out of RAM.

Does anybody know why rolling the persister log would cause the system
to run out of memory? It seems like a strange place to need to
allocate a lot of RAM, but I'm not at all familiar with how the
persister rollover works. I'm going to next try just doing basic_gets
without being in a transaction to see if that prevents the crash, but
I'd like a better understanding of how the log rollover works, and why
it seems to need a lot of memory to succeed.

Also, will the future (1.8?) persister still do the process described
in https://dev.rabbitmq.com/wiki/RabbitPersisterDesign ? Writing the
entire rabbit state to disk every so-often doesn't seem like it would
work terribly well when storing huge amounts of data, or am I
understanding how the persister log works?

Search Discussions

  • Matthias Radestock at Oct 19, 2009 at 12:31 pm
    Tsuraan,

    tsuraan wrote:
    One of our systems overfed the rabbit, and it's now a bit angry. The
    strange problem that we're having, though, is that it seems that the
    commit after the thousandth ack'd message is triggering the persister
    log rollover, which is then causing beam to run out of memory. The
    only thing we're doing is basic_get'ing messages, acking them, and
    then committing (the channel is transactional). We've also tried
    committing every thousand messages; either way, the first commit after
    the thousandth ack'd message is causing the persister log rollover,
    followed immediately by beam crashing due to being out of RAM.
    The logic for when the persister log is rolled over is quite complex,
    involving both time-based and message-count based heuristics.

    As an aside, why are you using transactions on the consuming side? I
    have yet to come across a use case where that is genuinely required.
    Does anybody know why rolling the persister log would cause the system
    to run out of memory? It seems like a strange place to need to
    allocate a lot of RAM
    When rolling the persister log, rabbit writes a snapshot of all
    currently persisted messages, and it needs to allocate memory for that
    entire snapshot.
    Also, will the future (1.8?) persister still do the process described
    in https://dev.rabbitmq.com/wiki/RabbitPersisterDesign ? No.
    Writing the entire rabbit state to disk every so-often doesn't seem
    like it would work terribly well when storing huge amounts of data
    Indeed.


    Regards,

    Matthias.
  • Tsuraan at Oct 19, 2009 at 4:10 pm

    As an aside, why are you using transactions on the consuming side? I
    have yet to come across a use case where that is genuinely required.
    Our consumers are almost always also producers, so we do the message
    ack and the creation of new messages in a transaction. I'm not sure
    that it's necessary; I'm used to databases, where I tend to do
    everything in a transaction to ensure that everything is consistent
    when I'm done, but our messaging stuff is designed to handle multiple
    copies of the same message, so it's probably a bit overkill to also
    use transactions.

    I seem to remember that I had some problem when I was first looking at
    rabbit that I could publish a stream of messages in non-transactional
    mode, and not all the messages would be delivered (or enqueued). I
    think I may have been doing something stupid like exiting my program
    without closing my channel, but moving to transactional mode fixed the
    problem, and I think I've been a bit religious about transactions
    since then.
    When rolling the persister log, rabbit writes a snapshot of all
    currently persisted messages, and it needs to allocate memory for that
    entire snapshot.
    Ok, that explains it.
    Also, will the future (1.8?) persister still do the process described
    in https://dev.rabbitmq.com/wiki/RabbitPersisterDesign ?
    No.
    Glad to hear it :)

    Thanks!
  • Matthias Radestock at Oct 19, 2009 at 4:54 pm
    Tsuraan,

    tsuraan wrote:
    As an aside, why are you using transactions on the consuming side? I
    have yet to come across a use case where that is genuinely required.
    Our consumers are almost always also producers, so we do the message
    ack and the creation of new messages in a transaction. I'm not sure
    that it's necessary; I'm used to databases, where I tend to do
    everything in a transaction to ensure that everything is consistent
    when I'm done, but our messaging stuff is designed to handle multiple
    copies of the same message, so it's probably a bit overkill to also
    use transactions.

    I seem to remember that I had some problem when I was first looking at
    rabbit that I could publish a stream of messages in non-transactional
    mode, and not all the messages would be delivered (or enqueued). I
    think I may have been doing something stupid like exiting my program
    without closing my channel, but moving to transactional mode fixed the
    problem, and I think I've been a bit religious about transactions
    since then.
    Transactions are currently the only way for a client to get an
    acknowledgment from the broker that sent messages have not only made it
    to the broker (a normal connection/channel.close, or indeed any other
    synchronous AMQP operation, is sufficient for that) but also will
    survive a broker restart (if they were marked as persistent).

    So use of tx mode on the sending side is quite common.

    Use of tx mode on the receiving side, for acks, is much less common,
    since the worst thing that can happen if an ack gets lost is that the
    message will get delivered again. That can happen in other situations
    anyway, e.g. client or connection failure, so apps need to be able to
    cope with it already.

    If your producers are also senders, and you want the aforementioned
    guarantees on sending, I would recommend using two channels - one in tx
    mode for sending and one on non-tx mode for receiving/ack'ing.


    Regard,

    Matthias.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedOct 15, '09 at 7:23p
activeOct 19, '09 at 4:54p
posts4
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Matthias Radestock: 2 posts Tsuraan: 2 posts

People

Translate

site design / logo © 2022 Grokbase