Guys,

There has been a lot of discussion about overflowing messages to disk
when memory is scarce (as opposed to throttling producers):

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2008-November/002386.html
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2008-September/001858.html
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2008-October/002212.html

The long and the short of it is that it would realistically take one
of the Rabbit dev team 3 working weeks to implement this properly
(with QA).

The unfortunate reality is that at LShift we have to do paid work in
order to fill the fridge.

So the situation on guaranteed store and forward is either:

1. We'll get around to it some time, or:
2. If everybody who wants disk overflow chips in with a fiver, then
when we get the money together, we can dedicate solid uninterrupted
time to this task.

Ben

Search Discussions

  • Ben Hood at Nov 16, 2008 at 8:41 pm
    Ez,
    On Sun, Nov 16, 2008 at 7:36 PM, Ezra Zygmuntowicz wrote:
    I got 5 on it ;)
    Maybe we need to get one of those Paypal Donate buttons........ ;-)

    Ben
  • Edwin Fine at Nov 17, 2008 at 12:15 am
    Count me in too. Heck, (he said generously), make it $10!

    Seriously, though, I don't want to be an ass about this, but I would think
    that there are now enough people (namely, non-paying customers ;) with
    enough valid use cases making enough noise about this particular issue, to
    warrant bumping it to the top of the Rabbit architectural issues list. When
    someone cannot even restart a node to drain the persister log after the node
    crashes with OOM, and has to delete the persister log, you know that there's
    a significant design deficiency to be remedied post-haste.

    Please believe me when I tell you that I am not trying to beat you up about
    it or be nasty or unfair - I have great regard for the
    Cohesive/LShift/Rabbit team - but I think it's reasonable to say that it's
    time to spend the estimated 3 work-weeks (I assume 120 work-hours) to remedy
    this. The cost of this should not be extreme. Let me put my project-manager
    hat on here... unless I am badly out of touch, one of your team shouldn't
    cost much more than, say, $80/hour at contract rates, so you are talking
    $10K to fix this (if I am wrong, and the rates are higher than that, PLEASE
    can I work for you? ;). Is there a way you could beg, borrow or
    venture-capital this to fast-track it? Or, being more constructive, how
    about 50 Rabbit users contributing $200 each? 25 @ $400? I would gladly
    contribute US$200-$400 to get this done (really). I wish I had $10K to just
    pay for this but sadly I don't.

    What do you say, Rabbit list? Would 49 of you PayPal $200 to get this done?

    I agree there is *maybe* a workaround by bumping up the swap space to a big
    number, but I see that as an emergency measure. I tried an experiment last
    night where I started an Erlang node in smp -disable mode (so that I had
    some free CPUs to kill it if things got hairy), and ran a tight loop that
    grew a list very quickly. I have a 20GB swap partition on Ubuntu x86_64,
    4-core, 8GB RAM. The Erlang node to got 8GB resident, and just before (and
    after) that, Linux starting trimming the working sets of the other processes
    to feed the hungry node. If I read the "top" display right, it managed to
    trim the working sets of most other processes down to around 5 - 10MB each
    (amazing!). The swap space used went up to 10GB, the Erlang node up to
    14.5GB of virtual memory, 6.5GB of resident memory. At that point the system
    ran like an anaesthetized snail, even though there were 3 almost idle CPUs.
    Command-line and GUI response times went sky-high, probably because
    everything was swapped half to death. The Erlang shell of the memory hog
    didn't even respond to Ctrl-C any more. It almost seemed as the node hung,
    but I can't swear to that. The good news is that I was able to kill the
    gluttonous node, and the system didn't crash (although it likely would have
    if it ran out of swap space) and it recovered perfectly. Did I mention I
    love Linux?

    I'd like to repeat this test using Rabbit, by feeding it a metric grillion
    persistent messages without draining, stopping it before it crashes, and
    seeing if I can get it to recover once the swap file is almost full (by
    starting consumers). Unless someone has done this already?

    I suppose if one did run out of swap and got a panic crash, one could add a
    big fat terabyte disc (or logical volume) to the machine and put a humongous
    swap partition there, then restart and let Rabbit try to recover the
    persister log. Theoretically, it should be able to do so - eventually -
    because the swap space should now be large enough to take the entire
    persister log's memory-resident bits. One might need to set up the Linux
    kernel flags suitably to prevent the process killer from killing the Rabbit
    process before it has a chance to drain the swamp, I mean queue.

    Just to reiterate: Rabbit is a great product and you are a great team. This
    is not a "moan".

    Regards,
    Edwin

    On Sun, Nov 16, 2008 at 3:41 PM, Ben Hood wrote:

    Ez,
    On Sun, Nov 16, 2008 at 7:36 PM, Ezra Zygmuntowicz wrote:
    I got 5 on it ;)
    Maybe we need to get one of those Paypal Donate buttons........ ;-)

    Ben

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081116/8ebbddec/attachment.htm
  • Ben Hood at Nov 17, 2008 at 1:00 am
    Ed,
    Ed,

    On Mon, Nov 17, 2008 at 12:15 AM, Edwin Fine
    wrote:
    Count me in too. Heck, (he said generously), make it $10!
    OK, I've added a donate button to the lettuce site so that you can
    take care of your Rabbit :-)
    Seriously, though, I don't want to be an ass about this, but I would think
    that there are now enough people (namely, non-paying customers ;) with
    enough valid use cases making enough noise about this particular issue, to
    warrant bumping it to the top of the Rabbit architectural issues list. When
    someone cannot even restart a node to drain the persister log after the node
    crashes with OOM, and has to delete the persister log, you know that there's
    a significant design deficiency to be remedied post-haste.
    That wasn't a design deficiency, that's just how we chose to implement
    it. Design-wise, it is reasonably easy to incorporate.
    - but I think it's reasonable to say that it's
    time to spend the estimated 3 work-weeks (I assume 120 work-hours) to remedy
    this. The cost of this should not be extreme. Let me put my project-manager
    hat on here... unless I am badly out of touch, one of your team shouldn't
    cost much more than, say, $80/hour at contract rates, so you are talking
    $10K to fix this (if I am wrong, and the rates are higher than that, PLEASE
    can I work for you? ;).
    10K would probably do it, but that's not our daily rate - we'd do it
    because of the cause.

    Yes, we could probably outsource some AS/400 stuff to you ;-)
    Is there a way you could beg, borrow or
    venture-capital this to fast-track it?
    :-)

    Ben
  • Valentino Volonghi at Nov 17, 2008 at 8:56 am

    On Nov 16, 2008, at 4:15 PM, Edwin Fine wrote:

    What do you say, Rabbit list? Would 49 of you PayPal $200 to get
    this done?

    I'll talk with my CTO tomorrow and see what we can put together. If it
    were just
    up to me I'd be in for a grand actually.

    --
    Valentino Volonghi aka Dialtone
    Now running MacOS X 10.5
    Home Page: http://www.twisted.it
    http://www.adroll.com

    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081117/d6afe429/attachment.htm
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: PGP.sig
    Type: application/pgp-signature
    Size: 194 bytes
    Desc: This is a digitally signed message part
    Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081117/d6afe429/attachment.pgp
  • Ben Hood at Nov 17, 2008 at 10:01 am
    Dear Rabbit Community,
    On Mon, Nov 17, 2008 at 8:56 AM, Valentino Volonghi wrote:
    I'll talk with my CTO tomorrow and see what we can put together. If it were
    just
    up to me I'd be in for a grand actually.
    Thanks for looking into this, you'll make a bunny happy :-)

    BTW - Many thanks to those who have already donated - I was surprised
    at the reaction that we got just after I added the link to the page.
    We'll have to see what comes together, but I will keep you informed as
    to how this is going to get publicized and more importantly, when
    feature X is going to be delivered.

    Thanks for the support!

    Ben
  • Ben Hood at Nov 18, 2008 at 5:56 pm
    Dear List,
    On Mon, Nov 17, 2008 at 8:56 AM, Valentino Volonghi wrote:

    On Nov 16, 2008, at 4:15 PM, Edwin Fine wrote:

    What do you say, Rabbit list? Would 49 of you PayPal $200 to get this done?

    I'll talk with my CTO tomorrow and see what we can put together. If it were
    just
    up to me I'd be in for a grand actually.
    I have been asked off-list whether they can send a cheque instead of
    using Paypal. The only reason why I went for a Paypal Donate button
    was because of the speed at which you can set this up, and I didn't
    really expect much response, so I am pleasantly surprised at this :-)

    I you do prefer to go down the paper route, send it to me at the office:

    LShift Ltd
    6 Rufus St,
    London N1 6PE
    UK

    BTW, I am still thinking about how to visualize the fund raiser -
    maybe as a progress bar a la curl? Any ideas or widgets?

    Ben
  • Valentino Volonghi at Nov 19, 2008 at 2:03 am

    On Nov 18, 2008, at 9:56 AM, Ben Hood wrote:

    Dear List,

    On Mon, Nov 17, 2008 at 8:56 AM, Valentino Volonghi <dialtone at gmail.com
    wrote:

    On Nov 16, 2008, at 4:15 PM, Edwin Fine wrote:

    What do you say, Rabbit list? Would 49 of you PayPal $200 to get
    this done?

    I'll talk with my CTO tomorrow and see what we can put together. If
    it were
    just
    up to me I'd be in for a grand actually.
    I have been asked off-list whether they can send a cheque instead of
    using Paypal. The only reason why I went for a Paypal Donate button
    was because of the speed at which you can set this up, and I didn't
    really expect much response, so I am pleasantly surprised at this :-)

    I you do prefer to go down the paper route, send it to me at the
    office:

    LShift Ltd
    6 Rufus St,
    London N1 6PE
    UK

    BTW, I am still thinking about how to visualize the fund raiser -
    maybe as a progress bar a la curl? Any ideas or widgets?

    The simplest possible:

    <html>
    <head>

    <style type="text/css" media="screen">
    .progressbar {
    width: 1000px;
    height: 20px;
    border: solid 1px #000;
    }
    .progressbar span {
    background: blue;
    height: 20px;
    display: block;
    }
    </style>
    </head>
    <body>
    <div class="progressbar"><span style="width: 30%"></span></div>
    </body>
    </html>

    Or if you want to be more fancy:

    http://cssglobe.com/lab/progress_bar/

    :)

    --
    Valentino Volonghi aka Dialtone
    Now running MacOS X 10.5
    Home Page: http://www.twisted.it
    http://www.adroll.com

    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081118/50cd9931/attachment.htm
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: PGP.sig
    Type: application/pgp-signature
    Size: 194 bytes
    Desc: This is a digitally signed message part
    Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081118/50cd9931/attachment.pgp

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedNov 15, '08 at 10:59p
activeNov 19, '08 at 2:03a
posts8
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase