David King wrote:
If you just let it run, what happens?
I don't know. I guess we can see, but my guess is that it will wake
me up at two in the morning to fix the site. That's what usually
happens when rabbit crashes, anyway.
As long as you are running rabbit 1.8.0, and given the usage pattern you
describe, it is unlikely that rabbit would actually crash, though it may
refuse to accept messages from producers (which, I guess, would still be
a problem). It would be informative to find out whether it actually
reaches that point.
What memory limit does rabbit think it has? Check the rabbit log for an
info report along the lines of
=INFO REPORT==== 29-Oct-2009::15:43:27 ===
Memory limit set to 2048MB.
Also, are there any memory alarms in the logs? Look for entries like
=INFO REPORT==== 3-Jul-2010::21:28:31 ===
As Marek mentioned your graphs would seem to indicate that rabbit is
consuming a few gigs of memory right after startup. That doesn't look
right. A healthy rabbit should eat much less memory than that. For
example, the rabbit on my development machine starts with a 80MB virtual
memory footprint, 25 of which is resident.
I am assuming the graph shows the memory usage of the entire machine.
You said that the memory "does belong to beam.smp", but do you have a
graph, or even just some data points for only that process?
If the above doesn't yield any clues, perhaps you could grant one of the
rabbit engineers temporary access to the machine so they can delve into
the innards of rabbit to figure out where the memory is going? This
shouldn't affect the operation of your service.
What's the average and max message size in your set up?
They're just IDs, so they'll be between 5 and 10 bytes each.
Are any of the messages published as persistent? Another reason to
upgrade to 1.8.0 is that there only messages published as persistent
that go to durable queues actually hit the persister - a change we made
partially as a result of the investigation into your previous problems.
How are you measuring the queue length? With 'rabbitmqctl
How frequently do you run rabbitmqctl?