After reading some posts on the list it seems like many problems with
performance is nothing new. We are currently reorganizing a cluster
consisting of a myriad of jetty instances to a rabbitmq and a message
driven system. The current component we are moving is shortly a staging
system for a relative slow consumer process..

Our use-case:.

1. We have currently a small stream of relatively large messages, that
varies in size. (1600 bytes -> 5000 bytes )
2. The incoming rate varies form 0 -> 58 per/s in our current test queues.
3. We have currently two brokers in a cluster
4. The queues are set up in mirrored (We can not lose any messages),
durable and non-exclusive.
5. We use Publisher Confirms/Consumer Cancellation in the client
implementations in Java
6. We use prefetch-count as a basic QOS parameter. We have tested with: 1,
10 , 15, 25
7. We ack per (deliveryTag % prefetch-count == 0) to minimize acking.
8. The client can nack a message but it is seldom happening (every 1
million message maybe)

Problem:
Over time we see a larger performance decrease in consuming the messages
from the broker. The consumer is deadly fast in the start. 1-2 ms per
message. Then the retrieval gradually decreases to 25-26 ms per message.
The velocity of the speed decrease is varying by the prefetch size. When
increasing the prefetch count to the time before it reaches 25ms is
exponentially increasing. Up to 12 hours. If we re-initiate the connection
it seem to recover and everything seem OK.

Versions:

Java client: 2.5.1
Broker: 2.5.1 (ubuntu build)

Regards


--
Tarjei Romtveit
Data Management Director
Integrasco A/S
Mobile: +47 990 20 074
Address: Hasseldalen 3, N-4878 Grimstad, Norway
www.integrasco.com // twitter.com/integrasco // facebook.com/integrasco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111108/1d5655be/attachment.htm>

Search Discussions

  • Matthias Radestock at Nov 9, 2011 at 5:46 pm
    Tarjei,
    On 08/11/11 14:11, Tarjei Romtveit wrote:
    3. We have currently two brokers in a cluster
    4. The queues are set up in mirrored (We can not lose any messages),
    durable and non-exclusive.
    What do you mean by "mirrored"? Presumably not rabbit's active-active HA
    queue mirroring, since that only became available in 2.6.0 and you are
    running 2.5.1.
    Over time we see a larger performance decrease in consuming the messages
    from the broker. The consumer is deadly fast in the start. 1-2 ms per
    message. Then the retrieval gradually decreases to 25-26 ms per message.
    At the risk of stating the obvious, the slow-down isn't just due to
    there being no more messages in the queue, right? Are you seeing a
    backlog build up? If so, is the server busy?


    Regards,

    Matthias.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedNov 8, '11 at 2:11p
activeNov 9, '11 at 5:46p
posts2
users2
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase