Hi,
I'm developing a stress test to compare and measure the performance between
RabbitMQ and ActiveMQ. Also the objective was to compare the standard AMQP
with the JMS specifications. There is something that I would like to share
with the community since I think in could be a possible bug or a memory leak
problem. The stress test is very easy; there are no exponential ore base
variables, just a linear test case. I'm going to describe the scenario.
The scenario is composed by two JAVA clients: a publisher and a subscriber,
and one instance of RabbitMQ (V. 2.2.0) running over a virtual machine 2GB
RAM. The clients, both publisher and subscriber are placed in two different
physical machines. The messages to be sent by the publisher and received by
the subscriber are described within a cluster. A cluster specifies the size
of the body message (in byte) and the total number of messages to send and
receive for that dimension. Ex: C1 = (512::10000). The publisher uses a list
of clusters, and for each one of them it starts a new loop in which messages
are sent. Es: C1 = (512::500000); C2 = (1024::100000); C3 = (2048::10000).
At begin and at the end of each cluster, the publisher sends a special
message used to inform the subscriber that a new cluster is started or is
just finished. Also a special message is sent to the subscriber when all the
clusters have been elaborated. The subscriber does nothing more than
receiving messages and calculates the elapsed time (in seconds) between the
start and the end message for each cluster. The Publisher uses a "direct"
exchange (non-durable, non auto-delete) to publish the messages while the
subscriber creates and binds a new queue over the same "direct" exchange.

This is the code used to initialize the publisher:

ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("guest");
factory.setPassword("guest");
factory.setVirtualHost("/");
factory.setHost("RabbitMQ_Srv_01");
factory.setPort(5672);
conn = factory.newConnection();
channel = conn.createChannel();
boolean durable = false;
boolean autoDelete = false;
channel.exchangeDeclare("linearTest", "direct", durable, autoDelete, null);
...
//Define some properties
AMQP.BasicProperties props = MessageProperties.PERSISTENT_TEXT_PLAIN;
Map<String, Object> headers = new HashMap<String, Object>();
headers.put("TestCommand", CURRENT_COMMAND);
...
props.setHeaders(headers);
//Basic publish
channel.basicPublish("linearExch", "linearKey", props,
CLUSTER[i].messageBodyBytes);
...

This is the code used to initialize the subscriber:

ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("guest");
factory.setPassword("guest");
factory.setVirtualHost("/");
factory.setHost("RabbitMQ_Srv_01");
factory.setPort(5672);
Connection conn = factory.newConnection();
Channel channel = conn.createChannel();
boolean durable = false;
boolean autoDelete = false;
boolean exclusive = false;
channel.exchangeDeclare("linearExch", "direct", durable, autoDelete, null);
channel.queueDeclare("linearQueue", durable, exclusive, autoDelete, null);
channel.queueBind("linearQueue", "linearExch", "linearKey", null);
QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false;
channel.basicConsume("linearQueue", autoAck, consumer);

while (true) {
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
onMessage(delivery);
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
//If shutdown from the publisher is received
if (shutdown) {
channel.close();
conn.close();
break;
}
}
catch (InterruptedException ie) {
continue;
}
}

This is how I've set up the test case:

1. Started the RabbitMQ broker by using the .bat "rubbitmq-server.bat"
(NO extra configuration used).
2. Open the task manager to view the RabbitMQ (.erl process) memory
usage.
3. Started the subscriber.
4. Started the publisher.
5. Defined Clusters As: {C1(512:10000); C2(512:10000); C3(512:10000);
C4(512:100000); C5(512:100000); C6(512:100000); C7(1024:10000);
C8(1024:10000); C9(1024:10000); C10(1024:100000); C11(1024:100000);
C12(1024:100000); C13(2048:10000); C14(2048:10000); C15(2048:10000);
C16(2048:100000); C17(2048:100000); C18(2048:100000) }

These are some results and notifications:

. I've started a first test case, I've noticed that RabbitMQ uses lots of
memory during the test. At the first step, where only the broker was
running, just 19 KB of memory were used. At the end of the process the
memory usage increases up to more than 850 KB. After stopping the publisher
and the subscriber, the memory usage was very slowly decreasing from
850-800KB down to 50-20 KB.

. I've started a second test case but this time using the publisher and the
subscriber over the same physical machine. After a few minutes the broker
crashed down showing the following message: "eheap_alloc: Cannot allocate
373662860 bytes of memory (of type "heap")". I've repeated the same test
more and more but every time the broker crashed down.

. I?ve started a third test case, similar to the first one (publisher and
subscriber placed in two different machines) but this time using the
following cluster list: {C1(512:10000); C2(512:10000); C3(512:10000);
C4(512:100000); C5(512:100000); C6(512:100000); C7(512:500000);
C8(512:500000); C9(512:500000)}. At the time of elaborating the last element
(C9) the broker crashed down showing the same message as the one in the
previous test.

I think that the reason for the second test case can be found in the network
usage, which causes the broker to store in the memory a lots of message
because publisher and subscriber are placed in the same machine and, it?s
clearly that the subscriber leads substantially the usage and the
performance of the process (maybe because of the acknowledgements). But I
don?t have a good idea for the third test case, which causes the same error.

Now I?m facing with the fact that, comparing to ActiveMQ, the second test
and the third test are successfully done, in fact, even if the subscriber
and the publisher are placed in the same machine (the ActiveMQ broker runs
in the same virtual machine as the one used for the RabbitMQ) the test case
comes successfully to the end, and the broke doesn?t crash down. However,
the first test case (described in the first point) seems to be more
efficient, in terms of seconds, by using RabbitMQ.

Here some questions:

. In general, what do you think about the test?
. Am I adopting a very strange test case which is not well handled by
RabbitMQ?
. Is there any connection with the solved bug described in the RabbitMQ 2.2
release notes (?fix memory leak when long-running channels consume and
cancel on many queues?)?
. How does RabbitMQ handle message flooding? As stated in ActiveMQ official
page, the flow control, in the current version means that: ?if the broker
detects that the memory limit for the destination, or the temp- or
file-store limits for the broker, have been exceeded, then the flow of
messages can be slowed down. The producer will be either blocked until
resources are available or will receive a JMSException?.

Thanks for any response.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110111/ae46a278/attachment.htm>

Search Discussions

  • Matthew Sackman at Jan 14, 2011 at 4:00 pm
    Hi,
    On Tue, Jan 11, 2011 at 12:44:54PM +0100, Crisoforo Seccia wrote:
    . Am I adopting a very strange test case which is not well handled by
    RabbitMQ?
    Not really, no. However, there are problems with RabbitMQ under Windows,
    and you may have to do some tweaking of the configuration. The fact that
    you crashed on an eheap allocation suggests that Rabbit tried to
    allocate memory that Windows refused it.
    http://www.rabbitmq.com/extensions.html#memsup

    Rabbit by default assumes that it's the only application running on a
    machine, and thus it can expect to be allocated all memory. You may
    wish, as suggested in the above page, to try reducing the
    vm_memory_high_watermark and repeat the tests.
    . Is there any connection with the solved bug described in the RabbitMQ 2.2
    release notes (?fix memory leak when long-running channels consume and
    cancel on many queues?)?
    Nope - that bug would have likely taken months of constant tripping to
    actually result in any sizable memory leak.
    . How does RabbitMQ handle message flooding? As stated in ActiveMQ official
    page, the flow control, in the current version means that: ?if the broker
    detects that the memory limit for the destination, or the temp- or
    file-store limits for the broker, have been exceeded, then the flow of
    messages can be slowed down. The producer will be either blocked until
    resources are available or will receive a JMSException?.
    Again, the above link will explain it all, but in short, Rabbit uses TCP
    back pressure to prevent producers flooding it with too many messages.
    Ultimately, you're always limited by the rate at which you can get rid
    of messages - whether to clients or to disk. In effect, RAM just acts as
    a huge (hopefully!) buffer.

    Matthew
  • Crisoforo Seccia at Jan 21, 2011 at 11:44 am
    Hi,

    I've fallowed your instructions and configured my RabbitMQ server under
    windows to use different configuration setup:

    1) [{rabbit, [{vm_memory_high_watermark, 0.80}]}].
    2) [{rabbit, [{vm_memory_high_watermark, 0.40}]}].
    3) [{rabbit, [{vm_memory_high_watermark, 0.25}]}].
    4) [{rabbit, [{vm_memory_high_watermark, 0.20}]}].
    5) [{rabbit, [{vm_memory_high_watermark, 0.10}]}].

    Every time I've started RabbiitMQ server with one of the previous
    configuration and then started my Test application but every time the
    RabbitMQ server crashed down after a few minutes.I don't sincerly find any
    problem in my application's code and, as I've stated in previous e-mail the
    test is very simple. Anyway I've attached my project and hope someone can
    test my application under different machine then window because for the
    moment I'm unable to operate this kind of test. Hope to have a response.
    Thanks, best regards

    P.S. (How to use the project):

    The attached file is a MAVEN project (platform independent) which produces
    as output an executable project. The project is composed by two sub-modules,
    one is use to test ActiveMQ and another one is used to test RabbitMQ.
    Basically the project is a ready to go for running, just configure the
    properties file (server connection, login, etc.). The configuration files
    are respectively placed under
    "${broker_type}mq/publisher/data/publisher.properties". Both the two
    sub-modules (and both publisher or subscriber applications) produces as
    output a .csv file under the folder
    "${broker_type}mq/${application_type}/results" where are printed some test
    results. Also the messages number and the size of the message can be
    configured to the publishers by using the file located under the folder
    "${broker_type}mq/${application_type}/data/${broker_type}MQTesterPublisher.txt"
    (an example is already provided).

    Cristoforo.

    2011/1/14 Matthew Sackman <matthew at rabbitmq.com>
    Hi,
    On Tue, Jan 11, 2011 at 12:44:54PM +0100, Crisoforo Seccia wrote:
    . Am I adopting a very strange test case which is not well handled by
    RabbitMQ?
    Not really, no. However, there are problems with RabbitMQ under Windows,
    and you may have to do some tweaking of the configuration. The fact that
    you crashed on an eheap allocation suggests that Rabbit tried to
    allocate memory that Windows refused it.
    http://www.rabbitmq.com/extensions.html#memsup

    Rabbit by default assumes that it's the only application running on a
    machine, and thus it can expect to be allocated all memory. You may
    wish, as suggested in the above page, to try reducing the
    vm_memory_high_watermark and repeat the tests.
    . Is there any connection with the solved bug described in the RabbitMQ 2.2
    release notes (?fix memory leak when long-running channels consume and
    cancel on many queues?)?
    Nope - that bug would have likely taken months of constant tripping to
    actually result in any sizable memory leak.
    . How does RabbitMQ handle message flooding? As stated in ActiveMQ official
    page, the flow control, in the current version means that: ?if the broker
    detects that the memory limit for the destination, or the temp- or
    file-store limits for the broker, have been exceeded, then the flow of
    messages can be slowed down. The producer will be either blocked until
    resources are available or will receive a JMSException?.
    Again, the above link will explain it all, but in short, Rabbit uses TCP
    back pressure to prevent producers flooding it with too many messages.
    Ultimately, you're always limited by the rate at which you can get rid
    of messages - whether to clients or to disk. In effect, RAM just acts as
    a huge (hopefully!) buffer.

    Matthew
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110121/3099811d/attachment-0001.htm>
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: MQTester.rar
    Type: application/rar
    Size: 41564 bytes
    Desc: not available
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110121/3099811d/attachment-0001.rar>
  • Jerry Kuch at Jan 24, 2011 at 9:30 pm
    Hi, Cristoforo...
    I've fallowed your instructions and configured my RabbitMQ server
    under windows to use different configuration setup:

    1) [{rabbit, [{vm_memory_high_watermark, 0.80}]}].
    2) [{rabbit, [{vm_memory_high_watermark, 0.40}]}].
    3) [{rabbit, [{vm_memory_high_watermark, 0.25}]}].
    4) [{rabbit, [{vm_memory_high_watermark, 0.20}]}].
    5) [{rabbit, [{vm_memory_high_watermark, 0.10}]}].
    A good set of values to try... I assume you were able to confirm in
    your Rabbit log at startup that the changed watermark value was being
    reflected? The messages announcing the broker's configured memory
    limit should look roughly like...

    =INFO REPORT==== 24-Jan-2011::13:16:22 ===
    Memory limit set to 2779MB.

    ...and you should see this value dropping as you decrease the
    watermark setting in the manner you describe. I ask only because
    we've had a couple of cases (they seem a bit more common on Windows
    for some reason), where the setting was changed incorrectly and didn't
    get picked up by the broker at startup, and want to make sure you
    didn't encounter that pitfall, and are in fact trying out the broker
    with different watermark values.
    Every time I've started RabbiitMQ server with one of the previous
    configuration and then started my Test application but every time the
    RabbitMQ server crashed down after a few minutes.I don't sincerly find
    any problem in my application's code and, as I've stated in previous
    e-mail the test is very simple. Anyway I've attached my project and
    hope someone can test my application under different machine then
    window because for the moment I'm unable to operate this kind of
    test. Hope to have a response. Thanks, best regards
    I just tried running it on Mac OS X and it was able to run to
    completion. I did notice that as it ran Unix 'top' showed the RPRVT
    value for the Erlang BEAM process in which the broker was running
    bouncing steadily around 2GB, a value that could be strenuous on a
    32-bit Windows system where only 2GB of user space are available to a
    single process, assuming the same sort of memory loading occurs there.

    I expect I'll be able to give it a try on a Windows system later today
    once I put one together with Maven, a JVM, etc. to run your stress
    test.
    The attached file is a MAVEN project (platform independent) which
    produces as output an executable project.
    Thanks for providing such a clean, well packaged and easy to run repro
    case. This is always helpful... I'll mail the list once I've had a
    chance to exercise it under Windows myself...

    Best regards,
    Jerry
  • Jerry Kuch at Feb 8, 2011 at 5:15 pm
    Hi, Cristofo...
    Hi Jerry, sorry to bother you again! if you remember our discussion
    about some rabbitmq crash problems. However I think i've found a
    solution wich seems for me to work. I've used all the previous
    configurations and test applications with the last version of Erlang
    Installed on my win 2003 machine (so I'm moved from the erlang
    "R13B03" to the "R14B01") and thanks to this version, my test
    application works fine. As I said before, my intentions were to
    compare RabbitMQ with ActiveMQ and now I finally got out some useful
    results. if you are interesting I can send you a Excel file which
    reports (thanks to some diagrams...) the differences between the two
    servers. Thanks. Cristoforo.
    No bother at all! I'm glad you've got a solution that's viable for
    you now. That said, there are still some issues with memory handling
    in the Windows version of Erlang, but as we discussed previously they
    tend to only manifest when the process is under significant memory
    stress, as it might get, for example, when a significant imbalance
    between producers and consumers exists for too long. We've yet to see
    those issues under Linux, so long term, at least until the Windows
    Erlang situation improves, deploying on that OS is always something to
    keep in mind.

    If you're able to share your data and test results, I think we'd
    really appreciate seeing that. Depending on what the data contain,
    someone on the team may even have some suggestions on optimization
    for your specific scenario.

    Best regards,
    Jerry

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJan 11, '11 at 11:44a
activeFeb 8, '11 at 5:15p
posts5
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase