Hi,

I've been trying to track down some performance issues in our
application since switching to RabbitMQ (previously used Spread), and it
seems latency for a single round trip message is an issue.

In a simple(ish) benchmark I've created, I have a daemon that blocks,
waiting for a message. When it receives one, it sends a message back to
the original sender. I then have a script that runs on another machine
(on the same LAN), which sends messages of about 5k in size, and records
the time for the round trip.

In these tests, I get an average of about 0.1s for the round trip. Does
this seem unreasonably slow? In an identical test using Spread, I get
times of around 0.01s (the application performance issue we're seeing is
showing RabbitMQ about 3-4 times slower than Spread).

I realise it's quite likely the problem is something odd I'm doing, but
I just want to rule out anything obvious at this stage. Could it be OS,
Erlang version, etc?

I'm running on Debian Lenny, with RabbitMQ 2.7.1, and Erlang 12b3 (from
the standard Lenny package). Are there any known issues with this
configuration, or any other factors I might be missing?

The client is written in Perl, using a home-grown client (but based on
Net::AMQP), as there wasn't really much suitable to our purposes on
CPAN. I hope to have this client on CPAN sometime soon, and will likely
ask for a review for people who might have comments. It's possible
(likely?) it's not as efficient as it could be, and would be nice to
rule that out somehow.

Would appreciate any help.. apologies if I haven't given all the
relevant information.

Thanks,
Sam Crawley.

Search Discussions

  • Chris Chew at Feb 7, 2012 at 4:50 am
    Hi Sam.

    It's definitely worth trying a new version of Erlang. Something in the 14B range or newer should perform noticeably better.

    Also, are you doing a basicGet or basicConsume when listening for the messages?

    Cheers,

    Chris
    On Feb 6, 2012, at 19:46, "Sam Crawley" wrote:

    Hi,

    I've been trying to track down some performance issues in our
    application since switching to RabbitMQ (previously used Spread), and it
    seems latency for a single round trip message is an issue.

    In a simple(ish) benchmark I've created, I have a daemon that blocks,
    waiting for a message. When it receives one, it sends a message back to
    the original sender. I then have a script that runs on another machine
    (on the same LAN), which sends messages of about 5k in size, and records
    the time for the round trip.

    In these tests, I get an average of about 0.1s for the round trip. Does
    this seem unreasonably slow? In an identical test using Spread, I get
    times of around 0.01s (the application performance issue we're seeing is
    showing RabbitMQ about 3-4 times slower than Spread).

    I realise it's quite likely the problem is something odd I'm doing, but
    I just want to rule out anything obvious at this stage. Could it be OS,
    Erlang version, etc?

    I'm running on Debian Lenny, with RabbitMQ 2.7.1, and Erlang 12b3 (from
    the standard Lenny package). Are there any known issues with this
    configuration, or any other factors I might be missing?

    The client is written in Perl, using a home-grown client (but based on
    Net::AMQP), as there wasn't really much suitable to our purposes on
    CPAN. I hope to have this client on CPAN sometime soon, and will likely
    ask for a review for people who might have comments. It's possible
    (likely?) it's not as efficient as it could be, and would be nice to
    rule that out somehow.

    Would appreciate any help.. apologies if I haven't given all the
    relevant information.

    Thanks,
    Sam Crawley.

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Sam Crawley at Feb 14, 2012 at 2:47 am
    Hi,

    Thanks for the feedback (and apologies for the delay in responding).

    OK, I've reduced the variables even further and tested this on a single
    machine, running Debian Squeeze. I've built Erlang version 15B. RabbitMQ
    is installed from the RabbitMQ debian repo without any configuration.
    rabbitmqctl status gives me:

    Status of node 'rabbit at xxxx' ...
    [{pid,22390},
    {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
    {os_mon,"CPO CXC 138 46","2.2.8"},
    {sasl,"SASL CXC 138 11","2.2"},
    {mnesia,"MNESIA CXC 138 12","4.6"},
    {stdlib,"ERTS CXC 138 10","1.18"},
    {kernel,"ERTS CXC 138 10","2.15"}]},
    {os,{unix,linux}},
    {erlang_version,"Erlang R15B (erts-5.9) [source] [64-bit]
    [async-threads:30] [kernel-poll:true]\n"},
    {memory,[{total,25669112},
    {processes,10124080},
    {processes_used,10124066},
    {system,15545032},
    {atom,471633},
    {atom_used,470423},
    {binary,151688},
    {code,11797130},
    {ets,750032}]},
    {vm_memory_high_watermark,0.4},
    {vm_memory_limit,844374016}]
    ...done.

    I use the following Perl script to send an receive messages (no longer
    using our own client for this test, instead using Net::RabbitMQ, a
    wrapper around librabbitmq-c, available from CPAN... also reduced the
    message size - it doesn't seem to be a factor):

    ---

    use strict;
    use warnings;

    use v5.10;
    use Time::HiRes qw(gettimeofday tv_interval);
    use Net::RabbitMQ;
    use Statistics::Basic qw(:all);
    use List::Util qw(max min);

    my $length = 5;
    my $queue = 'my_queue';

    my $mq = Net::RabbitMQ->new();
    $mq->connect("localhost", { user => "guest", password => "guest" });
    $mq->channel_open(1);
    $mq->queue_declare(1, $queue);

    my $payload = 'a' x $length;

    my @times;

    $mq->consume(1, $queue, {});

    for (1..1000) {
    $mq->publish(1, $queue, $payload);

    my @start = gettimeofday();

    my $msg = $mq->recv();

    push @times, tv_interval( \@start );
    }

    $mq->disconnect();

    my $longest = max @times;
    my $shortest = min @times;

    print "Longest request: $longest, Shortest Request: $shortest\n";
    say "Mean Req time: " . mean(@times);

    ---

    AFAIK, acks, durable queues, etc. are all turned off.

    Running this, I always get an average time (and I'm obviously only
    timing the reading of messages) of 0.04s. (FWIW, I get the same with our
    home grown client).

    Does it seem like there's something wrong here?

    Thanks,
    Sam.
    On Tue, 2012-02-07 at 04:50 +0000, Chris Chew wrote:
    Hi Sam.

    It's definitely worth trying a new version of Erlang. Something in the 14B range or newer should perform noticeably better.

    Also, are you doing a basicGet or basicConsume when listening for the messages?

    Cheers,

    Chris
    On Feb 6, 2012, at 19:46, "Sam Crawley" wrote:

    Hi,

    I've been trying to track down some performance issues in our
    application since switching to RabbitMQ (previously used Spread), and it
    seems latency for a single round trip message is an issue.

    In a simple(ish) benchmark I've created, I have a daemon that blocks,
    waiting for a message. When it receives one, it sends a message back to
    the original sender. I then have a script that runs on another machine
    (on the same LAN), which sends messages of about 5k in size, and records
    the time for the round trip.

    In these tests, I get an average of about 0.1s for the round trip. Does
    this seem unreasonably slow? In an identical test using Spread, I get
    times of around 0.01s (the application performance issue we're seeing is
    showing RabbitMQ about 3-4 times slower than Spread).

    I realise it's quite likely the problem is something odd I'm doing, but
    I just want to rule out anything obvious at this stage. Could it be OS,
    Erlang version, etc?

    I'm running on Debian Lenny, with RabbitMQ 2.7.1, and Erlang 12b3 (from
    the standard Lenny package). Are there any known issues with this
    configuration, or any other factors I might be missing?

    The client is written in Perl, using a home-grown client (but based on
    Net::AMQP), as there wasn't really much suitable to our purposes on
    CPAN. I hope to have this client on CPAN sometime soon, and will likely
    ask for a review for people who might have comments. It's possible
    (likely?) it's not as efficient as it could be, and would be nice to
    rule that out somehow.

    Would appreciate any help.. apologies if I haven't given all the
    relevant information.

    Thanks,
    Sam Crawley.

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Feb 14, 2012 at 1:24 pm

    On 14/02/12 02:47, Sam Crawley wrote:
    Running this, I always get an average time (and I'm obviously only
    timing the reading of messages) of 0.04s. (FWIW, I get the same with our
    home grown client).

    Does it seem like there's something wrong here?
    Yes. Very much so. I replicated your problem.

    The problem is that the old version of librabbitmq that is embedded in
    Net::RabbitMQ does not disable Nagle's algorithm. (I assume your
    homebrew client also does not.)

    Looking in wireshark we see that almost all of the delay happens after
    the basic.publish, when the TCP stack waits 0.04s (on my machine) for
    more data before *actually* sending the publish. Everything else is very
    fast in comparison, giving an almost perfect 25 msg/s for me.

    I hacked in a call to setsockopt() to set TCP_NODELAY in amqp_socket.c,
    and now I get:

    Longest request: 0.000537, Shortest Request: 0.000146
    Mean Req time: 0

    Which I think is more the performance you might be looking for.
    (Presumably the "mean req time" suffered underflow.)

    The current version of librabbitmq sets TCP_NODELAY, so maybe
    Net::RabbitMQ should be updated? (CCing maintainer...)

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Sam Crawley at Feb 14, 2012 at 10:39 pm
    Thanks, that certainly seems to have done the trick!
    On Tue, 2012-02-14 at 13:24 +0000, Simon MacMullen wrote:
    On 14/02/12 02:47, Sam Crawley wrote:
    Running this, I always get an average time (and I'm obviously only
    timing the reading of messages) of 0.04s. (FWIW, I get the same with our
    home grown client).

    Does it seem like there's something wrong here?
    Yes. Very much so. I replicated your problem.

    The problem is that the old version of librabbitmq that is embedded in
    Net::RabbitMQ does not disable Nagle's algorithm. (I assume your
    homebrew client also does not.)

    Looking in wireshark we see that almost all of the delay happens after
    the basic.publish, when the TCP stack waits 0.04s (on my machine) for
    more data before *actually* sending the publish. Everything else is very
    fast in comparison, giving an almost perfect 25 msg/s for me.

    I hacked in a call to setsockopt() to set TCP_NODELAY in amqp_socket.c,
    and now I get:

    Longest request: 0.000537, Shortest Request: 0.000146
    Mean Req time: 0

    Which I think is more the performance you might be looking for.
    (Presumably the "mean req time" suffered underflow.)

    The current version of librabbitmq sets TCP_NODELAY, so maybe
    Net::RabbitMQ should be updated? (CCing maintainer...)

    Cheers, Simon
  • Emile Joubert at Feb 7, 2012 at 3:23 pm
    Hi Sam,
    On 07/02/12 02:45, Sam Crawley wrote:
    In these tests, I get an average of about 0.1s for the round trip. Does
    this seem unreasonably slow? In an identical test using Spread, I get
    times of around 0.01s (the application performance issue we're seeing is
    showing RabbitMQ about 3-4 times slower than Spread).
    There are a few factors that will influence the latency on the broker.
    Auto-acknowledgement and asynchronous delivery should be used.
    Persistence, transactions, long queues and synchronous message delivery
    should be avoided. Reducing the maximum frame size may help latency.

    Make sure the difference you are measuring is due to the broker only and
    not a change in network conditions. A quick unscientific test shows that
    latencies of 0.001s are typical on my own workstation for messages of
    5kb when using localhost (so very little network delay) and when queues
    are short.

    That said, RabbitMQ is not specifically aimed at the ultra low-latency
    market. There are more suitable products available if latency is your
    overriding concern.


    -Emile

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedFeb 7, '12 at 2:45a
activeFeb 14, '12 at 10:39p
posts6
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase