RabbitMQ Stability Issues with large queue - 2011-12-28

Hi All,

I posted in the IRC channel a few nights ago, and they suggested that I bring this to the listserv.
Hopefully can get some suggestions on how to keep my servers from crashing.
Thanks,
-- DawgTool

Cluster Info:
Cluster status of node 'dc001 at rmquat-m01' ...
[{nodes,[{disc,['dc001 at rmquat-m04','dc001 at rmquat-m03','dc001 at rmquat-m02',
'dc001 at rmquat-m01']}]},
{running_nodes,['dc001 at rmquat-m04','dc001 at rmquat-m03','dc001 at rmquat-m02',
'dc001 at rmquat-m01']}]

Config Info:
==> enabled_plugins <==
[rabbitmq_management,rabbitmq_management_agent,rabbitmq_management_visualiser].

==> rabbitmq.config <==
[
{rabbit, [{vm_memory_high_watermark, 0.6},
{collect_statistics_interval, 5000},
{hipe_compile, true}
]
},
{rabbitmq_management, [ {http_log_dir, "/data/rabbitmq/dc001/rabbit-mgmt"} ] },
{rabbitmq_management_agent, [ {force_fine_statistics, true} ] }
].

==> rabbitmq-env.conf <==
NODENAME=dc001
BASE=/data/rabbitmq/dc001
MNESIA_BASE=/data/rabbitmq/dc001/mnesia
LOG_BASE=/data/rabbitmq/dc001/log
SERVER_START_ARGS="-smp enable"


IRC:
[17:32] <dawgtool> background. doing some testing on 2.7.0 : 4 servers 2vcpu, 8gb ram, 80gb disc.
[17:32] <dawgtool> cluser is setup all disc, currently one exchange durable fanout
[17:33] <dawgtool> one queue also durable bind to the exchange.
[17:33] <dawgtool> i'm pushing about 5M records, payload is ~500bytes each record
[17:34] <dawgtool> rate is about 14k/s (which seems pretty slow)
[17:35] <dawgtool> but my problem is, I'm testing a case where they consumers are busy or unavailable, so the queues would be filling up.
[17:35] <dawgtool> even after slowing the publish rate to about 4k/s the mirrored queue does not complete on any of the clusters nodes other then master.
[17:37] <dawgtool> memory seems to be the biggest issue here, as the servers will grow passed the high water mark, and eventually crash one at a time.
[17:37] <dawgtool> once they are restarted, most servers in the cluster will have about 200k to 300k of messages in their queue
[17:40] <dawgtool> so question is, why is so much memory being consumed (on disk these records are about 5.5GB) RabbitMQ pushes to 7.9 real, 11.9 virtual (swapping).
[17:40] <dawgtool> why is the queue not stopping the publishers (RAM based clusters seem to stop the publisher until it can be spilled to disk)
[17:41] <dawgtool> Why is mirroring unreliable in this test.
[17:41] <dawgtool> ok, i'm done with the backgroud. =)
[17:41] <dawgtool> lol
[17:42] <antares_> having 300K messages in one queue will result in RAM consumption like this
[17:42] <antares_> 30 queues with 10K is a better option
[17:43] <antares_> I can't say for mirroring but my understanding is that mirroring cannot decrease RAM usage
[17:43] <dawgtool> true, i need to make sure I don't loose any records, so disc with mirror
[17:44] <dawgtool> does performance get that bad after 300k message in a queue?
[17:44] <antares_> this kind of questions is better asked on the rabbitmq-discuss (mailing list)
[17:45] <antares_> the exact number will vary but yes, having queues with excessive # of messages that are not consumed will result in more or less this behavior
[17:45] <dawgtool> yea, just joined the mailing list, I can post it there. was hoping someone had a quick answer. =)
[17:45] <antares_> each queue is backed by one Erlang process and Erlang VM GC releases process memory all at once
[17:46] <dawgtool> even with the -smp set to true?
[17:46] <antares_> so having large amount of messages in your queues impedes that
[17:46] <antares_> dawgtool: I doubt that -smp affects GC behavior
[17:46] <dawgtool> hmmm
[17:47] <antares_> in this regard anyway, because there is still no shared heap
[17:47] <antares_> but rabbitmq team members definitely know more than I do
[17:47] <antares_> and they are all on rabbitmq-discuss
[17:48] <dawgtool> ok, I'll give the mailing list a shot. 300k is going to be hard to live under, one of my current systems is doing server times that a second. =(
[17:49] <dawgtool> which i was hoping to migrate to a more open system, at least parts of it. =(
[17:49] <antares_> dawgtool: total # of messages is not the point
[17:50] <antares_> the point is max # of messages in one queue
[17:50] <antares_> you can have 10K queues and one app consuming messages from them all
[17:50] <antares_> unless ordering is a serious concern for your case, it will work just as well as 1 queue
[17:50] <dawgtool> yea, understand, but some consumers might crash, and I will get a backlog
[17:51] <dawgtool> I need to make sure the MQ system can handle at least a 30 minute outage
[17:51] <antares_> again, you will have the same problem with just 1 queue
[17:51] <antares_> dawgtool: I see. And not lose anything?
[17:51] <dawgtool> right, =(
[17:51] <antares_> rabbitmq queues can have message TTL
[17:52] <dawgtool> yea, I have TTL on metric collection consumers.. usually 6 seconds.
[17:55] <dawgtool> in production, the idea would be to have two exchanges: input.exchange.fanout, metric.exchange.topic bind to input.exchange.fanout. queue.durable.mirror.prod bind to input.exchange.fanout no ttl, several queue.trans.nomirror.metric[1-x] ttl 6sec.
[17:56] <dawgtool> the input.exchange.fanout will have a second and third queue eventually, which is why there are two exchanges.
[17:57] <dawgtool> but the second and third will have a 30min ttl
[18:01] <dawgtool> anyway, thanks for the info.. I'll shot an email out to the listserv and see if I get any bites. =)
[18:01] <dawgtool> thanks again. =)
[18:01] <antares_> dawgtool: no problem

Search Discussions

  • DawgTool at Dec 28, 2011 at 10:20 pm
    Hi All,

    Ran the publisher against a single node (removed the clustering) and was able to grab the report just before crashing.
    Here are the details:

    ==> dc001.log <==
    =INFO REPORT==== 28-Dec-2011::17:00:00 ===
    vm_memory_high_watermark set. Memory used:8662988728 allowed:5153960755

    =INFO REPORT==== 28-Dec-2011::17:00:00 ===
    alarm_handler: {set,{{vm_memory_high_watermark,'dc001 at rmquat-m01'},[]}}

    =WARNING REPORT==== 28-Dec-2011::17:00:00 ===
    Mnesia('dc001 at rmquat-m01'): ** WARNING ** Mnesia is overloaded: {dump_log, time_threshold}




    Reporting server status on {{2011,12,28},{22,6,3}}

    Status of node 'dc001 at rmquat-m01' ...
    [{pid,13173},
    {running_applications,
    [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","2.7.0"},
    {rabbitmq_management,"RabbitMQ Management Console","2.7.0"},
    {rabbitmq_mochiweb,"RabbitMQ Mochiweb Embedding","2.7.0"},
    {webmachine,"webmachine","1.7.0-rmq2.7.0-hg"},
    {amqp_client,"RabbitMQ AMQP Client","2.7.0"},
    {rabbitmq_management_agent,"RabbitMQ Management Agent","2.7.0"},
    {rabbit,"RabbitMQ","2.7.0"},
    {mnesia,"MNESIA CXC 138 12","4.4.17"},
    {os_mon,"CPO CXC 138 46","2.2.5"},
    {sasl,"SASL CXC 138 11","2.1.9.3"},
    {mochiweb,"MochiMedia Web Server","1.3-rmq2.7.0-git"},
    {inets,"INETS CXC 138 49","5.5.2"},
    {stdlib,"ERTS CXC 138 10","1.17.3"},
    {kernel,"ERTS CXC 138 10","2.14.3"}]},
    {os,{unix,linux}},
    {erlang_version,
    "Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [hipe] [kernel-poll:true]\n"},
    {memory,
    [{total,8825539568},
    {processes,4061532008},
    {processes_used,4061085200},
    {system,4764007560},
    {atom,1677993},
    {atom_used,1659957},
    {binary,4448013416},
    {code,16452002},
    {ets,295062344}]},
    {vm_memory_high_watermark,0.5999999999767169},
    {vm_memory_limit,5153960755}]

    Cluster status of node 'dc001 at rmquat-m01' ...
    [{nodes,[{disc,['dc001 at rmquat-m01']}]},{running_nodes,['dc001 at rmquat-m01']}]

    Application environment of node 'dc001 at rmquat-m01' ...
    [{auth_backends,[rabbit_auth_backend_internal]},
    {auth_mechanisms,['PLAIN','AMQPLAIN']},
    {backing_queue_module,rabbit_variable_queue},
    {cluster_nodes,[]},
    {collect_statistics,fine},
    {collect_statistics_interval,5000},
    {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
    {default_user,<<"guest">>},
    {default_user_tags,[administrator]},
    {default_vhost,<<"/">>},
    {delegate_count,16},
    {error_logger,{file,"/data/rabbitmq/dc001/log/dc001.log"}},
    {frame_max,131072},
    {hipe_compile,true},
    {included_applications,[]},
    {msg_store_file_size_limit,16777216},
    {msg_store_index_module,rabbit_msg_store_ets_index},
    {queue_index_max_journal_entries,262144},
    {sasl_error_logger,{file,"/data/rabbitmq/dc001/log/dc001-sasl.log"}},
    {server_properties,[]},
    {ssl_listeners,[]},
    {ssl_options,[]},
    {tcp_listen_options,[binary,
    {packet,raw},
    {reuseaddr,true},
    {backlog,128},
    {nodelay,true},
    {exit_on_close,false}]},
    {tcp_listeners,[5672]},
    {trace_vhosts,[]},
    {vm_memory_high_watermark,0.6}]

    Connections:
    pid address port peer_address peer_port ssl peer_cert_subject peer_cert_issuer peer_cert_validity auth_mechanism ssl_protocol ssl_key_exchange ssl_cipher ssl_hash protocol user vhost timeout frame_max client_properties recv_oct recv_cnt send_oct send_cnt send_pend state channels
    <'dc001 at rmquat-m01'.1.4461.3> 192.168.0.100 5672 192.168.0.97 33792 false PLAIN {0,8,0} guest / 0 131072 [] 79447230 86718 292 4 0 blocked 1

    Channels:
    pid connection number user vhost transactional confirm consumer_count messages_unacknowledged messages_unconfirmed messages_uncommitted acks_uncommitted prefetch_count client_flow_blocked
    <'dc001 at rmquat-m01'.1.4465.3> <'dc001 at rmquat-m01'.1.4461.3> 1 guest / false false 0 0 0 0 0 0 false

    Queues on /:
    pid name durable auto_delete arguments owner_pid slave_pids synchronised_slave_pids exclusive_consumer_pid exclusive_consumer_tag messages_ready messages_unacknowledged messages consumers memory backing_queue_status
    <'dc001 at rmquat-m01'.1.9112.0> dc.data:durable:all true false [] 4589172 0 4589172 0 4041816264 [{q1,0}, {q2,0}, {delta,{delta,2667892,1921280,4589172}}, {q3,132544}, {q4,2535348}, {len,4589172}, {pending_acks,0}, {target_ram_count,0}, {ram_msg_count,2535348}, {ram_ack_count,0}, {next_seq_id,4589172}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]

    Exchanges on /:
    name type durable auto_delete internal arguments
    amq.direct direct true false false []
    dc.data:fanout fanout true false false []
    amq.topic topic true false false []
    amq.rabbitmq.trace topic true false false []
    amq.rabbitmq.log topic true false false []
    amq.fanout fanout true false false []
    amq.headers headers true false false []
    direct true false false []
    amq.match headers true false false []

    Bindings on /:
    source_name source_kind destination_name destination_kind routing_key arguments
    exchange dc.data:durable:all queue dc.data:durable:all []
    dc.data:fanout exchange dc.data:durable:all queue []

    Consumers on /:

    Permissions on /:
    user configure write read
    guest .* .* .*

    End of server status report
    ...done.



    On Dec 28, 2011, at 9:22 AM, DawgTool wrote:

    RabbitMQ Stability Issues with large queue - 2011-12-28

    Hi All,

    I posted in the IRC channel a few nights ago, and they suggested that I bring this to the listserv.
    Hopefully can get some suggestions on how to keep my servers from crashing.
    Thanks,
    -- DawgTool

    Cluster Info:
    Cluster status of node 'dc001 at rmquat-m01' ...
    [{nodes,[{disc,['dc001 at rmquat-m04','dc001 at rmquat-m03','dc001 at rmquat-m02',
    'dc001 at rmquat-m01']}]},
    {running_nodes,['dc001 at rmquat-m04','dc001 at rmquat-m03','dc001 at rmquat-m02',
    'dc001 at rmquat-m01']}]

    Config Info:
    ==> enabled_plugins <==
    [rabbitmq_management,rabbitmq_management_agent,rabbitmq_management_visualiser].

    ==> rabbitmq.config <==
    [
    {rabbit, [{vm_memory_high_watermark, 0.6},
    {collect_statistics_interval, 5000},
    {hipe_compile, true}
    ]
    },
    {rabbitmq_management, [ {http_log_dir, "/data/rabbitmq/dc001/rabbit-mgmt"} ] },
    {rabbitmq_management_agent, [ {force_fine_statistics, true} ] }
    ].

    ==> rabbitmq-env.conf <==
    NODENAME=dc001
    BASE=/data/rabbitmq/dc001
    MNESIA_BASE=/data/rabbitmq/dc001/mnesia
    LOG_BASE=/data/rabbitmq/dc001/log
    SERVER_START_ARGS="-smp enable"


    IRC:
    [17:32] <dawgtool> background. doing some testing on 2.7.0 : 4 servers 2vcpu, 8gb ram, 80gb disc.
    [17:32] <dawgtool> cluser is setup all disc, currently one exchange durable fanout
    [17:33] <dawgtool> one queue also durable bind to the exchange.
    [17:33] <dawgtool> i'm pushing about 5M records, payload is ~500bytes each record
    [17:34] <dawgtool> rate is about 14k/s (which seems pretty slow)
    [17:35] <dawgtool> but my problem is, I'm testing a case where they consumers are busy or unavailable, so the queues would be filling up.
    [17:35] <dawgtool> even after slowing the publish rate to about 4k/s the mirrored queue does not complete on any of the clusters nodes other then master.
    [17:37] <dawgtool> memory seems to be the biggest issue here, as the servers will grow passed the high water mark, and eventually crash one at a time.
    [17:37] <dawgtool> once they are restarted, most servers in the cluster will have about 200k to 300k of messages in their queue
    [17:40] <dawgtool> so question is, why is so much memory being consumed (on disk these records are about 5.5GB) RabbitMQ pushes to 7.9 real, 11.9 virtual (swapping).
    [17:40] <dawgtool> why is the queue not stopping the publishers (RAM based clusters seem to stop the publisher until it can be spilled to disk)
    [17:41] <dawgtool> Why is mirroring unreliable in this test.
    [17:41] <dawgtool> ok, i'm done with the backgroud. =)
    [17:41] <dawgtool> lol
    [17:42] <antares_> having 300K messages in one queue will result in RAM consumption like this
    [17:42] <antares_> 30 queues with 10K is a better option
    [17:43] <antares_> I can't say for mirroring but my understanding is that mirroring cannot decrease RAM usage
    [17:43] <dawgtool> true, i need to make sure I don't loose any records, so disc with mirror
    [17:44] <dawgtool> does performance get that bad after 300k message in a queue?
    [17:44] <antares_> this kind of questions is better asked on the rabbitmq-discuss (mailing list)
    [17:45] <antares_> the exact number will vary but yes, having queues with excessive # of messages that are not consumed will result in more or less this behavior
    [17:45] <dawgtool> yea, just joined the mailing list, I can post it there. was hoping someone had a quick answer. =)
    [17:45] <antares_> each queue is backed by one Erlang process and Erlang VM GC releases process memory all at once
    [17:46] <dawgtool> even with the -smp set to true?
    [17:46] <antares_> so having large amount of messages in your queues impedes that
    [17:46] <antares_> dawgtool: I doubt that -smp affects GC behavior
    [17:46] <dawgtool> hmmm
    [17:47] <antares_> in this regard anyway, because there is still no shared heap
    [17:47] <antares_> but rabbitmq team members definitely know more than I do
    [17:47] <antares_> and they are all on rabbitmq-discuss
    [17:48] <dawgtool> ok, I'll give the mailing list a shot. 300k is going to be hard to live under, one of my current systems is doing server times that a second. =(
    [17:49] <dawgtool> which i was hoping to migrate to a more open system, at least parts of it. =(
    [17:49] <antares_> dawgtool: total # of messages is not the point
    [17:50] <antares_> the point is max # of messages in one queue
    [17:50] <antares_> you can have 10K queues and one app consuming messages from them all
    [17:50] <antares_> unless ordering is a serious concern for your case, it will work just as well as 1 queue
    [17:50] <dawgtool> yea, understand, but some consumers might crash, and I will get a backlog
    [17:51] <dawgtool> I need to make sure the MQ system can handle at least a 30 minute outage
    [17:51] <antares_> again, you will have the same problem with just 1 queue
    [17:51] <antares_> dawgtool: I see. And not lose anything?
    [17:51] <dawgtool> right, =(
    [17:51] <antares_> rabbitmq queues can have message TTL
    [17:52] <dawgtool> yea, I have TTL on metric collection consumers.. usually 6 seconds.
    [17:55] <dawgtool> in production, the idea would be to have two exchanges: input.exchange.fanout, metric.exchange.topic bind to input.exchange.fanout. queue.durable.mirror.prod bind to input.exchange.fanout no ttl, several queue.trans.nomirror.metric[1-x] ttl 6sec.
    [17:56] <dawgtool> the input.exchange.fanout will have a second and third queue eventually, which is why there are two exchanges.
    [17:57] <dawgtool> but the second and third will have a 30min ttl
    [18:01] <dawgtool> anyway, thanks for the info.. I'll shot an email out to the listserv and see if I get any bites. =)
    [18:01] <dawgtool> thanks again. =)
    [18:01] <antares_> dawgtool: no problem
  • Emile Joubert at Jan 3, 2012 at 2:10 pm
    Hi,

    The logfile indicates that the memory alarm has triggered, preventing
    further publishes. A broker in this state will only allow fetching. Is
    this what you see? What are the symptoms of the crash you observed?

    According to the report your one queue alone uses 3.7Gb while the total
    memory limit is only 4.8Gb. I would suggest lowering the
    vm_memory_high_watermark to the default value of 0.4 or lower so that
    the broker starts conserving memory sooner.

    If you see segfaults then consider disabling HiPE. This feature is
    experimental and could be the cause of your problem.


    On 28/12/11 22:20, DawgTool wrote:
    ==> dc001.log <==
    =INFO REPORT==== 28-Dec-2011::17:00:00 ===
    vm_memory_high_watermark set. Memory used:8662988728 allowed:5153960755

    =INFO REPORT==== 28-Dec-2011::17:00:00 ===
    alarm_handler: {set,{{vm_memory_high_watermark,'dc001 at rmquat-m01'},[]}}
    {vm_memory_high_watermark,0.5999999999767169},
    {vm_memory_limit,5153960755}]
    {vm_memory_high_watermark,0.6}]
    {rabbit, [{vm_memory_high_watermark, 0.6},
    {collect_statistics_interval, 5000},
    {hipe_compile, true}
    ]
    },
    {rabbitmq_management, [ {http_log_dir, "/data/rabbitmq/dc001/rabbit-mgmt"} ] },
    {rabbitmq_management_agent, [ {force_fine_statistics, true} ] }
    [17:32] <dawgtool> background. doing some testing on 2.7.0 : 4 servers 2vcpu, 8gb ram, 80gb disc.
    [17:32] <dawgtool> cluser is setup all disc, currently one exchange durable fanout
    [17:33] <dawgtool> one queue also durable bind to the exchange.
    [17:33] <dawgtool> i'm pushing about 5M records, payload is ~500bytes each record
    [17:34] <dawgtool> rate is about 14k/s (which seems pretty slow)
    [17:35] <dawgtool> but my problem is, I'm testing a case where they consumers are busy or unavailable, so the queues would be filling up.
    [17:35] <dawgtool> even after slowing the publish rate to about 4k/s the mirrored queue does not complete on any of the clusters nodes other then master.
    [17:37] <dawgtool> memory seems to be the biggest issue here, as the servers will grow passed the high water mark, and eventually crash one at a time.
    [17:37] <dawgtool> once they are restarted, most servers in the cluster will have about 200k to 300k of messages in their queue
    [17:40] <dawgtool> so question is, why is so much memory being consumed (on disk these records are about 5.5GB) RabbitMQ pushes to 7.9 real, 11.9 virtual (swapping).
    [17:40] <dawgtool> why is the queue not stopping the publishers (RAM based clusters seem to stop the publisher until it can be spilled to disk)
    [17:41] <dawgtool> Why is mirroring unreliable in this test.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedDec 28, '11 at 2:22p
activeJan 3, '12 at 2:10p
posts3
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

DawgTool: 2 posts Emile Joubert: 1 post

People

Translate

site design / logo © 2022 Grokbase