I'm working on a system that currently has 2 rabbit nodes in a cluster. Ideally, one would be a disc node and one would be a ram node. The two nodes exist on individual machines on the same LAN segment. The ram node is on a machine that can come and go every now and again. Network connectivity between the nodes is solid.

On each machine, applications wishing to use rabbit talk to the local instance, secure in the knowledge that rabbit will get the data where it needs to go. I've a modest number of exchanges (say, 5ish), and a mix of long-lived and short-lived queues. There is a modest flow of messages (say, maybe 30-40 a minute, amounting to a couple meg at peak).

In a nutshell, the LATEST.LOG file on my ram clustered node grows, by all accounts, forever. I left it running over the weekend and it was well over 500meg. In the past I've let it run until it consumed all the space on the partition and the broker instance cratered. Running it as a ram node instead of a disc node obviates the problem, but is sub-optimal.

I'm using rabbit 2.5.0, though I've seen the problem on 2.4.0 and 2.4.1. The OS is linux based (2.6.32), and our Erlang version is R14A.

The googles (or my google-fu) seem to be silent beyond tweaking mnesia parameters that I don't fully understand. Any insight or assistance would be greatly appreciated!

Thanks!

--
Chris

Search Discussions

  • Matthew Sackman at Jun 20, 2011 at 10:30 pm

    On Mon, Jun 20, 2011 at 06:24:45PM -0400, Chris Madden wrote:
    In a nutshell, the LATEST.LOG file on my ram clustered node grows, by
    all accounts, forever. I left it running over the weekend and it was
    well over 500meg. In the past I've let it run until it consumed all
    the space on the partition and the broker instance cratered. Running
    it as a ram node instead of a disc node obviates the problem, but is
    sub-optimal.

    I'm using rabbit 2.5.0, though I've seen the problem on 2.4.0 and
    2.4.1. The OS is linux based (2.6.32), and our Erlang version is R14A.
    Very interesting. I've not heard of that one before and it certainly
    points to an mnesia issue. R14A is a beta release and I would strongly
    point an accusing finger at that first. Please try the latest Erlang
    R14B03.

    Matthew
  • Chris Madden at Jun 21, 2011 at 3:18 pm

    On Mon, Jun 20, 2011 at 06:24:45PM -0400, Chris Madden wrote:
    In a nutshell, the LATEST.LOG file on my ram clustered node grows, by
    all accounts, forever. I left it running over the weekend and it was
    well over 500meg. In the past I've let it run until it consumed all
    the space on the partition and the broker instance cratered. Running
    it as a ram node instead of a disc node obviates the problem, but is
    sub-optimal.

    I'm using rabbit 2.5.0, though I've seen the problem on 2.4.0 and
    2.4.1. The OS is linux based (2.6.32), and our Erlang version is R14A.
    Very interesting. I've not heard of that one before and it certainly
    points to an mnesia issue. R14A is a beta release and I would strongly
    point an accusing finger at that first. Please try the latest Erlang
    R14B03.

    Matthew
    I upgraded our erlang version to R14B03, but the LATEST.LOG still continues to be growing, albeit, more slowly perhaps?

    I'm kind of suspicious of one of the queue usage models I'm using. When one of the processes needs to make a one-off request (rpc like), it creates a queue dedicated to that request, and expects the reply to come back on that queue. After a timeout or the response is delivered, the queue is taken down. As a result, we have queues come and go a few times a minute... not really ideal, but it helps with debugging. Beyond this, I think the rabbit setup is pretty standard. Any ideas?


    Below is the status/cluster_status output, in case it points at something obvious:

    drone:0:~ # rabbitmqctl status
    Status of node rabbit at drone ...
    [{pid,12348},
    {running_applications,[{rabbit,"RabbitMQ","2.5.0"},
    {os_mon,"CPO CXC 138 46","2.2.6"},
    {sasl,"SASL CXC 138 11","2.1.9.4"},
    {mnesia,"MNESIA CXC 138 12","4.4.19"},
    {stdlib,"ERTS CXC 138 10","1.17.4"},
    {kernel,"ERTS CXC 138 10","2.14.4"}]},
    {os,{unix,linux}},
    {erlang_version,"Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
    {memory,[{total,30529744},
    {processes,10871688},
    {processes_used,10849544},
    {system,19658056},
    {atom,1114745},
    {atom_used,1105101},
    {binary,4144112},
    {code,11021554},
    {ets,1404144}]}]
    ...done.

    drone:0:~ # rabbitmqctl cluster_status
    Cluster status of node rabbit at drone ...
    [{nodes,[{disc,[rabbit at controller]},{ram,[rabbit at drone]}]},
    {running_nodes,[rabbit at controller,rabbit at drone]}]
    ...done.
  • Matthew Sackman at Jun 22, 2011 at 10:25 am
    Hi Chris,
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:
    I upgraded our erlang version to R14B03, but the LATEST.LOG still continues to be growing, albeit, more slowly perhaps? Curious.
    I'm kind of suspicious of one of the queue usage models I'm using. When one of the processes needs to make a one-off request (rpc like), it creates a queue dedicated to that request, and expects the reply to come back on that queue. After a timeout or the response is delivered, the queue is taken down. As a result, we have queues come and go a few times a minute... not really ideal, but it helps with debugging. Beyond this, I think the rabbit setup is pretty standard. Any ideas?
    How is the timeout done? Are you using our queue expiry timeout
    extension or some other mechanism? This might require some mnesia
    tuning, but I'm amazed if it does and you're the first person to run
    into this - it's such a common scenario I'm sure we'd have heard about
    it earlier.

    Matthew
  • Matthias Radestock at Jun 22, 2011 at 10:37 am
    Chris,
    On 22/06/11 11:25, Matthew Sackman wrote:
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:
    I upgraded our erlang version to R14B03, but the LATEST.LOG still
    continues to be growing, albeit, more slowly perhaps?
    Curious.
    Are you sure that you haven't perhaps got a growing list of queues?
    Check with 'rabbitmqctl list_queues'. Or, even better, use the fancy new
    'rabbitmqctl report' to produce a complete report of the server status
    and send that to us.
    I'm kind of suspicious of one of the queue usage models I'm using.
    When one of the processes needs to make a one-off request (rpc
    like), it creates a queue dedicated to that request, and expects
    the reply to come back on that queue.
    Generally, "queue per request" is a design anti pattern. Though if your
    processes are short-lived and don't make many requests it's ok.


    Matthias.
  • Chris Madden at Jun 22, 2011 at 2:11 pm
    Yeah, my queue list seems to be stable. Below I've attached my rabbitmqctl report output (which I had to change some names in, but should be the complete except for that).

    As for queue-per-request, it is on the agenda to get rid of... just need some more time.

    Currently, the ram node has been up for about 18 hours, LATEST.LOG is currently north of 45 megabytes

    Here is the rabbitmqctl report:

    Reporting server status on {{2011,6,22},{13,42,19}}
    Status of node rabbit at controller ...
    [{pid,1830},
    {running_applications,[{rabbit,"RabbitMQ","2.5.0"},
    {os_mon,"CPO CXC 138 46","2.2.6"},
    {sasl,"SASL CXC 138 11","2.1.9.4"},
    {mnesia,"MNESIA CXC 138 12","4.4.19"},
    {stdlib,"ERTS CXC 138 10","1.17.4"},
    {kernel,"ERTS CXC 138 10","2.14.4"}]},
    {os,{unix,linux}},
    {erlang_version,"Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
    {memory,[{total,71146096},
    {processes,25544656},
    {processes_used,25505984},
    {system,45601440},
    {atom,1416329},
    {atom_used,1393242},
    {binary,6167824},
    {code,10984170},
    {ets,25034304}]}]
    Cluster status of node rabbit at controller ...
    [{nodes,[{disc,[rabbit at controller]},{ram,[rabbit at drone]}]},
    {running_nodes,[rabbit at drone,rabbit at controller]}]
    Application environment of node rabbit at controller ...
    [{auth_backends,[rabbit_auth_backend_internal]},
    {auth_mechanisms,['PLAIN','AMQPLAIN']},
    {backing_queue_module,rabbit_variable_queue},
    {cluster_nodes,[]},
    {collect_statistics,none},
    {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
    {default_user,<<"guest">>},
    {default_user_is_admin,true},
    {default_vhost,<<"/">>},
    {delegate_count,16},
    {frame_max,131072},
    {included_applications,[]},
    {msg_store_file_size_limit,16777216},
    {msg_store_index_module,rabbit_msg_store_ets_index},
    {persister_hibernate_after,10000},
    {persister_max_wrap_entries,500},
    {queue_index_max_journal_entries,262144},
    {server_properties,[]},
    {ssl_listeners,[]},
    {ssl_options,[]},
    {tcp_listen_options,[binary,
    {packet,raw},
    {reuseaddr,true},
    {backlog,128},
    {nodelay,true},
    {exit_on_close,false}]},
    {tcp_listeners,[5672]},
    {trace_vhosts,[]},
    {vm_memory_high_watermark,0.08}]
    Status of node rabbit at drone ...
    [{pid,1816},
    {running_applications,[{rabbit,"RabbitMQ","2.5.0"},
    {os_mon,"CPO CXC 138 46","2.2.6"},
    {sasl,"SASL CXC 138 11","2.1.9.4"},
    {mnesia,"MNESIA CXC 138 12","4.4.19"},
    {stdlib,"ERTS CXC 138 10","1.17.4"},
    {kernel,"ERTS CXC 138 10","2.14.4"}]},
    {os,{unix,linux}},
    {erlang_version,"Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]\n"},
    {memory,[{total,30815144},
    {processes,11450224},
    {processes_used,11427656},
    {system,19364920},
    {atom,1334153},
    {atom_used,1326101},
    {binary,3202208},
    {code,10984170},
    {ets,1866904}]}]
    Cluster status of node rabbit at drone ...
    [{nodes,[{disc,[rabbit at controller]},{ram,[rabbit at drone]}]},
    {running_nodes,[rabbit at controller,rabbit at drone]}]
    Application environment of node rabbit at drone ...
    [{auth_backends,[rabbit_auth_backend_internal]},
    {auth_mechanisms,['PLAIN','AMQPLAIN']},
    {backing_queue_module,rabbit_variable_queue},
    {cluster_nodes,[]},
    {collect_statistics,none},
    {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
    {default_user,<<"guest">>},
    {default_user_is_admin,true},
    {default_vhost,<<"/">>},
    {delegate_count,16},
    {frame_max,131072},
    {included_applications,[]},
    {msg_store_file_size_limit,16777216},
    {msg_store_index_module,rabbit_msg_store_ets_index},
    {persister_hibernate_after,10000},
    {persister_max_wrap_entries,500},
    {queue_index_max_journal_entries,262144},
    {server_properties,[]},
    {ssl_listeners,[]},
    {ssl_options,[]},
    {tcp_listen_options,[binary,
    {packet,raw},
    {reuseaddr,true},
    {backlog,128},
    {nodelay,true},
    {exit_on_close,false}]},
    {tcp_listeners,[5672]},
    {trace_vhosts,[]},
    {vm_memory_high_watermark,0.14}]
    Connections:
    pid address port peer_address peer_port ssl peer_cert_subject peer_cert_issuer peer_cert_validity auth_mechanism ssl_protocol ssl_key_exchange ssl_cipher ssl_hash protocol user vhost timeout frame_max client_properties recv_oct recv_cnt send_oct send_cnt send_pend state channels
    <rabbit at controller.2.1597.6> 127.0.0.1 5672 127.0.0.1 37268 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 679040714 480320 384 4 0 running 1
    <rabbit at controller.2.968.6> 127.0.0.1 5672 127.0.0.1 34908 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 93883820 402448 73175294 384910 0 running 0
    <rabbit at controller.2.937.6> 127.0.0.1 5672 127.0.0.1 34904 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 105280 452 384 4 0 running 1
    <rabbit at controller.2.1416.6> 127.0.0.1 5672 127.0.0.1 37264 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 388 4 384 4 0 running 1
    <rabbit at controller.2.1137.6> 127.0.0.1 5672 127.0.0.1 37254 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 74069739 158463 83606920 134436 0 running 1
    <rabbit at controller.2.1369.6> 127.0.0.1 5672 127.0.0.1 37261 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 6095432 33922 9124252 36203 0 running 0
    <rabbit at controller.2.13952.0> 127.0.0.1 5672 127.0.0.1 58419 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"platform","Java"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"copyright","Copyright (C) 2007-2008 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd."},{"version","2.2.0"}] 3939165 21115 14575925 19596 0 running 0
    <rabbit at controller.2.1075.6> 127.0.0.1 5672 127.0.0.1 34918 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 388 5 384 4 0 running 1
    <rabbit at controller.2.4001.6> 127.0.0.1 5672 127.0.0.1 38002 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 437 5 1432152649 24684 0 running 1
    <rabbit at controller.2.941.6> 127.0.0.1 5672 127.0.0.1 34905 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 643 5 384 4 0 running 1
    <rabbit at controller.2.1420.6> 127.0.0.1 5672 127.0.0.1 37265 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 432 6 430 5 0 running 1
    <rabbit at controller.2.1295.6> 127.0.0.1 5672 127.0.0.1 37260 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 245539007 801610 2465991959 754123 0 running 0
    <rabbit at controller.2.1144.6> 127.0.0.1 5672 127.0.0.1 37255 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 8132196 13220 4287324 10869 0 running 1
    <rabbit at controller.2.1082.6> 127.0.0.1 5672 127.0.0.1 34919 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 435 6 430 5 0 running 1
    <rabbit at controller.2.1050.6> 127.0.0.1 5672 127.0.0.1 34915 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 30929757 23307 384 4 0 running 1
    <rabbit at controller.2.980.6> 127.0.0.1 5672 127.0.0.1 34909 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 1426635583 988608 4662902 10098 0 running 1
    <rabbit at controller.2.1395.6> 127.0.0.1 5672 127.0.0.1 37262 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 12306728 57153 10270868 54363 0 running 0
    <rabbit at controller.2.933.6> 127.0.0.1 5672 127.0.0.1 34903 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 927 10 400 5 0 running 2
    <rabbit at controller.2.5492.6> 127.0.0.1 5672 127.0.0.1 38182 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 1542496 8532 1624362 9033 0 running 0
    <rabbit at controller.2.998.6> 127.0.0.1 5672 127.0.0.1 34910 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 901791 3488 384 4 0 running 1
    <rabbit at controller.2.1054.6> 127.0.0.1 5672 127.0.0.1 34916 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 440 6 2086512417 35278 0 running 1
    <rabbit at drone.3.396.0> 127.0.0.1 5672 127.0.0.1 21308 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 7589272 39572 1714718956 56320 0 running 0
    <rabbit at drone.3.532.0> 127.0.0.1 5672 127.0.0.1 21314 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 388 4 384 4 0 running 1
    <rabbit at drone.3.550.0> 127.0.0.1 5672 127.0.0.1 21317 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 1684538810 1169551 384 4 0 running 1
    <rabbit at drone.3.336.0> 127.0.0.1 5672 127.0.0.1 21302 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 690 7 384 4 0 running 1
    <rabbit at drone.3.554.0> 127.0.0.1 5672 127.0.0.1 21318 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 437 5 335809059 8567 0 running 1
    <rabbit at drone.3.388.0> 127.0.0.1 5672 127.0.0.1 21307 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 1554367 6720 2466953 6422 0 running 0
    <rabbit at drone.3.324.0> 127.0.0.1 5672 127.0.0.1 21299 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 3161712253 2411550 253176665 268551 0 running 1
    <rabbit at drone.3.328.0> 127.0.0.1 5672 127.0.0.1 21300 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 53151 195 384 4 0 running 1
    <rabbit at drone.3.528.0> 127.0.0.1 5672 127.0.0.1 21313 false PLAIN {0,9,1} u / 0 131072 [{"product","RabbitMQ"},{"information","Licensed under the MPL. See http://www.rabbitmq.com/"},{"platform","Java"},{"capabilities",[{"exchange_exchange_bindings",true},{"consumer_cancel_notify",true},{"basic.nack",true},{"publisher_confirms",true}]},{"copyright","Copyright (C) 2007-2011 VMware, Inc."},{"version","2.5.0"}] 432 5 430 5 0 running 1
    Channels:
    pid connection number user vhost transactional confirm consumer_count messages_unacknowledged messages_unconfirmed acks_uncommitted prefetch_count client_flow_blocked
    <rabbit at drone.3.340.0> <rabbit at drone.3.324.0> 1 u / false false 1 0 0 0 0 false
    <rabbit at drone.3.342.0> <rabbit at drone.3.328.0> 1 u / false false 0 0 0 0 0 false
    <rabbit at drone.3.348.0> <rabbit at drone.3.336.0> 1 u / false false 0 0 0 0 0 false
    <rabbit at drone.3.535.0> <rabbit at drone.3.528.0> 1 u / false false 1 0 0 0 0 false
    <rabbit at drone.3.542.0> <rabbit at drone.3.532.0> 1 u / false false 0 0 0 0 0 false
    <rabbit at drone.3.557.0> <rabbit at drone.3.550.0> 1 u / false false 0 0 0 0 0 false
    <rabbit at drone.3.560.0> <rabbit at drone.3.554.0> 1 u / false false 1 0 0 0 0 false
    Queues on /:
    pid name durable auto_delete arguments owner_pid exclusive_consumer_pid exclusive_consumer_tag messages_ready messages_unacknowledged messages consumers memory backing_queue_status
    <rabbit at controller.2.665.0> broadcaster1.module1 false false [{"x-message-ttl",3600000}] 0 0 0 1 68192 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,50}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.1145.6> controllerServiceQueue false false [] 0 0 0 1 34744 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.1149.6> exchange2ServiceQueue false false [] 0 0 0 1 68192 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at drone.3.349.0> exchange1ServiceQueue false false [] 0 0 0 1 21968 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.639.0> broadcaster2.module1 false false [{"x-message-ttl",3600000}] 0 0 0 1 21968 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.664.0> broadcaster1.module3 false false [{"x-message-ttl",3600000}] 0 0 0 1 68192 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,27}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.666.0> broadcaster1.module2 false false [{"x-message-ttl",3600000}] 0 0 0 1 29944 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.984.6> exchange3ServiceQueue false false [] 0 0 0 1 21968 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.640.0> broadcaster2.module3 false false [{"x-message-ttl",3600000}] 0 0 0 1 21968 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    <rabbit at controller.2.641.0> broadcaster2.module2 false false [{"x-message-ttl",3600000}] 0 0 0 1 22048 [{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {outstanding_txns,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {ram_index_count,0}, {next_seq_id,0}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]
    Exchanges on /:
    name type durable auto_delete internal arguments
    amq.direct direct true false false []
    amq.topic topic true false false []
    appController topic false true false []
    broadcaster1 topic false false false []
    amq.rabbitmq.trace topic true false false []
    exchange1 topic false true false []
    amq.rabbitmq.log topic true false false []
    amq.fanout fanout true false false []
    amq.headers headers true false false []
    exchange2 topic false true false []
    direct true false false []
    amq.match headers true false false []
    exchange3 topic false true false []
    broadcaster2 topic false false false []
    Bindings on /:
    source_name source_kind destination_name destination_kind routing_key arguments
    exchange exchange3ServiceQueue queue exchange3ServiceQueue []
    exchange exchange2ServiceQueue queue exchange2ServiceQueue []
    exchange controllerServiceQueue queue controllerServiceQueue []
    exchange exchange1ServiceQueue queue exchange1ServiceQueue []
    exchange broadcaster1.module1 queue broadcaster1.module1 []
    exchange broadcaster1.module2 queue broadcaster1.module2 []
    exchange broadcaster1.module3 queue broadcaster1.module3 []
    exchange broadcaster2.module1 queue broadcaster2.module1 []
    exchange broadcaster2.module2 queue broadcaster2.module2 []
    exchange broadcaster2.module3 queue broadcaster2.module3 []
    appController exchange controllerServiceQueue queue controller.request []
    exchange2 exchange exchange2ServiceQueue queue exchange2.request []
    exchange1 exchange exchange1ServiceQueue queue exchange1.request []
    broadcaster1 exchange broadcaster1.module1 queue values.basedata []
    broadcaster1 exchange broadcaster1.module3 queue values.basedata []
    broadcaster1 exchange broadcaster1.module1 queue values.calculated_flush []
    broadcaster1 exchange broadcaster1.module2 queue values.calculated_flush []
    broadcaster1 exchange broadcaster1.module1 queue values.calculated_intermediate []
    broadcaster1 exchange broadcaster1.module2 queue values.pending_flush []
    broadcaster1 exchange broadcaster1.module3 queue values.pending_flush []
    broadcaster1 exchange broadcaster1.module3 queue values.pending_intermediate []
    broadcaster2 exchange broadcaster2.module2 queue system.options []
    broadcaster2 exchange broadcaster2.module3 queue system.options []
    exchange3 exchange exchange3ServiceQueue queue module3.request []
    Consumers on /:
    queue_name channel_pid consumer_tag ack_required
    broadcaster1.module1 <rabbit at controller.2.4004.6> amq.ctag-PVN5XyJ4QPorqqyE/5HnBA== false
    controllerServiceQueue <rabbit at controller.2.1140.6> amq.ctag-XxBFNOFkhrDcPbtptZ7Nvw== false
    exchange2ServiceQueue <rabbit at controller.2.1148.6> amq.ctag-dqh1qbfXhYypDRv8X67CCQ== false
    exchange1ServiceQueue <rabbit at drone.3.340.0> amq.ctag-3ZofayOMYay35WDNCKrH9A== false
    broadcaster2.module1 <rabbit at controller.2.1423.6> amq.ctag-TIPAseQ+E8s/rW11C4WJ6Q== false
    broadcaster1.module3 <rabbit at controller.2.1060.6> amq.ctag-ASACcGoydZ+UfZhdYE8IQA== false
    broadcaster1.module2 <rabbit at drone.3.560.0> amq.ctag-VGPw77M0WptkPvwBO519kQ== false
    exchange3ServiceQueue <rabbit at controller.2.983.6> amq.ctag-t3pbYjoIx4fjZ/ep8KFkyA== false
    broadcaster2.module3 <rabbit at controller.2.1085.6> amq.ctag-kIkZatK5xzXpcy8M2FL0nw== false
    broadcaster2.module2 <rabbit at drone.3.535.0> amq.ctag-6bcgsQVrQGK/hP9kIMbaBw== false
    Permissions on /:
    user configure write read
    u .* .* .*
    End of server status report
    ...done.

    --
    Chris Madden
    On Wednesday, June 22, 2011 at 6:37 AM, Matthias Radestock wrote:

    Chris,
    On 22/06/11 11:25, Matthew Sackman wrote:
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:
    I upgraded our erlang version to R14B03, but the LATEST.LOG still
    continues to be growing, albeit, more slowly perhaps?
    Curious.
    Are you sure that you haven't perhaps got a growing list of queues?
    Check with 'rabbitmqctl list_queues'. Or, even better, use the fancy new
    'rabbitmqctl report' to produce a complete report of the server status
    and send that to us.
    I'm kind of suspicious of one of the queue usage models I'm using.
    When one of the processes needs to make a one-off request (rpc
    like), it creates a queue dedicated to that request, and expects
    the reply to come back on that queue.
    Generally, "queue per request" is a design anti pattern. Though if your
    processes are short-lived and don't make many requests it's ok.


    Matthias.
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com (mailto:rabbitmq-discuss at lists.rabbitmq.com)
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Chris Madden at Jun 28, 2011 at 1:55 am
    I've whipped up a pair of python scripts that demonstrate the issue.
    They're using the queue-per-request anti-pattern, as that is what my
    production code is using (if I can only get more time...)

    My test environment is 2 ubuntu 11.04 server boxes with erlang R14b03. I
    have no plugins on a clean install of rabbit 2.5.0. There are 2 nodes in
    my cluster, A and B. A is a disc node, B is a ram node. I run echo.py on
    the ram node, and spew.py on the disc node. Each python script connects to
    its local broker. I'm using python-amqplib 0.6.1-1 for these scripts,
    though my production code is java based. The "guest" account is set as an
    administrator on vhost /. On my setup, the LATEST.LOG grows decently fast,
    maybe 15 meg an hour.

    If there is something pathological in my scripts, I'd love to hear it
    (beyond the aforementioned anti-pattern)... it is entirely possible I'm not
    releasing a resource I guess. Also, I'm using basic_get() as it avoids the
    complexity of threading logic in the example, in production I use consumers.

    tarball md5 is 1591ebbb1e9cffa06e6aec1c1ecb7b1d



    On Wed, Jun 22, 2011 at 6:37 AM, Matthias Radestock
    wrote:
    Chris,

    On 22/06/11 11:25, Matthew Sackman wrote:
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:

    I upgraded our erlang version to R14B03, but the LATEST.LOG still
    continues to be growing, albeit, more slowly perhaps?
    Curious.
    Are you sure that you haven't perhaps got a growing list of queues? Check
    with 'rabbitmqctl list_queues'. Or, even better, use the fancy new
    'rabbitmqctl report' to produce a complete report of the server status and
    send that to us.


    I'm kind of suspicious of one of the queue usage models I'm using.
    When one of the processes needs to make a one-off request (rpc
    like), it creates a queue dedicated to that request, and expects
    the reply to come back on that queue.
    Generally, "queue per request" is a design anti pattern. Though if your
    processes are short-lived and don't make many requests it's ok.


    Matthias.

    ______________________________**_________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
    https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110627/b49fdf42/attachment-0001.htm>
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: repro.tar.gz
    Type: application/x-gzip
    Size: 952 bytes
    Desc: not available
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110627/b49fdf42/attachment-0001.bin>
  • Matthias Radestock at Jun 28, 2011 at 4:18 am
    Chris,
    On 28/06/11 02:55, Chris Madden wrote:
    I've whipped up a pair of python scripts that demonstrate the issue.
    [...] the LATEST.LOG grows decently fast, maybe 15 meg an hour.
    I can reproduce this.
    If there is something pathological in my scripts, I'd love to hear it
    (beyond the aforementioned anti-pattern)...
    There is nothing pathological in the scripts, though they are an
    exercise in AMQP anti-patterns ;) (you really shouldn't be using a topic
    exchange; a direct exchange, in fact the default exchange, should do
    just fine; take a look at
    http://www.rabbitmq.com/tutorials/tutorial-six-python.html when you have
    a spare moment).

    I've filed a bug. My investigation so far indicates that this may well
    be a problem in mnesia. We have a somewhat unusual configuration in that
    that our 'ram' nodes aren't really ram nodes in the mnesia sense - they
    are fully-fledged disk nodes which happen to have no local disk copies
    of tables. It is conceivable that this induces a faulty code path in the
    log handling code.

    Until we fix that... I am pretty sure that in your setup the disk usage
    growth rate will be proportional to the queue creation rate. So if you
    eliminated the "one queue per request" anti-pattern that would probably
    give you plenty of headroom before bumping into any disk space limits.

    Thanks for reporting this problem and taking the time constructing a
    reproducible test. Much appreciated.


    Regards,

    Matthias.
  • Alexandru Scvorţov at Jul 29, 2011 at 1:49 pm
    Chris,

    A fix for this has landed on default earlier today and should be in the
    next release.

    Basically, we re-enabled disk-less nodes. So, 'ram' nodes will not have
    any files on disk anymore.

    Thanks for reporting the problem.

    Cheers,
    Alex
    On Mon, Jun 27, 2011 at 09:55:26PM -0400, Chris Madden wrote:
    I've whipped up a pair of python scripts that demonstrate the issue.
    They're using the queue-per-request anti-pattern, as that is what my
    production code is using (if I can only get more time...)

    My test environment is 2 ubuntu 11.04 server boxes with erlang R14b03. I
    have no plugins on a clean install of rabbit 2.5.0. There are 2 nodes in
    my cluster, A and B. A is a disc node, B is a ram node. I run echo.py on
    the ram node, and spew.py on the disc node. Each python script connects to
    its local broker. I'm using python-amqplib 0.6.1-1 for these scripts,
    though my production code is java based. The "guest" account is set as an
    administrator on vhost /. On my setup, the LATEST.LOG grows decently fast,
    maybe 15 meg an hour.

    If there is something pathological in my scripts, I'd love to hear it
    (beyond the aforementioned anti-pattern)... it is entirely possible I'm not
    releasing a resource I guess. Also, I'm using basic_get() as it avoids the
    complexity of threading logic in the example, in production I use consumers.

    tarball md5 is 1591ebbb1e9cffa06e6aec1c1ecb7b1d



    On Wed, Jun 22, 2011 at 6:37 AM, Matthias Radestock
    wrote:
    Chris,

    On 22/06/11 11:25, Matthew Sackman wrote:
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:

    I upgraded our erlang version to R14B03, but the LATEST.LOG still
    continues to be growing, albeit, more slowly perhaps?
    Curious.
    Are you sure that you haven't perhaps got a growing list of queues? Check
    with 'rabbitmqctl list_queues'. Or, even better, use the fancy new
    'rabbitmqctl report' to produce a complete report of the server status and
    send that to us.


    I'm kind of suspicious of one of the queue usage models I'm using.
    When one of the processes needs to make a one-off request (rpc
    like), it creates a queue dedicated to that request, and expects
    the reply to come back on that queue.
    Generally, "queue per request" is a design anti pattern. Though if your
    processes are short-lived and don't make many requests it's ok.


    Matthias.

    ______________________________**_________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
    https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Alexandru Scvortov at Jul 29, 2011 at 1:53 pm

    Basically, we re-enabled disk-less nodes. So, 'ram' nodes will not have
    any files on disk anymore.
    'Ram' nodes will not have any *mnesia* files on disk anymore. Message
    store files will still be there.

    Alex
    On Fri, Jul 29, 2011 at 02:49:54PM +0100, Alexandru Scvor?ov wrote:
    Chris,

    A fix for this has landed on default earlier today and should be in the
    next release.

    Basically, we re-enabled disk-less nodes. So, 'ram' nodes will not have
    any files on disk anymore.

    Thanks for reporting the problem.

    Cheers,
    Alex
    On Mon, Jun 27, 2011 at 09:55:26PM -0400, Chris Madden wrote:
    I've whipped up a pair of python scripts that demonstrate the issue.
    They're using the queue-per-request anti-pattern, as that is what my
    production code is using (if I can only get more time...)

    My test environment is 2 ubuntu 11.04 server boxes with erlang R14b03. I
    have no plugins on a clean install of rabbit 2.5.0. There are 2 nodes in
    my cluster, A and B. A is a disc node, B is a ram node. I run echo.py on
    the ram node, and spew.py on the disc node. Each python script connects to
    its local broker. I'm using python-amqplib 0.6.1-1 for these scripts,
    though my production code is java based. The "guest" account is set as an
    administrator on vhost /. On my setup, the LATEST.LOG grows decently fast,
    maybe 15 meg an hour.

    If there is something pathological in my scripts, I'd love to hear it
    (beyond the aforementioned anti-pattern)... it is entirely possible I'm not
    releasing a resource I guess. Also, I'm using basic_get() as it avoids the
    complexity of threading logic in the example, in production I use consumers.

    tarball md5 is 1591ebbb1e9cffa06e6aec1c1ecb7b1d



    On Wed, Jun 22, 2011 at 6:37 AM, Matthias Radestock
    wrote:
    Chris,

    On 22/06/11 11:25, Matthew Sackman wrote:
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:

    I upgraded our erlang version to R14B03, but the LATEST.LOG still
    continues to be growing, albeit, more slowly perhaps?
    Curious.
    Are you sure that you haven't perhaps got a growing list of queues? Check
    with 'rabbitmqctl list_queues'. Or, even better, use the fancy new
    'rabbitmqctl report' to produce a complete report of the server status and
    send that to us.


    I'm kind of suspicious of one of the queue usage models I'm using.
    When one of the processes needs to make a one-off request (rpc
    like), it creates a queue dedicated to that request, and expects
    the reply to come back on that queue.
    Generally, "queue per request" is a design anti pattern. Though if your
    processes are short-lived and don't make many requests it's ok.


    Matthias.

    ______________________________**_________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
    https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Chris Madden at Jun 22, 2011 at 2:14 pm

    On Wed, Jun 22, 2011 at 6:25 AM, Matthew Sackman wrote:

    Hi Chris,
    On Tue, Jun 21, 2011 at 11:18:36AM -0400, Chris Madden wrote:
    I upgraded our erlang version to R14B03, but the LATEST.LOG still
    continues to be growing, albeit, more slowly perhaps?

    Curious.
    I'm kind of suspicious of one of the queue usage models I'm using. When
    one of the processes needs to make a one-off request (rpc like), it creates
    a queue dedicated to that request, and expects the reply to come back on
    that queue. After a timeout or the response is delivered, the queue is taken
    down. As a result, we have queues come and go a few times a minute... not
    really ideal, but it helps with debugging. Beyond this, I think the rabbit
    setup is pretty standard. Any ideas?

    How is the timeout done? Are you using our queue expiry timeout
    extension or some other mechanism? This might require some mnesia
    tuning, but I'm amazed if it does and you're the first person to run
    into this - it's such a common scenario I'm sure we'd have heard about
    it earlier.

    I use the rabbit max message lifetime option on some queues, but in general,
    the timeout is in the application layer, where if a response isn't received
    in a certain amount of time, the application assumes it is lost
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110622/2296059e/attachment.htm>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJun 20, '11 at 10:24p
activeJul 29, '11 at 1:53p
posts11
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase