FAQ
When running a rabbit cluster, the first node that includes the plugin
starts up fine.

Any additional nodes crash on startup. Moving the plugin out of the plugins
folder allows rabbit to start fine. Running the agent alone is not a
problem.

I have seen this mentioned in the mailing lists since 2.4.1 came out, but I
have not seen any follow-up providing a fix. Any feedback welcome.

Thanks,

--

C. Ian Patton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/9cccc074/attachment.htm>

Search Discussions

  • Simon MacMullen at May 31, 2011 at 2:39 pm
    Hi Ian.
    On 31/05/11 15:22, Ian Patton wrote:
    When running a rabbit cluster, the first node that includes the plugin
    starts up fine.

    Any additional nodes crash on startup. Moving the plugin out of the
    plugins folder allows rabbit to start fine. Running the agent alone is
    not a problem.
    Are all these nodes on the same machine? If so, the problem is as
    described here:

    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010703.html

    If not, then what error messages are you seeing in the logs?

    Cheers, Simon
    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware
  • Ian Patton at May 31, 2011 at 2:54 pm
    sorry, forgot to loop the list in my last reply.

    I looked at http://www.rabbitmq.com/releases/plugins/v2.4.1/
    and there is a "-mem-fix" version. I tried this and it seems to do the same
    thing. Not sure that this version is for, but it is not linked from the main
    plugins page.

    Ian
    On Tue, May 31, 2011 at 10:50 AM, Simon MacMullen wrote:
    On 31/05/11 15:44, Ian Patton wrote:

    =SUPERVISOR REPORT==== 31-May-2011::14:41:44 ===
    Supervisor: {local,rabbit_sup}
    Context: child_terminated
    Reason: killed
    Offender: [{pid,<0.241.0>},
    {name,rabbit_mgmt_global_sup},
    {mfargs,{rabbit_mgmt_global_sup,start_link,[]}},
    {restart_type,transient},
    {shutdown,4294967295},
    {child_type,worker}]


    =SUPERVISOR REPORT==== 31-May-2011::14:41:44 ===
    Supervisor: {local,rabbit_sup}
    Context: shutdown
    Reason: reached_max_restart_intensity
    Offender: [{pid,<0.241.0>},
    {name,rabbit_mgmt_global_sup},
    {mfargs,{rabbit_mgmt_global_sup,start_link,[]}},
    {restart_type,transient},
    {shutdown,4294967295},
    {child_type,worker}]
    Ah. That is interesting. Not sure what the problem is yet, but this gives
    me something to get my teeth into.


    Cheers, Simon

    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

    --

    C. Ian Patton
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/9858b54d/attachment-0001.htm>
  • Simon MacMullen at May 31, 2011 at 2:58 pm

    On 31/05/11 15:54, Ian Patton wrote:
    sorry, forgot to loop the list in my last reply.

    I looked at http://www.rabbitmq.com/releases/plugins/v2.4.1/
    and there is a "-mem-fix" version. I tried this and it seems to do the
    same thing. Not sure that this version is for, but it is not linked from
    the main plugins page.
    That was fixing the bug reported here:

    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-April/012352.html

    but that's a different thing; you need a high rate of queue churn to see
    much chance of seeing it.

    Your problem hasn't been fixed yet I'm afraid, but now I have something
    to go on.

    Cheers, Simon

    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware
  • Ian Patton at May 31, 2011 at 4:08 pm
    I can create a virtual machine image and ssh it to some server for you to
    grab... if you have such a place to put it. If so, let me know which
    hypervisor you prefer (ESX, Xen, KVM, etc)

    Ian

    On Tue, May 31, 2011 at 11:57 AM, Simon MacMullen wrote:

    I don't suppose there's any chance you could give me access to this
    machine? I'm not having any luck reproducing this...

    Cheers, Simon

    On 31/05/11 15:44, Ian Patton wrote:

    The nodes are on separate machines.

    Here is the log:

    cat /var/log/rabbitmq/rabbit\@mc-messages2.log

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Limiting to approx 999900 file handles (899908 sockets)

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Memory limit set to 1584MB.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Management plugin upgraded statistics to fine.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Statistics database started.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    msg_store_transient: using rabbit_msg_store_ets_index to provide index

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    msg_store_persistent: using rabbit_msg_store_ets_index to provide index

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    started TCP Listener on [::]:5672

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    node 'rabbit at mc-messages1' up

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Management agent started.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Management plugin started.
    HTTP API: http://mc-messages2:55672/api/
    Management UI: http://mc-messages2:55672/mgmt/

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    stopped TCP Listener on [::]:5672

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    application: rabbit
    exited: shutdown
    type: permanent

    and here is the sasl log:

    cat rabbit\@mc-messages2-sasl.log

    =SUPERVISOR REPORT==== 31-May-2011::14:41:44 ===
    Supervisor: {local,rabbit_sup}
    Context: child_terminated
    Reason: killed
    Offender: [{pid,<0.241.0>},
    {name,rabbit_mgmt_global_sup},
    {mfargs,{rabbit_mgmt_global_sup,start_link,[]}},
    {restart_type,transient},
    {shutdown,4294967295},
    {child_type,worker}]


    =SUPERVISOR REPORT==== 31-May-2011::14:41:44 ===
    Supervisor: {local,rabbit_sup}
    Context: shutdown
    Reason: reached_max_restart_intensity
    Offender: [{pid,<0.241.0>},
    {name,rabbit_mgmt_global_sup},
    {mfargs,{rabbit_mgmt_global_sup,start_link,[]}},
    {restart_type,transient},
    {shutdown,4294967295},
    {child_type,worker}]

    --

    C. Ian Patton


    On Tue, May 31, 2011 at 10:39 AM, Simon MacMullen <simon at rabbitmq.com
    wrote:

    Hi Ian.


    On 31/05/11 15:22, Ian Patton wrote:

    When running a rabbit cluster, the first node that includes the
    plugin
    starts up fine.

    Any additional nodes crash on startup. Moving the plugin out of the
    plugins folder allows rabbit to start fine. Running the agent
    alone is
    not a problem.


    Are all these nodes on the same machine? If so, the problem is as
    described here:


    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010703.html

    If not, then what error messages are you seeing in the logs?

    Cheers, Simon
    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    <mailto:rabbitmq-discuss at lists.rabbitmq.com>

    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss





    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

    --

    C. Ian Patton
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/ebd1412f/attachment.htm>
  • Ian Patton at May 31, 2011 at 4:19 pm
    Does this help?

    mc-messages2:/var/log/rabbitmq # sudo -u rabbitmq rabbitmqctl status
    Status of node 'rabbit at mc-messages2' ...
    [{pid,23029},
    {running_applications,
    [{webmachine,"webmachine","1.7.0"},
    {rabbit_management_agent,"RabbitMQ Management Agent","2.4.1"},
    {amqp_client,"RabbitMQ AMQP Client","2.4.1"},
    {rabbit,"RabbitMQ","2.4.1"},
    {os_mon,"CPO CXC 138 46","2.2.5"},
    {sasl,"SASL CXC 138 11","2.1.9.3"},
    {rabbit_mochiweb,"RabbitMQ Mochiweb Embedding","2.4.1"},
    {mochiweb,"MochiMedia Web Server","1.3"},
    {inets,"INETS CXC 138 49","5.5.2"},
    {mnesia,"MNESIA CXC 138 12","4.4.17"},
    {stdlib,"ERTS CXC 138 10","1.17.3"},
    {kernel,"ERTS CXC 138 10","2.14.3"}]},
    {nodes,[{disc,['rabbit at mc-messages2','rabbit at mc-messages1']}]},
    {running_nodes,['rabbit at mc-messages1','rabbit at mc-messages2']}]
    ...done.

    mc-messages2:/var/log/rabbitmq # erl
    Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:4:4] [rq:4]
    [async-threads:0] [hipe] [kernel-poll:false]


    Ian
    On Tue, May 31, 2011 at 12:08 PM, Ian Patton wrote:

    I can create a virtual machine image and ssh it to some server for you to
    grab... if you have such a place to put it. If so, let me know which
    hypervisor you prefer (ESX, Xen, KVM, etc)

    Ian

    On Tue, May 31, 2011 at 11:57 AM, Simon MacMullen wrote:

    I don't suppose there's any chance you could give me access to this
    machine? I'm not having any luck reproducing this...

    Cheers, Simon

    On 31/05/11 15:44, Ian Patton wrote:

    The nodes are on separate machines.

    Here is the log:

    cat /var/log/rabbitmq/rabbit\@mc-messages2.log

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Limiting to approx 999900 file handles (899908 sockets)

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Memory limit set to 1584MB.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Management plugin upgraded statistics to fine.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Statistics database started.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    msg_store_transient: using rabbit_msg_store_ets_index to provide index

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    msg_store_persistent: using rabbit_msg_store_ets_index to provide index

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    started TCP Listener on [::]:5672

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    node 'rabbit at mc-messages1' up

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Management agent started.

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    Management plugin started.
    HTTP API: http://mc-messages2:55672/api/
    Management UI: http://mc-messages2:55672/mgmt/

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    stopped TCP Listener on [::]:5672

    =INFO REPORT==== 31-May-2011::14:41:44 ===
    application: rabbit
    exited: shutdown
    type: permanent

    and here is the sasl log:

    cat rabbit\@mc-messages2-sasl.log

    =SUPERVISOR REPORT==== 31-May-2011::14:41:44 ===
    Supervisor: {local,rabbit_sup}
    Context: child_terminated
    Reason: killed
    Offender: [{pid,<0.241.0>},
    {name,rabbit_mgmt_global_sup},
    {mfargs,{rabbit_mgmt_global_sup,start_link,[]}},
    {restart_type,transient},
    {shutdown,4294967295},
    {child_type,worker}]


    =SUPERVISOR REPORT==== 31-May-2011::14:41:44 ===
    Supervisor: {local,rabbit_sup}
    Context: shutdown
    Reason: reached_max_restart_intensity
    Offender: [{pid,<0.241.0>},
    {name,rabbit_mgmt_global_sup},
    {mfargs,{rabbit_mgmt_global_sup,start_link,[]}},
    {restart_type,transient},
    {shutdown,4294967295},
    {child_type,worker}]

    --

    C. Ian Patton


    On Tue, May 31, 2011 at 10:39 AM, Simon MacMullen <simon at rabbitmq.com
    wrote:

    Hi Ian.


    On 31/05/11 15:22, Ian Patton wrote:

    When running a rabbit cluster, the first node that includes the
    plugin
    starts up fine.

    Any additional nodes crash on startup. Moving the plugin out of
    the
    plugins folder allows rabbit to start fine. Running the agent
    alone is
    not a problem.


    Are all these nodes on the same machine? If so, the problem is as
    described here:


    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010703.html

    If not, then what error messages are you seeing in the logs?

    Cheers, Simon
    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    <mailto:rabbitmq-discuss at lists.rabbitmq.com>

    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss





    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

    --

    C. Ian Patton

    --

    C. Ian Patton
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/f3f0b483/attachment-0001.htm>
  • Ian Patton at May 31, 2011 at 5:46 pm
    That worked! Do you want to see any output?

    Ian
    On Tue, May 31, 2011 at 1:02 PM, Simon MacMullen wrote:
    On 31/05/11 17:35, Simon MacMullen wrote:
    On 31/05/11 17:08, Ian Patton wrote:

    I can create a virtual machine image and ssh it to some server for you
    to grab... if you have such a place to put it. If so, let me know which
    hypervisor you prefer (ESX, Xen, KVM, etc)
    I'd prefer ESX but anything really. I'm afraid I don't have a place for
    you to oput it; could you put it somewhere I could retrieve it?
    Actually, could you give this a try:


    http://www.rabbitmq.com/releases/plugins/test/rabbitmq-management-2.4.1-globals-debug.ez

    It contains some additional logging code, as well as a stab-in-the-dark
    attempt at a fix. Please install it instead of rabbitmq-management-2.4.1.ez
    on all the cluster nodes, and then send me the logs from *all* the nodes.

    Thanks for your help.


    Cheers, Simon

    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

    --

    C. Ian Patton
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/9193b394/attachment.htm>
  • Simon MacMullen at May 31, 2011 at 9:42 pm
    Yes please! That would help confirm exactly what's happening.

    Cheers, Simon
    --
    Sent from my Android phone with K-9 Mail. Please excuse my brevity.

    Ian Patton wrote:

    That worked! Do you want to see any output?


    Ian

    On Tue, May 31, 2011 at 1:02 PM, Simon MacMullen wrote:

    On 31/05/11 17:35, Simon MacMullen wrote:

    On 31/05/11 17:08, Ian Patton wrote:

    I can create a virtual machine image and ssh it to some server for you
    to grab... if you have such a place to put it. If so, let me know which
    hypervisor you prefer (ESX, Xen, KVM, etc)


    I'd prefer ESX but anything really. I'm afraid I don't have a place for
    you to oput it; could you put it somewhere I could retrieve it?


    Actually, could you give this a try:

    http://www.rabbitmq.com/releases/plugins/test/rabbitmq-management-2.4.1-globals-debug.ez

    It contains some additional logging code, as well as a stab-in-the-dark attempt at a fix. Please install it instead of rabbitmq-management-2.4.1.ez on all the cluster nodes, and then send me the logs from *all* the nodes.

    Thanks for your help.



    Cheers, Simon

    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware




    --

    C. Ian Patton


    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/8e87c7e3/attachment.htm>
  • Charly Hamy at May 31, 2011 at 2:46 pm
    Hi,

    you seem to be running your nodes on the same server. Maybe the problem is
    that only one management plugin can run on a single machine, because only on
    plugin may be listening to the port where the UI is avaiblable.

    You could try to load that plugin on one node only.

    For this, set different "*RABBITMQ_PLUGINS_DIR" for your nodes: for exemple
    you could have one "plugins" directory with the management plugin (and its
    dependencies) - used by one of your nodes - and another with the "management
    agent" plugin only - for the other nodes.*
    *
    *
    *Hope this will help.*
    *
    *
    *Regards,*
    *Charly*

    On Tue, May 31, 2011 at 4:22 PM, Ian Patton wrote:

    When running a rabbit cluster, the first node that includes the plugin
    starts up fine.

    Any additional nodes crash on startup. Moving the plugin out of the plugins
    folder allows rabbit to start fine. Running the agent alone is not a
    problem.

    I have seen this mentioned in the mailing lists since 2.4.1 came out, but I
    have not seen any follow-up providing a fix. Any feedback welcome.

    Thanks,

    --

    C. Ian Patton


    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110531/10356161/attachment.htm>
  • Simon MacMullen at May 31, 2011 at 2:49 pm

    On 31/05/11 15:46, Charly Hamy wrote:
    you seem to be running your nodes on the same server. Maybe the problem
    is that only one management plugin can run on a single machine, because
    only on plugin may be listening to the port where the UI is avaiblable.
    No, Ian has been in touch by private email and he's seeing a real
    problem. Not sure what the problem is yet but at least I have a
    meaningful error now.

    Cheers, Simon

    --
    Simon MacMullen
    Staff Engineer, RabbitMQ
    SpringSource, a division of VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedMay 31, '11 at 2:22p
activeMay 31, '11 at 9:42p
posts10
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase