Hello!

(Reposted+edited since I forgot to join the list first)

I'm trying to move my actual (working) RabbitMQ node to another, less
loaded, server (actually another VM, hidden from internet, which will
also have a memcache).

So after setting up a barebone 64b Debian6 from a netinst and adding
RabbitMQ repo and installing it I got a working node.

Problem is, the management plugin seems to die, bringing down the
daemon with it after a couple of seconds (immediatly after start,
rabbitmq_ctl status does list plugins, two seconds later it's dead
already):

[... end of output launched from cli http://pastebin.com/rQGBDbF8 ]
broker running
** Found 0 name clashes in code paths
Erlang has closed
{"Kernel pid
terminated",application_controller,"{application_start_failure,rabbitmq_management,
{bad_return,{{rabbit_mgmt_app,start,[normal,[]]},{'EXIT',{timeout,
{gen_server,call,[rabbit_mochiweb_registry,
{add,rabbit_mgmt,#Fun<rabbit_mochiweb.2.78160903>,#Fun<rabbit_mochiweb.
5.23044403>,{[109,103,109,116],
[77,97,110,97,103,101,109,101,110,116,58,32,87,101,98,32,85,73]}}]}}}}}}"}
_____________
[... end of rabbit at mqcache.log ]
=INFO REPORT==== 18-Jul-2011::17:57:27 ===
Management agent started.

=INFO REPORT==== 18-Jul-2011::17:57:27 ===
Management plugin started.
HTTP API: http://mqcache:55672/api/
Management UI: http://mqcache:55672/mgmt/

=INFO REPORT==== 18-Jul-2011::17:57:32 ===
application: rabbitmq_management
exited: {bad_return,
{{rabbit_mgmt_app,start,[normal,[]]},
{'EXIT',
{timeout,
{gen_server,call,
[rabbit_mochiweb_registry,
{add,rabbit_mgmt,
#Fun<rabbit_mochiweb.2.78160903>,
#Fun<rabbit_mochiweb.5.23044403>,
{"mgmt","Management: Web UI"}}]}}}}}
type: permanent

=INFO REPORT==== 18-Jul-2011::17:57:32 ===
stopped TCP Listener on [::]:5672

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller)
({application_start_failure,rabbitmq_management,{bad_return,
{{rabbit_mgmt_app,start,[normal,[]]},{'EXIT',{timeout,{gen_server,call,
[rabbit_mochiweb_reg


All plugin files are there, I even compared the CRCs with my actual
RabbitMQ installation (which is another VM, just internet wise +
apache/mysql/etc...) . It's an independant node, no clustering,
replication, nothing special involved. Installing plugins one after
another boils it down to rabbitmq_management alone being the culprit.

As said, it's a fresh and updated Deb6 64b 2 CPUs, 2Gigs of RAM.
Erlang R14A (erts-5.8)
RabbitMQ 2.5.1
# md5sum *.ez
2602dd3df87fa4c7dedd6eb3192b0c19 amqp_client-2.5.1.ez
07f3b87ccb1d03d479a4277014a7e017 mochiweb-1.3-rmq2.5.1-git9a53dbd.ez
69aa2c17ebfff7bdf141d8df0b358631 rabbitmq_management-2.5.1.ez
4165b7d2aeeeeae9239b173b5e803c89 rabbitmq_management_agent-2.5.1.ez
0cd0b60bd8b9019fd9dbea08f667a697 rabbitmq_mochiweb-2.5.1.ez
2c7fddc210317a054c97b43e13d38ced webmachine-1.7.0-rmq2.5.1-
hg0c4b60a.ez

The only diff I can imagine is that no queues/exchanges have ever been
created yet, and there's already an admin user created on it.

Has anyone encountered this? Erlang is a bit chinese to me, I might
have missed an obvious message somewhere, but I can't figure it out
yet :-)

Thanks in advance,

JaXX
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110718/65e39e16/attachment.htm>

Search Discussions

  • Julien Banchet at Jul 18, 2011 at 8:24 pm
    release built from mercurial gives the same results, only ditching out
    the rabbitmq_managment plugin gets RabbitMQ to run (tested some
    twitter streams against it)... and I don't see any big differences
    between the one working perfectly and this one, could it be a secret
    dependency somewhere?

    I am ready to help as much as possible, and since the VM is empty, I
    could even give access to it if needed

    I could be the perfect noob at times, but I don't understand what's
    going on (and Erlang being total abscon to me I don't know how to
    debug it, even having played with ejabberd back in my teen years ;-) )
    Though I'm sure I hit a bug, I'm just scared to have missed something
    obvious and turn tomato-red-ashamed :-)
    On Jul 18, 4:39?pm, Julien Banchet wrote:
    Hello!
    [...]
  • Simon MacMullen at Jul 18, 2011 at 8:57 pm
    Hi Julien.
    On 18/07/2011 9:24PM, Julien Banchet wrote:
    release built from mercurial gives the same results, only ditching out
    the rabbitmq_managment plugin gets RabbitMQ to run (tested some
    twitter streams against it)... and I don't see any big differences
    between the one working perfectly and this one, could it be a secret
    dependency somewhere?
    I should hope not - the Debian rabbitmq-server package is supposed to
    pull in all the dependencies needed by all the plugins.
    I am ready to help as much as possible, and since the VM is empty, I
    could even give access to it if needed

    I could be the perfect noob at times, but I don't understand what's
    going on (and Erlang being total abscon to me I don't know how to
    debug it, even having played with ejabberd back in my teen years ;-) )
    Though I'm sure I hit a bug, I'm just scared to have missed something
    obvious and turn tomato-red-ashamed :-)
    This does look like a bug. Somehow the mgmt plugin is timing out trying
    to contact the web plugin registry. Odd. Do you have any unusual config?
    (Can you post your /etc/rabbitmq/rabbitmq.config if you have one?) Also,
    can you post or pastebin the rabbit at mqcache-sasl.log?

    Cheers, Simon
  • Julien Banchet at Jul 18, 2011 at 10:28 pm
    Hi Simon,
    On Jul 18, 5:57?pm, Simon MacMullen wrote:
    Hi Julien.

    On 18/07/2011 9:24PM, Julien Banchet wrote:
    [...]

    I should hope not - the Debian rabbitmq-server package is supposed to
    pull in all the dependencies needed by all the plugins.
    Yeah, I imagine so too :-) Plus, looks like it barely needs anything
    out of the erlang scope which is long time packaged with mainstream
    debian.

    I am ready to help as much as possible, and since the VM is empty, I
    could even give access to it if needed
    I could be the perfect noob at times, but I don't understand what's
    going on (and Erlang being total abscon to me I don't know how to
    debug it, even having played with ejabberd back in my teen years ;-) )
    Though I'm sure I hit a bug, I'm just scared to have missed something
    obvious and turn tomato-red-ashamed :-)
    This does look like a bug. Somehow the mgmt plugin is timing out trying
    to contact the web plugin registry. Odd. Do you have any unusual config?
    (Can you post your /etc/rabbitmq/rabbitmq.config if you have one?) Also,
    can you post or pastebin the rab... at mqcache-sasl.log?
    rabbitmq.config inexistant or with
    [
    {rabbit, [{tcp_listeners, [{"0.0.0.0", 5672}]}]}
    ].

    (tried in case it was related to networking and to simplify it... the
    VM (vmware) has two interfaces, eth0 on NAT, eth1 on HostOnly, no need
    for IPv6 nor bridging)

    Nothing special otherwise, debian 6.0.2.1 amd64 netinstall, selected
    only SSH server (the damn thing still puts Exim as an MTA, yuk :-) ),
    added "deb http://www.rabbitmq.com/debian/ testing main" to the
    sources, updated apt, install rabbitmq-server, wget'd the 6 plugin
    files, restarted, et voil? :-(


    PasteBined on http://pastebin.com/42Ptzq8R rabbit at mqcache-sasl.log +
    rabbit at mqcache.log + startup_log (loaded with the precompiled
    plugins)
    Cheers, Simon
    Thanks,

    Julien (JaXX)
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Jul 19, 2011 at 10:58 am

    On 18/07/11 23:28, Julien Banchet wrote:
    I am ready to help as much as possible, and since the VM is empty, I
    could even give access to it if needed
    Yeah, it may come to that. I'm afraid none of the logs say anything
    useful. I'll send you an ssh key via private mail if you don't mind.
    Nothing special otherwise, debian 6.0.2.1 amd64 netinstall, selected
    only SSH server (the damn thing still puts Exim as an MTA, yuk :-) ),
    added "deb http://www.rabbitmq.com/debian/ testing main" to the
    sources, updated apt, install rabbitmq-server, wget'd the 6 plugin
    files, restarted, et voil? :-(
    I've replicated this as best I can (VMware / Debian 6 / amd64 / minimal
    debs / 2 networks), no dice.

    Cheers, Simon
    --
    Simon MacMullen
    RabbitMQ, VMware
  • Julien Banchet at Jul 19, 2011 at 1:56 pm

    Le 19 juil. 2011 ? 07:58, Simon MacMullen a ?crit :
    On 18/07/11 23:28, Julien Banchet wrote:
    I am ready to help as much as possible, and since the VM is empty, I
    could even give access to it if needed
    Yeah, it may come to that. I'm afraid none of the logs say anything useful. I'll send you an ssh key via private mail if you don't mind.
    catted to authorized_keys everywhere needed :-)
    Nothing special otherwise, debian 6.0.2.1 amd64 netinstall, selected
    only SSH server (the damn thing still puts Exim as an MTA, yuk :-) ),
    added "deb http://www.rabbitmq.com/debian/ testing main" to the
    sources, updated apt, install rabbitmq-server, wget'd the 6 plugin
    files, restarted, et voil? :-(
    I've replicated this as best I can (VMware / Debian 6 / amd64 / minimal debs / 2 networks), no dice.
    I possibly messed up somehow then, but innocently enough to declare this "unclear" then; you'll probably understand the guts of Erlang and the management plugin better than me though...

    (btw, replaced it with the "visualizer" version I saw on the mailing list, same result, it dies the same way)

    Best,

    JaXX
  • Simon MacMullen at Jul 19, 2011 at 3:21 pm

    On 18/07/11 20:39, Julien Banchet wrote:
    Problem is, the management plugin seems to die, bringing down the
    daemon with it after a couple of seconds (immediatly after start,
    rabbitmq_ctl status does list plugins, two seconds later it's dead
    already):
    So this turns out to be a bug which can cause startup for the whole
    broker to fail when mochiweb takes too long (more than five seconds) to
    start an HTTP listening socket.

    On Julien's machine this was happening because mochiweb tries to look up
    "localhost" as an IPv6 address, and it was taking a while to time out.

    In case anybody else is bitten by this, you can download a version of
    rabbitmq_mochiweb which will wait longer than 5s from:

    http://www.rabbitmq.com/releases/plugins/v2.5.1/rabbitmq_mochiweb-2.5.1-registry-timeout-fix.ez

    ...although the "correct" fix is probably to have IPv6 working properly,
    or turned off.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Julien Banchet at Jul 19, 2011 at 3:50 pm

    Le 19 juil. 2011 ? 12:21, Simon MacMullen a ?crit :
    On 18/07/11 20:39, Julien Banchet wrote:
    Problem is, the management plugin seems to die, bringing down the
    daemon with it after a couple of seconds (immediatly after start,
    rabbitmq_ctl status does list plugins, two seconds later it's dead
    already):
    So this turns out to be a bug which can cause startup for the whole broker to fail when mochiweb takes too long (more than five seconds) to start an HTTP listening socket.

    On Julien's machine this was happening because mochiweb tries to look up "localhost" as an IPv6 address, and it was taking a while to time out.

    In case anybody else is bitten by this, you can download a version of rabbitmq_mochiweb which will wait longer than 5s from:

    http://www.rabbitmq.com/releases/plugins/v2.5.1/rabbitmq_mochiweb-2.5.1-registry-timeout-fix.ez

    ...although the "correct" fix is probably to have IPv6 working properly, or turned off.
    Hopefully you know it inside-out too, and being the only thing affected by an incomplete (though stock) v6 stack, on an isolated machine, I would have never figured that alone

    So as we say "Don't need it? Remove it!"
    echo net.ipv6.conf.all.disable_ipv6=1 > /etc/sysctl.d/noipv6.conf

    Thanks again,

    JaXX
    arg, missed the reply to all button

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJul 18, '11 at 7:39p
activeJul 19, '11 at 3:50p
posts8
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Julien Banchet: 5 posts Simon MacMullen: 3 posts

People

Translate

site design / logo © 2017 Grokbase