I upgraded from 2.6.0 to 2.7.0 using the .deb from the rabbit website.
(64bit debian)
Enabled management_plugin using rabbitmq-plugins as per docs, crashed
with the following on startup (pasted below).
Same crash with just rabbitmq_management_agent plugin enabled.

Search Discussions

  • Matthias Radestock at Nov 11, 2011 at 10:42 am
    Richard,
    On 10/11/11 16:15, Richard Jones wrote:
    I upgraded from 2.6.0 to 2.7.0 using the .deb from the rabbit website.
    (64bit debian)
    Enabled management_plugin using rabbitmq-plugins as per docs, crashed
    with the following on startup (pasted below).
    Same crash with just rabbitmq_management_agent plugin enabled.

    From what i can see, the get_used_fd function just tries to ls a
    directory in proc, not sure what that would fail, perms seem ok.
    Permissions is indeed the likely cause.

    Try running rabbit w/o the management plug-in enabled and then do an 'ls
    /proc/<rabbit-process-id>/fd' as the 'rabbitmq' user.

    Matthias.
  • Richard Jones at Nov 11, 2011 at 11:07 am
    The /proc/<pid> dir is owned by rabbitmq, but the /proc/<pid>/fd dir
    is owned (and only readable) by root, along with a bunch of other
    stuff in /proc/<pid>.

    I start rabbit using "sudo /etc/init.d/rabbitmq-server start"

    Did something change in how rabbit setuids or something since 2.6? It
    worked fine prior to installing the new deb.

    Regards,
    RJ

    On 11 November 2011 10:42, Matthias Radestock wrote:
    Richard,
    On 10/11/11 16:15, Richard Jones wrote:

    I upgraded from 2.6.0 to 2.7.0 using the .deb from the rabbit website.
    (64bit debian)
    Enabled management_plugin using rabbitmq-plugins as per docs, crashed
    with the following on startup (pasted below).
    Same crash with just rabbitmq_management_agent plugin enabled.

    ?From what i can see, the get_used_fd function just tries to ls a
    directory in proc, not sure what that would fail, perms seem ok.
    Permissions is indeed the likely cause.

    Try running rabbit w/o the management plug-in enabled and then do an 'ls
    /proc/<rabbit-process-id>/fd' as the 'rabbitmq' user.

    Matthias.
  • Simon MacMullen at Nov 11, 2011 at 11:49 am

    On 11/11/11 11:07, Richard Jones wrote:
    The /proc/<pid> dir is owned by rabbitmq, but the /proc/<pid>/fd dir
    is owned (and only readable) by root, along with a bunch of other
    stuff in /proc/<pid>.
    Hmm. Needless to say, that's not what I'm seeing here. I'm on Ubuntu
    10.04 - you?
    I start rabbit using "sudo /etc/init.d/rabbitmq-server start"

    Did something change in how rabbit setuids or something since 2.6? It
    worked fine prior to installing the new deb.
    2.6.1 did something like 'setsid sh -c "/usr/sbin/rabbitmq-server"'
    2.7.0 does 'setsid /usr/sbin/rabbitmq-server'

    (in both cases with come redirection and environment munging.)

    This doesn't feel like something that could cause this failure though.

    I'll patch rabbit_management_agent to fall back to lsof if it can't open
    /proc/<pid>/fd, but I'd like to find out why you're seeing this - I've
    googled a bit but can't see anything that would indicate why permissions
    are as they are.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Richard Jones at Nov 11, 2011 at 12:19 pm
    I'm running debian lenny: 2.6.26-2-amd64 #1 SMP Thu Nov 25 04:30:55
    UTC 2010 x86_64 GNU/Linux

    Same if I start rabbit manually, in the foreground, like so (as per
    /usr/sbin/rabbitmq-server):
    su rabbitmq -s /bin/sh -c "/usr/lib/rabbitmq/bin/rabbitmq-server"

    Here's the /proc/PID for the relevant beam process:

    dr-xr-xr-x 7 rabbitmq rabbitmq 0 2011-11-11 13:06 .
    dr-xr-xr-x 527 root root 0 2011-09-22 09:05 ..
    dr-xr-xr-x 2 rabbitmq rabbitmq 0 2011-11-11 13:06 attr
    -r-------- 1 root root 0 2011-11-11 13:06 auxv
    -r--r--r-- 1 root root 0 2011-11-11 13:06 cgroup
    --w------- 1 root root 0 2011-11-11 13:06 clear_refs
    -r--r--r-- 1 root root 0 2011-11-11 13:06 cmdline
    -rw-r--r-- 1 root root 0 2011-11-11 13:06 coredump_filter
    -r--r--r-- 1 root root 0 2011-11-11 13:06 cpuset
    lrwxrwxrwx 1 root root 0 2011-11-11 13:06 cwd -> /home/rj
    -r-------- 1 root root 0 2011-11-11 13:06 environ
    lrwxrwxrwx 1 root root 0 2011-11-11 13:06 exe ->
    /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp
    dr-x------ 2 root root 0 2011-11-11 13:06 fd
    dr-x------ 2 root root 0 2011-11-11 13:06 fdinfo
    -r--r--r-- 1 root root 0 2011-11-11 13:06 io
    -r-------- 1 root root 0 2011-11-11 13:06 limits
    -rw-r--r-- 1 root root 0 2011-11-11 13:06 loginuid
    -r--r--r-- 1 root root 0 2011-11-11 13:06 maps
    -rw------- 1 root root 0 2011-11-11 13:06 mem
    -r--r--r-- 1 root root 0 2011-11-11 13:06 mountinfo
    -r--r--r-- 1 root root 0 2011-11-11 13:06 mounts
    -r-------- 1 root root 0 2011-11-11 13:06 mountstats
    dr-xr-xr-x 5 rabbitmq rabbitmq 0 2011-11-11 13:06 net
    -r--r--r-- 1 root root 0 2011-11-11 13:06 numa_maps
    -rw-r--r-- 1 root root 0 2011-11-11 13:06 oom_adj
    -r--r--r-- 1 root root 0 2011-11-11 13:06 oom_score
    -r-------- 1 root root 0 2011-11-11 13:06 pagemap
    lrwxrwxrwx 1 root root 0 2011-11-11 13:06 root -> /
    -rw-r--r-- 1 root root 0 2011-11-11 13:06 sched
    -r--r--r-- 1 root root 0 2011-11-11 13:06 sessionid
    -r--r--r-- 1 root root 0 2011-11-11 13:06 smaps
    -r--r--r-- 1 root root 0 2011-11-11 13:06 stat
    -r--r--r-- 1 root root 0 2011-11-11 13:06 statm
    -r--r--r-- 1 root root 0 2011-11-11 13:06 status
    dr-xr-xr-x 49 rabbitmq rabbitmq 0 2011-11-11 13:06 task
    -r--r--r-- 1 root root 0 2011-11-11 13:06 wchan

    Conversely, if I just start vim, like so:
    su rabbitmq -s /bin/sh -c "vim"
    then everything in the /proc/PID dir *is* owned by rabbitmq.

    If I just start erl, it's the same as starting rabbit (proc/PID/fd
    owned by root)

    At a bit of a loss.

    RJ



    On 11 November 2011 11:49, Simon MacMullen wrote:
    On 11/11/11 11:07, Richard Jones wrote:

    The /proc/<pid> ?dir is owned by rabbitmq, but the /proc/<pid>/fd dir
    is owned (and only readable) by root, along with a bunch of other
    stuff in /proc/<pid>.
    Hmm. Needless to say, that's not what I'm seeing here. I'm on Ubuntu 10.04 -
    you?
    I start rabbit using "sudo /etc/init.d/rabbitmq-server start"

    Did something change in how rabbit setuids or something since 2.6? ?It
    worked fine prior to installing the new deb.
    2.6.1 did something like 'setsid sh -c "/usr/sbin/rabbitmq-server"'
    2.7.0 does 'setsid /usr/sbin/rabbitmq-server'

    (in both cases with come redirection and environment munging.)

    This doesn't feel like something that could cause this failure though.

    I'll patch rabbit_management_agent to fall back to lsof if it can't open
    /proc/<pid>/fd, but I'd like to find out why you're seeing this - I've
    googled a bit but can't see anything that would indicate why permissions are
    as they are.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Nov 11, 2011 at 12:52 pm

    On 11/11/11 12:19, Richard Jones wrote:
    Conversely, if I just start vim, like so:
    su rabbitmq -s /bin/sh -c "vim"
    then everything in the /proc/PID dir *is* owned by rabbitmq.

    If I just start erl, it's the same as starting rabbit (proc/PID/fd
    owned by root)

    At a bit of a loss.
    Yeah, that doesn't make sense to me either.
    /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp
    This is the only thing to leap out - you have a self compiled Erlang but
    you installed RabbitMQ using the .deb? I still don't see how that would
    be the issue though.

    Anyway, I've put a management agent plugin that should handle failure to
    open /proc/<pid>/fd at:

    http://www.rabbitmq.com/releases/plugins/v2.7.0/rabbitmq_management_agent-2.7.0.proc-fix.ez

    - could you download this to
    /usr/lib/rabbitmq/lib/rabbitmq_server-2.7.0/plugins/, delete the other
    management-agent, and see if this fixes the problem?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Richard Jones at Nov 11, 2011 at 1:08 pm
    Ah, got it - bit of a weird one:

    This erlang install has the beam.smp process setcap'ed to allow it to
    bind ports <1024 without running as root.

    # getcap /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp
    /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp = cap_net_bind_service+ep

    Once I do this to remove the capability:
    # setcap cap_net_bind_service=-ep /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp

    this fixes the problem, and everything in proc/pid is owned by
    rabbitmq as expected.

    If i "setcap cap_net_bind_service=+ep vim" the same thing happens, so
    looks like a general setcap issue.

    Setcap isn't like suid, the process wasn't running as root or really
    with root perms (just that it can bind ports <1024), but apparently it
    was confused enough to break proc ownership. Don't know what the
    intended behaviour is, can't find any warnings in the setcap docs
    (which are sparse).

    Sorry for the wild goose chase.. As it happens, I'm using an iptables
    redirect now anyway, so don't need the setcap anymore.

    I put the cap back on, and tested your fixed plugin: it boots up fine
    with the fixed management_agent plugin installed. Might be worth
    leaving that in as a fallback (and anywhere else that checks
    /proc/self)


    No idea why this happened after the upgrade - that setcap has been set
    for ages, long before I upgraded to 2.7.0.

    Thanks,
    RJ


    On 11 November 2011 12:52, Simon MacMullen wrote:
    On 11/11/11 12:19, Richard Jones wrote:

    Conversely, if I just start vim, like so:
    ? su rabbitmq -s /bin/sh -c "vim"
    then everything in the /proc/PID dir *is* owned by rabbitmq.

    If I just start erl, it's the same as starting rabbit (proc/PID/fd
    owned by root)

    At a bit of a loss.
    Yeah, that doesn't make sense to me either.
    /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp
    This is the only thing to leap out - you have a self compiled Erlang but you
    installed RabbitMQ using the .deb? I still don't see how that would be the
    issue though.

    Anyway, I've put a management agent plugin that should handle failure to
    open /proc/<pid>/fd at:

    http://www.rabbitmq.com/releases/plugins/v2.7.0/rabbitmq_management_agent-2.7.0.proc-fix.ez

    - could you download this to
    /usr/lib/rabbitmq/lib/rabbitmq_server-2.7.0/plugins/, delete the other
    management-agent, and see if this fixes the problem?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • Simon MacMullen at Nov 11, 2011 at 1:22 pm
    Cool. Thanks for getting back to the list. So yes, the fix will probably
    go in some future release, but I'm glad to hear it's not a priority...

    Cheers, Simon
    On 11/11/11 13:08, Richard Jones wrote:
    Ah, got it - bit of a weird one:

    This erlang install has the beam.smp process setcap'ed to allow it to
    bind ports<1024 without running as root.

    # getcap /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp
    /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp = cap_net_bind_service+ep

    Once I do this to remove the capability:
    # setcap cap_net_bind_service=-ep /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp

    this fixes the problem, and everything in proc/pid is owned by
    rabbitmq as expected.

    If i "setcap cap_net_bind_service=+ep vim" the same thing happens, so
    looks like a general setcap issue.

    Setcap isn't like suid, the process wasn't running as root or really
    with root perms (just that it can bind ports<1024), but apparently it
    was confused enough to break proc ownership. Don't know what the
    intended behaviour is, can't find any warnings in the setcap docs
    (which are sparse).

    Sorry for the wild goose chase.. As it happens, I'm using an iptables
    redirect now anyway, so don't need the setcap anymore.

    I put the cap back on, and tested your fixed plugin: it boots up fine
    with the fixed management_agent plugin installed. Might be worth
    leaving that in as a fallback (and anywhere else that checks
    /proc/self)


    No idea why this happened after the upgrade - that setcap has been set
    for ages, long before I upgraded to 2.7.0.

    Thanks,
    RJ



    On 11 November 2011 12:52, Simon MacMullenwrote:
    On 11/11/11 12:19, Richard Jones wrote:

    Conversely, if I just start vim, like so:
    su rabbitmq -s /bin/sh -c "vim"
    then everything in the /proc/PID dir *is* owned by rabbitmq.

    If I just start erl, it's the same as starting rabbit (proc/PID/fd
    owned by root)

    At a bit of a loss.
    Yeah, that doesn't make sense to me either.
    /usr/local/lib/erlang/erts-5.8.1/bin/beam.smp
    This is the only thing to leap out - you have a self compiled Erlang but you
    installed RabbitMQ using the .deb? I still don't see how that would be the
    issue though.

    Anyway, I've put a management agent plugin that should handle failure to
    open /proc/<pid>/fd at:

    http://www.rabbitmq.com/releases/plugins/v2.7.0/rabbitmq_management_agent-2.7.0.proc-fix.ez

    - could you download this to
    /usr/lib/rabbitmq/lib/rabbitmq_server-2.7.0/plugins/, delete the other
    management-agent, and see if this fixes the problem?

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware

    --
    Simon MacMullen
    RabbitMQ, VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedNov 10, '11 at 4:15p
activeNov 11, '11 at 1:22p
posts8
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase