FAQ
Greetings!

I am having a problem with puppet mounting a device and am hoping someone
can help. Here is the short version, if you have questions or need more
detail, please feel free to ask.

I have a Panasas storage device on my network on which my home directory
resides. The Panasas device mounts the filesystem via a kernel module
(which they call DirectFlow). Thus, when I run as root ` mount /home` a
kernel module is loaded and then the filesystem is loaded. I have certain
restrictions in place on my servers which I have to keep in line (Puppet is
AMAZING for this!) and the mounting of /home is one of those restrictions.

I have in my puppet manifest this:
mount { "/home" :
ensure => mounted,
atboot => true,
device => "panfs://192.168.1.20/home",
fstype => "panfs",
options => "defaults,nodev",
remounts => true,
pass => 2,
dump => 1,
}

If I run, as root, `puppet agent --test` then the /home filesystem is
mounted and everything is wonderful. However, if I let the puppet agent
daemon try to mount /home I get errors in the log files without the mount
ever happening.

Apr 2 13:01:08 testnode puppet-agent[29955]:
(/Stage[main]/mount::Homefilesystem/Mount[/home]/ensure) ensure changed
'unmounted' to 'mounted'
Apr 2 13:01:08 testnode puppet-agent[29955]:
(/Stage[main]/mount::Homefilesystem/Mount[/home]) Could not evaluate:
Execution of '/bin/mount -o defaults,nodev /home' returned 1: mount.panfs
error: cannot init pan_sock_ping 0x239d (pan_sock: protected socket,
permission denied)

Neither Panasas representatives I talked to seemed to have any idea what
Puppet was before I spoke to them. My coworkers, the Panasas reps, and I
brainstormed a few ideas but only three seemed to "work":
* Have the puppet daemon run as root instead of the puppet user (which is
an obvious issue)
* Use auto-mount (which "works" but is causing some oddities in a few of my
jobs which I am fairly sure is due to the latency of the mount)
* Have Puppet call a script with the setuid bit configured which can mount
/home (which doesn't 100% address my needs of puppet being able to remount
if one of those parameters is wrong/missing/changed/whatever without that
script getting complicated).

Before I commit towards one option, I thought I would ask the other Puppet
masters out there for ideas. Given the popularity of Puppet in datacenters
as well as the popularity of SAN devices in datacenters I figure someone
out there has probably solved this problem. I am hoping that their solution
is better then the ones we have come up with. :-D

Can anyone help me out with this?

Thank you in advance!!

Stack

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Steven Nemetz at Apr 9, 2013 at 2:36 pm
    This is an OS privilege issue. So puppet has to working within what can be done on the OS.
    By default mount requires root privileges to make changes. This is why it will work as root but not as puppet.
    So you either run as root or set it up so a non-root user can mount

    Some OS's have options to allow mount to work for non-root users. Check your mount man page and test.
    Load a different app to handle user mounts. There are other apps available for this type of thing. Fuse and apps based on fuse are the most common, but not the only ones available.

    Hope that helps a little,

    Steven

    Date: Mon, 8 Apr 2013 21:02:26 -0700
    From: i.am.stack@gmail.com
    To: puppet-users@googlegroups.com
    Subject: [Puppet Users] Puppet mount fails due to kernel module

    Greetings!
    I am having a problem with puppet mounting a device and am hoping someone can help. Here is the short version, if you have questions or need more detail, please feel free to ask.
    I have a Panasas storage device on my network on which my home directory resides. The Panasas device mounts the filesystem via a kernel module (which they call DirectFlow). Thus, when I run as root ` mount /home` a kernel module is loaded and then the filesystem is loaded. I have certain restrictions in place on my servers which I have to keep in line (Puppet is AMAZING for this!) and the mounting of /home is one of those restrictions.
    I have in my puppet manifest this:mount { "/home" :


    ensure => mounted,


    atboot => true,


    device => "panfs://192.168.1.20/home",


    fstype => "panfs",


    options => "defaults,nodev",


    remounts => true,


    pass => 2,


    dump => 1,


    }

    If I run, as root, `puppet agent --test` then the /home filesystem is mounted and everything is wonderful. However, if I let the puppet agent daemon try to mount /home I get errors in the log files without the mount ever happening.
    Apr 2 13:01:08 testnode puppet-agent[29955]: (/Stage[main]/mount::Homefilesystem/Mount[/home]/ensure) ensure changed 'unmounted' to 'mounted'Apr 2 13:01:08 testnode puppet-agent[29955]: (/Stage[main]/mount::Homefilesystem/Mount[/home]) Could not evaluate: Execution of '/bin/mount -o defaults,nodev /home' returned 1: mount.panfs error: cannot init pan_sock_ping 0x239d (pan_sock: protected socket, permission denied)
    Neither Panasas representatives I talked to seemed to have any idea what Puppet was before I spoke to them. My coworkers, the Panasas reps, and I brainstormed a few ideas but only three seemed to "work":
    * Have the puppet daemon run as root instead of the puppet user (which is an obvious issue)* Use auto-mount (which "works" but is causing some oddities in a few of my jobs which I am fairly sure is due to the latency of the mount)* Have Puppet call a script with the setuid bit configured which can mount /home (which doesn't 100% address my needs of puppet being able to remount if one of those parameters is wrong/missing/changed/whatever without that script getting complicated).
    Before I commit towards one option, I thought I would ask the other Puppet masters out there for ideas. Given the popularity of Puppet in datacenters as well as the popularity of SAN devices in datacenters I figure someone out there has probably solved this problem. I am hoping that their solution is better then the ones we have come up with. :-D
    Can anyone help me out with this?
    Thank you in advance!!
    Stack



    --

    You received this message because you are subscribed to the Google Groups "Puppet Users" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

    To post to this group, send email to puppet-users@googlegroups.com.

    Visit this group at http://groups.google.com/group/puppet-users?hl=en.

    For more options, visit https://groups.google.com/groups/opt_out.






    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jonathan Stanton at Apr 9, 2013 at 2:56 pm
    Hello,
    On Apr 9, 2013, at 12:02 AM, Stack Kororā wrote:

    Greetings!

    I am having a problem with puppet mounting a device and am hoping someone
    can help. Here is the short version, if you have questions or need more
    detail, please feel free to ask.

    I have a Panasas storage device on my network on which my home directory
    resides. The Panasas device mounts the filesystem via a kernel module
    (which they call DirectFlow). Thus, when I run as root ` mount /home` a
    kernel module is loaded and then the filesystem is loaded. I have certain
    restrictions in place on my servers which I have to keep in line (Puppet is
    AMAZING for this!) and the mounting of /home is one of those restrictions.

    I have in my puppet manifest this:
    mount { "/home" :
    ensure => mounted,
    atboot => true,
    device => "panfs://192.168.1.20/home",
    fstype => "panfs",
    options => "defaults,nodev",
    remounts => true,
    pass => 2,
    dump => 1,
    }

    If I run, as root, `puppet agent --test` then the /home filesystem is
    mounted and everything is wonderful. However, if I let the puppet agent
    daemon try to mount /home I get errors in the log files without the mount
    ever happening.
    The puppet agent that runs on your server is normally running as root (it has to be to have privileges to make all of the change it can do). However a number of Linux operating systems have further security protection beyond just root which can restrict what daemons can do but does not restrict what a 'human' logged in as root may be able to do. For example all of the Redhat Enterprise Linux (and Fedora) use Selinux which will normally limit what even root executing dameon processes can do in order to protect the system from attacks. Other OS's like OpenBSD/FreeBSD or Linux distributions have similar capabilities that go by different names.

    If those are enabled, you may find that things you can do as a root user don't work when run from cron or from a daemon process.

    If this is RHEL/Centos then try putting SElinux in permissive mode (as root user run 'setenforce Permissive' ) and see if you have the same problem. If so then that identifies the issue and you can either generate a custom selinux policy for puppet, run in permissive, or change the way the mount happens. What to do depends on your organization security policy.

    Since it works when you run puppet agent --test the manifest itself is probably fine and the puppet mount code is able to load your module ok.
    Apr 2 13:01:08 testnode puppet-agent[29955]:
    (/Stage[main]/mount::Homefilesystem/Mount[/home]/ensure) ensure changed
    'unmounted' to 'mounted'
    Apr 2 13:01:08 testnode puppet-agent[29955]:
    (/Stage[main]/mount::Homefilesystem/Mount[/home]) Could not evaluate:
    Execution of '/bin/mount -o defaults,nodev /home' returned 1: mount.panfs
    error: cannot init pan_sock_ping 0x239d (pan_sock: protected socket,
    permission denied)

    Neither Panasas representatives I talked to seemed to have any idea what
    Puppet was before I spoke to them. My coworkers, the Panasas reps, and I
    brainstormed a few ideas but only three seemed to "work":
    * Have the puppet daemon run as root instead of the puppet user (which is
    an obvious issue)
    This should already be happening. Only the 'puppet master' runs as a regular 'puppet' user, the agent normally runs as root.
    * Use auto-mount (which "works" but is causing some oddities in a few of my
    jobs which I am fairly sure is due to the latency of the mount)
    * Have Puppet call a script with the setuid bit configured which can mount
    /home (which doesn't 100% address my needs of puppet being able to remount
    if one of those parameters is wrong/missing/changed/whatever without that
    script getting complicated).
    I would not recommend this. It is fragile and as you say loses much of the benefit of puppet.
    Before I commit towards one option, I thought I would ask the other Puppet
    masters out there for ideas. Given the popularity of Puppet in datacenters
    as well as the popularity of SAN devices in datacenters I figure someone
    out there has probably solved this problem. I am hoping that their solution
    is better then the ones we have come up with. :-D

    Can anyone help me out with this?

    Thank you in advance!!

    Stack
    Hope this helps.

    Jonathan
    -------------------------------------------------------------------------------
    Jonathan Stanton jonathan@spreadconcepts.com
    Spread Group Messaging www.spread.org
    Spread Concepts LLC www.spreadconcepts.com
    -------------------------------------------------------------------------------




    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Stack Kororā at Apr 9, 2013 at 11:07 pm

    On Tuesday, April 9, 2013 9:56:11 AM UTC-5, Jonathan Stanton wrote:
    [snip]
    If those are enabled, you may find that things you can do as a root user
    don't work when run from cron or from a daemon process.

    If this is RHEL/Centos then try putting SElinux in permissive mode (as
    root user run 'setenforce Permissive' ) and see if you have the same
    problem. If so then that identifies the issue and you can either generate a
    custom selinux policy for puppet, run in permissive, or change the way the
    mount happens. What to do depends on your organization security policy.
    Thanks Jonathan!
    That was exactly the problem. SELinux runs on these systems and I was so
    focused on the puppet part that I missed the SELinux part. Doh!

    [snip]
    This should already be happening. Only the 'puppet master' runs as a
    regular 'puppet' user, the agent normally runs as root.
    Thanks for letting me know. I thought both ran as puppet before now.



    I put SELinux into permissive mode and let the puppet agent do its thing
    successfully! Hooray!! But I really need SELinux...Any suggestions on
    getting this to work through SELinux* ?

    * I completely understand that this is not a Puppet problem anymore so a
    response of 'Go harass the SELinux list' won't hurt my feelings any. But it
    is worth it to ask as I am sure there are others who deal with Puppet and
    SELinux.
    :-)

    Since SELinux is in permissive mode, I piped the relevant information from
    audit.log into audit2allow.

    $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -m panfs
    module panfs 1.0;

    require {
    type node_t;
    type sysctl_vm_t;
    type mount_t;
    class capability net_raw;
    class dir search;
    class file read;
    class rawip_socket { ioctl shutdown bind create getattr node_bind };
    }

    #============= mount_t ==============
    allow mount_t node_t:rawip_socket node_bind;
    allow mount_t self:capability net_raw;
    allow mount_t self:rawip_socket { bind create ioctl shutdown getattr };
    allow mount_t sysctl_vm_t:dir search;
    allow mount_t sysctl_vm_t:file read;

    Since that looked good, I updated the module in SELinux
    $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -M panfs
    $ semodule -i panfs
    $semodule -l | grep panfs
    panfs 1.0

    Then I turned SELinux back on with setenforce and reset puppet with
    `service puppet restart`. I didn't get any SELinux audit messages, but it
    still doesn't mount. It looks like (to me anyway) that the mounting process
    is still trying to get to resources that it can't access because they are
    being blocked by SELinux. However, I was really hoping that it would put
    something in to the audit.log file, but nothing changed. Any ideas as to
    why it didn't work?

    Apr 9 16:22:00 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]/ensure) ensure changed
    'unmounted' to 'mounted'
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) Could not evaluate:
    Execution of '/bin/mount -o defaults,nodev /home' returned 1: mount.panfs
    warning: couldn't ping address 192.168.1.20:3095 using 192.168.1.11:1,
    0x239d (pan_sock: protected socket, permission denied)
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs warning:
    This mount still may succeed, but one or more local interfaces (listed
    below) failed communicate with the Panasas realm during mount. This
    suggests that a route cannot be established between these local
    interface(s) and the system. A client sends a list of IP addresses on
    which the Panasas storage system may establish a connection. If any one of
    these addresses should be excluded from the mount time check, use the
    'callback-network-disallow' or 'callback-address-disallow' mount options.
    See 'man 8 mount.panfs' for more details on PanFS mount options.
    Excluding the interface from the check at mount time will avoid long
    running mount commands.
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs: failed
    local addresses: 192.168.1.11:1
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs:
    successful local addresses:
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error:
    couldn't ping realm servers for mount
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error:
    cannot process mount options in postinit step 0x7 (Invalid argument)

    Thanks again! I appreciate the help!

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Sergio Jimenez at Apr 10, 2013 at 1:35 pm
    Hi!

    just a thought, I don't have any place to test it and it's been a while
    since I had to deal with SELinux...but check SELinux booleans you might
    find something there that is not allowing puppet agent to use mount.

    IIRC you could list all of them with "semanage boolean -l".

    sjr
    On Wednesday, April 10, 2013 1:07:04 AM UTC+2, Stack Kororā wrote:


    On Tuesday, April 9, 2013 9:56:11 AM UTC-5, Jonathan Stanton wrote:

    [snip]
    If those are enabled, you may find that things you can do as a root user
    don't work when run from cron or from a daemon process.

    If this is RHEL/Centos then try putting SElinux in permissive mode (as
    root user run 'setenforce Permissive' ) and see if you have the same
    problem. If so then that identifies the issue and you can either generate a
    custom selinux policy for puppet, run in permissive, or change the way the
    mount happens. What to do depends on your organization security policy.
    Thanks Jonathan!
    That was exactly the problem. SELinux runs on these systems and I was so
    focused on the puppet part that I missed the SELinux part. Doh!

    [snip]
    This should already be happening. Only the 'puppet master' runs as a
    regular 'puppet' user, the agent normally runs as root.
    Thanks for letting me know. I thought both ran as puppet before now.



    I put SELinux into permissive mode and let the puppet agent do its thing
    successfully! Hooray!! But I really need SELinux...Any suggestions on
    getting this to work through SELinux* ?

    * I completely understand that this is not a Puppet problem anymore so a
    response of 'Go harass the SELinux list' won't hurt my feelings any. But it
    is worth it to ask as I am sure there are others who deal with Puppet and
    SELinux.
    :-)

    Since SELinux is in permissive mode, I piped the relevant information from
    audit.log into audit2allow.

    $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -m panfs
    module panfs 1.0;

    require {
    type node_t;
    type sysctl_vm_t;
    type mount_t;
    class capability net_raw;
    class dir search;
    class file read;
    class rawip_socket { ioctl shutdown bind create getattr node_bind
    };
    }

    #============= mount_t ==============
    allow mount_t node_t:rawip_socket node_bind;
    allow mount_t self:capability net_raw;
    allow mount_t self:rawip_socket { bind create ioctl shutdown getattr };
    allow mount_t sysctl_vm_t:dir search;
    allow mount_t sysctl_vm_t:file read;

    Since that looked good, I updated the module in SELinux
    $ tail -50 /var/log/audit/audit.log | grep -i panfs | audit2allow -M panfs
    $ semodule -i panfs
    $semodule -l | grep panfs
    panfs 1.0

    Then I turned SELinux back on with setenforce and reset puppet with
    `service puppet restart`. I didn't get any SELinux audit messages, but it
    still doesn't mount. It looks like (to me anyway) that the mounting process
    is still trying to get to resources that it can't access because they are
    being blocked by SELinux. However, I was really hoping that it would put
    something in to the audit.log file, but nothing changed. Any ideas as to
    why it didn't work?

    Apr 9 16:22:00 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]/ensure) ensure changed
    'unmounted' to 'mounted'
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) Could not evaluate:
    Execution of '/bin/mount -o defaults,nodev /home' returned 1: mount.panfs
    warning: couldn't ping address 192.168.1.20:3095 using 192.168.1.11:1,
    0x239d (pan_sock: protected socket, permission denied)
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs warning:
    This mount still may succeed, but one or more local interfaces (listed
    below) failed communicate with the Panasas realm during mount. This
    suggests that a route cannot be established between these local
    interface(s) and the system. A client sends a list of IP addresses on
    which the Panasas storage system may establish a connection. If any one of
    these addresses should be excluded from the mount time check, use the
    'callback-network-disallow' or 'callback-address-disallow' mount options.
    See 'man 8 mount.panfs' for more details on PanFS mount options.
    Excluding the interface from the check at mount time will avoid long
    running mount commands.
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs: failed
    local addresses: 192.168.1.11:1
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs:
    successful local addresses:
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error:
    couldn't ping realm servers for mount
    Apr 9 16:22:01 test puppet-agent[32086]:
    (/Stage[main]/Cisaudit::Homefilesystem/Mount[/home]) mount.panfs error:
    cannot process mount options in postinit step 0x7 (Invalid argument)

    Thanks again! I appreciate the help!
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Stack Kororā at Apr 12, 2013 at 10:22 pm
    Greetings!

    Thanks for all the pointers in the right direction. I was finally able to
    figure it out today. I figured I would post my solution in case someone
    else finds this thread later on and might be able to use it. I am running
    this on a few servers successfully, but again, I just figured it out today
    so this is what I would call "beta" (at best). :-D

    This is just the module needed in SELinux for puppet to mount a Panasas
    device + the commands I used to create the module. After the module is
    loaded either restart the puppet service or wait till its next go and it
    should auto mount the device.

    Hope it helps! And thanks again!


    $ cat panfs.te
    module panfs 1.4;
    require {
    type node_t;
    type sysctl_vm_t;
    type mount_t;
    class capability net_raw;
    class dir search;
    class file read;
    class rawip_socket { read bind create getattr write ioctl shutdown
    node_bind };
    }
    #============= mount_t ==============
    allow mount_t node_t:rawip_socket node_bind;
    allow mount_t self:rawip_socket { bind create ioctl shutdown write read
    getattr };
    allow mount_t self:capability net_raw;
    allow mount_t sysctl_vm_t:dir search;
    allow mount_t sysctl_vm_t:file read;

    $ checkmodule -M -m panfs.te -o panfs.mod
    $ semodule_package -o panfs.pp -m panfs.mod
    $ semodule -i panfs.pp

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedApr 9, '13 at 2:06p
activeApr 12, '13 at 10:22p
posts6
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2021 Grokbase