FAQ
Hi All,

We are seeing an unfortunate side effect using puppet to manage a
directory of symlinks pointing to automounted filesystems. It seems
that as well as verifying that the link matches the desired target
(readlink/lstat), it is also performing an open() on the links and
causing the filesystems to be automounted.

The pattern here is:

file { "/data":
ensure => directory,
source => "/nfsmount/links/data",
recurse => true,
force => true,
}

Imagine everything is in the steady state, i.e.:

symlink /data/fs1 -> /net/server1/fs1 exists
symlink /nfsmount/links/data/fs1 -> /net/server1/fs1 exists

Then at some point in the run (after doing lstat("/data/fs1"),
readlink("/data/fs1")), puppet appears to attempt an open("/data/fs1",
O_RDONLY).

IANAE here, but it seems that this open is unnecessary? Once the
target of the link in the managed directory is verified to match the
target of the link on the source, that should be sufficient?

(just had an idea as I was writing this) it *appears* I can work
around it by coding all of the symlinks individually as ensure=>link
into puppet itself. However I still think the behaviour is
incorrect. Thoughts?

Thanks,
Tim

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Krzysztof Wilczynski at Apr 23, 2012 at 10:41 am
    Hi Tim,

    On Monday, 23 April 2012 05:41:31 UTC+1, Tim wrote:

    [...]
    IANAE here, but it seems that this open is unnecessary?
    [...]

    This is not Puppet doing open on this directory (not explicitly anyway).
    Have a look on this:

    kwilczynski@desktop:~/Development/Sandbox/Test$ mkdir -p a/{b,c/d}
    kwilczynski@desktop:~/Development/Sandbox/Test$ tree
    .
    └── a
    ├── b
    └── c
    └── d

    4 directories, 0 files

    kwilczynski@desktop:~/Development/Sandbox/Test$ strace -e
    open,lstat,getdents ruby -e "Dir['./**/*']"
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/usr/lib/libruby1.8.so.1.8", O_RDONLY) = 3
    open("/lib/libpthread.so.0", O_RDONLY) = 3
    open("/lib/librt.so.1", O_RDONLY) = 3
    open("/lib/libdl.so.2", O_RDONLY) = 3
    open("/lib/libcrypt.so.1", O_RDONLY) = 3
    open("/lib/libm.so.6", O_RDONLY) = 3
    open("/lib/libc.so.6", O_RDONLY) = 3
    open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    getdents(3, /* 3 entries */, 32768) = 72
    lstat("./a", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
    open("./a", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
    getdents(4, /* 4 entries */, 32768) = 96
    lstat("./a/b", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
    open("./a/b", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
    getdents(5, /* 2 entries */, 32768) = 48
    getdents(5, /* 0 entries */, 32768) = 0
    lstat("./a/c", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
    open("./a/c", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
    getdents(5, /* 3 entries */, 32768) = 72
    lstat("./a/c/d", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
    open("./a/c/d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
    getdents(6, /* 2 entries */, 32768) = 48
    getdents(6, /* 0 entries */, 32768) = 0
    getdents(5, /* 0 entries */, 32768) = 0
    getdents(4, /* 0 entries */, 32768) = 0
    getdents(3, /* 0 entries */, 32768) = 0
    kwilczynski@desktop:~/Development/Sandbox/Test$

    And then ...

    kwilczynski@desktop:~/Development/Sandbox/Test$ strace -e
    open,lstat,getdents ruby -e "Dir['./*']"
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/usr/lib/libruby1.8.so.1.8", O_RDONLY) = 3
    open("/lib/libpthread.so.0", O_RDONLY) = 3
    open("/lib/librt.so.1", O_RDONLY) = 3
    open("/lib/libdl.so.2", O_RDONLY) = 3
    open("/lib/libcrypt.so.1", O_RDONLY) = 3
    open("/lib/libm.so.6", O_RDONLY) = 3
    open("/lib/libc.so.6", O_RDONLY) = 3
    open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    getdents(3, /* 3 entries */, 32768) = 72
    getdents(3, /* 0 entries */, 32768) = 0

    Or, an equivalent:

    kwilczynski@desktop:~/Development/Sandbox/Test$ strace -e
    open,lstat,getdents ruby -e "Dir.entries('.')"
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/usr/lib/libruby1.8.so.1.8", O_RDONLY) = 3
    open("/lib/libpthread.so.0", O_RDONLY) = 3
    open("/lib/librt.so.1", O_RDONLY) = 3
    open("/lib/libdl.so.2", O_RDONLY) = 3
    open("/lib/libcrypt.so.1", O_RDONLY) = 3
    open("/lib/libm.so.6", O_RDONLY) = 3
    open("/lib/libc.so.6", O_RDONLY) = 3
    open("/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    getdents(3, /* 3 entries */, 32768) = 72
    getdents(3, /* 0 entries */, 32768) = 0
    kwilczynski@desktop:~/Development/Sandbox/Test$

    Also, two more examples:

    kwilczynski@desktop:~/Development/Sandbox/Test$ cat > ftw.c
    #include <ftw.h>
    #include <stdio.h>
    #include <stdlib.h>

    int callback(const char *name, const struct stat *status, int type);

    int main(int argc, char *argv[])
    {
    char *root = ".";

    ftw((char *) root, callback, 1);

    return 0;
    }

    int callback(const char *name, const struct stat *status, int type)
    {
    if (type == FTW_NS)
    return 0;

    if (type == FTW_F || type == FTW_SL || type == FTW_D)
    printf("%s\n", name);

    return 0;
    }
    kwilczynski@desktop:~/Development/Sandbox/Test$ gcc -o ftw ftw.c
    kwilczynski@desktop:~/Development/Sandbox/Test$ ./ftw
    .
    ./a
    ./a/b
    ./a/c
    ./a/c/d
    ./readdir.c
    ./readdir
    ./ftw.c
    ./ftw

    kwilczynski@desktop:~/Development/Sandbox/Test$ strace -e
    open,lstat,getdents ./ftw
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/lib/libc.so.6", O_RDONLY) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    .
    getdents(3, /* 7 entries */, 32768) = 192
    getdents(3, /* 0 entries */, 32768) = 0
    open("./a", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    ./a
    getdents(3, /* 4 entries */, 32768) = 96
    getdents(3, /* 0 entries */, 32768) = 0
    open("./a/b", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    ./a/b
    getdents(3, /* 2 entries */, 32768) = 48
    getdents(3, /* 0 entries */, 32768) = 0
    open("./a/c", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    ./a/c
    getdents(3, /* 3 entries */, 32768) = 72
    getdents(3, /* 0 entries */, 32768) = 0
    open("./a/c/d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    ./a/c/d
    getdents(3, /* 2 entries */, 32768) = 48
    getdents(3, /* 0 entries */, 32768) = 0
    ./readdir.c
    ./readdir
    ./ftw.c
    ./ftw
    kwilczynski@desktop:~/Development/Sandbox/Test$

    And a non-recursive one:

    kwilczynski@desktop:~/Development/Sandbox/Test$ cat > readdir.c
    #include <stdio.h>
    #include <stdlib.h>
    #include <dirent.h>

    struct dirent *d_ent;

    int main(int argc, char *argv[])
    {
    char *root = ".";

    DIR *d_ptr;

    if((d_ptr = opendir((char *) root)) == NULL) {
    printf("Error");
    exit(1);
    }

    while(d_ent = readdir(d_ptr))
    printf("%s\n", d_ent->d_name);

    closedir(d_ptr);

    return 0;
    }
    kwilczynski@desktop:~/Development/Sandbox/Test$ gcc -o readdir readdir.c
    kwilczynski@desktop:~/Development/Sandbox/Test$ ./readdir
    .
    ..
    a
    readdir.c
    readdir

    kwilczynski@desktop:~/Development/Sandbox/Test$ strace -e
    open,lstat,getdents ./readdir
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/lib/libc.so.6", O_RDONLY) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    getdents(3, /* 5 entries */, 32768) = 136
    .
    ..
    a
    readdir.c
    readdir
    getdents(3, /* 0 entries */, 32768) = 0
    kwilczynski@desktop:~/Development/Sandbox/Test$

    Can you see what is going on when you request recursive directory
    traversal? :)

    You'd have to probably remove "recurse => true" from your File resource
    that is looking after a directory structure in this case.

    Therefore, your open() is a needed to get the file hand (file descriptor)
    on a new directory of interest when traversing. Almost everything is a file
    in Unix / Linux :)

    KW

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/HSnqpx2nF38J.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Krzysztof Wilczynski at Apr 23, 2012 at 10:53 am
    [...]

    I forgot to add ...

    You probably do not want Puppet to follow symbolic links which in
    conjunction with the "recurse => true" is leading to open() done on each
    new directory during traversal. Perhaps setting "links => ignore" will
    help, not sure, thought.

    KW

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/-ZEF7r2mrzAJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jcbollinger at Apr 23, 2012 at 12:58 pm

    On Apr 22, 11:41 pm, Tim wrote:
    Hi All,

    We are seeing an unfortunate side effect using puppet to manage a
    directory of symlinks pointing to automounted filesystems.  It seems
    that as well as verifying that the link matches the desired target
    (readlink/lstat), it is also performing an open() on the links and
    causing the filesystems to be automounted.

    The pattern here is:

    file { "/data":
    ensure => directory,
    source => "/nfsmount/links/data",
    recurse => true,
    force => true,

    }

    Imagine everything is in the steady state, i.e.:

    symlink /data/fs1 -> /net/server1/fs1 exists
    symlink /nfsmount/links/data/fs1 -> /net/server1/fs1 exists

    Then at some point in the run (after doing lstat("/data/fs1"),
    readlink("/data/fs1")), puppet appears to attempt an open("/data/fs1",
    O_RDONLY).

    IANAE here, but it seems that this open is unnecessary?  Once the
    target of the link in the managed directory is verified to match the
    target of the link on the source, that should be sufficient?

    That sounds sensible, but I don't know whether it's practical; some of
    Puppet's behaviors are constrained by the underlying Ruby libraries.
    In any case, the first thing to try is to set "links => manage" on
    your File resource (not "links => ignore", which cannot work when the
    desired links are not yet present).

    The bottom line, however, is that recursive File management has always
    been a sore spot for Puppet, and it probably always will be. It is
    not remotely possible to provide enough information in the parameters
    of a single File resource to cover all the possible permutations of
    desired behavior. Puppet does a fairly good job of coping with that
    when asked to do so, but generally you are better off finding a
    different approach.

    If you have a large number of files, then you may be better off
    wrapping the whole thing into a custom package (e.g. an RPM), putting
    that in a local repository or on a network filesystem, and managing
    that package instead of the individual files. If there are only a few
    files (that you want to manage) then by all means set up individual
    file resources for them.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Tim at Apr 24, 2012 at 3:43 am
    I've ended up doing as John said and creating file resources for all
    of these directories and links instead of syncing from an existing
    source.

    Works a treat :) Thanks for your help all.

    Cheers,
    Tim
    On Apr 23, 10:58 pm, jcbollinger wrote:
    On Apr 22, 11:41 pm, Tim wrote:








    Hi All,
    We are seeing an unfortunate side effect using puppet to manage a
    directory of symlinks pointing to automounted filesystems.  It seems
    that as well as verifying that the link matches the desired target
    (readlink/lstat), it is also performing an open() on the links and
    causing the filesystems to be automounted.
    The pattern here is:
    file { "/data":
    ensure => directory,
    source => "/nfsmount/links/data",
    recurse => true,
    force => true,
    }
    Imagine everything is in the steady state, i.e.:
    symlink /data/fs1 -> /net/server1/fs1 exists
    symlink /nfsmount/links/data/fs1 -> /net/server1/fs1 exists
    Then at some point in the run (after doing lstat("/data/fs1"),
    readlink("/data/fs1")), puppet appears to attempt an open("/data/fs1",
    O_RDONLY).
    IANAE here, but it seems that this open is unnecessary?  Once the
    target of the link in the managed directory is verified to match the
    target of the link on the source, that should be sufficient?
    That sounds sensible, but I don't know whether it's practical; some of
    Puppet's behaviors are constrained by the underlying Ruby libraries.
    In any case, the first thing to try is to set "links => manage" on
    your File resource (not "links => ignore", which cannot work when the
    desired links are not yet present).

    The bottom line, however, is that recursive File management has always
    been a sore spot for Puppet, and it probably always will be.  It is
    not remotely possible to provide enough information in the parameters
    of a single File resource to cover all the possible permutations of
    desired behavior.  Puppet does a fairly good job of coping with that
    when asked to do so, but generally you are better off finding a
    different approach.

    If you have a large number of files, then you may be better off
    wrapping the whole thing into a custom package (e.g. an RPM), putting
    that in a local repository or on a network filesystem, and managing
    that package instead of the individual files.  If there are only a few
    files (that you want to manage) then by all means set up individual
    file resources for them.

    John
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedApr 23, '12 at 9:18a
activeApr 24, '12 at 3:43a
posts5
users3
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase