FAQ
Greetings,

In my multiple hundred servers, I have <10 that are Red Hat based. We
recently brought them under the same management as the rest of the servers
utilizing Puppet. Then we ran into issues because we were hitting RHN too
frequently and we got our servers banned. :-(

I went digging for the culprit and found it in a section we wrote because
audit is insistent that some packages should never be installed and they
want regular checks that the packages are not installed. I rather liked my
original solution (below) as I have dozens of packages that shouldn't be
installed and occasionally I get another one to add to the list. This code
made it really simple to add a new package.

class audit::software_remove (
) {
     $removethesepackages = [
                 'telnet-server',
                 'telnet',
                 # Dozens removed for sanity :-)
                 ]
     case $operatingsystem {
         'SLES' : { package {$removethesepackages : ensure=>absent,} }
         'Scientific', 'CentOS', 'RedHat' : { package {$removethesepackages
: ensure=>purged,} }
         default : {}
     }
}

Now SLES runs this code amazingly well because zypper just does a check
against the local install before trying to remove a package and if it isn't
installed it doesn't do anything at all. One of the many shortcomings of
yum is that it always hits the repos. Since we have a local repo set up for
SLES, Scientific, and CentOS we don't care if we beat up on them. However,
we discussed the local repo caching with Red Hat and it didn't work out
(too much complexity for too few servers; won't go into details). Not only
that, but because every package is a separate transaction, we hit the repo
/multiple/ times every puppet run. Thus, our servers hit Red Hat repos
frequently and we get banned. :-(

So I went thinking about how to solve both the problem of slamming the repo
and checking for the package before trying to do a removal.

We have and utilize the pkginventory module[1] (which we have applied the
waiting patches plus done some of our own since that module hasn't been
updated in a long while). This gives me a fact of pkg_telnet or
pkg_telnet_server if those packages are installed. So I decided to utilize
that and I came up with the code below.

[1] https://forge.puppetlabs.com/ody/pkginventory

class audit::software_remove (
) {
     if $pkg_telnet_server {
         case $operatingsystem {
             'SLES' : { package { 'telnet-server': ensure=>absent,} }
             'Scientific', 'CentOS', 'RedHat' : { package {'telnet-server' :
ensure=>purged,} }
             default : {}
         }
     }
     else { notify {"Package telnet-server is not installed":}}
#----------------
     if $::pkg_telnet {
         case $operatingsystem {
             'SLES' : { package { 'telnet': ensure=>absent,} }
             'Scientific', 'CentOS', 'RedHat' : { package {'telnet' :
ensure=>purged,} }
             default : {}
         }
     }
     else { notify {"Package telnet is not installed":}}
}

Hrm. It works, but it isn't a good solution in my opinion. Not at all. Too
much code is being duplicated here. Well, there is some clean up I can do
though now. Previously I used the case statement because audit wanted us to
do a "yum purge" if any of the packages are found installed but SLES
doesn't support "purged" and I don't really see a difference. That is a
pointless case statement in my opinion with far too much code duplication.

class audit::software_remove (
) {
     if $pkg_telnet_server { package { 'telnet-server': ensure=>absent,} }
     else { notify {"Package telnet-server is not installed":}}
#----------------
     if $pkg_telnet { package { 'telnet': ensure=>absent,} }
     else { notify {"Package telnet is not installed":}}
}

Ah. Better. Now there are only 3 lines I have to duplicate (if,else,my
comment separator I use for my own sanity). Plus, there is a package check
/before/ the yum command even has a chance to run. I *still* get hit with
those REALLY annoying double notify messages in the logs (why twice? I
still do not understand why twice! Ugh...another gripe for another day) but
at least I don't hit the repo a million times every puppet run and that was
the real goal here.

So far in my testing environment, things are going well. Puppet runs are
MUCH faster not having to hit yum all the time. Logs are a bit more crowded
but at least they don't say "Software_remove/Package[telnet]/ensure:
created" which was always a confusing statement in the puppet logs (another
thing I never understood about puppet logging; how is a remove
"created"...).

Yet even still, there is something bothering me about how much code
duplication just got added. I feel that there is a simpler and better way
of doing this. I played with a few things I found online, but either they
were more complex or they still hit the repo really hard.

So I thought I would ask: Does anyone have any tips/suggestions on how I
might improve this code a bit more? Any thoughts? Anything jump out at you
that I could be doing better?

Thanks! I appreciate it!
~Stack~

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • José Luis Ledesma at Jul 23, 2014 at 6:37 am
    Hi,

    I dont know if it will work, but have you tried adding provider => rpm in
    the package resource?

    Regards,
    El 23/07/2014 02:37, "Stack Kororā" <i.am.stack@gmail.com> escribió:
    Greetings,

    In my multiple hundred servers, I have <10 that are Red Hat based. We
    recently brought them under the same management as the rest of the servers
    utilizing Puppet. Then we ran into issues because we were hitting RHN too
    frequently and we got our servers banned. :-(

    I went digging for the culprit and found it in a section we wrote because
    audit is insistent that some packages should never be installed and they
    want regular checks that the packages are not installed. I rather liked my
    original solution (below) as I have dozens of packages that shouldn't be
    installed and occasionally I get another one to add to the list. This code
    made it really simple to add a new package.

    class audit::software_remove (
    ) {
    $removethesepackages = [
    'telnet-server',
    'telnet',
    # Dozens removed for sanity :-)
    ]
    case $operatingsystem {
    'SLES' : { package {$removethesepackages : ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {$removethesepackages
    : ensure=>purged,} }
    default : {}
    }
    }

    Now SLES runs this code amazingly well because zypper just does a check
    against the local install before trying to remove a package and if it isn't
    installed it doesn't do anything at all. One of the many shortcomings of
    yum is that it always hits the repos. Since we have a local repo set up for
    SLES, Scientific, and CentOS we don't care if we beat up on them. However,
    we discussed the local repo caching with Red Hat and it didn't work out
    (too much complexity for too few servers; won't go into details). Not only
    that, but because every package is a separate transaction, we hit the repo
    /multiple/ times every puppet run. Thus, our servers hit Red Hat repos
    frequently and we get banned. :-(

    So I went thinking about how to solve both the problem of slamming the
    repo and checking for the package before trying to do a removal.

    We have and utilize the pkginventory module[1] (which we have applied the
    waiting patches plus done some of our own since that module hasn't been
    updated in a long while). This gives me a fact of pkg_telnet or
    pkg_telnet_server if those packages are installed. So I decided to utilize
    that and I came up with the code below.

    [1] https://forge.puppetlabs.com/ody/pkginventory

    class audit::software_remove (
    ) {
    if $pkg_telnet_server {
    case $operatingsystem {
    'SLES' : { package { 'telnet-server': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet-server'
    : ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $::pkg_telnet {
    case $operatingsystem {
    'SLES' : { package { 'telnet': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet' :
    ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet is not installed":}}
    }

    Hrm. It works, but it isn't a good solution in my opinion. Not at all. Too
    much code is being duplicated here. Well, there is some clean up I can do
    though now. Previously I used the case statement because audit wanted us to
    do a "yum purge" if any of the packages are found installed but SLES
    doesn't support "purged" and I don't really see a difference. That is a
    pointless case statement in my opinion with far too much code duplication.

    class audit::software_remove (
    ) {
    if $pkg_telnet_server { package { 'telnet-server': ensure=>absent,} }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $pkg_telnet { package { 'telnet': ensure=>absent,} }
    else { notify {"Package telnet is not installed":}}
    }

    Ah. Better. Now there are only 3 lines I have to duplicate (if,else,my
    comment separator I use for my own sanity). Plus, there is a package check
    /before/ the yum command even has a chance to run. I *still* get hit with
    those REALLY annoying double notify messages in the logs (why twice? I
    still do not understand why twice! Ugh...another gripe for another day) but
    at least I don't hit the repo a million times every puppet run and that was
    the real goal here.

    So far in my testing environment, things are going well. Puppet runs are
    MUCH faster not having to hit yum all the time. Logs are a bit more crowded
    but at least they don't say "Software_remove/Package[telnet]/ensure:
    created" which was always a confusing statement in the puppet logs (another
    thing I never understood about puppet logging; how is a remove
    "created"...).

    Yet even still, there is something bothering me about how much code
    duplication just got added. I feel that there is a simpler and better way
    of doing this. I played with a few things I found online, but either they
    were more complex or they still hit the repo really hard.

    So I thought I would ask: Does anyone have any tips/suggestions on how I
    might improve this code a bit more? Any thoughts? Anything jump out at you
    that I could be doing better?

    Thanks! I appreciate it!
    ~Stack~

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com
    <https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAF_B3ddAyrZOE2fQv1yG226it1NFbFwCNaAfyVDxMzCCk1Dh_Q%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Martin Willemsma at Jul 23, 2014 at 8:38 am
    Hi Stack,

    Not sure if this is doable for you, but maybe you can create a schedule to
    schedule your RedHat package removals.

    http://docs.puppetlabs.com/references/latest/type.html#schedule

    For example create a schedule to run this once a day, or not more than x
    times a day and set the package resource defaults in your class
    audit::software_remove to this schedule.

    class audit::software_remove{

       schedule{ 'rhel_repos':
         period => daily,
         repeat => 3,
       }

       Package{ schedule => 'rhel_repos', }

    }

    This ties the schedule to the package resource and will apply it only 3
    times a day. I use this in our setup to something similar with auditing
    resources and the requirements are not to tight that this needs to be
    applied in every run.

    What can I do when yum or up2date command fails with RHN account has been
    disabled for 'Abuse of Service' ?

    This occurs when a machine has been flagged as connecting to Red Hat
    Network (RHN) more frequently than 100 times per day after the first 1500
    check-ins since the system was first registered with Red Hat Network. A
    machine that has been registered for a week would be rejected after 2200
    check-ins.
    There is no advantage to be gained by checking in every 5 minutes that
    cannot be achieved by checking in every 60 or 120 minutes. If the default
    check-in interval of 240 minutes is not enough, then you can configure cron
    jobs or the Red Hat Update Agent daemon to connect with Red Hat Network
    every hour or every other hour, which should be sufficient, and is
    recommended.
    From: https://access.redhat.com/solutions/8165





    2014-07-23 8:37 GMT+02:00 José Luis Ledesma <joseluis.ledesma@gmail.com>:
    Hi,

    I dont know if it will work, but have you tried adding provider => rpm in
    the package resource?

    Regards,
    El 23/07/2014 02:37, "Stack Kororā" <i.am.stack@gmail.com> escribió:

    Greetings,
    In my multiple hundred servers, I have <10 that are Red Hat based. We
    recently brought them under the same management as the rest of the servers
    utilizing Puppet. Then we ran into issues because we were hitting RHN too
    frequently and we got our servers banned. :-(

    I went digging for the culprit and found it in a section we wrote because
    audit is insistent that some packages should never be installed and they
    want regular checks that the packages are not installed. I rather liked my
    original solution (below) as I have dozens of packages that shouldn't be
    installed and occasionally I get another one to add to the list. This code
    made it really simple to add a new package.

    class audit::software_remove (
    ) {
    $removethesepackages = [
    'telnet-server',
    'telnet',
    # Dozens removed for sanity :-)
    ]
    case $operatingsystem {
    'SLES' : { package {$removethesepackages : ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package
    {$removethesepackages : ensure=>purged,} }
    default : {}
    }
    }

    Now SLES runs this code amazingly well because zypper just does a check
    against the local install before trying to remove a package and if it isn't
    installed it doesn't do anything at all. One of the many shortcomings of
    yum is that it always hits the repos. Since we have a local repo set up for
    SLES, Scientific, and CentOS we don't care if we beat up on them. However,
    we discussed the local repo caching with Red Hat and it didn't work out
    (too much complexity for too few servers; won't go into details). Not only
    that, but because every package is a separate transaction, we hit the repo
    /multiple/ times every puppet run. Thus, our servers hit Red Hat repos
    frequently and we get banned. :-(

    So I went thinking about how to solve both the problem of slamming the
    repo and checking for the package before trying to do a removal.

    We have and utilize the pkginventory module[1] (which we have applied the
    waiting patches plus done some of our own since that module hasn't been
    updated in a long while). This gives me a fact of pkg_telnet or
    pkg_telnet_server if those packages are installed. So I decided to utilize
    that and I came up with the code below.

    [1] https://forge.puppetlabs.com/ody/pkginventory

    class audit::software_remove (
    ) {
    if $pkg_telnet_server {
    case $operatingsystem {
    'SLES' : { package { 'telnet-server': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet-server'
    : ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $::pkg_telnet {
    case $operatingsystem {
    'SLES' : { package { 'telnet': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet' :
    ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet is not installed":}}
    }

    Hrm. It works, but it isn't a good solution in my opinion. Not at all.
    Too much code is being duplicated here. Well, there is some clean up I can
    do though now. Previously I used the case statement because audit wanted us
    to do a "yum purge" if any of the packages are found installed but SLES
    doesn't support "purged" and I don't really see a difference. That is a
    pointless case statement in my opinion with far too much code duplication.

    class audit::software_remove (
    ) {
    if $pkg_telnet_server { package { 'telnet-server': ensure=>absent,} }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $pkg_telnet { package { 'telnet': ensure=>absent,} }
    else { notify {"Package telnet is not installed":}}
    }

    Ah. Better. Now there are only 3 lines I have to duplicate (if,else,my
    comment separator I use for my own sanity). Plus, there is a package check
    /before/ the yum command even has a chance to run. I *still* get hit with
    those REALLY annoying double notify messages in the logs (why twice? I
    still do not understand why twice! Ugh...another gripe for another day) but
    at least I don't hit the repo a million times every puppet run and that was
    the real goal here.

    So far in my testing environment, things are going well. Puppet runs are
    MUCH faster not having to hit yum all the time. Logs are a bit more crowded
    but at least they don't say "Software_remove/Package[telnet]/ensure:
    created" which was always a confusing statement in the puppet logs (another
    thing I never understood about puppet logging; how is a remove
    "created"...).

    Yet even still, there is something bothering me about how much code
    duplication just got added. I feel that there is a simpler and better way
    of doing this. I played with a few things I found online, but either they
    were more complex or they still hit the repo really hard.

    So I thought I would ask: Does anyone have any tips/suggestions on how I
    might improve this code a bit more? Any thoughts? Anything jump out at you
    that I could be doing better?

    Thanks! I appreciate it!
    ~Stack~

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com
    <https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/CAF_B3ddAyrZOE2fQv1yG226it1NFbFwCNaAfyVDxMzCCk1Dh_Q%40mail.gmail.com
    <https://groups.google.com/d/msgid/puppet-users/CAF_B3ddAyrZOE2fQv1yG226it1NFbFwCNaAfyVDxMzCCk1Dh_Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.


    --
    Met vriendelijke groet, Kind Regards,

    Martin Willemsma

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFcmyyp3r1tTxPVo16oE5rgSO7ozF2AYw4M6Y-cHx%3DCcR4qCUA%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stack Kororā at Jul 23, 2014 at 12:04 pm

    On Wednesday, July 23, 2014 3:38:26 AM UTC-5, mawi wrote:
    Hi Stack,

    Not sure if this is doable for you, but maybe you can create a schedule to
    schedule your RedHat package removals.

    http://docs.puppetlabs.com/references/latest/type.html#schedule

    For example create a schedule to run this once a day, or not more than x
    times a day and set the package resource defaults in your class
    audit::software_remove to this schedule.

    class audit::software_remove{

    schedule{ 'rhel_repos':
    period => daily,
    repeat => 3,
    }

    Package{ schedule => 'rhel_repos', }

    }

    This ties the schedule to the package resource and will apply it only 3
    times a day. I use this in our setup to something similar with auditing
    resources and the requirements are not to tight that this needs to be
    applied in every run.
    This is a good idea. I wish I had thought of that sooner. Thanks!

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/edf10f6c-4cf0-476f-bedb-dbe340a618ce%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stack Kororā at Jul 23, 2014 at 11:59 am
    Greetings,

    I did try this but ran into issues in my testing when a package had
    dependencies. I don't remember what the problem was though...I should
    probably explore this a bit more.

    Thanks!
    On Wednesday, July 23, 2014 1:37:41 AM UTC-5, Jose Luis Ledesma wrote:

    Hi,

    I dont know if it will work, but have you tried adding provider => rpm in
    the package resource?

    Regards,
    El 23/07/2014 02:37, "Stack Kororā" <i.am....@gmail.com <javascript:>>
    escribió:
    Greetings,

    In my multiple hundred servers, I have <10 that are Red Hat based. We
    recently brought them under the same management as the rest of the servers
    utilizing Puppet. Then we ran into issues because we were hitting RHN too
    frequently and we got our servers banned. :-(

    I went digging for the culprit and found it in a section we wrote because
    audit is insistent that some packages should never be installed and they
    want regular checks that the packages are not installed. I rather liked my
    original solution (below) as I have dozens of packages that shouldn't be
    installed and occasionally I get another one to add to the list. This code
    made it really simple to add a new package.

    class audit::software_remove (
    ) {
    $removethesepackages = [
    'telnet-server',
    'telnet',
    # Dozens removed for sanity :-)
    ]
    case $operatingsystem {
    'SLES' : { package {$removethesepackages : ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package
    {$removethesepackages : ensure=>purged,} }
    default : {}
    }
    }

    Now SLES runs this code amazingly well because zypper just does a check
    against the local install before trying to remove a package and if it isn't
    installed it doesn't do anything at all. One of the many shortcomings of
    yum is that it always hits the repos. Since we have a local repo set up for
    SLES, Scientific, and CentOS we don't care if we beat up on them. However,
    we discussed the local repo caching with Red Hat and it didn't work out
    (too much complexity for too few servers; won't go into details). Not only
    that, but because every package is a separate transaction, we hit the repo
    /multiple/ times every puppet run. Thus, our servers hit Red Hat repos
    frequently and we get banned. :-(

    So I went thinking about how to solve both the problem of slamming the
    repo and checking for the package before trying to do a removal.

    We have and utilize the pkginventory module[1] (which we have applied the
    waiting patches plus done some of our own since that module hasn't been
    updated in a long while). This gives me a fact of pkg_telnet or
    pkg_telnet_server if those packages are installed. So I decided to utilize
    that and I came up with the code below.

    [1] https://forge.puppetlabs.com/ody/pkginventory

    class audit::software_remove (
    ) {
    if $pkg_telnet_server {
    case $operatingsystem {
    'SLES' : { package { 'telnet-server': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet-server'
    : ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $::pkg_telnet {
    case $operatingsystem {
    'SLES' : { package { 'telnet': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet' :
    ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet is not installed":}}
    }

    Hrm. It works, but it isn't a good solution in my opinion. Not at all.
    Too much code is being duplicated here. Well, there is some clean up I can
    do though now. Previously I used the case statement because audit wanted us
    to do a "yum purge" if any of the packages are found installed but SLES
    doesn't support "purged" and I don't really see a difference. That is a
    pointless case statement in my opinion with far too much code duplication.

    class audit::software_remove (
    ) {
    if $pkg_telnet_server { package { 'telnet-server': ensure=>absent,} }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $pkg_telnet { package { 'telnet': ensure=>absent,} }
    else { notify {"Package telnet is not installed":}}
    }

    Ah. Better. Now there are only 3 lines I have to duplicate (if,else,my
    comment separator I use for my own sanity). Plus, there is a package check
    /before/ the yum command even has a chance to run. I *still* get hit with
    those REALLY annoying double notify messages in the logs (why twice? I
    still do not understand why twice! Ugh...another gripe for another day) but
    at least I don't hit the repo a million times every puppet run and that was
    the real goal here.

    So far in my testing environment, things are going well. Puppet runs are
    MUCH faster not having to hit yum all the time. Logs are a bit more crowded
    but at least they don't say "Software_remove/Package[telnet]/ensure:
    created" which was always a confusing statement in the puppet logs (another
    thing I never understood about puppet logging; how is a remove
    "created"...).

    Yet even still, there is something bothering me about how much code
    duplication just got added. I feel that there is a simpler and better way
    of doing this. I played with a few things I found online, but either they
    were more complex or they still hit the repo really hard.

    So I thought I would ask: Does anyone have any tips/suggestions on how I
    might improve this code a bit more? Any thoughts? Anything jump out at you
    that I could be doing better?

    Thanks! I appreciate it!
    ~Stack~

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users...@googlegroups.com <javascript:>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com
    <https://groups.google.com/d/msgid/puppet-users/548784ca-1894-4a00-b98d-b501815c9d0d%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/bc625f51-8704-43c1-80da-c674fe4b4b12%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jcbollinger at Jul 23, 2014 at 1:08 pm

    On Wednesday, July 23, 2014 6:59:19 AM UTC-5, Stack Kororā wrote:
    Greetings,

    I did try this but ran into issues in my testing when a package had
    dependencies. I don't remember what the problem was though...I should
    probably explore this a bit more.
    Yes, if you need to ensure 'purged' rather than just absent then the 'rpm'
    provider is not an option.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/71ccd66c-56c6-4054-b7ce-32b0da7ddc3c%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jcbollinger at Jul 23, 2014 at 1:53 pm

    On Tuesday, July 22, 2014 7:37:30 PM UTC-5, Stack Kororā wrote:
    Greetings,

    In my multiple hundred servers, I have <10 that are Red Hat based. We
    recently brought them under the same management as the rest of the servers
    utilizing Puppet. Then we ran into issues because we were hitting RHN too
    frequently and we got our servers banned. :-(

    I went digging for the culprit and found it in a section we wrote because
    audit is insistent that some packages should never be installed and they
    want regular checks that the packages are not installed. I rather liked my
    original solution (below) as I have dozens of packages that shouldn't be
    installed and occasionally I get another one to add to the list. This code
    made it really simple to add a new package.

    class audit::software_remove (
    ) {
    $removethesepackages = [
    'telnet-server',
    'telnet',
    # Dozens removed for sanity :-)
    ]
    case $operatingsystem {
    'SLES' : { package {$removethesepackages : ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {$removethesepackages
    : ensure=>purged,} }
    default : {}
    }
    }

    Now SLES runs this code amazingly well because zypper just does a check
    against the local install before trying to remove a package and if it isn't
    installed it doesn't do anything at all. One of the many shortcomings of
    yum is that it always hits the repos.

    But that's what's curious about your issue: it's not normal for yum to hit
    the repos to service every request. Yum (by default) maintains local
    caches of repo metadata to avoid that. At least on CentOS. I am not
    positive that *bona fide* RHEL has the same default, but it certainly has
    the same capability.


    Since we have a local repo set up for SLES, Scientific, and CentOS we
    don't care if we beat up on them. However, we discussed the local repo
    caching with Red Hat and it didn't work out (too much complexity for too
    few servers; won't go into details).

    Have you considered setting up a caching proxy between you and them?


    Not only that, but because every package is a separate transaction, we hit
    the repo /multiple/ times every puppet run. Thus, our servers hit Red Hat
    repos frequently and we get banned. :-(

    Again, this doesn't make sense. What per-package request(s) is yum
    actually making? It does not even request repo metadata for me when I ask
    it to remove a package that is not installed (much less any
    package-specific data).


    So I went thinking about how to solve both the problem of slamming the
    repo and checking for the package before trying to do a removal.

    We have and utilize the pkginventory module[1] (which we have applied the
    waiting patches plus done some of our own since that module hasn't been
    updated in a long while). This gives me a fact of pkg_telnet or
    pkg_telnet_server if those packages are installed. So I decided to utilize
    that and I came up with the code below.

    [1] https://forge.puppetlabs.com/ody/pkginventory

    class audit::software_remove (
    ) {
    if $pkg_telnet_server {
    case $operatingsystem {
    'SLES' : { package { 'telnet-server': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet-server'
    : ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $::pkg_telnet {
    case $operatingsystem {
    'SLES' : { package { 'telnet': ensure=>absent,} }
    'Scientific', 'CentOS', 'RedHat' : { package {'telnet' :
    ensure=>purged,} }
    default : {}
    }
    }
    else { notify {"Package telnet is not installed":}}
    }

    Hrm. It works, but it isn't a good solution in my opinion. Not at all. Too
    much code is being duplicated here. Well, there is some clean up I can do
    though now. Previously I used the case statement because audit wanted us to
    do a "yum purge" if any of the packages are found installed but SLES
    doesn't support "purged" and I don't really see a difference. That is a
    pointless case statement in my opinion with far too much code duplication.

    class audit::software_remove (
    ) {
    if $pkg_telnet_server { package { 'telnet-server': ensure=>absent,} }
    else { notify {"Package telnet-server is not installed":}}
    #----------------
    if $pkg_telnet { package { 'telnet': ensure=>absent,} }
    else { notify {"Package telnet is not installed":}}
    }
    How about this:

    define audit::forbidden_package () {
       $is_present = inline_template(
         '<%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %>')

       if $is_present {
         package { $title: ensure => absent }
       } else {
         notify { "Package $title is not installed": }
       }
    }

    class audit::software_remove (
    ) {
         $forbidden_packages = [
                     'telnet-server',
                     'telnet',
                     # Dozens removed for sanity :-)
                     ]
       audit::forbidden_package { $forbidden_packages: }
    }


    Now you're back to having *no* lines to duplicate.


    Ah. Better. Now there are only 3 lines I have to duplicate (if,else,my
    comment separator I use for my own sanity). Plus, there is a package check
    /before/ the yum command even has a chance to run. I *still* get hit with
    those REALLY annoying double notify messages in the logs (why twice? I
    still do not understand why twice! Ugh...another gripe for another day)

    I'm uncertain what you're talking about, but maybe it's that the log
    records when the notify resource is synced, and the result of syncing it is
    to output a message to the log. If you do not specify anything else via
    its 'message' parameter, then a notify's message is the same as its title.
    You might be able to improve that particular issue by adding "loglevel =>
    'debug'" to your Notifies (the theory being that it would suppress logging
    the sync, but not the message).


    but at least I don't hit the repo a million times every puppet run and
    that was the real goal here.

    So far in my testing environment, things are going well. Puppet runs are
    MUCH faster not having to hit yum all the time. Logs are a bit more crowded
    but at least they don't say "Software_remove/Package[telnet]/ensure:
    created" which was always a confusing statement in the puppet logs (another
    thing I never understood about puppet logging; how is a remove
    "created"...).

    Yet even still, there is something bothering me about how much code
    duplication just got added. I feel that there is a simpler and better way
    of doing this. I played with a few things I found online, but either they
    were more complex or they still hit the repo really hard.

    So I thought I would ask: Does anyone have any tips/suggestions on how I
    might improve this code a bit more? Any thoughts? Anything jump out at you
    that I could be doing better?
    See above.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/83848bc3-ed80-4fcd-8604-689da2a5d303%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Stack Kororā at Jul 23, 2014 at 11:26 pm
    Greetings!

    Thank you so much John. I just learned something new about Puppet. Utilizing inline_template is a heck of a lot easier then how I first attempted that variable substitution. I might have to go back and fix some of my older code later....


    Here are a few other notes in response to your email:
    Have you considered setting up a caching proxy between you and them?
    We have discussed doing a caching proxy, but haven't ever had the time/inclination to implement one yet.

    ----
    What per-package request(s) is yum actually making?
    I explored the yum thing a bit more. Running puppet-3.6.2-1.el6.noarch on both server and client using CentOS6 as my test systems. I started a puppet run in one terminal and ran this code in a second:
    $ while [ "$(pgrep puppet)" != "" ]; do pgrep yum; done | uniq

    If I just do this:
    package { 'telnet' : ensure=>absent,}

    Nothing triggers.

    If I do it this way:
    $removethesepackages = [
                     'telnet-server',
                     'telnet',
                     ]
    package {$removethesepackages : ensure=>absent,}

    Then I get a yum PID per package. For every PID I get a line in the puppet log like this:
    Notice: /Stage[main]/audit::Software_disabled/Package[telnet]/ensure: created

    (there is that weird error message again where an absent is "created").

    I don't know why. Both work as expected, but the second triggers a yum call the first doesn't.

    So I thought, 'Maybe it is hitting local cache and not actually going out to the repo'. I dug around in the logs on our local repo and found this:
    [IP REMOVED] - - [23/Jul/2014:14:07:58 -0500] "GET /puppetlabs/6/products/x86_64/repodata/repomd.xml HTTP/1.1" 200 2529 "-" "urlgrabber/3.9.1 yum/3.2.29"

    It isn't one per package, but it is one per puppet run. Something about that method calls yum differently I guess. Not sure why.

    ---

    The double notice I was referring to is this:
    Notice: Package telnet is not installed
    Notice: /Stage[main]/audit::Software_disabled/audit::Forbidden_package[telnet]/Notify[Package telnet is not installed]/message: defined 'message' as 'Package telnet is not installed'

    I am told three times in two lines (more with wrap around on a console) that telnet isn't installed. I find it annoying and haven't found a solution to removing it yet and leaving just the first Notice. If you know of one I would be /very/ grateful.

    ----
    I implemented your code and it is working brilliantly. I made two changes.
    1) I placed the define in init.pp so I can reference it anywhere in the audit class easily.
    2) I changed:
      '<%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %>')
    to:
      "<%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %>")

    Using the single quotes gave me the error:
    Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Syntax error at '::pkg_'; expected ')' at /etc/puppet/modules/audit/manifests/software_disabled.pp:8 on node centos6.testing.puppet
    Warning: Not using cache on failed catalog
    Error: Could not retrieve catalog; skipping run

    But now it is working really well in my dev environment! I push to production tomorrow...We will see how pleased I am with my code changes at the end of the day after this fix + 4 other "minor" changes roll out. :-D

    Thank you to everyone who has chimed in. These responses are exactly what I was looking for. I have learned more about puppet and have a few new tricks to use. I really do appreciate it.

    Thanks!
    ~Stack~


    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1d09b25c-55ab-486b-a6c4-e31803b9b813%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jcbollinger at Jul 24, 2014 at 1:38 pm

    On Wednesday, July 23, 2014 6:26:00 PM UTC-5, Stack Kororā wrote:
    Greetings!

    Thank you so much John. I just learned something new about Puppet. Utilizing inline_template is a heck of a lot easier then how I first attempted that variable substitution. I might have to go back and fix some of my older code later....


    Here are a few other notes in response to your email:
    Have you considered setting up a caching proxy between you and them?
    We have discussed doing a caching proxy, but haven't ever had the time/inclination to implement one yet.

    ----
    What per-package request(s) is yum actually making?
    I explored the yum thing a bit more. Running puppet-3.6.2-1.el6.noarch on both server and client using CentOS6 as my test systems. I started a puppet run in one terminal and ran this code in a second:
    $ while [ "$(pgrep puppet)" != "" ]; do pgrep yum; done | uniq

    If I just do this:
    package { 'telnet' : ensure=>absent,}

    Nothing triggers.

    If I do it this way:
    $removethesepackages = [
    'telnet-server',
    'telnet',
    ]
    package {$removethesepackages : ensure=>absent,}

    Then I get a yum PID per package. For every PID I get a line in the puppet log like this:
    Notice: /Stage[main]/audit::Software_disabled/Package[telnet]/ensure: created

    (there is that weird error message again where an absent is "created").

    I don't know why. Both work as expected, but the second triggers a yum call the first doesn't.
    I find it surprising -- and in fact unlikely -- that using an array
    resource title produces different behavior than does a sequence of separate
    resource declaration for each of the packages. If that is in fact the case
    then I encourage you to file a bug report, but I'm pretty sure you're mixed
    up. In particular, Puppet will *never* invoke 'yum' to ensure a package
    'absent' -- the 'yum' provider intentionally uses 'rpm -e' for that.
    Puppet uses 'yum remove' only to ensure a package 'purged'.

    In any case, the issue is not so much whether Puppet spawns yum processes;
    rather it is what yum *does* once spawned. Yum should not be hitting RHN
    to service a request to remove a package, whether that package is in fact
    installed or not. Furthermore, the number of RHN requests needed to serve
    all other package management needs should be minimized by yum's metadata
    caching. Even if you instructed Puppet to clear yum metadata on each run,
    you should hit RHN only once per Puppet run, except when you install or
    update a package. If you see different behavior then you should
    investigate yum's own configuration.


    So I thought, 'Maybe it is hitting local cache and not actually going out to the repo'. I dug around in the logs on our local repo and found this:
    [IP REMOVED] - - [23/Jul/2014:14:07:58 -0500] "GET /puppetlabs/6/products/x86_64/repodata/repomd.xml HTTP/1.1" 200 2529 "-" "urlgrabber/3.9.1 yum/3.2.29"

    It isn't one per package, but it is one per puppet run. Something about that method calls yum differently I guess. Not sure why.
    Puppet executes a "yum list --all" once per run so it can tell what to do
    with packages ensured anything other than 'absent' or 'purged'. This is
    part of a strategy to avoid needlessly running yum to install packages that
    aren't available or to update a packages that are already at the latest
    available version. But it should be served from yum's cache most of the
    time. If you see these frequently going out to RHN, then again, something
    is weird about your yum configuration.


    ---

    The double notice I was referring to is this:
    Notice: Package telnet is not installed
    Notice: /Stage[main]/audit::Software_disabled/audit::Forbidden_package[telnet]/Notify[Package telnet is not installed]/message: defined 'message' as 'Package telnet is not installed'

    I am told three times in two lines (more with wrap around on a console) that telnet isn't installed. I find it annoying and haven't found a solution to removing it yet and leaving just the first Notice. If you know of one I would be /very/ grateful.
    So that's just what I supposed you might be talking about. Did you try my
    suggestion to set "loglevel => 'debug'" on the Notice resources?


    ----
    I implemented your code and it is working brilliantly. I made two changes.
    1) I placed the define in init.pp so I can reference it anywhere in the audit class easily.
    Sorry, I didn't intend to imply a specific file to put the define in. The
    standard place for a define named audit::forbidden_package would be <module
    path>/audit/manifests/forbidden_package.pp (same rule as for classes).
    Putting it there is even more likely to ensure that Puppet can always find
    it. If you follow that pattern systematically then it also helps *you*
    find it.


    2) I changed:
    '<%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %>')
    to:
    "<%= scope.lookupvar('::pkg_' + @title.gsub('-', '_')) %>")

    Using the single quotes gave me the error:
    Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Syntax error at '::pkg_'; expected ')' at /etc/puppet/modules/audit/manifests/software_disabled.pp:8 on node centos6.testing.puppet
    Warning: Not using cache on failed catalog
    Error: Could not retrieve catalog; skipping run
    Oops. Sorry about that.


    But now it is working really well in my dev environment! I push to production tomorrow...We will see how pleased I am with my code changes at the end of the day after this fix + 4 other "minor" changes roll out. :-D
    Great! I'm glad it's working for you.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7a770d55-3200-4972-bc39-3f0ec5e59725%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedJul 23, '14 at 12:37a
activeJul 24, '14 at 1:38p
posts9
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase