[Puppet Users] defined() implicitly requiring resource reference

Does it do that when Package[package-434] is already declared elsewhere, or
only when it is not?


Both.

Supposing that the target package is not declared elsewhere (so that the
!defined() condition is true) the definition will declare the package
itself to ensure it absent, and in that case you would expect a
relationship between the defined-type instance and the resource declared by
it. If elsewhere you have specific references to that package, applicable
resource parameter defaults, or collectors that will match that package,
then you can get relationships with it that are not evident from the
defined type body.

On the other hand, defined() is evil. Do not use it. Ever. I usually
attribute its malignancy to the parse-order dependency it inherently
creates -- which is indeed a serious problem -- but in this case I think
trying to use it to approach your problem it has also obfuscated your
manifests enough to confuse you about the scope and nature of some of your
other declarations.

Instead of using defined(), you can apply logic farther upstream to make
the correct declaration in the first (one) place or to apply resource
parameter overrides to the correct resources. Alternatively, you can
simply determine by other means what packages need to be ensured absent,
such as by filtering a list of possible packages against a list of packages
that are supposed to be installed. Some of those options may still
susceptible to the problem you observed, however, if relevant relationships
spring from declarations elsewhere, as I described they may do.

For the record, however, no order-of-application relationship should be
implied by the reference itself. Therefore, when the referenced Package is
declared elsewhere (so that the !defined() condition is false), there
should be a relationship between
App::Package::Cleaner::Check_and_remove[foo] and Package[foo] only if that
relationship is declared somewhere else.


John

Hi John,

Thanks for the reply.
Sorry, I should have been clearer that this occurs when Package[package-434] IS declared elsewhere. "!defined(Package[package-434])" therefore is false, so just by referencing the existing declaration within the defined() call it seems to incite an implicit dependency.

I've tried this other ways, but here's an example of why farther upstream logic doesn't work:

define myapp ($requested_package){

   package { $requested_package:
     ensure => present
   }

   define package_cleanup {
     $installed_package = $title

     if $installed_package != $requested_package {
       package { $installed_package:
         ensure => purged
       }
     }
   }

   # assuming a facter fact named 'installed_packages'
   package_cleanup { split($::installed_packages, ','): }
}

# now in the case of:
# $::installed_packages = 'one,two,three'
# with:
myapp { 'oneA': requested_package => 'one' }
myapp { 'twoA': requested_package => 'two' }
myapp { 'oneB': requested_package => 'one' }

# we'd end up with package conflicts because
# Myapp[oneA] will define Package[one] (present)
# then define Package[two], Package[three] (absent),
# and Myapp[twoA] will try and define Package[two]
# (present) and fail with a non-uniqueness error.

I don't see how this is doable without defined() or some other check of the catalog to see what packages are "needed" elsewhere. Do you have any suggestions?

I'd hoped that using a Stage to run after everything else would sort this all out, but I'm now not sure this is correct and also see your earlier post about never using stages [with this and "defined() is evil", which pieces of Puppet /can/ we use safely?].

Thanks,
Tom

If that's really what's happening then you should be able to create a
simple test case that demonstrates it. That would be a worthy subject for
a bug report.



No. defined() is never what you want. It may at times seem expedient, but
it's bad news every time.



I don't much like that general approach in the first place on account of
the $requested_package parameter. That you encounter difficulty when you
try something a bit dodgy should not be surprising.


In fact, despite my dissatisfaction with your approach, you can indeed do
this without defined(), and without even disrupting your current structure
very much. Here's one way I think would work:

# This class ensures all known app packages are
# by default purged
class app::packages {
     $apps = split($::app_packages, ',')
     package { $apps:
         ensure => 'purged'
     }
}

# Overrides the requested package to be declared
# present instead of purged.
define app::myapp($requested_package) {
     include 'app::packages'
     Package<| title == $requested_package |> {
         ensure => 'present'
     }
}

# no separate package_cleanup required

It's not uncommon for stages to cause cycles, but they will not resolve
cycles that are already present without them, because stages only add
relationships to those that are otherwise there. They are not evil in the
same way that defined() is, however. Although I disfavor stages myself and
generally do not recommend them to folks, I do not consider a manifest set
that successfully uses them to be inherently flawed (unlike one that relies
on defined()).

Puppet is broad in scope and rich in features, so it should not be
surprising that a few of its features are of questionable value. Pretty
much any system of comparable size has such. To Puppetlabs's credit, they
do deprecate and eventually remove features that they conclude are obsolete
or irretrievably broken. They have floated the idea of doing this with
defined(), in fact, and that may eventually happen.

Other things I recommend you avoid:

    - most uses of 'import'. Bringing node definitions into site.pp is
    about the only appropriate use of 'import' that comes to mind. Most other
    potential uses are better served by the autoloader.
    - Lexical nesting of classes and/or defined types. This is partly for
    cooperating with the autoloader and partly just a matter of style, but it
    is widely considered a best practice to put each class and definition in
    its own file.
    - parameterized-style class declarations. (That is, declarations of the
    form "class { 'foo': param1 => bar }".) This one is controversial. I
    have written on it frequently in the past, so I will omit the recap unless
    you insist. Parameterized classes themselves are ok in Puppet 3, provided
    that you rely on hiera data bindings or maybe an ENC when you need to
    customize their parameters.

John

"Parameterized classes: the best thing since decaffeinated coffee"

Hi John,

Sorry for the delayed reply.
I'll see what I can do.
OK, understood.
Can you explain this further so I can understand the issue?



OK, I wondered whether we could do something like this however - forgive my naivety - I still can't see how this could be a complete solution without something like defined().

As an example... your above snippet works fine to ensure already installed packages remain installed, but what if we wanted to install a brand new version of app::myapp? Because a 'package' resource with title $requested_package does not yet exist, the Package<||> collector matches no resources and the new package is not installed. The only solution that I can come up with is to check whether such a resource is already defined and, if not, define one.

Your guidance is appreciated.

Tom

Initially, it was mostly a gut feeling. After having had time to step back
from the issue and return to it fresh, I think it's a combination of
things, mostly revolving around what you're actually modeling, and how
you're modeling it.

Basically, the 'myapp' definition represents one package chosen from a list
of mutually exclusive packages. If that's all it is, then its name is
misleading -- it should be more generic -- and it should probably take the
exclusive list as a second parameter. On the other hand, if it is indeed
supposed to be something specific, then it doesn't take much advantage of
that. In particular -- and here's where my previous comment came from --
if it supposed to represent something specific to your application, then
why doesn't it know anything about the application's package names?

Also, if the point is supposed to be that only one version of the
application can be installed at a time, and the definition is specific to
that application, then it really ought to be a class instead.


You appear to have a serious misunderstanding. Resource collectors have no
direct relationship with or dependency on which resources are already
installed on the target system. They work exclusively with resource *
declarations* in your manifests, and they do so at catalog compilation
time. Moreover, they are independent of parse order (though the example
anyway ensures a parse order that would work if collectors were parse-order
dependent).

Explanation of the example:

    - class app::packages declares all of the possible application packages,
    specifying the intended state for each one as 'purged'. If that is the
    only thing applied to the target node then it will cause the removal of
    each and every one of those packages that is installed. ('purged' is
    stronger than 'absent'. The former is more sure to remove the specified
    package, but the latter takes care to avoid causing any other packages to
    be removed, and therefore fails if any other package depends on the target
    package.) It is necessary that the list of possible packages include every
    one that you may want to have installed, so it needs to be updated whenever
    you introduce a new one that you want to manage. That was already a
    requirement for you, however, whether you recognized it or not.
    - Resources of defined type app::myapp ensure that class app::packages
    is declared on the target node (by declaring it itself)
    - Resources of defined type app::myapp override the declaration of the
    target package (made by class app::packages) so that its target state is
    'present' instead of 'purged'. This will cause it to be installed if it is
    not already present, and will avoid removing it if it is present.
    - To use the example, simply declare one or more instances of app::myapp
    for the target node, either in a node block or in some class assigned to
    the node. You may also declare class app::packages directly for any node,
    whether or not they declare any app::myapp instances. That is useful if
    you have nodes on which you want to ensure that no version of the
    application is installed.


John

Hi John
Alas, the whole intention of this is that multiple versions of the package must be installed at the same time. This is where the problem lies, in that we have no way to clean up old unused versions once they're no longer needed.

We have instances of an application MyApp, which utilise a shared Package in many to one relationship:

     Package[one] ___ MyApp[app1]

     Package[two] ___ MyApp[app2]
                 \___ MyApp[app3]

     Package[three] _ MyApp[app4]
                   \_ MyApp[app5]
                   \_ MyApp[app6]

In the above example, if we upgraded MyApp[app2] and MyApp[app3] to Package[three], we'd still have Package[two] installed on all of the hosts but no MyApp instances would be using it. This is the unused Package we're trying to clean up with Puppet (there's no longer a MyApp resource definition which references Package[two]).

There is no mutually exclusive packages or any other such conflicts.

This is where it fell apart for me before, I didn't realise that you assumed $::app_packages was every possible package; in my first post I'd specified that $::app_packages was a fact generated on each node from the list of *currently installed* packages, so it contains all of the packages which have been previously been installed but not packages which may be installed in the future.

I think it's going to be a managerial nightmare to update some such list of all possible packages, so might look for an alternate solution. I don't think Puppet really meets our needs here, unless there's a way we can programmatically parse out all of the versions of the package specified in our manifests to include them in the $::app_packages (so they get defined as 'purged') before building the rest of the catalog which would override them as 'present'. This feels like something that might be possible with the use of some combination of defined() or stages, but I'm keen to not run up against any caveats you've mentioned with those features.
Thanks for your help.

Cheers,
Tom

My apologies for the confusion.




I think you're looking at it the wrong way around. It is a fundamental
problem for you if you don't know what all the packages you (may) want to
manage are. This is not information that ought to be gleaned from your
manifest set, nor even information that you ought to be collecting
dynamically from clients, but rather information which you ought to have a
priori. How you store it and keep it updated is a separate matter that I
have heretofore avoided, but Puppet does have a capable external data
interface, Hiera, that could reasonably be applied to the problem.

Nevertheless, if it is not feasible for you to determine up front all the
packages your apps may be using or may want to use, then you can get by
with a smaller list, consisting of just the union of those packages
currently installed and those you want to be installed. You already have
those data at hand, and it is possible to merge them into the needed list.
For example, rewriting my app::packages this way should help:

class app::packages {
     $apps = unique(concat(split($::app_packages, ','), <wanted_packages>))
     package { $apps:
         ensure => 'purged'
     }
}

That might be paired with app::myapp declarations there or elsewhere of the
form

app::myapp { <wanted_packages>: }

The <wanted_packages> represents an array of the names of the packages you
want installed, which could be a variable (including a parameter) of some
class in the 'app' module (even app::packages itself), or might be a
function call (e.g. to an external data service) or maybe something else.
The unique() and concat() functions are available from PuppetLabs's popular
"stdlib" add-in module.


John

Do note, by the way, that this general approach of relying on a dynamic,
client-supplied list of packages that may need to be removed is inherently
more risky than one based on a centrally managed, static list of packages.
If your custom fact malfunctions (or is subverted) then you could end up
removing packages you did not intend to remove, including packages
essential to the client's operation.


John