On Friday, January 25, 2013 9:29:00 AM UTC-6, asq wrote:
W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger
To be clear, it looks like you are talking about counts of defined type *
instances*, not counts of distinct defined types. Is that right?
yes. it's test::rule() and test::rule_inline() in linked manifest.
I am inclined to suspect that the problem is not defined types in
general, but rather the specific defined types you are using. Indeed, your
data seem to support that inasmuch as inlining definitions doesn't cut your
compilation times very much.
we're talking about 5m33s and 4m13s difference in compile. this *is* a lot.
The version using more defined-type instances has more total resources.
The time per resource is very nearly the same. Your compile times
certainly will scale with the total number of resources declared.
Do not confuse defined types with macros. They are bona fide resource
types, functioning like native types in most ways.
I can think of several things might cause long compilation times:
- evaluating many templates, especially complex ones
- making many hiera lookups if your hierarchy definition uses
interpolated non-global variables
- repetition of anything requiring network access, such as syncing
files with a version-control repository, accessing files on a remote file
system, or performing name resolutions.
- calling generate() a lot
That's not an exhaustive list.
please check define rule_inline() in linked pp file - it's nothing fancy.
Fanciness is not the question. Even very simple manifests can make
declarations that are costly to compile.
- 3 static files
- 1 erb (35 lines with 5 variables substitution and 1 if/else clause)
The erb evaluation is probably more costly than everything else about that
particular File declaration, combined. Especially so in that it requires
(in that particular case) up to four separate probes of the file system
just to find the template file.
- 1 hiera call
I believe hiera caches data to the extent that it can do, but depending on
how your hierarchy is defined and how your data are arranged, it could
still be that each one of those hiera() calls involves filesystem probes
and possibly reading and parsing YAML. That's not particularly cheap even
if your data are simple.
- 1 exported resource
I must have overlooked that, but it's not especially cheap, either,
inasmuch as it involves connecting to and querying a DB. It's practically
expensive if the underlying DB is not local, as that then would involve
- 4 calls to custom type (firewall)
Which are probably cheap(ish), but that's not certain.
no generates or other puppetmaster functions, no network stuff, no
dns-related stuff or anything i haven't already hashed-out to test if it
will affect compile times significantly.
Anyway, I think filing a bug now would be premature. It may be that there
is a bona fide performance problem, but if so then I don't think it has
been pinpointed yet.
i filed a bug #18880 anyway, as i sense it might be some design error in
puppet - i can't really see why number of define() calls should relate to
compile times. as they are not "real" resources, there should be no reason
for them to be included in catalog, which also might relate to apply times.
Definition instances certainly are real resources in most ways that
matter. In any event, your compilation times even for the "inline" version
of your manifests seem unreasonably high, which is why I suggested that you
look at other possible performance drags.
In addition to the manifest-specific issues I discussed above, you should
also look at how your master is provisioned and loaded. Puppet is a bit of
a memory hog, so if the available physical RAM is low then Puppet may
perform a lot of expensive swaps between physical and virtual RAM. If the
master is sharing hardware with other services (even other masters in, for
example, a passenger setup), then contention for the disk subsystem could
slow everything, especially if you have SATA (or worse) disks instead of
SCSI. There are many other such possibilities.
On the other hand, I don't mean to suggest that declaration of defined-type
instances is free or cheap. As I said, defined types are not macros, but
rather bona fide resource types. Declaring a defined-type instance
involves more work than simply making all the declarations that appear in
the definition body.