[Puppet Users] system users class hogging whole cpu

On Mon, Jun 11, 2012 at 11:46 AM, Azfar Hashmi wrote:
Any idea why it always hanging at this class?


Without some code to look at it would be hard for anybody to take a guess...


Peter L. Berghold salty.cowdawg@gmail.com

http://blog.berghold.net
“"If A equals success, then the formula is A equals X plus Y and Z, with X
being work, Y play, and Z keeping your mouth shut." - Albert
Einstein<http://www.quotesdaddy.com/quote/1184670/albert-einstein/if-a-equals-success-then-the-formula-is-a-equals-x>
” Get this email app!
<http://www.wisestamp.com/apps/quotes?utm_source=extension&utm_medium=email&utm_term=quotes&utm_campaign=apps>

Want a signature like mine?
<http://r1.wisestamp.com/r/landing?promo=16&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_16>
Click
here.<http://r1.wisestamp.com/r/landing?promo=16&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_16>




>

The NSS configuration on that node is different, causing an adduser call
to run through the complete user directory to find a free uid.


Also, what Peter said.

Best Regards, David

Hi,
Sorry I forgot to add the code, below is my class.

class std_debian_users {

#We create users

users::useraccount { "user1":
ensure => present,
fullname => "user",
password =>
'$6$UCKmcpzC$.bgwFSgNNaauzWk7WUyO9dohTRRy5Li.Xd0KddJxICEECMESAbt1XThDc770V1hYiyGBdc5vekq5N4njQv6bZ1',
shell => '/bin/bash'
}

users::useraccount { "user2":
ensure => present,
fullname => "user2",
password =>
'$6$qBa42BcV$oKXSwPgUKlvDTX71U5Of42wP9G8CgXUn/UYQei5bKiN5xXowtQaFr/n1wiwe/BTpykrFkaOngBPSrHen7rVmV1',
shell => '/bin/bash'
}

users::useraccount { "user3":
ensure => present,
fullname => "user3",
password =>
'$6$atO.haGi$28ZSmzB/fcHm3wcqFWxjUcGvEha0Ea0h2RYHUwvqjjbxjIw881GJxgPScI.baoE38iEWwrAaZfUtgu5pCS9NJ/',
shell => '/bin/bash'
}

users::useraccount { "user4":
ensure => present,
fullname => "user4",
password =>
'$6$RdmLIN/Q$g2PpxbYrmv4sBRHIVn3k8xgc0ZbbaCux3RiBKqArM7A.RkGzuJajaD3cbZUYvHAz63iTEdA1y88SDtGz/h8nH/',
shell => '/bin/bash'
}

#users::useraccount { "user5":
# ensure => present,
# fullname => "user5",
# password =>
'$6$998jabhr$yEDSXs4/cdZxAmRjj2H7XHIPngRYsBycGfhFyrcfSeaTnaFuUIcYkZu7bMThbN5Q9p5zlCmzu1PpZXpt3yqIe/',
# shell => '/bin/bash'
#}

users::useraccount { "user6":
ensure => present,
fullname => "user6",
password =>
'$6$5O7svvCF$o6IyT0mTIhd8GeS/C4.Cp.UR4dXo.nwJwWzGVXtN6vGe8AvzF/y.XrbkFR2XLkFt87VauI9z.U6Qtm0pWmOdm/',
shell => '/bin/bash'
}

users::useraccount { "user7":
ensure => present,
fullname => "user7",
password =>
'$6$PUi11dq7$DBjgWj461bZwNv4hKguwownEsU.wmwpuMbAidMGceVs/cr18rrVW.O8ZV2/dwiDLQsZGV38uPQvAfRvcNhaIA/',
shell => '/bin/bash'
}

#SSH Public Keys for users

file { '/home/user1/.ssh/authorized_keys2':
owner => user1,
group => user1,
mode => 0600,
require => Users::Useraccount['user1'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user1/.ssh/authorized_keys2",
}

file { '/home/user2/.ssh/authorized_keys2':
owner => user2,
group => user2,
mode => 0600,
require => Users::Useraccount['user2'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user2/.ssh/authorized_keys2",
}

file { '/home/user3/.ssh/authorized_keys2':
owner => user3,
group => user3,
mode => 0600,
require => Users::Useraccount['user3'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user3/.ssh/authorized_keys2",
}

file { '/home/user4/.ssh/authorized_keys2':
owner => user4,
group => user4,
mode => 0600,
require => Users::Useraccount['user4'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user4/.ssh/authorized_keys2",
}

file { '/home/user5/.ssh/authorized_keys2':
owner => user5,
group => user5,
mode => 0600,
require => Users::Useraccount['user5'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user5/.ssh/authorized_keys2",
}

file { '/home/user6/.ssh/authorized_keys2':
owner => user6,
group => user6,
mode => 0600,
require => Users::Useraccount['user6'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user6/.ssh/authorized_keys2",
}
#file { '/home/user7/.ssh/authorized_keys2':
# owner => user7,
# group => user7,
# mode => 0600,
# require => Users::Useraccount['user7'],
# source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user7/.ssh/authorized_keys2",
# }

}

Peter,

What should I change in nns config to change its behavior without breaking
any other function. BTW all are Debian 6 but on different clouds.

Anyone please

I don't see anything inherently wrong with the class, and the behavior you
describe is not normal for Puppet, so the issue likely arises from a
combination of the node's current configuration with the configuration
details you are applying.

Even with the class, you really haven't given us much to go on. David's
suggestion is as plausible as any; it supposes that there is a very large
number of users already defined to the system, perhaps in an external
directory, so that the system has to perform a lot of work to find
acceptable UIDs for the new users. If he's right, then you should see
similarly high CPU usage when you try to manually add the same users via
the "adduser" command. Also, you should be able to sidestep the problem by
specifying the UIDs that these users should have (via the User resources'
'uid' properties).

The contents of /etc/nsswitch.conf might also provide a clue.


John

Thanks Jhon for the response but I am still in black-hole.
Btw total no of users on the system are only below which are in the class.
So I don't think puppet should take much time to find free UIDs. Regarding
other classess if I disable all classes and and use only this class the
behavior is same no matter how long I remain the process running puppet
just do nothing and consume 100% cpu then I have to kill the process
eventually.

You may get an idea of what it's doing with strace.

strace puppet

Of course, this can be very boring such as millions of calls to poll(),
in which case you need to look at open filedescriptors etc.

HTH,
Felix

After doing what, exactly? Did you actually try adding these users
manually via adduser and/or useradd?


Please forgive me if I am skeptical. Does the command 'puppet resource
user' agree about how many users there are? (And / or does it exhibit
similar behavior?)


Define "eventually". How long have you let it run before killing the job?
What changes has it succeeded in making before you kill it?

Also, is there a particular user or users in that class that trigger the
behavior, or do they all do so individually? And I don't see where you
presented the code for your users::useraccount definition. Perhaps there's
a problem with that. For instance, is it by any chance managing very large
files or a large number of files? Do you have the same problem with plain
User resources that you have with Users::Useraccount?

You need to determine what is actually happening before you can fix it.
Felix's suggestion to strace the puppet run is a good one, but before that
I would look for clues in the agent's and master's logs. Running one or
both with --debug output turned on would make clues more likely to appear
there.


John

After doing what, exactly? Did you actually try adding these users
manually via adduser and/or useradd?

No I didn't add them manually some times puppet works and add them if not
existed some time it just hang.

Please forgive me if I am skeptical. Does the command 'puppet resource
user' agree about how many users there are? (And / or does it exhibit
similar behavior?)

"puppet resource user" finishing timely in 2-3 seconds.

Define "eventually". How long have you let it run before killing the job?
What changes has it succeeded in making before you kill it?

I waited even more then 24 hours.

Also, is there a particular user or users in that class that trigger the
behavior, or do they all do so individually? And I don't see where you
presented the code for your users::useraccount definition. Perhaps there's
a problem with that. For instance, is it by any chance managing very large
files or a large number of files? Do you have the same problem with plain
User resources that you have with Users::Useraccount?
You need to determine what is actually happening before you can fix it.
Felix's suggestion to strace the puppet run is a good one, but before that
I would look for clues in the agent's and master's logs. Running one or
both with --debug output turned on would make clues more likely to appear
there.

I will try that and let you know.

Hello there,

I've done the strace as suggested and I found that puppet is doing an open
and lstat of all files in a user's directory which contains a large number
of files.

I was reading about it and I found about the "recursive => false" used when
dealing with this kind of problem, as it happened before, however I could
not find a way to implement it, assuming this is the solution.

I tried to put it this way:

file { '/home/user5/.ssh/authorized_keys2':
owner => user5,
group => user5,
mode => 0600,
require => Users::Useraccount['user5'],
source => "puppet://
sw-zabbix.abc.com/files/standard_debian/home/user5/.ssh/authorized_keys2",
recurse => remote,
}


but it doesn't make any difference and puppet starts scanning all the
billion of files in the home directory of that user.

Can you please help on this?

Thanks a lot in advance,

Marco

Hello,

After I wrote the other message I actually found where to put the recursion
option, on

./modules/users/manifests/definitions/useraccount.pp

So now it's sorted, or at least the process is not taking ages anymore.

I guess that was it, please confirm :-)

Thanks in advance,

Marco

We couldn't possibly, nor have we been in any position to guess this
reason from the information given so far.

Glad you came through with the debugging.

Please note that puppet agent -dv or failing that puppet agent
--evaltrace would probably have made this even easier for you to spot.

Cheers,
Felix