Two load situations, one pretty high, one relatively low. Usually it's
somewhere in between these.
top - 12:43:32 up 65 days, 3:27, 3 users, load average: 12.89, 9.29, 8.00
Tasks: 249 total, 15 running, 234 sleeping, 0 stopped, 0 zombie
Cpu0 : 83.7%us, 11.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 4.3%hi, 0.3%si,
0.0%st
Cpu1 : 87.7%us, 12.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu2 : 93.0%us, 7.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu3 : 89.4%us, 10.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu4 : 92.4%us, 7.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu5 : 87.1%us, 12.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu6 : 91.4%us, 8.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu7 : 89.4%us, 10.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 12191256k total, 10148368k used, 2042888k free, 156196k buffers
Swap: 2031608k total, 32704k used, 1998904k free, 4762472k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20989 puppet 20 0 296m 195m 2832 R 70.7 1.6 1:21.05 ruby
14894 puppet 20 0 301m 201m 2832 R 68.7 1.7 5:10.94 ruby
22920 puppet 20 0 201m 101m 2832 R 66.4 0.8 0:22.10 ruby
17551 puppet 20 0 284m 183m 2832 R 61.7 1.5 2:17.69 ruby
17555 puppet 20 0 306m 204m 2832 R 57.7 1.7 3:15.24 ruby
22930 puppet 20 0 202m 101m 2820 S 51.8 0.9 0:07.88 ruby
16171 puppet-d 20 0 173m 73m 1788 R 51.4 0.6 5749:13 ruby
15194 puppet-d 20 0 227m 128m 1788 R 44.8 1.1 5743:53 ruby
16159 puppet-d 20 0 227m 128m 1788 R 43.8 1.1 5697:15 ruby
16166 puppet-d 20 0 213m 114m 1788 R 38.8 1.0 5742:11 ruby
16199 puppet-d 20 0 220m 121m 1788 R 36.2 1.0 5766:36 ruby
16216 puppet-d 20 0 216m 117m 1788 R 33.9 1.0 5719:14 ruby
16158 puppet-d 20 0 216m 117m 1788 R 31.5 1.0 5766:53 ruby
23391 root 20 0 188m 84m 3296 S 30.2 0.7 0:03.85 puppet
22140 puppet 20 0 282m 181m 2832 R 29.2 1.5 0:48.71 ruby
23543 root 20 0 442m 31m 7520 R 14.9 0.3 0:00.45 python
15049 mysql 20 0 1170m 20m 4420 S 5.6 0.2 1551:47 mysqld
20997 puppet 20 0 286m 186m 2832 S 5.3 1.6 1:23.59 ruby
674 root 20 0 0 0 0 S 3.3 0.0 43:29.96 flush-253:0
6765 apache 20 0 202m 5264 2464 S 3.3 0.0 1:24.64 httpd
6850 apache 20 0 202m 5264 2464 S 3.3 0.0 1:26.29 httpd
21001 puppet 20 0 207m 107m 2832 S 3.3 0.9 0:48.17 ruby
6751 root 20 0 1461m 4476 2796 S 3.0 0.0 1:29.47
PassengerHelper
6766 apache 20 0 202m 5264 2464 S 3.0 0.0 1:21.27 httpd
6767 apache 20 0 202m 5272 2464 S 3.0 0.0 1:22.50 httpd
6769 apache 20 0 202m 5264 2464 S 2.7 0.0 1:24.32 httpd
6770 apache 20 0 202m 5264 2464 S 2.7 0.0 1:23.05 httpd
6771 apache 20 0 202m 5264 2464 S 2.7 0.0 1:22.95 httpd
7192 apache 20 0 202m 5272 2464 S 2.7 0.0 1:21.21 httpd
9109 apache 20 0 202m 5264 2464 S 2.7 0.0 1:19.35 httpd
22924 apache 20 0 202m 5264 2464 S 2.7 0.0 0:01.04 httpd
6768 apache 20 0 202m 5264 2464 S 2.3 0.0 1:22.45 httpd
6798 apache 20 0 202m 5272 2464 S 2.3 0.0 1:24.98 httpd
7175 apache 20 0 202m 5264 2464 S 1.7 0.0 1:22.83 httpd
7191 apache 20 0 202m 5264 2464 S 1.7 0.0 1:22.30 httpd
15501 puppet-d 20 0 225m 121m 3728 S 1.7 1.0 271:12.00 ruby
6851 apache 20 0 202m 5264 2464 S 1.3 0.0 1:22.33 httpd
16025 puppet 20 0 301m 199m 2832 S 0.7 1.7 4:10.28 ruby
102 root 39 19 0 0 0 S 0.3 0.0 26:32.33 khugepaged
1165 root 20 0 192m 1352 800 S 0.3 0.0 19:24.76 snmpd
6091 puppet-d 20 0 219m 120m 1788 S 0.3 1.0 2814:11 ruby
12485 puppet 20 0 301m 201m 2832 S 0.3 1.7 5:45.50 ruby
23503 root 20 0 15164 1460 1012 R 0.3 0.0 0:00.04 top
1 root 20 0 19228 952 820 S 0.0 0.0 0:07.93 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 2:59.01 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 3487:07 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:14.73 watchdog/0
and
top - 11:51:21 up 65 days, 2:35, 3 users, load average: 6.86, 6.95, 5.70
Tasks: 246 total, 10 running, 236 sleeping, 0 stopped, 0 zombie
Cpu0 : 88.8%us, 8.6%sy, 0.0%ni, 1.3%id, 0.0%wa, 0.7%hi, 0.7%si,
0.0%st
Cpu1 : 85.4%us, 11.6%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu2 : 84.1%us, 15.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu3 : 89.0%us, 9.0%sy, 0.0%ni, 2.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu4 : 88.4%us, 11.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu5 : 83.8%us, 15.9%sy, 0.0%ni, 0.3%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu6 : 85.1%us, 12.3%sy, 0.0%ni, 2.6%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu7 : 86.1%us, 13.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 12191256k total, 10695256k used, 1496000k free, 159008k buffers
Swap: 2031608k total, 36528k used, 1995080k free, 4940364k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6963 puppet 20 0 289m 189m 2832 R 98.0 1.6 3:42.39 ruby
7200 puppet 20 0 296m 195m 2832 R 98.0 1.6 5:16.97 ruby
6806 puppet 20 0 287m 187m 2832 R 90.7 1.6 2:51.12 ruby
16199 puppet-d 20 0 220m 121m 1788 R 90.7 1.0 5756:26 ruby
6815 puppet 20 0 300m 200m 2832 R 90.1 1.7 5:23.84 ruby
9111 puppet 20 0 284m 183m 2832 R 89.1 1.5 2:10.29 ruby
6949 puppet 20 0 298m 197m 2832 S 61.9 1.7 4:16.82 ruby
16067 root 20 0 189m 85m 3296 S 44.0 0.7 0:03.89 puppet
6772 apache 20 0 202m 5264 2464 S 6.0 0.0 0:17.48 httpd
6765 apache 20 0 202m 5264 2464 R 5.6 0.0 0:17.48 httpd
6768 apache 20 0 202m 5264 2464 S 5.3 0.0 0:17.72 httpd
7192 apache 20 0 202m 5272 2464 R 5.3 0.0 0:15.75 httpd
6798 apache 20 0 202m 5272 2464 S 4.6 0.0 0:17.14 httpd
6751 root 20 0 1332m 4308 2796 S 4.3 0.0 0:17.61
PassengerHelper
6766 apache 20 0 202m 5264 2464 S 4.3 0.0 0:17.32 httpd
6771 apache 20 0 202m 5264 2464 S 4.3 0.0 0:16.38 httpd
6850 apache 20 0 202m 5264 2464 R 4.3 0.0 0:16.98 httpd
7175 apache 20 0 202m 5260 2464 S 4.3 0.0 0:15.63 httpd
7180 puppet 20 0 301m 200m 2832 S 4.3 1.7 4:01.09 ruby
10049 apache 20 0 202m 5268 2464 S 4.3 0.0 0:10.43 httpd
10050 apache 20 0 202m 5268 2464 S 4.3 0.0 0:10.90 httpd
6767 apache 20 0 202m 5268 2464 S 4.0 0.0 0:15.59 httpd
6851 apache 20 0 202m 5264 2464 S 3.6 0.0 0:15.80 httpd
7191 apache 20 0 202m 5264 2464 S 3.6 0.0 0:16.16 httpd
15049 mysql 20 0 1170m 20m 4424 S 3.3 0.2 1548:50 mysqld
7064 puppet 20 0 296m 194m 2832 S 2.3 1.6 3:22.13 ruby
9109 apache 20 0 202m 5264 2464 S 1.3 0.0 0:12.50 httpd
15501 puppet-d 20 0 225m 121m 3728 S 1.0 1.0 270:45.64 ruby
13527 root 20 0 15164 1464 1012 R 0.7 0.0 0:00.62 top
4 root 20 0 0 0 0 S 0.3 0.0 3485:56 ksoftirqd/0
10 root RT 0 0 0 0 S 0.3 0.0 0:22.64 watchdog/1
29 root 20 0 0 0 0 S 0.3 0.0 2726:41 ksoftirqd/6
6811 puppet 20 0 298m 199m 2832 S 0.3 1.7 4:01.83 ruby
6908 puppet 20 0 291m 191m 2832 S 0.3 1.6 2:41.72 ruby
7072 puppet 20 0 298m 198m 2832 S 0.3 1.7 3:58.77 ruby
15194 puppet-d 20 0 227m 128m 1788 S 0.3 1.1 5733:11 ruby
1 root 20 0 19228 952 820 S 0.0 0.0 0:07.93 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 2:58.92 migration/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:14.72 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 2:13.03 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 8879:22 ksoftirqd/1
11 root RT 0 0 0 0 S 0.0 0.0 2:14.30 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 6996:32 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:19.37 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 2:18.66 migration/3
Btw: have already upgraded to 3.2.2 - doesn't change a thing.