[Puppet Users] Puppet/Passenger :: Could not retrieve catalog from remote server:Error 403 on server

Check the owner of config.ru. The owner of this file is who passenger will run the puppetmaster daemon as. I'm guessing that it's not owned by puppet.

Thanks for the reply.

I have checked permissions per the master puppet.conf excerpt below
.
My understanding is that Passenger does not really install anything or
copies files around.
You only create a directory and copy the config.ru into it and change
permissions to puppet.
The only thing that passenger does is to install a Apache module, then you
configure your vhost with that info.

I don't know whether I could blame the problem on any of the other packages
(I.e. ruby), because things work perfectly fine with WEBrick.

Below I added more information. Please let me know If anyone spots
something out of place.

*[[email protected] puppet]# cat puppet.conf*
[main]
user = puppet
group = puppet


*[[email protected] ]# ls -l /var/lib/puppetmaster/*
-rw-r--r-- 1 puppet puppet 431 Sep 27 21:51 config.ru
drwxr-xr-x 2 puppet puppet 4096 Sep 27 21:31 public
drwxr-xr-x 2 puppet puppet 4096 Sep 27 21:31 tmp

*[[email protected] ~]# ps -ef | grep puppet*
avahi 1989 1 0 09:34 ? 00:00:00 avahi-daemon: running
[puppetm01.local]
root 2666 1 0 09:34 ? 00:00:01 /usr/bin/ruby /usr/sbin/puppetd
puppet 9734 9541 2 12:35 ? 00:00:00
master

puppet 9769 1 0 12:35 ? 00:00:00 Rack: /var/lib/puppetmaster



*
[[email protected] ]# grep puppet /etc/passwd*
puppet:x:52:52:Puppet:/var/lib/puppet:/sbin/nologin
puppetdb:x:494:488:PuppetDB daemon:/usr/share/puppetdb:/sbin/nologin
puppet-dashboard:x:492:489:Puppet
Dashboard:/usr/share/puppet-dashboard:/sbin/nologin

*[[email protected] ]# id -a puppet*
uid=52(puppet) gid=52(puppet) groups=52(puppet)
*
[[email protected] ~]# passenger-memory-stats *

-------- Apache processes ---------
PID PPID VMSize Private Name
-----------------------------------
9534 1 26.8 MB 0.3 MB /usr/sbin/httpd
9551 9534 26.7 MB 0.2 MB /usr/sbin/httpd
9552 9534 26.8 MB 0.2 MB /usr/sbin/httpd
9553 9534 27.0 MB 0.5 MB /usr/sbin/httpd
9554 9534 27.0 MB 0.5 MB /usr/sbin/httpd
9555 9534 26.8 MB 0.3 MB /usr/sbin/httpd
9556 9534 26.8 MB 0.2 MB /usr/sbin/httpd
9557 9534 26.9 MB 0.3 MB /usr/sbin/httpd
9558 9534 26.8 MB 0.2 MB /usr/sbin/httpd
9559 9534 26.8 MB 0.2 MB /usr/sbin/httpd
### Processes: 10
### Total private dirty RSS: 3.00 MB


-------- Nginx processes --------

### Processes: 0
### Total private dirty RSS: 0.00 MB


---- Passenger processes ----
PID VMSize Private Name
-----------------------------
9536 6.7 MB 0.2 MB PassengerWatchdog
9539 17.8 MB 0.4 MB PassengerHelperAgent
9541 18.7 MB 4.9 MB Passenger spawn server
9544 13.2 MB 0.4 MB PassengerLoggingAgent
9769 51.8 MB 26.0 MB Rack: /var/lib/puppetmaster
9802 60.6 MB 36.6 MB Passenger ApplicationSpawner:
/usr/share/puppet-dashboard
9808 61.1 MB 37.2 MB Rails: /usr/share/puppet-dashboard
### Processes: 7
### Total private dirty RSS: 105.69 MB

*
[[email protected] ~]# passenger-status --verbose*
----------- General information -----------
max = 12
count = 2
active = 0
inactive = 2
Waiting on global queue: 0

----------- Application groups -----------
/usr/share/puppet-dashboard:
App root: /usr/share/puppet-dashboard
* PID: 9808 Sessions: 0 Processed: 2 Uptime: 58s
URL : http://127.0.0.1:50447
Password: xxxxxxxxxxxxxx

/var/lib/puppetmaster:
App root: /var/lib/puppetmaster
* PID: 9769 Sessions: 0 Processed: 2 Uptime: 1m 56s
URL : http://127.0.0.1:55087
Password: xxxxxxxxxxxxxx

*[[email protected] ~]# tail -f /var/log/httpd/access_log
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:20 -0700] "POST
/production/catalog/puppetm01.example.com HTTP/1.1" 403 138 "-" "-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:20 -0700] "PUT
/production/report/puppetm01.example.com HTTP/1.1" 500 635 "-" "-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:30 -0700] "POST
/production/catalog/puppetm01.example.com HTTP/1.1" 403 138 "-" "-"
xxx.xxx.xxx.xxx - - [28/Sep/2012:12:39:33 -0700] "PUT
/production/report/puppetm01.example.com HTTP/1.1" 403 137 "-" "-"


[root@puppetm01 ~]# find /var/lib/puppet | grep catalog | xargs ls -l
-rw-r-----. 1 root root 13150 Sep 27 21:00
/var/lib/puppet/client_yaml/catalog/puppetm01.example.com.yaml

/var/lib/puppet/client_yaml/catalog:
total 16
-rw-r-----. 1 root root 13150 Sep 27 21:00 puppetm01.example.com.yaml*



Thanks,
LL
-----

Adding more troubleshooting info at the link below.

http://pastebin.com/AvCJSQgk

I recreated the certificates and rebooted the system, but still same result.
I really hope to get to the bottom of this. I cannot find a meaningful
reference anywhere.

----

Does anyone have a hint to address this problem?

Or,

Is this destined to stump many a puppet enthusiast?
If this is a bug, where does one notify puppet labs of it?

LL
----

This is a trivial problem to solve, but only you can do it. tcpdump is your friend.

I don't think this is trivial. If it were, I would have already found the
problem by looking at the obvious things.
What I have seen from several posts is that there's other error similar to
the one I've seen. I even came across a bug report filed a while back with
the same error I see, but I lost the link and cannot find it.

The problem is not even from a client to the master. The testing I've done
is all in the master.

I'll try strace instead of tcpdump, being that this is not a TCP
communication problem over the wire but rather a file or directory access
problem.


LL
----

Um, no. Puppet client talks to the server over the network, even on the same host. You really should listen to advice we provide.

So if the server responds with a 403 error over the network, what exactly
do you think a tcpdump will show? The exact same error message.

This is why you would use strace, to see what is happening inside the
actual process.

Try not being so condescending, particularly when you're wrong.