[Puppet Users] puppet master REST API returns 403 when running under passenger works when running from command line

Hi! Everyone,

puppet agent is not able to fetch any files, plugins or post catalog,
reports to the master. both puppet agent and master are on version 3.0.l,
passenger version 3.0.18 ,

nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/apps/nginx
--conf-path=/apps/nginx/nginx.conf --pid-path=/apps/nginx/run/nginx.pid
--error-log-path=/apps/nginx/logs/error.log
--http-log-path=/apps/nginx/logs/access.log --with-http_ssl_module
--with-http_gzip_static_module
--add-module=/usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/nginx
--add-module=/apps/Downloads/nginx/nginx-auth-ldap-master/

the agent command shows this output

[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose
--server bangvmpllda02.XXXXXX.com
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will
continue:
Warning: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31)
access to /certificate_revocation_list/ca [find] at :106
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources
using 'eval_generate: Error 403 on SERVER: Forbidden request:
10.209.47.31(10.209.47.31) access to /file_metadata/plugins [search] at :106
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER:
Forbidden request: 10.209.47.31(10.209.47.31) access to
/file_metadata/plugins [find] at :106 Could not retrieve file metadata for
puppet://bangvmpllda02.XXXXXX.com/plugins: Error 403 on SERVER: Forbidden
request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find]
at :106
Error: Could not retrieve catalog from remote server: Error 403 on SERVER:
Forbidden request: 10.209.47.31(10.209.47.31) access to
/catalog/blramisr195602.XXXXXX.com [find] at :106
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request:
10.209.47.31(10.209.47.31) access to /report/blramisr195602.XXXXXX.com
[save] at :106

and on master logs I see

[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose
--server bangvmpllda02.XXXXXX.com
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will
continue:
Warning: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31)
access to /certificate_revocation_list/ca [find] at :106
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources
using 'eval_generate: Error 403 on SERVER: Forbidden request:
10.209.47.31(10.209.47.31) access to /file_metadata/plugins [search] at :106
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER:
Forbidden request: 10.209.47.31(10.209.47.31) access to
/file_metadata/plugins [find] at :106 Could not retrieve file metadata for
puppet://bangvmpllda02.XXXXXX.com/plugins: Error 403 on SERVER: Forbidden
request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find]
at :106
Error: Could not retrieve catalog from remote server: Error 403 on SERVER:
Forbidden request: 10.209.47.31(10.209.47.31) access to
/catalog/blramisr195602.XXXXXX.com [find] at :106
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request:
10.209.47.31(10.209.47.31) access to /report/blramisr195602.XXXXXX.com
[save] at :106


I am not sure why is it evaluating things on IP?

I also changed agent setup to following

[main]
# The Puppet log directory.
# The default value is '$vardir/log'.
logdir = /var/log/puppet

# Where Puppet PID files are kept.
# The default value is '$vardir/run'.
rundir = /var/run/puppet

# Where SSL certificates are kept.
# The default value is '$confdir/ssl'.
ssldir = $vardir/ssl
report = true
pluginsync = true
server = devops.XXXXXX.com
certname = blramisr195602.XXXXXX.com
dns_alt_names = 10.209.47.31
modulepath = /etc/puppet/modules

and resigned certifcates on master after clean up, but the puppet master
still blocks it. However If I run through puppet master daemon (without
nginx + passenger) all requests go through.

Is there any specific configuration for Nginx host header etc or in
passenger that I am missing?

BR/
Anadi Misra.