FAQ
Hello,

Stuck in a weird place here. I am trying to do 'puppet master
--compile foo.com', however I am not getting the catalog json. So far,
I have noticed two sort of outputs:

1) The above command results in :
notice: Compiled catalog for foo.com in environment production in 10.60 seconds
Failed to submit 'replace catalog' command for foo.com to PuppetDB at
puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
finished A

Notice that I don't actually get any json & of course, the ssl error.

2) Other output:
Failed when searching for node bar.com: Could not retrieve facts for
bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

Disabling storeconfig allows me to get a proper catalog json.

Regarding the ssl errors, we've established the correctness of
certificates in the relevant places - and done tests using curl and
using ruby net/http. Here's a gist of for the curl and ruby snippet.
https://gist.github.com/alcy/5262866

Any suggestions what could be happening here ? Perhaps in
indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
params or something ?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Mohit Chawla at Mar 28, 2013 at 12:55 pm
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    Notice that I don't actually get any json & of course, the ssl error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Mar 28, 2013 at 1:12 pm
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?

    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?

    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?

    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?

    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to the
    puppetdb host itself?

    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    Notice that I don't actually get any json & of course, the ssl error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Mar 28, 2013 at 1:18 pm
    Hello Ken,

    Thanks for the response.
    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber wrote:
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    Notice that I don't actually get any json & of course, the ssl error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Mar 28, 2013 at 1:37 pm
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.
    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber wrote:
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    Notice that I don't actually get any json & of course, the ssl error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Mar 28, 2013 at 8:54 pm
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need to
    confirm this.
    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.
    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber wrote:
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    Notice that I don't actually get any json & of course, the ssl error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Mar 28, 2013 at 9:44 pm
    Yeah, it does seem very odd though ... if agent works - and the master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need to
    confirm this.
    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.
    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber wrote:
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    Notice that I don't actually get any json & of course, the ssl error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Apr 1, 2013 at 7:49 am
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't happen during
    standard puppet runs as opposed to doing a --compile. Here's the puppet
    trace when doing --compile - https://gist.github.com/alcy/5283712.



    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber wrote:

    Yeah, it does seem very odd though ... if agent works - and the master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need to
    confirm this.
    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.
    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber wrote:
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So
    far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in
    10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to PuppetDB
    at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the ssl
    error.
    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts
    for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en
    .
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Apr 1, 2013 at 8:33 am
    Hi,

    Got it working by passing --certname param (the ca server hostname, instead
    of this master).



    On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla
    wrote:
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't happen
    during standard puppet runs as opposed to doing a --compile. Here's the
    puppet trace when doing --compile - https://gist.github.com/alcy/5283712.



    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber wrote:

    Yeah, it does seem very odd though ... if agent works - and the master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need to
    confirm this.
    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.

    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    So I have some questions, as the error could mean a number of things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to
    the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So
    far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in
    10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the ssl
    error.
    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts
    for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl and
    using ruby net/http. Here's a gist of for the curl and ruby
    snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en
    .
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Apr 1, 2013 at 8:57 am
    Slightly OT here - shouldn't setting ca_name be enough for --compile to not
    fail ? Why does it need the --certname param ?


    On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla
    wrote:
    Hi,

    Got it working by passing --certname param (the ca server hostname,
    instead of this master).



    On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla <
    mohit.chawla.binary@gmail.com> wrote:
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't happen
    during standard puppet runs as opposed to doing a --compile. Here's the
    puppet trace when doing --compile - https://gist.github.com/alcy/5283712.




    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber wrote:

    Yeah, it does seem very odd though ... if agent works - and the master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need to
    confirm this.

    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.

    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    So I have some questions, as the error could mean a number of
    things:
    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their
    fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf to
    talk to your puppetdb server match the SSL certificate assigned to
    the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json.
    So far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production
    in 10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the ssl
    error.
    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts
    for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl
    and
    using ruby net/http. Here's a gist of for the curl and ruby
    snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct
    ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com
    .
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en
    .
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Apr 2, 2013 at 10:16 am
    If you specify 'certname' it will use the local certificate with that
    name as apposed to using the certificate with the same name as the
    boxes fqdn (ie. facter fqdn).

    For example, if I use 'puppet master --compile puppetdb1.vm' on its
    own, it uses my local fqdn (determined with `facter fqdn`) as an
    indicator of where the certificate is:
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname
    foo it uses /var/lib/puppet/ssl/certs/foo.pem.

    I presume that the certificate that matches the fqdn on the host you
    are testing is probably _not_ signed by the CA that is loaded into
    PuppetDB. Its just a guess, but we should confirm this.

    Lets take a step back a bit and do an exercise. On your PuppetDB node,
    in /etc/puppetdb/ssl/ you have three files:

    * truststore.jks
    * keystore.jks
    * puppetdb_keystore_pw.txt

    If you look inside truststore.jks, you'll see that it contains the CA
    certificate:

    # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
    Enter keystore password: (this is contained inside puppetdb_keystore_pw.txt)

    Keystore type: JKS
    Keystore provider: SUN

    Your keystore contains 1 entry

    puppetdb ca, 28-Mar-2013, trustedCertEntry,
    Certificate fingerprint (SHA1):
    B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    This CA certificate is the same as what matches my CA file here, we
    can test this by matching the fingerprint:

    # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint
    SHA1 Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    For client validation, this is really what matters - the client
    certificate must be signed by this CA certificate to be accepted by
    PuppetDB. It doesn't matter what the client thinks its CA is in Puppet
    configuration or whatever (so setting --ca_name is not enough here),
    all that matters is the client certificate must have been signed by
    that CA, and it is verified using OpenSSL techniques on the PuppetDB
    server during SSL negotiation.

    You can do this manually for example. Lets confirm this on my master -
    we know that my local CA is confirmed as the thing loaded into the
    PuppetDB trust store, so now we just run the verify routine against my
    certificate:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet
    master --configprint hostcert`
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK

    You must be sure to test this host certificate against the exact same
    CA loaded into PuppetDB ... not just the local CA (in case they differ
    - you can check this with the -fingerprint command).

    If the certificate on the other hand is NOT signed by the CA loaded
    into PuppetDB you get something like:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm
    error 20 at 0 depth lookup:unable to get local issuer certificate

    Which is by design for this example - that certificate 'puppetdb2.vm'
    was a certificate I copied from another Puppetmaster which has a
    separate CA, and thus it fails verification. In the real world,
    puppetdb2.vm would get reject by PuppetDB - which is the correct
    behaviour.

    ken.

    On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla
    wrote:
    Slightly OT here - shouldn't setting ca_name be enough for --compile to not
    fail ? Why does it need the --certname param ?

    On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla wrote:

    Hi,

    Got it working by passing --certname param (the ca server hostname,
    instead of this master).



    On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla
    wrote:
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't happen during
    standard puppet runs as opposed to doing a --compile. Here's the puppet
    trace when doing --compile - https://gist.github.com/alcy/5283712.



    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber wrote:

    Yeah, it does seem very odd though ... if agent works - and the master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need to
    confirm this.

    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to
    run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.

    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    So I have some questions, as the error could mean a number of
    things:

    What version of PuppetDB are you running? And what exact version of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you
    getting
    any trouble with replace facts/replace catalogs generally? ie. Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their
    fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf
    to
    talk to your puppetdb server match the SSL certificate assigned to
    the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json. So
    far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production in
    10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the ssl
    error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve facts
    for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished
    A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl
    and
    using ruby net/http. Here's a gist of for the curl and ruby
    snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct
    ssl
    params or something ?
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Apr 3, 2013 at 6:41 am
    Hey Ken, that definitely cleared some misconceptions, thanks !

    I now know what the problem was. First, I assumed that the client must've
    the same certificate as in puppetdb's truststore. I didn't realize that any
    cert signed by that CA should be able to make calls. In our scenario, where
    we have separate certs for the agent and master component on the same
    server (under /etc/puppet/ssl & /var/lib/puppet/ssl respectively), invoking
    --compile without giving the certname meant no certificate at all ! And
    so, passing --ssldir to use the agent's certs (signed by the same CA)
    worked.
    On Tue, Apr 2, 2013 at 3:40 PM, Ken Barber wrote:

    If you specify 'certname' it will use the local certificate with that
    name as apposed to using the certificate with the same name as the
    boxes fqdn (ie. facter fqdn).

    For example, if I use 'puppet master --compile puppetdb1.vm' on its
    own, it uses my local fqdn (determined with `facter fqdn`) as an
    indicator of where the certificate is:
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname
    foo it uses /var/lib/puppet/ssl/certs/foo.pem.

    I presume that the certificate that matches the fqdn on the host you
    are testing is probably _not_ signed by the CA that is loaded into
    PuppetDB. Its just a guess, but we should confirm this.

    Lets take a step back a bit and do an exercise. On your PuppetDB node,
    in /etc/puppetdb/ssl/ you have three files:

    * truststore.jks
    * keystore.jks
    * puppetdb_keystore_pw.txt

    If you look inside truststore.jks, you'll see that it contains the CA
    certificate:

    # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
    Enter keystore password: (this is contained inside
    puppetdb_keystore_pw.txt)

    Keystore type: JKS
    Keystore provider: SUN

    Your keystore contains 1 entry

    puppetdb ca, 28-Mar-2013, trustedCertEntry,
    Certificate fingerprint (SHA1):
    B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    This CA certificate is the same as what matches my CA file here, we
    can test this by matching the fingerprint:

    # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint
    SHA1
    Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    For client validation, this is really what matters - the client
    certificate must be signed by this CA certificate to be accepted by
    PuppetDB. It doesn't matter what the client thinks its CA is in Puppet
    configuration or whatever (so setting --ca_name is not enough here),
    all that matters is the client certificate must have been signed by
    that CA, and it is verified using OpenSSL techniques on the PuppetDB
    server during SSL negotiation.

    You can do this manually for example. Lets confirm this on my master -
    we know that my local CA is confirmed as the thing loaded into the
    PuppetDB trust store, so now we just run the verify routine against my
    certificate:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet
    master --configprint hostcert`
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK

    You must be sure to test this host certificate against the exact same
    CA loaded into PuppetDB ... not just the local CA (in case they differ
    - you can check this with the -fingerprint command).

    If the certificate on the other hand is NOT signed by the CA loaded
    into PuppetDB you get something like:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm
    error 20 at 0 depth lookup:unable to get local issuer certificate

    Which is by design for this example - that certificate 'puppetdb2.vm'
    was a certificate I copied from another Puppetmaster which has a
    separate CA, and thus it fails verification. In the real world,
    puppetdb2.vm would get reject by PuppetDB - which is the correct
    behaviour.

    ken.

    On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla
    wrote:
    Slightly OT here - shouldn't setting ca_name be enough for --compile to not
    fail ? Why does it need the --certname param ?


    On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla <
    mohit.chawla.binary@gmail.com>
    wrote:
    Hi,

    Got it working by passing --certname param (the ca server hostname,
    instead of this master).



    On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla
    wrote:
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't happen
    during
    standard puppet runs as opposed to doing a --compile. Here's the puppet
    trace when doing --compile - https://gist.github.com/alcy/5283712.



    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber wrote:

    Yeah, it does seem very odd though ... if agent works - and the master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of
    having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need
    to
    confirm this.

    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer environment.

    What does your puppet.conf look like on the host you are trying to
    run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at
    all?
    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.

    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    So I have some questions, as the error could mean a number of
    things:

    What version of PuppetDB are you running? And what exact version
    of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you
    getting
    any trouble with replace facts/replace catalogs generally? ie.
    Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their
    fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in /etc/puppet/puppetdb.conf
    to
    talk to your puppetdb server match the SSL certificate assigned
    to
    the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog
    json. So
    far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment
    production in
    10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0
    state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the ssl
    error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve
    facts
    for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished
    A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using curl
    and
    using ruby net/http. Here's a gist of for the curl and ruby
    snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the correct
    ssl
    params or something ?
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    .
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com
    .
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en
    .
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Apr 3, 2013 at 9:43 am
    Yes, in this way it behaves a lot like the puppet masters. Don't
    forget to check out certificate-whitelist
    (http://docs.puppetlabs.com/puppetdb/1.1/configure.html#certificate-whitelist)
    this gives you the ability to only allow certain hosts to connect if
    you desire it.

    On Wed, Apr 3, 2013 at 7:41 AM, Mohit Chawla
    wrote:
    Hey Ken, that definitely cleared some misconceptions, thanks !

    I now know what the problem was. First, I assumed that the client must've
    the same certificate as in puppetdb's truststore. I didn't realize that any
    cert signed by that CA should be able to make calls. In our scenario, where
    we have separate certs for the agent and master component on the same server
    (under /etc/puppet/ssl & /var/lib/puppet/ssl respectively), invoking
    --compile without giving the certname meant no certificate at all ! And so,
    passing --ssldir to use the agent's certs (signed by the same CA) worked.
    On Tue, Apr 2, 2013 at 3:40 PM, Ken Barber wrote:

    If you specify 'certname' it will use the local certificate with that
    name as apposed to using the certificate with the same name as the
    boxes fqdn (ie. facter fqdn).

    For example, if I use 'puppet master --compile puppetdb1.vm' on its
    own, it uses my local fqdn (determined with `facter fqdn`) as an
    indicator of where the certificate is:
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname
    foo it uses /var/lib/puppet/ssl/certs/foo.pem.

    I presume that the certificate that matches the fqdn on the host you
    are testing is probably _not_ signed by the CA that is loaded into
    PuppetDB. Its just a guess, but we should confirm this.

    Lets take a step back a bit and do an exercise. On your PuppetDB node,
    in /etc/puppetdb/ssl/ you have three files:

    * truststore.jks
    * keystore.jks
    * puppetdb_keystore_pw.txt

    If you look inside truststore.jks, you'll see that it contains the CA
    certificate:

    # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
    Enter keystore password: (this is contained inside
    puppetdb_keystore_pw.txt)

    Keystore type: JKS
    Keystore provider: SUN

    Your keystore contains 1 entry

    puppetdb ca, 28-Mar-2013, trustedCertEntry,
    Certificate fingerprint (SHA1):
    B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    This CA certificate is the same as what matches my CA file here, we
    can test this by matching the fingerprint:

    # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint
    SHA1
    Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    For client validation, this is really what matters - the client
    certificate must be signed by this CA certificate to be accepted by
    PuppetDB. It doesn't matter what the client thinks its CA is in Puppet
    configuration or whatever (so setting --ca_name is not enough here),
    all that matters is the client certificate must have been signed by
    that CA, and it is verified using OpenSSL techniques on the PuppetDB
    server during SSL negotiation.

    You can do this manually for example. Lets confirm this on my master -
    we know that my local CA is confirmed as the thing loaded into the
    PuppetDB trust store, so now we just run the verify routine against my
    certificate:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet
    master --configprint hostcert`
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK

    You must be sure to test this host certificate against the exact same
    CA loaded into PuppetDB ... not just the local CA (in case they differ
    - you can check this with the -fingerprint command).

    If the certificate on the other hand is NOT signed by the CA loaded
    into PuppetDB you get something like:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm
    error 20 at 0 depth lookup:unable to get local issuer certificate

    Which is by design for this example - that certificate 'puppetdb2.vm'
    was a certificate I copied from another Puppetmaster which has a
    separate CA, and thus it fails verification. In the real world,
    puppetdb2.vm would get reject by PuppetDB - which is the correct
    behaviour.

    ken.

    On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla
    wrote:
    Slightly OT here - shouldn't setting ca_name be enough for --compile to
    not
    fail ? Why does it need the --certname param ?


    On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla
    <mohit.chawla.binary@gmail.com>
    wrote:
    Hi,

    Got it working by passing --certname param (the ca server hostname,
    instead of this master).



    On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla
    wrote:
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't happen
    during
    standard puppet runs as opposed to doing a --compile. Here's the
    puppet
    trace when doing --compile - https://gist.github.com/alcy/5283712.




    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com>
    wrote:
    Yeah, it does seem very odd though ... if agent works - and the
    master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3
    read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen it
    is when there was an SSL bug, and the transport is cut short - which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb
    server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want
    to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I cant
    think of any peculiar settings apart from this being one of the two
    puppet masters apart from a separate ca server (we took care of
    having
    the ca server's certs being available at these masters). And afaik
    right now, there wasn't any ~/.puppet dir for root, however I need
    to
    confirm this.

    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer
    environment.

    What does your puppet.conf look like on the host you are trying to
    run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at
    all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.

    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    So I have some questions, as the error could mean a number of
    things:

    What version of PuppetDB are you running? And what exact version
    of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you
    getting
    any trouble with replace facts/replace catalogs generally? ie.
    Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their
    fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in
    /etc/puppet/puppetdb.conf
    to
    talk to your puppetdb server match the SSL certificate assigned
    to
    the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet master
    --compile foo.com', however I am not getting the catalog json.
    So
    far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment production
    in
    10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0
    state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the ssl
    error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve
    facts
    for
    bar.com: Failed to find facts from PuppetDB at puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished
    A

    Disabling storeconfig allows me to get a proper catalog json.

    Regarding the ssl errors, we've established the correctness of
    certificates in the relevant places - and done tests using
    curl
    and
    using ruby net/http. Here's a gist of for the curl and ruby
    snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the
    correct
    ssl
    params or something ?
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Mohit Chawla at Apr 3, 2013 at 9:56 am
    Ah nice, thanks for pointing that out.

    On Wed, Apr 3, 2013 at 3:13 PM, Ken Barber wrote:

    Yes, in this way it behaves a lot like the puppet masters. Don't
    forget to check out certificate-whitelist
    (
    http://docs.puppetlabs.com/puppetdb/1.1/configure.html#certificate-whitelist
    )
    this gives you the ability to only allow certain hosts to connect if
    you desire it.

    On Wed, Apr 3, 2013 at 7:41 AM, Mohit Chawla
    wrote:
    Hey Ken, that definitely cleared some misconceptions, thanks !

    I now know what the problem was. First, I assumed that the client must've
    the same certificate as in puppetdb's truststore. I didn't realize that any
    cert signed by that CA should be able to make calls. In our scenario, where
    we have separate certs for the agent and master component on the same server
    (under /etc/puppet/ssl & /var/lib/puppet/ssl respectively), invoking
    --compile without giving the certname meant no certificate at all ! And so,
    passing --ssldir to use the agent's certs (signed by the same CA) worked.
    On Tue, Apr 2, 2013 at 3:40 PM, Ken Barber wrote:

    If you specify 'certname' it will use the local certificate with that
    name as apposed to using the certificate with the same name as the
    boxes fqdn (ie. facter fqdn).

    For example, if I use 'puppet master --compile puppetdb1.vm' on its
    own, it uses my local fqdn (determined with `facter fqdn`) as an
    indicator of where the certificate is:
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname
    foo it uses /var/lib/puppet/ssl/certs/foo.pem.

    I presume that the certificate that matches the fqdn on the host you
    are testing is probably _not_ signed by the CA that is loaded into
    PuppetDB. Its just a guess, but we should confirm this.

    Lets take a step back a bit and do an exercise. On your PuppetDB node,
    in /etc/puppetdb/ssl/ you have three files:

    * truststore.jks
    * keystore.jks
    * puppetdb_keystore_pw.txt

    If you look inside truststore.jks, you'll see that it contains the CA
    certificate:

    # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
    Enter keystore password: (this is contained inside
    puppetdb_keystore_pw.txt)

    Keystore type: JKS
    Keystore provider: SUN

    Your keystore contains 1 entry

    puppetdb ca, 28-Mar-2013, trustedCertEntry,
    Certificate fingerprint (SHA1):
    B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    This CA certificate is the same as what matches my CA file here, we
    can test this by matching the fingerprint:

    # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint
    SHA1
    Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1

    For client validation, this is really what matters - the client
    certificate must be signed by this CA certificate to be accepted by
    PuppetDB. It doesn't matter what the client thinks its CA is in Puppet
    configuration or whatever (so setting --ca_name is not enough here),
    all that matters is the client certificate must have been signed by
    that CA, and it is verified using OpenSSL techniques on the PuppetDB
    server during SSL negotiation.

    You can do this manually for example. Lets confirm this on my master -
    we know that my local CA is confirmed as the thing loaded into the
    PuppetDB trust store, so now we just run the verify routine against my
    certificate:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet
    master --configprint hostcert`
    /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK

    You must be sure to test this host certificate against the exact same
    CA loaded into PuppetDB ... not just the local CA (in case they differ
    - you can check this with the -fingerprint command).

    If the certificate on the other hand is NOT signed by the CA loaded
    into PuppetDB you get something like:

    # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem
    /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm
    error 20 at 0 depth lookup:unable to get local issuer certificate

    Which is by design for this example - that certificate 'puppetdb2.vm'
    was a certificate I copied from another Puppetmaster which has a
    separate CA, and thus it fails verification. In the real world,
    puppetdb2.vm would get reject by PuppetDB - which is the correct
    behaviour.

    ken.

    On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla
    wrote:
    Slightly OT here - shouldn't setting ca_name be enough for --compile
    to
    not
    fail ? Why does it need the --certname param ?


    On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla
    <mohit.chawla.binary@gmail.com>
    wrote:
    Hi,

    Got it working by passing --certname param (the ca server hostname,
    instead of this master).



    On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla
    wrote:
    Hi Ken,

    Got a trace from running puppetdb-foreground --debug -
    https://gist.github.com/alcy/5283661. Weird that this doesn't
    happen
    during
    standard puppet runs as opposed to doing a --compile. Here's the
    puppet
    trace when doing --compile - https://gist.github.com/alcy/5283712.




    On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com>
    wrote:
    Yeah, it does seem very odd though ... if agent works - and the
    master
    is able to talk to PuppetDB no problem, then its weird that running
    puppet master on the command line doesn't seem to work.

    What is strange is that the SSL error is very very unspecific:

    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3
    read
    finished A

    It doesn't speak of a particular problem, ie. remote host doesn't
    match certificate, CA not verified etc. etc. it just says 'read
    finished A'. I think this is significant, the last time I've seen
    it
    is when there was an SSL bug, and the transport is cut short -
    which
    is why I wanted to see what you had in your logs.

    If you try running the command with --debug --trace do you get any
    more information? If you tcpdump the connection to the puppetdb
    server
    do you see the TCP packets hit the server on port 8081? Try running
    puppetdb-foreground --debug on your puppetdb server and see what
    happens when you attempt to compile like this. You'd obviously want
    to
    try these things when the other activity on the server is disabled
    :-).

    ken.

    On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla
    wrote:
    Hi,

    Not at the workstation right now, but regarding puppet.conf I
    cant
    think of any peculiar settings apart from this being one of the
    two
    puppet masters apart from a separate ca server (we took care of
    having
    the ca server's certs being available at these masters). And
    afaik
    right now, there wasn't any ~/.puppet dir for root, however I
    need
    to
    confirm this.

    On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com>
    wrote:
    I'm just trying to run up the same environment so I can try to
    replicate it, as yet I can't replicate it on the newer
    environment.

    What does your puppet.conf look like on the host you are trying
    to
    run
    puppet master --compile btw? I presume you are trying to run the
    command as root, is there a ~/.puppet directory for that user at
    all?

    ken.

    On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla
    wrote:
    Hello Ken,

    Thanks for the response.

    On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <
    ken@puppetlabs.com>
    wrote:
    So I have some questions, as the error could mean a number of
    things:

    What version of PuppetDB are you running? And what exact
    version
    of
    Java is it using?
    puppetdb version is puppetdb-1.0.4-1.el6.noarch.

    $ java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.5)
    (rhel-1.50.1.11.5.el6_3-x86_64)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    Can you take a look at puppetdb.log and tell me if you see any
    meaningful error messages?
    I took a look, no error messages, only info messages.
    Without trying to compile a catalog in this manner - are you
    getting
    any trouble with replace facts/replace catalogs generally? ie.
    Just
    trying to running puppet agent -t for example?
    Nope, those runs are working fine.
    Have you tried re-initialisation your SSL setup with
    puppetdb-ssl-setup at all yet?
    Yes, and confirmed the validity of certificates and their
    fingerprints
    in the truststore and master ca.
    Does the hostname you have specified in
    /etc/puppet/puppetdb.conf
    to
    talk to your puppetdb server match the SSL certificate
    assigned
    to
    the
    puppetdb host itself? Yes.
    On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla
    wrote:
    Forgot mentioning the env details:

    [user@puppetmaster ~]# rpm -qa | grep puppet
    puppetlabs-release-6-6.noarch
    puppetdb-terminus-1.0.4-1.el6.noarch
    mcollective-puppet-agent-1.4.1-1.noarch
    puppet-2.7.20-1.el6.noarch
    hiera-puppet-1.0.0-1.el6.noarch
    puppet-server-2.7.20-1.el6.noarch
    mcollective-puppet-common-1.4.1-1.noarch

    [user@puppetmaster ~]# cat /etc/centos-release
    CentOS release 6.3 (Final)

    On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla
    wrote:
    Hello,

    Stuck in a weird place here. I am trying to do 'puppet
    master
    --compile foo.com', however I am not getting the catalog
    json.
    So
    far,
    I have noticed two sort of outputs:

    1) The above command results in :
    notice: Compiled catalog for foo.com in environment
    production
    in
    10.60 seconds
    Failed to submit 'replace catalog' command for foo.com to
    PuppetDB at
    puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0
    state=SSLv3
    read
    finished A

    Notice that I don't actually get any json & of course, the
    ssl
    error.

    2) Other output:
    Failed when searching for node bar.com: Could not retrieve
    facts
    for
    bar.com: Failed to find facts from PuppetDB at
    puppetdb:8081:
    SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read
    finished
    A

    Disabling storeconfig allows me to get a proper catalog
    json.
    Regarding the ssl errors, we've established the correctness
    of
    certificates in the relevant places - and done tests using
    curl
    and
    using ruby net/http. Here's a gist of for the curl and ruby
    snippet.
    https://gist.github.com/alcy/5262866

    Any suggestions what could be happening here ? Perhaps in
    indirector/facts/puppetdb.rb, http_get doesn't get the
    correct
    ssl
    params or something ?
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit
    https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to
    puppet-users@googlegroups.com.
    .
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com
    .
    --
    You received this message because you are subscribed to the Google
    Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en
    .
    --
    You received this message because you are subscribed to the Google
    Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedMar 28, '13 at 12:53p
activeApr 3, '13 at 9:56a
posts14
users2
websitepuppetlabs.com

2 users in discussion

Mohit Chawla: 9 posts Ken Barber: 5 posts

People

Translate

site design / logo © 2022 Grokbase