FAQ
Greetings,

I am having an issue with Hiera again. Bangin my head on it for a few hours
and not getting anywhere. Since this group was able to help last time, I
have my fingers crossed you can help me again. :-D

I am trying to use JSON with Hiera this time.

First, fully updated dev box.
$ cat /etc/redhat-release
Scientific Linux release 6.7 (Carbon)

$ sudo yum update
Loaded plugins: refresh-packagekit, security
Setting up Update Process
No Packages marked for Update

$ rpm -qa | egrep '(puppet|fact|hiera|ruby)'
facter-2.4.4-1.el6.x86_64
puppet-3.8.2-1.el6.noarch
hiera-1.3.4-1.el6.noarch
puppet-server-3.8.2-1.el6.noarch
ruby-libs-1.8.7.374-4.el6_6.x86_64
ruby-rdoc-1.8.7.374-4.el6_6.x86_64
ruby-shadow-2.2.0-2.el6.x86_64
ruby-1.8.7.374-4.el6_6.x86_64
rubygems-1.3.7-5.el6.noarch
ruby-augeas-0.4.1-3.el6.x86_64
rubygem-deep_merge-1.0.0-2.el6.noarch
libselinux-ruby-2.0.94-5.8.el6.x86_64
rubygem-json-1.5.5-3.el6.x86_64
ruby-irb-1.8.7.374-4.el6_6.x86_64


Let's use the puppet documentation to get started, shall we?
https://docs.puppetlabs.com/hiera/1/data_sources.html

$ cat /etc/puppet/hiera.yaml
---
:backends:
    - json
:hierarchy:
    - common
:merge_behavior: "deeper"
:json:
    :datadir: /etc/puppet/hieradata

BTW: the doc's actual JSON example is broken as it is missing the [ ]
around it. I included it. below and if you don't you fail the parser and
Hiera just ignores everything instead of giving an error.

$ cat /etc/puppet/hieradata/common.json
[
{
    "apache-packages" : [
    "apache2",
    "apache2-common",
    "apache2-utils"
    ],

     "hosts_entry" : "sandbox.%{fqdn}",

     "sshd_settings" : {
                        "root_allowed" : "no",
                        "password_allowed" : "no"
                      }
}
]

Verify valid JSON
$ cat /etc/puppet/hieradata/common.json | jq '.[0]'
{
   "sshd_settings": {
     "password_allowed": "no",
     "root_allowed": "no"
   },
   "hosts_entry": "sandbox.%{fqdn}",
   "apache-packages": [
     "apache2",
     "apache2-common",
     "apache2-utils"
   ]
}

Awesome! Lets validate in Hiera:
$ hiera --version
1.3.4
$ hiera -c /etc/puppet/hiera.yaml hosts_entry
/usr/lib/ruby/site_ruby/1.8/hiera/filecache.rb:56:in `read_file': Data
retrieved from /etc/puppet/hieradata/common.json is String not Hash
(TypeError)
from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:24:in
`lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:76:in `datasources'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `map'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `datasources'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:17:in
`lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:206:in `lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `each'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera.rb:60:in `lookup'
from /usr/bin/hiera:225



And here is where things went wrong. The worst part is that it is such a
generic error I can't find anything of real value in my online searching.
Even worse, the only directly relevant hit I found basically said to stop
using JSON and use YAML instead. Not really a solution is it?

But whatever. It is probably good to verify that YAML is at least working
with Heira in the dev environment. Using the puppet documentation again for
guidance.
$ mv /etc/puppet/hieradata/common.json /etc/puppet/hieradata/common.yaml
$ vim /etc/puppet/hieradata/common.yaml
$ cat /etc/puppet/hieradata/common.yaml
---
# array
apache-packages:
     - apache2
     - apache2-common
     - apache2-utils

# string
apache-service: apache2

# interpolated facter variable
hosts_entry: "sandbox.%{fqdn}"

# hash
sshd_settings:
     root_allowed: "no"
     password_allowed: "yes"

# alternate hash notation
sshd_settings: {root_allowed: "no", password_allowed: "yes"}

# to return "true" or "false"
sshd_settings: {root_allowed: no, password_allowed: yes}


$ vim /etc/puppet/hiera.yaml
$ cat /etc/puppet/hiera.yaml
---
:backends:
     - yaml
:hierarchy:
     - common
:merge_behavior: "deeper"
:yaml:
     :datadir: /etc/puppet/hieradata


$ hiera -c /etc/puppet/hiera.yaml hosts_entry
sandbox.

Awesome! So YAML works.

Back to the JSON version....

$ hiera -c /etc/puppet/hiera.yaml hosts_entry
/usr/lib/ruby/site_ruby/1.8/hiera/filecache.rb:56:in `read_file': Data
retrieved from /etc/puppet/hieradata/common.json is String not Hash
(TypeError)
from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:24:in
`lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:76:in `datasources'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `map'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `datasources'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:17:in
`lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:206:in `lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `each'
from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `lookup'
from /usr/lib/ruby/site_ruby/1.8/hiera.rb:60:in `lookup'
from /usr/bin/hiera:225


Dah!

No matter how simple I make it or what tweaks I do to the common.json file,
I always get that exact error. If I run puppet agent, I get a very similar
error too!

"So just use YAML like the other post said"
1) That doesn't fix JSON and it avoids the problem ("Hey doc, my legs are
broken!" "Have you tried walking on your hands instead? Why not just do
that?" :-P)

2) The fancy new tool that my security team is using to manage and monitor
system wide variables reads/writes JSON but doesn't do YAML. Meaning we
either have to keep doing the manual checks we are doing, or stick a parser
in between. Why? Puppet and Heira are supposed to work with JSON, it
read/writes JSON, this should theoretically just work.

Does anyone know why hiera isn't working with JSON? I really feel like this
is probably a simple solution that I am just not seeing. I am hoping
someone else might be able to recognize the answer.

Thanks!
~Stack~

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/2703e86d-f1e8-40a3-9d0c-24fd85bbbfe4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Matthaus Owens at Sep 17, 2015 at 10:24 pm
    Stack,


    On Thu, Sep 17, 2015 at 12:41 PM, Stack Kororā wrote:
    Greetings,

    I am having an issue with Hiera again. Bangin my head on it for a few hours
    and not getting anywhere. Since this group was able to help last time, I
    have my fingers crossed you can help me again. :-D

    I am trying to use JSON with Hiera this time.

    First, fully updated dev box.
    $ cat /etc/redhat-release
    Scientific Linux release 6.7 (Carbon)

    $ sudo yum update
    Loaded plugins: refresh-packagekit, security
    Setting up Update Process
    No Packages marked for Update

    $ rpm -qa | egrep '(puppet|fact|hiera|ruby)'
    facter-2.4.4-1.el6.x86_64
    puppet-3.8.2-1.el6.noarch
    hiera-1.3.4-1.el6.noarch
    puppet-server-3.8.2-1.el6.noarch
    ruby-libs-1.8.7.374-4.el6_6.x86_64
    ruby-rdoc-1.8.7.374-4.el6_6.x86_64
    ruby-shadow-2.2.0-2.el6.x86_64
    ruby-1.8.7.374-4.el6_6.x86_64
    rubygems-1.3.7-5.el6.noarch
    ruby-augeas-0.4.1-3.el6.x86_64
    rubygem-deep_merge-1.0.0-2.el6.noarch
    libselinux-ruby-2.0.94-5.8.el6.x86_64
    rubygem-json-1.5.5-3.el6.x86_64
    ruby-irb-1.8.7.374-4.el6_6.x86_64


    Let's use the puppet documentation to get started, shall we?
    https://docs.puppetlabs.com/hiera/1/data_sources.html

    $ cat /etc/puppet/hiera.yaml
    ---
    :backends:
    - json
    :hierarchy:
    - common
    :merge_behavior: "deeper"
    :json:
    :datadir: /etc/puppet/hieradata

    BTW: the doc's actual JSON example is broken as it is missing the [ ] around
    it. I included it. below and if you don't you fail the parser and Hiera just
    ignores everything instead of giving an error.
    Adding [ ] around your json is what is causing the problem you're
    seeing. It turns the data you are writing into an Array instead of a
    Hash. Hiera is expecting it to be a hash, as the error message
    indicates. The error message here may not be very helpful though,
    because the data isn't a string, but is an Array. What parse error do
    you get without the [ ]?

    I was able to get a correct return from hiera by removing the [ ].
    $ cat /etc/puppet/hieradata/common.json
    [
    {
    "apache-packages" : [
    "apache2",
    "apache2-common",
    "apache2-utils"
    ],

    "hosts_entry" : "sandbox.%{fqdn}",

    "sshd_settings" : {
    "root_allowed" : "no",
    "password_allowed" : "no"
    }
    }
    ]

    Verify valid JSON
    $ cat /etc/puppet/hieradata/common.json | jq '.[0]'
    {
    "sshd_settings": {
    "password_allowed": "no",
    "root_allowed": "no"
    },
    "hosts_entry": "sandbox.%{fqdn}",
    "apache-packages": [
    "apache2",
    "apache2-common",
    "apache2-utils"
    ]
    }

    Awesome! Lets validate in Hiera:
    $ hiera --version
    1.3.4
    $ hiera -c /etc/puppet/hiera.yaml hosts_entry
    /usr/lib/ruby/site_ruby/1.8/hiera/filecache.rb:56:in `read_file': Data
    retrieved from /etc/puppet/hieradata/common.json is String not Hash
    (TypeError)
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:24:in
    `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:76:in `datasources'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `map'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `datasources'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:17:in
    `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:206:in `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `each'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera.rb:60:in `lookup'
    from /usr/bin/hiera:225



    And here is where things went wrong. The worst part is that it is such a
    generic error I can't find anything of real value in my online searching.
    Even worse, the only directly relevant hit I found basically said to stop
    using JSON and use YAML instead. Not really a solution is it?

    But whatever. It is probably good to verify that YAML is at least working
    with Heira in the dev environment. Using the puppet documentation again for
    guidance.
    $ mv /etc/puppet/hieradata/common.json /etc/puppet/hieradata/common.yaml
    $ vim /etc/puppet/hieradata/common.yaml
    $ cat /etc/puppet/hieradata/common.yaml
    ---
    # array
    apache-packages:
    - apache2
    - apache2-common
    - apache2-utils

    # string
    apache-service: apache2

    # interpolated facter variable
    hosts_entry: "sandbox.%{fqdn}"

    # hash
    sshd_settings:
    root_allowed: "no"
    password_allowed: "yes"

    # alternate hash notation
    sshd_settings: {root_allowed: "no", password_allowed: "yes"}

    # to return "true" or "false"
    sshd_settings: {root_allowed: no, password_allowed: yes}


    $ vim /etc/puppet/hiera.yaml
    $ cat /etc/puppet/hiera.yaml
    ---
    :backends:
    - yaml
    :hierarchy:
    - common
    :merge_behavior: "deeper"
    :yaml:
    :datadir: /etc/puppet/hieradata


    $ hiera -c /etc/puppet/hiera.yaml hosts_entry
    sandbox.

    Awesome! So YAML works.

    Back to the JSON version....

    $ hiera -c /etc/puppet/hiera.yaml hosts_entry
    /usr/lib/ruby/site_ruby/1.8/hiera/filecache.rb:56:in `read_file': Data
    retrieved from /etc/puppet/hieradata/common.json is String not Hash
    (TypeError)
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:24:in
    `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:76:in `datasources'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `map'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `datasources'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:17:in
    `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:206:in `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `each'
    from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `lookup'
    from /usr/lib/ruby/site_ruby/1.8/hiera.rb:60:in `lookup'
    from /usr/bin/hiera:225


    Dah!

    No matter how simple I make it or what tweaks I do to the common.json file,
    I always get that exact error. If I run puppet agent, I get a very similar
    error too!

    "So just use YAML like the other post said"
    1) That doesn't fix JSON and it avoids the problem ("Hey doc, my legs are
    broken!" "Have you tried walking on your hands instead? Why not just do
    that?" :-P)

    2) The fancy new tool that my security team is using to manage and monitor
    system wide variables reads/writes JSON but doesn't do YAML. Meaning we
    either have to keep doing the manual checks we are doing, or stick a parser
    in between. Why? Puppet and Heira are supposed to work with JSON, it
    read/writes JSON, this should theoretically just work.

    Does anyone know why hiera isn't working with JSON? I really feel like this
    is probably a simple solution that I am just not seeing. I am hoping someone
    else might be able to recognize the answer.

    Thanks!
    ~Stack~

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/2703e86d-f1e8-40a3-9d0c-24fd85bbbfe4%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    Matthaus Owens
    Puppet Labs

    PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.
    Register now to take advantage of the Early Bird discount —save $249!

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACD%3DwAde7ZJS5ECkFEZaeMG1M-XukW5HKtJnCSRo8-9Sw3--AA%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • ~Stack~ at Sep 17, 2015 at 10:57 pm
    Greetings Matthaus!
    On 09/17/2015 05:24 PM, Matthaus Owens wrote:
    Stack,
    [snip]
    BTW: the doc's actual JSON example is broken as it is missing the [ ] around
    it. I included it. below and if you don't you fail the parser and Hiera just
    ignores everything instead of giving an error.
    Adding [ ] around your json is what is causing the problem you're
    seeing. It turns the data you are writing into an Array instead of a
    Hash. Hiera is expecting it to be a hash, as the error message
    indicates. The error message here may not be very helpful though,
    because the data isn't a string, but is an Array. What parse error do
    you get without the [ ]?

    I was able to get a correct return from hiera by removing the [ ].
    On one hand, I feel really dumb. That should have been obvious. I was
    comparing against the output I got from that other program and I see how
    and why it is wrapping it with [] now. Plus the example I was told to
    use to verify the JSON output was "jq[0]". 15 seconds on the jq tutorial
    website shows why that is broken. Some how that fact just didn't click
    in my head. Duh.

    On the other hand, scrolling back through my terminal history I see the
    pre-[] testing where hiera simply just didn't print anything or returned
    nil. I am failing to see the difference between then and now...My guess
    is that I had another hiera error, and in my struggles for fixing this
    issue, I fixed that one...but I'm not 100% on that...

    On the bright side, it is all working now!! :-)

    I must be dense or have done something to make Hiera not like me. It
    seems we brawl for a long time, then suddenly everything starts
    working... :-D

    Thank you so much for pointing out my silly goof up. I really REALLY
    appreciate the second pair of eyes pointing out my mistake.

    ~Stack~

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/55FB4568.6070302%40gmail.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedSep 17, '15 at 7:41p
activeSep 17, '15 at 10:57p
posts3
users2
websitepuppetlabs.com

2 users in discussion

~Stack~: 2 posts Matthaus Owens: 1 post

People

Translate

site design / logo © 2022 Grokbase