Hi Thomas,

Am 10. Februar 2016 21:33:43 MEZ, schrieb "Thomas Güttler" <guettliml@thomas-guettler.de>:
Hi Florian,

I guess you are too fast here.
Oops, sorry. I got this topic in the back of my head for 2 or 3
years now so I have a lot of thoughts on the topic ^^"
My topic is about the great fixed schema definition
which you get when you use a relational database.
See a few comments below.
Storing JSON in postgres is in my eyes the same
fuzzy-data-soup like YAML in a file system.

Or I have not understood what you mean with
"PostgreSQL's JSON features".
PostgreSQL can store JSON, evaluate attributes
and index them. I think also validate the structure.

*But the important part here is:*
You can make an query and ask to get JSON back.

You can have a nice relational schema and let PostgreSQL
handle the data manipulation to fit whatever structure salt
Thus the mentioned view.
What do you mean with "database's schema surely
won't be simple"?

AFAIK there is no common schema definition for pillar data.
Our custom pillar data schema is up to now very simple.
I guess we would need 5 tables. But that are our
custom tables. AFAIK everybody is on his own in this context.
At least up to now.
When you're using salt to manage users, salt itself, ssh,
monitoring tools, on workstations tools for the devs, on
servers databases & backup clients & webservers & all
the little infrastructure bits and utilize half a dozen
formulas your pillar starts to get complicated.
Defining the schema you then have to map into a YAML/
JSON treelike structure won't be too simple.

One could see the format of the pillar data like formulas
expect it to be as a common schema. But of course that's
the resulting JSON.

Based on that there may be some SQL schemas showing up
but the default will stay YAML+Jinja.
What do you mean with "topfile" table? I read the top of this:
I think topfile is not part of pillar. It is part of the
A way of grouping the minions based on different attributes
still allowing to target single hosts.
Everyone matching 'web*' gets the webserver stuff, 'xsql*'
gets the database stuff, here you get our defaults for RedHat systems, …
A kind of metatable like `top.sls` provides for `file_roots`
and `pillar_roots`.
But maybe you are right. If it groups the infrastructure it
could be good candiate for belonging into a database.

My background:

- Code needs to live in version control (I use git)
- Data belongs into the database (I use postgres and
You're halfway there. Use Django-CRM to model your
DB-schemas and then add a view returning JSON.
Up to now nothing new.

- Config is data, and belongs into the database. I came
across this during the last years. This thinking is unfortunately
not wide spread.

With salt I am unsure. What is code and what is config? Up
to now I am unsure.
States are code, pillar consists of specific config values
thus data. Its structure might be code-ish but it's just
defining what goes where.
Pillars look like config, like something that belongs into a
And it's way easier to have input validation on those than
on a bunch of plain text files! A missing '.' (or ':') in an IP?
PostgreSQL will be like "dude, that's no `inet`!" :D

Anyone had to restructure half their YAML files to fit a new
pillar structure? Like when you dump a custom state to use
a more feature-rich formula?
Not fun. Even with the more cumbersome structures
wrapped in macros.

And yes, you can add commit hooks checking you've written
proper YAML. But checking for IPs proper for a certain
subnet or validating usernames defined somewhere else?
I'd rather write the schema for a relational DB I can later
build a frontend for. Even if it's just curses providing some

And for versioning the actual data: Just look at all the wikis
with a RDBMS backend.
Thomas Güttler
Regards, Florian
*turning down rantiness, going for breakfast*
Am Mittwoch, 10. Februar 2016 09:41:26 UTC+1 schrieb Florian Ermisch:
Hi Thomas,

If I just had the time I'd build something like this based on
Using PostgreSQL's JSON features you could have a
(materialized) view where you just select the minion ID from
and get the minion's whole pillar as JSON. This would make
the ext_pillar's implementation trivial.
The database's schema surely won't be simple but when
you get that one right the view for the JSON shouldn't be to
hard to figure out.

Of course you could also take a topfile like approach,
make multiple queries and place the JSON from the "users"
view in `{"users": …}` and so on.
You could have a "topfile" table! :D

Hm, might be useful to pass some grains to the ext_pillar, too.

Regards, Florian

Am 8. Februar 2016 22:11:30 MEZ, schrieb "Thomas Güttler" <guet...@thomas-guettler.de>:
I like to create the dependency-net in salt with YAML.

But I don't like to create the input-data (pillars) in YAML.
There is no fixed schema and typos will happen sooner
There or later.

I like to specify data structures in relational databases.

I saw that there is a mysql module to define pillars:


I am unsure how to works, maybe it does not at all what
I want.

Here is my idea. The following data is from the pillar docs:

thatch: 1000
shouse: 1001
utahdave: 1002
redbeard: 1003

What does it look like? For me it looks like a database dump.

There is a table users with columns: username, user_id.

Since the above data is only the data from one host, we
need to add a column "host_id"
which is a ForeignKey to a table of all hosts.

The most basic implementation would be to dump the
data from the DB to YAML. This way no single line in salt
would need a change.

What do you think?

Tell me what's wrong and why this is a bad idea :-)

For those people unfamiliar with relational databases:
It is not hard like years ago. It is fun and professional.
I personally like to specify my data structures with
This gives you an admin interface and schema-migration
tool for free

Thomas Güttler
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 8 of 18 | next ›
Discussion Overview
groupsalt-users @
postedFeb 8, '16 at 9:11p
activeMar 7, '16 at 10:00a



site design / logo © 2021 Grokbase