FAQ
Dear all,

We have a few reasons to have various PuppetDBs (and Masters, one per
cluster) in our datacenter and, eventually, there may be a need to share
*only* a few exported resources between clusters, like a few ssh-keys or
nagios tests.

Is there a mechanism to pull and push some exported resources from one
PuppetDB to another? I guess this could be done with the API, but has
anybody tried it, or will it work at all?

Thanks a lot,
Pablo Fernandez

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Pablo Fernandez at Oct 10, 2013 at 7:08 am
    Hi,

    I have found a PuppetDB Hiera backend
    (https://github.com/dalen/puppet-puppetdbquery) that could (potentially)
    be used to query other PuppetDBs... but that would return hiera data,
    and not exported resources like a normal PuppetDB does.

    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.

    Thanks!
    BR/Pablo

    On 10/07/2013 01:07 PM, Pablo Fernandez wrote:
    Dear all,

    We have a few reasons to have various PuppetDBs (and Masters, one per
    cluster) in our datacenter and, eventually, there may be a need to share
    *only* a few exported resources between clusters, like a few ssh-keys or
    nagios tests.

    Is there a mechanism to pull and push some exported resources from one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?

    Thanks a lot,
    Pablo Fernandez
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Oct 10, 2013 at 7:16 am

    Is there a mechanism to pull and push some exported resources from one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 10, 2013 at 12:13 pm
    Thanks Ken,

    I took a look at the Command API and seems to be quite interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change a whole
    catalog, or facts. Do they need to be existing catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of a node get
    introduced into the DB? And moreover, how do I query a whole catalog?

    Another possibility would be to include the resource inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported resources don't
    really mater what node do they belong, as long as they have the right
    attributes, right?), but the Catalog wire format seems to require the
    whole set of resources, so, if I define the resources of Node_B with
    some exported resources from Node_A, I believe I would lose the original
    Node_B's resources. The only possibility would be to query Node_B full
    catalog, and *add also* those resources from Node_A before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported resources from one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Wolf Noble at Oct 10, 2013 at 2:31 pm
    I'd be interested in accomplishing a similar thing. I want to have a
    central reporting puppetdb which I can (outside of puppet) query for
    multisite information without tying isolated environments together via a
    single postgres db, or using multi-master replication.

    my needs do not involve making each puppet site aware of each other's data,
    but it would be nice to have one db which has all the data for my digging
    pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at site A (what
    updates have you had in the past N minutes/ since the last time I checked
    in with you/ since timestamp(N)), and then iterate through the collected
    data and feed it into a reporting puppetdb instance, however I've not gone
    beyond thinking about it.




    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change a whole
    catalog, or facts. Do they need to be existing catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of a node get
    introduced into the DB? And moreover, how do I query a whole catalog?

    Another possibility would be to include the resource inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported resources don't
    really mater what node do they belong, as long as they have the right
    attributes, right?), but the Catalog wire format seems to require the
    whole set of resources, so, if I define the resources of Node_B with
    some exported resources from Node_A, I believe I would lose the original
    Node_B's resources. The only possibility would be to query Node_B full
    catalog, and *add also* those resources from Node_A before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported resources from one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Throwe, Jesse at Oct 10, 2013 at 2:36 pm
    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321

    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble wrote:

    I'd be interested in accomplishing a similar thing. I want to have a
    central reporting puppetdb which I can (outside of puppet) query for
    multisite information without tying isolated environments together via a
    single postgres db, or using multi-master replication.

    my needs do not involve making each puppet site aware of each other's
    data, but it would be nice to have one db which has all the data for my
    digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at site A (what
    updates have you had in the past N minutes/ since the last time I checked
    in with you/ since timestamp(N)), and then iterate through the collected
    data and feed it into a reporting puppetdb instance, however I've not gone
    beyond thinking about it.




    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change a whole
    catalog, or facts. Do they need to be existing catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of a node get
    introduced into the DB? And moreover, how do I query a whole catalog?

    Another possibility would be to include the resource inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported resources don't
    really mater what node do they belong, as long as they have the right
    attributes, right?), but the Catalog wire format seems to require the
    whole set of resources, so, if I define the resources of Node_B with
    some exported resources from Node_A, I believe I would lose the original
    Node_B's resources. The only possibility would be to query Node_B full
    catalog, and *add also* those resources from Node_A before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported resources from one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 10, 2013 at 3:07 pm
    Yes, I saw that... this may be a solution for replicating full PuppetDBs
    (if you tackle the backlog during a long shortage), but that does not
    scale very well if you have 5-10 PuppetDBs all being replicated to a
    single, bigger one.

    Besides, it would not cover cases when you want to have only some nodes
    replicated, or even only some resources.

    On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321


    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble wrote:

    I'd be interested in accomplishing a similar thing. I want to have
    a central reporting puppetdb which I can (outside of puppet) query
    for multisite information without tying isolated environments
    together via a single postgres db, or using multi-master
    replication.

    my needs do not involve making each puppet site aware of each
    other's data, but it would be nice to have one db which has all
    the data for my digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at site A
    (what updates have you had in the past N minutes/ since the last
    time I checked in with you/ since timestamp(N)), and then iterate
    through the collected data and feed it into a reporting puppetdb
    instance, however I've not gone beyond thinking about it.





    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
    wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite
    interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to
    add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is
    unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change
    a whole
    catalog, or facts. Do they need to be existing catalogs, or
    could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of
    a node get
    introduced into the DB? And moreover, how do I query a whole
    catalog?

    Another possibility would be to include the resource inside an
    existing
    Node_B catalog on the PuppetDB_B (after all, exported
    resources don't
    really mater what node do they belong, as long as they have
    the right
    attributes, right?), but the Catalog wire format seems to
    require the
    whole set of resources, so, if I define the resources of
    Node_B with
    some exported resources from Node_A, I believe I would lose
    the original
    Node_B's resources. The only possibility would be to query
    Node_B full
    catalog, and *add also* those resources from Node_A before
    sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to implemen
    what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported
    resources from one
    PuppetDB to another? I guess this could be done with the
    API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions
    internally
    about this kind of thing. How about you outline your
    requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API
    does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to puppet-users@googlegroups.com
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Throwe, Jesse at Oct 10, 2013 at 3:23 pm
    That would really depend on the approach ultimately taken for the
    multi-master. If the puppet master (or agent in masterless mode) forwards
    the same fact/report/node/etc data to multiple puppetdb instances then yes
    the scale breaks horribly and does not meet part of this request. If there
    is some kind of store-and-forward mechanism in puppetdb itself, it would be
    possible to extend the forward to say 'only if it matches these rules'.
      This could definitely be useful in a variety of ways beyond simple HA
    purposes.



    On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
    wrote:
    Yes, I saw that... this may be a solution for replicating full PuppetDBs
    (if you tackle the backlog during a long shortage), but that does not scale
    very well if you have 5-10 PuppetDBs all being replicated to a single,
    bigger one.

    Besides, it would not cover cases when you want to have only some nodes
    replicated, or even only some resources.



    On 10/10/2013 04:35 PM, Throwe, Jesse wrote:

    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321

    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble wrote:

    I'd be interested in accomplishing a similar thing. I want to have a
    central reporting puppetdb which I can (outside of puppet) query for
    multisite information without tying isolated environments together via a
    single postgres db, or using multi-master replication.

    my needs do not involve making each puppet site aware of each other's
    data, but it would be nice to have one db which has all the data for my
    digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at site A (what
    updates have you had in the past N minutes/ since the last time I checked
    in with you/ since timestamp(N)), and then iterate through the collected
    data and feed it into a reporting puppetdb instance, however I've not gone
    beyond thinking about it.





    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez <pablo.fernandez@cscs.ch
    wrote:
    Thanks Ken,

    I took a look at the Command API and seems to be quite interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change a whole
    catalog, or facts. Do they need to be existing catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of a node get
    introduced into the DB? And moreover, how do I query a whole catalog?

    Another possibility would be to include the resource inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported resources don't
    really mater what node do they belong, as long as they have the right
    attributes, right?), but the Catalog wire format seems to require the
    whole set of resources, so, if I define the resources of Node_B with
    some exported resources from Node_A, I believe I would lose the original
    Node_B's resources. The only possibility would be to query Node_B full
    catalog, and *add also* those resources from Node_A before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported resources from
    one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 15, 2013 at 9:00 am
    Ok, so I finally opted for inserting a full node from one DB into
    another, since getting the resources should be quite easy with a GET.

    But then I run into a problem when running the replace catalog command
    (with an empty catalog):

    curl -vv -G -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode
    'payload={"command":"replace catalog","version": 2,"payload":"{
    \"metadata\": {\"api_version\": 1}, \"data\": {\"name\":
    \"puppet09.cscs.ch\",\"version\": \"1111\", \"edges\": [],
    \"resources\": [] } }"}'

    Which returns ok. The puppetdb.log shows:

    2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
    [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying after
    attempt 8, due to: org.postgresql.util.PSQLException: ERROR: syntax
    error at or near ")"
       Position: 66

    I have looked again and again into the data, and syntax seems to be correct.

    Anybody tried this? Hints?

    Thanks a lot,
    Pablo Fernandez



    On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
    That would really depend on the approach ultimately taken for the
    multi-master. If the puppet master (or agent in masterless mode)
    forwards the same fact/report/node/etc data to multiple puppetdb
    instances then yes the scale breaks horribly and does not meet part of
    this request. If there is some kind of store-and-forward mechanism in
    puppetdb itself, it would be possible to extend the forward to say
    'only if it matches these rules'. This could definitely be useful in
    a variety of ways beyond simple HA purposes.



    On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
    wrote:

    Yes, I saw that... this may be a solution for replicating full
    PuppetDBs (if you tackle the backlog during a long shortage), but
    that does not scale very well if you have 5-10 PuppetDBs all being
    replicated to a single, bigger one.

    Besides, it would not cover cases when you want to have only some
    nodes replicated, or even only some resources.


    On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321


    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com
    wrote:

    I'd be interested in accomplishing a similar thing. I want to
    have a central reporting puppetdb which I can (outside of
    puppet) query for multisite information without tying
    isolated environments together via a single postgres db, or
    using multi-master replication.

    my needs do not involve making each puppet site aware of each
    other's data, but it would be nice to have one db which has
    all the data for my digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at
    site A (what updates have you had in the past N minutes/
    since the last time I checked in with you/ since
    timestamp(N)), and then iterate through the collected data
    and feed it into a reporting puppetdb instance, however I've
    not gone beyond thinking about it.





    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
    wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite
    interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was
    to add exported
    resources from Node_A (managed by PuppetMaster_A that has
    its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is
    unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be
    aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to
    change a whole
    catalog, or facts. Do they need to be existing catalogs,
    or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported
    resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog",
    no "add
    catalog". Would it be rejected? How does the first
    catalog of a node get
    introduced into the DB? And moreover, how do I query a
    whole catalog?

    Another possibility would be to include the resource
    inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported
    resources don't
    really mater what node do they belong, as long as they
    have the right
    attributes, right?), but the Catalog wire format seems to
    require the
    whole set of resources, so, if I define the resources of
    Node_B with
    some exported resources from Node_A, I believe I would
    lose the original
    Node_B's resources. The only possibility would be to
    query Node_B full
    catalog, and *add also* those resources from Node_A
    before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to
    implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported
    resources from one
    PuppetDB to another? I guess this could be done with
    the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions
    internally
    about this kind of thing. How about you outline your
    requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand?
    The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The
    documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to
    the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails
    from it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at
    http://groups.google.com/group/puppet-users.
    For more options, visit
    https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com > Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to puppet-users@googlegroups.com
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 15, 2013 at 9:25 am
    If I try to add a fictitious resource:

    {\"type\": \"Service\",
      \"title\": \"iptables\",
      \"exported\": false,
      \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\",
      \"line\": 121,
      \"tags\": [\"node\", \"puppetdb\"],
      \"parameters\": {
                       \"ensure\" : \"running\"}
    }

    I get:

    2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command]
    [15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying after
    attempt 8, due to: org.postgresql.util.PSQLException: No value specified
    for parameter 1.


    This way of issuing commands is quite nasty...

    BR/Pablo

    On 10/15/2013 11:00 AM, Pablo Fernandez wrote:
    Ok, so I finally opted for inserting a full node from one DB into
    another, since getting the resources should be quite easy with a GET.

    But then I run into a problem when running the replace catalog command
    (with an empty catalog):

    curl -vv -G -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode
    'payload={"command":"replace catalog","version": 2,"payload":"{
    \"metadata\": {\"api_version\": 1}, \"data\": {\"name\":
    \"puppet09.cscs.ch\",\"version\": \"1111\", \"edges\": [],
    \"resources\": [] } }"}'

    Which returns ok. The puppetdb.log shows:

    2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
    [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying
    after attempt 8, due to: org.postgresql.util.PSQLException: ERROR:
    syntax error at or near ")"
    Position: 66

    I have looked again and again into the data, and syntax seems to be
    correct.

    Anybody tried this? Hints?

    Thanks a lot,
    Pablo Fernandez



    On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
    That would really depend on the approach ultimately taken for the
    multi-master. If the puppet master (or agent in masterless mode)
    forwards the same fact/report/node/etc data to multiple puppetdb
    instances then yes the scale breaks horribly and does not meet part
    of this request. If there is some kind of store-and-forward
    mechanism in puppetdb itself, it would be possible to extend the
    forward to say 'only if it matches these rules'. This could
    definitely be useful in a variety of ways beyond simple HA purposes.



    On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
    wrote:

    Yes, I saw that... this may be a solution for replicating full
    PuppetDBs (if you tackle the backlog during a long shortage), but
    that does not scale very well if you have 5-10 PuppetDBs all
    being replicated to a single, bigger one.

    Besides, it would not cover cases when you want to have only some
    nodes replicated, or even only some resources.


    On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321


    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <wolf@wolfspyre.com
    wrote:

    I'd be interested in accomplishing a similar thing. I want
    to have a central reporting puppetdb which I can (outside of
    puppet) query for multisite information without tying
    isolated environments together via a single postgres db, or
    using multi-master replication.

    my needs do not involve making each puppet site aware of
    each other's data, but it would be nice to have one db which
    has all the data for my digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at
    site A (what updates have you had in the past N minutes/
    since the last time I checked in with you/ since
    timestamp(N)), and then iterate through the collected data
    and feed it into a reporting puppetdb instance, however I've
    not gone beyond thinking about it.





    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
    <pablo.fernandez@cscs.ch >> wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite
    interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was
    to add exported
    resources from Node_A (managed by PuppetMaster_A that
    has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is
    unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be
    aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to
    change a whole
    catalog, or facts. Do they need to be existing catalogs,
    or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported
    resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog",
    no "add
    catalog". Would it be rejected? How does the first
    catalog of a node get
    introduced into the DB? And moreover, how do I query a
    whole catalog?

    Another possibility would be to include the resource
    inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported
    resources don't
    really mater what node do they belong, as long as they
    have the right
    attributes, right?), but the Catalog wire format seems
    to require the
    whole set of resources, so, if I define the resources of
    Node_B with
    some exported resources from Node_A, I believe I would
    lose the original
    Node_B's resources. The only possibility would be to
    query Node_B full
    catalog, and *add also* those resources from Node_A
    before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to
    implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported
    resources from one
    PuppetDB to another? I guess this could be done with
    the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of
    discussions internally
    about this kind of thing. How about you outline your
    requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand?
    The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The
    documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to
    the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails
    from it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at
    http://groups.google.com/group/puppet-users.
    For more options, visit
    https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails
    from it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    For more options, visit
    https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com > Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 15, 2013 at 12:51 pm
    Hehe, sorry for the auto-reply... but I made it work, and I wanted to
    post it for reference.

    The key was to add also the edges: it can't be an empty list.

    For the record, doing:

    # curl -vv -G -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode 'payload={
       "command": "replace catalog",
       "version": 2,
       "payload": "{
         \"metadata\": {
           \"api_version\": 1
         },
         \"data\": {
           \"name\": \"myhost.blabla.bla\",
           \"version\": \"1111\",
           \"edges\": [
             {\"source\": {\"type\": \"Service\", \"title\": \"iptables\"},
              \"target\": {\"type\": \"Service\", \"title\": \"iptables\"},
              \"relationship\": \"before\"}
           ],
           \"resources\": [
             {\"type\": \"Service\",
              \"title\": \"iptables\",
              \"exported\": false,
              \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\",
              \"line\": 121,
              \"tags\": [\"node\", \"puppetdb\"],
              \"parameters\": {\"hasrestart\": true, \"ensure\" : \"running\"}
             }
           ]
         }
       }"
    }'

    Works like a charm, and add a new host into the PuppetDB out of the blue.
    Next: automatically serialize those escaped quotes and try to see if it
    exported resources work as they should.

    Thanks, and sorry again for the noise,
    BR/Pablo


    On 10/15/2013 11:25 AM, Pablo Fernandez wrote:
    If I try to add a fictitious resource:

    {\"type\": \"Service\",
    \"title\": \"iptables\",
    \"exported\": false,
    \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\",
    \"line\": 121,
    \"tags\": [\"node\", \"puppetdb\"],
    \"parameters\": {
    \"ensure\" : \"running\"}
    }

    I get:

    2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command]
    [15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying
    after attempt 8, due to: org.postgresql.util.PSQLException: No value
    specified for parameter 1.


    This way of issuing commands is quite nasty...

    BR/Pablo

    On 10/15/2013 11:00 AM, Pablo Fernandez wrote:
    Ok, so I finally opted for inserting a full node from one DB into
    another, since getting the resources should be quite easy with a GET.

    But then I run into a problem when running the replace catalog
    command (with an empty catalog):

    curl -vv -G -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode
    'payload={"command":"replace catalog","version": 2,"payload":"{
    \"metadata\": {\"api_version\": 1}, \"data\": {\"name\":
    \"puppet09.cscs.ch\",\"version\": \"1111\", \"edges\": [],
    \"resources\": [] } }"}'

    Which returns ok. The puppetdb.log shows:

    2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
    [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying
    after attempt 8, due to: org.postgresql.util.PSQLException: ERROR:
    syntax error at or near ")"
    Position: 66

    I have looked again and again into the data, and syntax seems to be
    correct.

    Anybody tried this? Hints?

    Thanks a lot,
    Pablo Fernandez



    On 10/10/2013 05:23 PM, Throwe, Jesse wrote:
    That would really depend on the approach ultimately taken for the
    multi-master. If the puppet master (or agent in masterless mode)
    forwards the same fact/report/node/etc data to multiple puppetdb
    instances then yes the scale breaks horribly and does not meet part
    of this request. If there is some kind of store-and-forward
    mechanism in puppetdb itself, it would be possible to extend the
    forward to say 'only if it matches these rules'. This could
    definitely be useful in a variety of ways beyond simple HA purposes.



    On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez
    wrote:

    Yes, I saw that... this may be a solution for replicating full
    PuppetDBs (if you tackle the backlog during a long shortage),
    but that does not scale very well if you have 5-10 PuppetDBs all
    being replicated to a single, bigger one.

    Besides, it would not cover cases when you want to have only
    some nodes replicated, or even only some resources.


    On 10/10/2013 04:35 PM, Throwe, Jesse wrote:
    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321


    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble
    wrote:

    I'd be interested in accomplishing a similar thing. I want
    to have a central reporting puppetdb which I can (outside
    of puppet) query for multisite information without tying
    isolated environments together via a single postgres db, or
    using multi-master replication.

    my needs do not involve making each puppet site aware of
    each other's data, but it would be nice to have one db
    which has all the data for my digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at
    site A (what updates have you had in the past N minutes/
    since the last time I checked in with you/ since
    timestamp(N)), and then iterate through the collected data
    and feed it into a reporting puppetdb instance, however
    I've not gone beyond thinking about it.





    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez
    <pablo.fernandez@cscs.ch >>> wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite
    interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do
    was to add exported
    resources from Node_A (managed by PuppetMaster_A that
    has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A
    is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be
    aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to
    change a whole
    catalog, or facts. Do they need to be existing
    catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported
    resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog",
    no "add
    catalog". Would it be rejected? How does the first
    catalog of a node get
    introduced into the DB? And moreover, how do I query a
    whole catalog?

    Another possibility would be to include the resource
    inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported
    resources don't
    really mater what node do they belong, as long as they
    have the right
    attributes, right?), but the Catalog wire format seems
    to require the
    whole set of resources, so, if I define the resources
    of Node_B with
    some exported resources from Node_A, I believe I would
    lose the original
    Node_B's resources. The only possibility would be to
    query Node_B full
    catalog, and *add also* those resources from Node_A
    before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to
    implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported
    resources from one
    PuppetDB to another? I guess this could be done
    with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of
    discussions internally
    about this kind of thing. How about you outline your
    requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand?
    The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The
    documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to
    the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving
    emails from it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at
    http://groups.google.com/group/puppet-users.
    For more options, visit
    https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails
    from it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at
    http://groups.google.com/group/puppet-users.
    For more options, visit
    https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Deepak Giridharagopal at Oct 15, 2013 at 4:51 pm

    On Tue, Oct 15, 2013 at 6:51 AM, Pablo Fernandez wrote:

    Hehe, sorry for the auto-reply... but I made it work, and I wanted to
    post it for reference.

    The key was to add also the edges: it can't be an empty list.

    For the record, doing:

    # curl -vv -G -H "Accept: application/json" '
    http://localhost:8080/v2/commands' --data-urlencode 'payload={

    "command": "replace catalog",
    "version": 2,
    "payload": "{
    \"metadata\": {
    \"api_version\": 1
    },
    \"data\": {
    \"name\": \"myhost.blabla.bla\",
    \"version\": \"1111\",
    \"edges\": [
    {\"source\": {\"type\": \"Service\", \"title\": \"iptables\"},
    \"target\": {\"type\": \"Service\", \"title\": \"iptables\"},
    \"relationship\": \"before\"}
    ],
    \"resources\": [

    {\"type\": \"Service\",
    \"title\": \"iptables\",
    \"exported\": false,
    \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\",
    \"line\": 121,
    \"tags\": [\"node\", \"puppetdb\"],
    \"parameters\": {\"hasrestart\": true, \"ensure\" : \"running\"}
    }
    ]
    }
    }"
    }'

    Works like a charm, and add a new host into the PuppetDB out of the blue.
    Next: automatically serialize those escaped quotes and try to see if it
    exported resources work as they should.

    Thanks, and sorry again for the noise,
    BR/Pablo
    Hi Pablo, I'm glad you got this working and I apologize it took so long. A
    few comments:

    1) You don't need to formulate the "payload" as a pre-serialized JSON
    string (with all the requisite escaping and whatnot) anymore. As of
    "replace catalog" version 2, you can just inline the raw JSON object itself:

    "payload": {"data": ..., "metadata": ..., ...}

    The entire command is a single JSON object that way, and the payload is
    just a normal value in that hash.

    2) You are correct that the edges key is required, and should contain every
    relationship specified in the supplied catalog:
    http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html#main-data-type-catalog

    I do agree that the error message isn't helpful at all, though, and we
    should fix that. It's likely that we aren't doing proper validation on that
    and letting things get to the storage code, which throws an opaque,
    low-level error that is difficult to debug. That said, as there aren't any
    relationships specified in your above set of resource (no require/subscribe
    params, for example) I'm curious why the empty list of edges fails. That
    smells like a bug to me; would you mind filing a ticket against PuppetDB
    with the above command included? We can take a look and see what's up.

    3) With curl, you can supply an argument of "--data-urlencode
    payload@mycatalog.json". Then you can just put the payload in a
    mycatalog.json file, and curl should automatically load the content and
    urlencode it for you. See
    http://curl.haxx.se/docs/manpage.html#--data-urlencode

    deepak




    On 10/15/2013 11:25 AM, Pablo Fernandez wrote:

    If I try to add a fictitious resource:

    {\"type\": \"Service\",
    \"title\": \"iptables\",
    \"exported\": false,
    \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\",
    \"line\": 121,
    \"tags\": [\"node\", \"puppetdb\"],
    \"parameters\": {
    \"ensure\" : \"running\"}
    }

    I get:

    2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command]
    [15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying after
    attempt 8, due to: org.postgresql.util.PSQLException: No value specified
    for parameter 1.


    This way of issuing commands is quite nasty...

    BR/Pablo


    On 10/15/2013 11:00 AM, Pablo Fernandez wrote:

    Ok, so I finally opted for inserting a full node from one DB into another,
    since getting the resources should be quite easy with a GET.

    But then I run into a problem when running the replace catalog command
    (with an empty catalog):

    curl -vv -G -H "Accept: application/json" '
    http://localhost:8080/v2/commands' --data-urlencode
    'payload={"command":"replace catalog","version": 2,"payload":"{
    \"metadata\": {\"api_version\": 1}, \"data\": {\"name\": \"
    puppet09.cscs.ch\",\"version\": \"1111\", \"edges\": [], \"resources\":
    [] } }"}'

    Which returns ok. The puppetdb.log shows:

    2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command]
    [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying after
    attempt 8, due to: org.postgresql.util.PSQLException: ERROR: syntax error
    at or near ")"
    Position: 66

    I have looked again and again into the data, and syntax seems to be
    correct.

    Anybody tried this? Hints?

    Thanks a lot,
    Pablo Fernandez




    On 10/10/2013 05:23 PM, Throwe, Jesse wrote:

    That would really depend on the approach ultimately taken for the
    multi-master. If the puppet master (or agent in masterless mode) forwards
    the same fact/report/node/etc data to multiple puppetdb instances then yes
    the scale breaks horribly and does not meet part of this request. If there
    is some kind of store-and-forward mechanism in puppetdb itself, it would be
    possible to extend the forward to say 'only if it matches these rules'.
    This could definitely be useful in a variety of ways beyond simple HA
    purposes.



    On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez <pablo.fernandez@cscs.ch
    wrote:
    Yes, I saw that... this may be a solution for replicating full PuppetDBs
    (if you tackle the backlog during a long shortage), but that does not scale
    very well if you have 5-10 PuppetDBs all being replicated to a single,
    bigger one.

    Besides, it would not cover cases when you want to have only some nodes
    replicated, or even only some resources.



    On 10/10/2013 04:35 PM, Throwe, Jesse wrote:

    It sounds like a shade of #19321 -
    https://projects.puppetlabs.com/issues/19321

    On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble wrote:

    I'd be interested in accomplishing a similar thing. I want to have a
    central reporting puppetdb which I can (outside of puppet) query for
    multisite information without tying isolated environments together via a
    single postgres db, or using multi-master replication.

    my needs do not involve making each puppet site aware of each other's
    data, but it would be nice to have one db which has all the data for my
    digging pleasure.

    from my understanding this is not currently supported.

    I was thinking something could be done to ask puppetdb at site A (what
    updates have you had in the past N minutes/ since the last time I checked
    in with you/ since timestamp(N)), and then iterate through the collected
    data and feed it into a reporting puppetdb instance, however I've not gone
    beyond thinking about it.





    On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez <
    pablo.fernandez@cscs.ch> wrote:
    Thanks Ken,

    I took a look at the Command API and seems to be quite interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change a whole
    catalog, or facts. Do they need to be existing catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of a node get
    introduced into the DB? And moreover, how do I query a whole catalog?

    Another possibility would be to include the resource inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported resources don't
    really mater what node do they belong, as long as they have the right
    attributes, right?), but the Catalog wire format seems to require the
    whole set of resources, so, if I define the resources of Node_B with
    some exported resources from Node_A, I believe I would lose the original
    Node_B's resources. The only possibility would be to query Node_B full
    catalog, and *add also* those resources from Node_A before sending them
    to the DB... and this seems to me like a bit dirty :)

    Am I missing anything? Or is there another way to implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported resources from
    one
    PuppetDB to another? I guess this could be done with the API, but
    has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 16, 2013 at 7:16 am
    Dear Deepak,

    On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:

    1) You don't need to formulate the "payload" as a pre-serialized JSON
    string (with all the requisite escaping and whatnot) anymore. As of
    "replace catalog" version 2, you can just inline the raw JSON object
    itself:

    "payload": {"data": ..., "metadata": ..., ...}

    The entire command is a single JSON object that way, and the payload
    is just a normal value in that hash
    3) With curl, you can supply an argument of "--data-urlencode
    payload@mycatalog.json". Then you can just put the payload in a
    mycatalog.json file, and curl should automatically load the content
    and urlencode it for you.
    See http://curl.haxx.se/docs/manpage.html#--data-urlencode

    This RAW version is quite interesting (escaping part of the thing is a
    bit nasty), I will play with it :)

    I have a question, though... if you can insert the RAW payload... what
    happens with the command name?
    I guess you mean:
    "payload": { "command": "replace catalog", "version": 2, "payload":
    {"data": ..., "metadata": ..., ...} }
    Without any escape sequence at all, right?

    And then send it with the --data-urlencode


    That said, as there aren't any relationships specified in your above
    set of resource (no require/subscribe params, for example) I'm curious
    why the empty list of edges fails. That smells like a bug to me; would
    you mind filing a ticket against PuppetDB with the above command
    included? We can take a look and see what's up.
    Submitted!


    Thanks a lot,
    Pablo Fernandez

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 17, 2013 at 3:09 pm
    Hi,

    I tried to upload a full catalog with the command:
    # curl -vv -G -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode payload@catalog.json

    Where catalog.json would start with:
    {"version": 2, "command": "replace catalog", "payload": {"data":
    {"version ...etc...

    But I get a:
    HTTP/1.1 413 FULL head

    This seems to be the Jetty server not accepting big POST. This one is
    112 KB, which is not really big for a catalog.
    I probably hit a configuration issue, but I don't see the option in the
    [jetty] section... bug maybe?

    Thanks!
    BR/Pablo

    On 10/16/2013 09:16 AM, Pablo Fernandez wrote:
    Dear Deepak,

    On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:

    1) You don't need to formulate the "payload" as a pre-serialized JSON
    string (with all the requisite escaping and whatnot) anymore. As of
    "replace catalog" version 2, you can just inline the raw JSON object
    itself:

    "payload": {"data": ..., "metadata": ..., ...}

    The entire command is a single JSON object that way, and the payload
    is just a normal value in that hash
    3) With curl, you can supply an argument of "--data-urlencode
    payload@mycatalog.json". Then you can just put the payload in a
    mycatalog.json file, and curl should automatically load the content
    and urlencode it for you.
    See http://curl.haxx.se/docs/manpage.html#--data-urlencode

    This RAW version is quite interesting (escaping part of the thing is a
    bit nasty), I will play with it :)

    I have a question, though... if you can insert the RAW payload... what
    happens with the command name?
    I guess you mean:
    "payload": { "command": "replace catalog", "version": 2, "payload":
    {"data": ..., "metadata": ..., ...} }
    Without any escape sequence at all, right?

    And then send it with the --data-urlencode


    That said, as there aren't any relationships specified in your above
    set of resource (no require/subscribe params, for example) I'm
    curious why the empty list of edges fails. That smells like a bug to
    me; would you mind filing a ticket against PuppetDB with the above
    command included? We can take a look and see what's up.
    Submitted!


    Thanks a lot,
    Pablo Fernandez
    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Deepak Giridharagopal at Oct 17, 2013 at 4:02 pm

    On Thu, Oct 17, 2013 at 9:09 AM, Pablo Fernandez wrote:

    Hi,

    I tried to upload a full catalog with the command:
    # curl -vv -G -H "Accept: application/json" '
    http://localhost:8080/v2/commands' --data-urlencode payload@catalog.json

    Where catalog.json would start with:
    {"version": 2, "command": "replace catalog", "payload": {"data": {"version
    ...etc...

    But I get a:
    HTTP/1.1 413 FULL head

    This seems to be the Jetty server not accepting big POST. This one is 112
    KB, which is not really big for a catalog.
    I probably hit a configuration issue, but I don't see the option in the
    [jetty] section... bug maybe?
    (resending as I forgot to reply to the list)

    The "-G" command you're giving to curl forces it to use a GET for the
    data...can you try after removing that option? That should default it to
    POST.

    deepak


    Thanks!
    BR/Pablo



    On 10/16/2013 09:16 AM, Pablo Fernandez wrote:

    Dear Deepak,


    On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:


    1) You don't need to formulate the "payload" as a pre-serialized JSON
    string (with all the requisite escaping and whatnot) anymore. As of
    "replace catalog" version 2, you can just inline the raw JSON object
    itself:

    "payload": {"data": ..., "metadata": ..., ...}

    The entire command is a single JSON object that way, and the payload is
    just a normal value in that hash

    3) With curl, you can supply an argument of "--data-urlencode
    payload@mycatalog.json". Then you can just put the payload in a
    mycatalog.json file, and curl should automatically load the content and
    urlencode it for you. See
    http://curl.haxx.se/docs/manpage.html#--data-urlencode



    This RAW version is quite interesting (escaping part of the thing is a bit
    nasty), I will play with it :)

    I have a question, though... if you can insert the RAW payload... what
    happens with the command name?
    I guess you mean:
    "payload": { "command": "replace catalog", "version": 2, "payload":
    {"data": ..., "metadata": ..., ...} }
    Without any escape sequence at all, right?

    And then send it with the --data-urlencode




    That said, as there aren't any relationships specified in your above set
    of resource (no require/subscribe params, for example) I'm curious why the
    empty list of edges fails. That smells like a bug to me; would you mind
    filing a ticket against PuppetDB with the above command included? We can
    take a look and see what's up.


    Submitted!


    Thanks a lot,
    Pablo Fernandez
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 17, 2013 at 4:22 pm
    That worked quite well, thanks!! That's what happens when you copy-paste
    without understanding everything :)

    Let me then share with the rest the little script to extract (and hint
    to insert) a full catalog into another PuppetDB:

    #!/usr/bin/python
    # Echo the output to a file (e.g. catalog.json), and use this way:
    # curl -vv -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode payload@catalog.json
    import json
    import urllib2
    response = urllib2.urlopen('http://localhost:8080/v3/catalogs/<node_name>')
    data = json.load(response)
    object = { "command": "replace catalog", "version": 2, "payload": data }
    print json.dumps(object, indent=2)


    I guess you could potentiate this script by running the curl itself, but
    that would require permissions on both PuppetDBs.

    BR/Pablo

    On 10/17/2013 06:02 PM, Deepak Giridharagopal wrote:
    On Thu, Oct 17, 2013 at 9:09 AM, Pablo Fernandez
    wrote:

    Hi,

    I tried to upload a full catalog with the command:
    # curl -vv -G -H "Accept: application/json"
    'http://localhost:8080/v2/commands' --data-urlencode
    payload@catalog.json
    Where catalog.json would start with:
    {"version": 2, "command": "replace catalog", "payload": {"data":
    {"version ...etc...

    But I get a:
    HTTP/1.1 413 FULL head

    This seems to be the Jetty server not accepting big POST. This one
    is 112 KB, which is not really big for a catalog.
    I probably hit a configuration issue, but I don't see the option
    in the [jetty] section... bug maybe?


    (resending as I forgot to reply to the list)

    The "-G" command you're giving to curl forces it to use a GET for the
    data...can you try after removing that option? That should default it
    to POST.

    deepak



    Thanks!
    BR/Pablo


    On 10/16/2013 09:16 AM, Pablo Fernandez wrote:
    Dear Deepak,

    On 10/15/2013 06:51 PM, Deepak Giridharagopal wrote:

    1) You don't need to formulate the "payload" as a pre-serialized
    JSON string (with all the requisite escaping and whatnot)
    anymore. As of "replace catalog" version 2, you can just inline
    the raw JSON object itself:

    "payload": {"data": ..., "metadata": ..., ...}

    The entire command is a single JSON object that way, and the
    payload is just a normal value in that hash
    3) With curl, you can supply an argument of "--data-urlencode
    payload@mycatalog.json ". Then
    you can just put the payload in a mycatalog.json file, and curl
    should automatically load the content and urlencode it for you.
    See http://curl.haxx.se/docs/manpage.html#--data-urlencode

    This RAW version is quite interesting (escaping part of the thing
    is a bit nasty), I will play with it :)

    I have a question, though... if you can insert the RAW payload...
    what happens with the command name?
    I guess you mean:
    "payload": { "command": "replace catalog", "version": 2,
    "payload": {"data": ..., "metadata": ..., ...} }
    Without any escape sequence at all, right?

    And then send it with the --data-urlencode


    That said, as there aren't any relationships specified in your
    above set of resource (no require/subscribe params, for example)
    I'm curious why the empty list of edges fails. That smells like
    a bug to me; would you mind filing a ticket against PuppetDB
    with the above command included? We can take a look and see
    what's up.
    Submitted!


    Thanks a lot,
    Pablo Fernandez
    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com > Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Deepak Giridharagopal at Oct 15, 2013 at 7:39 pm

    On Thu, Oct 10, 2013 at 6:12 AM, Pablo Fernandez wrote:

    Thanks Ken,

    I took a look at the Command API and seems to be quite interesting, but
    it triggers a few more questions.

    As a reminder (also for others), what I wanted to do was to add exported
    resources from Node_A (managed by PuppetMaster_A that has its own
    PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in
    order for other Nodes_B (managed by PuppetMaster_B) be aware and
    possibly realize those exported resources.

    Taking a look the Commands API seem to capabilities to change a whole
    catalog, or facts. Do they need to be existing catalogs, or could I add
    new ones?
    I am thinking: If I want to add Node_A's exported resource(s) to
    PuppetDB_B, I could "artificially" add its whole catalog to
    PuppetDB_B... but the Command API is "replace catalog", no "add
    catalog". Would it be rejected? How does the first catalog of a node get
    introduced into the DB? And moreover, how do I query a whole catalog?
    Lots of great questions here! :)

    * You can certainly add the whole catalog for a node to a secondary PuppetDB

    * The command is called "replace catalog" to indicate that it's, well, a
    replacement operation. So if you already have a catalog for node
    foo.comstored, issuing a "replace catalog" with a new catalog for
    foo.com will cause PuppetDB to delete the old catalog and store the new one
    for foo.com. It does this atomically, of course. This is "normal" operation
    for PuppetDB, as it only stores a single catalog for a given node.

    * A "replace catalog" command won't be rejected if there's already a
    catalog for that node; it'll just tell PuppetDB to store new data in place
    of the old data. In fact, this is how PuppetDB works with puppetmasters
    today...an agent requests a catalog from the master, the master compiles it
    for the agent, and the master issues a "replace catalog" command to
    puppetdb accordingly.

    * There is an endpoint to get the catalog for a node:
    /v3/catalogs/foo.com. The wire format of the result should be the
    same, documented catalog wire
    format as during insertion:
    http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html.
    I do believe, though, that we overlooked adding documentation for this
    new endpoint...apologies! A pull req adding those docs to the
    documentation/ directory of puppetdb would be much appreciated. :)

    Another possibility would be to include the resource inside an existing
    Node_B catalog on the PuppetDB_B (after all, exported resources don't
    really mater what node do they belong, as long as they have the right
    attributes, right?), but the Catalog wire format seems to require the
    whole set of resources, so, if I define the resources of Node_B with
    some exported resources from Node_A, I believe I would lose the original
    Node_B's resources. The only possibility would be to query Node_B full
    catalog, and *add also* those resources from Node_A before sending them
    to the DB... and this seems to me like a bit dirty :)
    Yes, you'd have to get the catalog for a node, modify it, and resubmit it.
    Remember that command submission is asynchronous, commands can be retried,
    etc. If you have a command of the form "add some resources to the currently
    stored catalog", the definition of "currently stored" isn't a static thing.
    It's thus subject to race conditions and other confusion. By requiring an
    entire catalog be submitted, and by making it clear through the name of the
    command itself ("replace catalog") a large class of problems go away while
    still keeping throughput high and retry-handling manageable.

    I'm not opposed to the idea of a "merge" type of command for catalogs, of
    course...but I think it's something that should be approached cautiously,
    with a good understanding of how it will impact (and be impacted by) the
    rest of the system. Maybe doing something like a command that merged a set
    of resources into a known, specific catalog (identified by a stable hash or
    id) would work?

    Am I missing anything? Or is there another way to implemen what I need
    with Puppet 3.2?

    Thanks again,
    BR/Pablo

    On 10/10/2013 09:16 AM, Ken Barber wrote:
    Is there a mechanism to pull and push some exported resources from one
    PuppetDB to another? I guess this could be done with the API, but has
    anybody tried it, or will it work at all?
    Not provided by us, yet. We've had a lot of discussions internally
    about this kind of thing. How about you outline your requirements in
    the form of a feature request here:

    http://projects.puppetlabs.com/projects/puppetdb/issues
    Did anybody try to push data to a PuppetDB by hand? The API does not
    seem to have the PUT or POST methods.
    The client obviously pushes uses POST :-). The documentation for the
    mechanism is here:

    http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html

    ken.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pablo Fernandez at Oct 16, 2013 at 7:22 am
    Thanks again for your answers, they are quite useful!

    On 10/15/2013 09:39 PM, Deepak Giridharagopal wrote:
    * You can certainly add the whole catalog for a node to a secondary
    PuppetDB
    I actually think this is the way to go, at least in my case.
    * There is an endpoint to get the catalog for a node:
    /v3/catalogs/foo.com <http://foo.com> . The wire format of the result
    should be the same, documented catalog wire format as during
    insertion: http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html
    . I do believe, though, that we overlooked adding documentation for
    this new endpoint...apologies! A pull req adding those docs to the
    documentation/ directory of puppetdb would be much appreciated. :)
    This is quite useful indeed! It even has the metadata part embedded.

    It is quite strange that the wire format differs from the GET output (
    /v3/nodes/<node>/resources ) in the sense that the replace catalog
    command requires resources to have the "sourcefile" (from the GET) named
    as "file", and "sourceline" as "line". The rest of the keys have the
    same name and are compatible... it's a pity this small asymmetry. I
    submitted a refactor request for this.


    Thanks again,
    Pablo

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedOct 7, '13 at 11:08a
activeOct 17, '13 at 4:22p
posts18
users5
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase