Dear Ansiblists,

I need to integrate application-level tests with ansible runs. These test
are quite specific to that applications we are deploying and go quite
beyond the existing test support in ansible. The purpose of these tests is
not to test the ansible code but the entire deployment stack. Running these
tests as part as part of the ansible playbook makes sense as the ansible
run knows the location of all created resources. I am happy to go off and
create this on my own, but I would really like to collaborate if there are
others working on similar problems. I want to write the tests using plain
python unittest so it will be familiar to my developers.

here is an example where i want to test a Consul cluster of 5 nodes. I want
to verify that the Consul API functions as expected.

here is a code example of what I am looking to test

# tests/test_consul.py
import unittest
import os.environ
import requests
import consul
import time


class TestConsul(unittest.TestCase):

   def setUp(self):
     self.members = os.environ['CONSUL_CLUSTER'].split(',')

   def test_nodes_return_same_membership_list(self):
     results = []
     for member in self.members:
       member_list = requests.get('http://member:4444/v1/cluster/members' %
member)
       member_list.sort()
       results.append(member_list)
     self.assertTrue(all([True for result in results if result ==
results[0]))

   def test_kv_put(self):
     c =
consul.Consul(self.members[0])

     c.kv.put('foo', 'bar')
     # wait for
replication

     time.sleep(5)
     results = []
     for member in self.members:
       c =
consul.Consul(member)

       results.append(c.kv.get('foo'))
     self.assertTrue(all([True for result in results if result == 'bar']))


as you can see, I plan on passing the list of nodes as an environment
variable, but I would love to hear alternate approaches.

I currently plan to trigger the running of these tests by an environment
variable.

$ RUN_TESTS=TRUE ansible-playbook my-plays.yml

- name: run tests
   when: "{{ lookoup('ENV', 'RUN_TESTS')|bool}}"
   roles:
      - execute-tests # executes tests in tests/test_*.py using unittest
module, yet to be written

It's really key that my developers be able to write tests using plain
python code rather than Ansible's YAML. This decouples the tests from
ansible. AThis could we helpful later if we every choose to run the exact
same tests later separately from ansible, for example executing them as
part of regular monitoring.

Thoughts?

Regards,

Bryan W. Berry



--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Michael DeHaan at Dec 15, 2014 at 12:10 pm
    This is a good read if you haven't seen it already -
    http://docs.ansible.com/test_strategies.html
    On Mon, Dec 15, 2014 at 4:33 AM, Bryan Berry wrote:

    Dear Ansiblists,

    I need to integrate application-level tests with ansible runs. These test
    are quite specific to that applications we are deploying and go quite
    beyond the existing test support in ansible. The purpose of these tests is
    not to test the ansible code but the entire deployment stack. Running these
    tests as part as part of the ansible playbook makes sense as the ansible
    run knows the location of all created resources. I am happy to go off and
    create this on my own, but I would really like to collaborate if there are
    others working on similar problems. I want to write the tests using plain
    python unittest so it will be familiar to my developers.

    here is an example where i want to test a Consul cluster of 5 nodes. I
    want to verify that the Consul API functions as expected.

    here is a code example of what I am looking to test

    # tests/test_consul.py
    import unittest
    import os.environ
    import requests
    import consul
    import time


    class TestConsul(unittest.TestCase):

    def setUp(self):
    self.members = os.environ['CONSUL_CLUSTER'].split(',')

    def test_nodes_return_same_membership_list(self):
    results = []
    for member in self.members:
    member_list = requests.get('http://member:4444/v1/cluster/members'
    % member)
    member_list.sort()
    results.append(member_list)
    self.assertTrue(all([True for result in results if result ==
    results[0]))

    def test_kv_put(self):
    c =
    consul.Consul(self.members[0])

    c.kv.put('foo', 'bar')
    # wait for
    replication

    time.sleep(5)
    results = []
    for member in self.members:
    c =
    consul.Consul(member)

    results.append(c.kv.get('foo'))
    self.assertTrue(all([True for result in results if result == 'bar']))



    as you can see, I plan on passing the list of nodes as an environment
    variable, but I would love to hear alternate approaches.

    I currently plan to trigger the running of these tests by an environment
    variable.

    $ RUN_TESTS=TRUE ansible-playbook my-plays.yml

    - name: run tests
    when: "{{ lookoup('ENV', 'RUN_TESTS')|bool}}"
    roles:
    - execute-tests # executes tests in tests/test_*.py using unittest
    module, yet to be written

    It's really key that my developers be able to write tests using plain
    python code rather than Ansible's YAML. This decouples the tests from
    ansible. AThis could we helpful later if we every choose to run the exact
    same tests later separately from ansible, for example executing them as
    part of regular monitoring.

    Thoughts?

    Regards,

    Bryan W. Berry



    --
    You received this message because you are subscribed to the Google Groups
    "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com
    <https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyNwsvqJOhoikq3hiDjJn%2BRsNG1VD7M%3DabwHX4-EH1MRA%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Bryan Berry at Dec 15, 2014 at 12:15 pm
    Hi Michael,

    I have read it and I believe we will use some of those testing strategies.
    Those tests are good for testing an ansible playbook while I am looking to
    test the actual API of the services I am spinning up, and in a far more
    flexible way. From that page:

    "Something like an integration test battery should be written by your QA
    team if you are a production webservice. This would include things like
    Selenium tests or automated API tests and would usually not be something
    embedded into your Ansible playbooks."

    This is my exact use case, though not using Selenium specifically. We have
    a QA team that will be writing integration tests. I would like run them at
    the end of the ansible run.
    On Mon, Dec 15, 2014 at 1:10 PM, Michael DeHaan wrote:

    This is a good read if you haven't seen it already -
    http://docs.ansible.com/test_strategies.html
    On Mon, Dec 15, 2014 at 4:33 AM, Bryan Berry wrote:

    Dear Ansiblists,

    I need to integrate application-level tests with ansible runs. These test
    are quite specific to that applications we are deploying and go quite
    beyond the existing test support in ansible. The purpose of these tests is
    not to test the ansible code but the entire deployment stack. Running these
    tests as part as part of the ansible playbook makes sense as the ansible
    run knows the location of all created resources. I am happy to go off and
    create this on my own, but I would really like to collaborate if there are
    others working on similar problems. I want to write the tests using plain
    python unittest so it will be familiar to my developers.

    here is an example where i want to test a Consul cluster of 5 nodes. I
    want to verify that the Consul API functions as expected.

    here is a code example of what I am looking to test

    # tests/test_consul.py
    import unittest
    import os.environ
    import requests
    import consul
    import time


    class TestConsul(unittest.TestCase):

    def setUp(self):
    self.members = os.environ['CONSUL_CLUSTER'].split(',')

    def test_nodes_return_same_membership_list(self):
    results = []
    for member in self.members:
    member_list = requests.get('http://member:4444/v1/cluster/members'
    % member)
    member_list.sort()
    results.append(member_list)
    self.assertTrue(all([True for result in results if result ==
    results[0]))

    def test_kv_put(self):
    c =
    consul.Consul(self.members[0])

    c.kv.put('foo', 'bar')
    # wait for
    replication

    time.sleep(5)
    results = []
    for member in self.members:
    c =
    consul.Consul(member)

    results.append(c.kv.get('foo'))
    self.assertTrue(all([True for result in results if result == 'bar']))



    as you can see, I plan on passing the list of nodes as an environment
    variable, but I would love to hear alternate approaches.

    I currently plan to trigger the running of these tests by an environment
    variable.

    $ RUN_TESTS=TRUE ansible-playbook my-plays.yml

    - name: run tests
    when: "{{ lookoup('ENV', 'RUN_TESTS')|bool}}"
    roles:
    - execute-tests # executes tests in tests/test_*.py using
    unittest module, yet to be written

    It's really key that my developers be able to write tests using plain
    python code rather than Ansible's YAML. This decouples the tests from
    ansible. AThis could we helpful later if we every choose to run the exact
    same tests later separately from ansible, for example executing them as
    part of regular monitoring.

    Thoughts?

    Regards,

    Bryan W. Berry



    --
    You received this message because you are subscribed to the Google Groups
    "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com
    <https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyNwsvqJOhoikq3hiDjJn%2BRsNG1VD7M%3DabwHX4-EH1MRA%40mail.gmail.com
    <https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyNwsvqJOhoikq3hiDjJn%2BRsNG1VD7M%3DabwHX4-EH1MRA%40mail.gmail.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANsX4xTz9QsXLtTVJOtHGKy%3D%2BGCJ2eG-en0hfgUSLFuT%2BCFa%3DA%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael DeHaan at Dec 15, 2014 at 1:15 pm
    Yeah in that case, it probably doesn't make sense to have ansible launch
    that job, but it could.

    "Those tests are good for testing an ansible playbook while I am looking to
    test the actual API of the services I am spinning up, and in a far more
    flexible way. From that page:"

    In our case, we'll fire off an install of Ansible in Jenkins against a
    QA/Stage environment, and then a dependent Jenkins job will launch
    automated API and Selenium tests.


    On Mon, Dec 15, 2014 at 7:15 AM, Bryan Berry wrote:

    Hi Michael,

    I have read it and I believe we will use some of those testing strategies.
    Those tests are good for testing an ansible playbook while I am looking to
    test the actual API of the services I am spinning up, and in a far more
    flexible way. From that page:

    "Something like an integration test battery should be written by your QA
    team if you are a production webservice. This would include things like
    Selenium tests or automated API tests and would usually not be something
    embedded into your Ansible playbooks."

    This is my exact use case, though not using Selenium specifically. We have
    a QA team that will be writing integration tests. I would like run them at
    the end of the ansible run.
    On Mon, Dec 15, 2014 at 1:10 PM, Michael DeHaan wrote:

    This is a good read if you haven't seen it already -
    http://docs.ansible.com/test_strategies.html

    On Mon, Dec 15, 2014 at 4:33 AM, Bryan Berry <bryan.berry@gmail.com>
    wrote:
    Dear Ansiblists,

    I need to integrate application-level tests with ansible runs. These
    test are quite specific to that applications we are deploying and go quite
    beyond the existing test support in ansible. The purpose of these tests is
    not to test the ansible code but the entire deployment stack. Running these
    tests as part as part of the ansible playbook makes sense as the ansible
    run knows the location of all created resources. I am happy to go off and
    create this on my own, but I would really like to collaborate if there are
    others working on similar problems. I want to write the tests using plain
    python unittest so it will be familiar to my developers.

    here is an example where i want to test a Consul cluster of 5 nodes. I
    want to verify that the Consul API functions as expected.

    here is a code example of what I am looking to test

    # tests/test_consul.py
    import unittest
    import os.environ
    import requests
    import consul
    import time


    class TestConsul(unittest.TestCase):

    def setUp(self):
    self.members = os.environ['CONSUL_CLUSTER'].split(',')

    def test_nodes_return_same_membership_list(self):
    results = []
    for member in self.members:
    member_list = requests.get('http://member:4444/v1/cluster/members'
    % member)
    member_list.sort()
    results.append(member_list)
    self.assertTrue(all([True for result in results if result ==
    results[0]))

    def test_kv_put(self):
    c =
    consul.Consul(self.members[0])

    c.kv.put('foo', 'bar')
    # wait for
    replication

    time.sleep(5)
    results = []
    for member in self.members:
    c =
    consul.Consul(member)

    results.append(c.kv.get('foo'))
    self.assertTrue(all([True for result in results if result == 'bar']))



    as you can see, I plan on passing the list of nodes as an environment
    variable, but I would love to hear alternate approaches.

    I currently plan to trigger the running of these tests by an environment
    variable.

    $ RUN_TESTS=TRUE ansible-playbook my-plays.yml

    - name: run tests
    when: "{{ lookoup('ENV', 'RUN_TESTS')|bool}}"
    roles:
    - execute-tests # executes tests in tests/test_*.py using
    unittest module, yet to be written

    It's really key that my developers be able to write tests using plain
    python code rather than Ansible's YAML. This decouples the tests from
    ansible. AThis could we helpful later if we every choose to run the exact
    same tests later separately from ansible, for example executing them as
    part of regular monitoring.

    Thoughts?

    Regards,

    Bryan W. Berry



    --
    You received this message because you are subscribed to the Google
    Groups "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com
    <https://groups.google.com/d/msgid/ansible-project/02e8c1c1-cb08-416c-9bf5-907607b529f5%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyNwsvqJOhoikq3hiDjJn%2BRsNG1VD7M%3DabwHX4-EH1MRA%40mail.gmail.com
    <https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgyNwsvqJOhoikq3hiDjJn%2BRsNG1VD7M%3DabwHX4-EH1MRA%40mail.gmail.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups
    "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/ansible-project/CANsX4xTz9QsXLtTVJOtHGKy%3D%2BGCJ2eG-en0hfgUSLFuT%2BCFa%3DA%40mail.gmail.com
    <https://groups.google.com/d/msgid/ansible-project/CANsX4xTz9QsXLtTVJOtHGKy%3D%2BGCJ2eG-en0hfgUSLFuT%2BCFa%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>
    .

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Ansible Project" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
    To post to this group, send email to ansible-project@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgwWZb1O1D%2BC2f7oNo8iYunytR0RC%2BSO-W00yjfX3o8ryw%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupansible-project @
postedDec 15, '14 at 12:09p
activeDec 15, '14 at 1:15p
posts4
users2

2 users in discussion

Bryan Berry: 2 posts Michael DeHaan: 2 posts

People

Translate

site design / logo © 2022 Grokbase