FAQ
Hello!

I'm currently working on a testsuite using Python's unittest library. This
works all good and fine, but there's one thing where I haven't seen an
elegant solution to yet, and that is the ordering. Currently, it takes all
classes and orders them alphabetically and then takes all test functions
therein and runs those alphabetically, too. However, sometimes it doesn't
make sense to run test_bar() if test_foo() already failed, because they
basically build upon each other. However, test_bar() is still run first,
and test_foo() second.

What I sometimes do is to simply number them, like test_1_foo() and
test_2_bar(), but that seems ugly. I'm not even looking for a way to
express complicated relations between tests, but is there a less ugly way
to explicitly order them?

Cheers!

Uli

--
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

Search Discussions

  • Peter Otten at Oct 7, 2010 at 6:37 pm

    Ulrich Eckhardt wrote:

    I'm currently working on a testsuite using Python's unittest library. This
    works all good and fine, but there's one thing where I haven't seen an
    elegant solution to yet, and that is the ordering. Currently, it takes all
    classes and orders them alphabetically and then takes all test functions
    therein and runs those alphabetically, too. However, sometimes it doesn't
    make sense to run test_bar() if test_foo() already failed, because they
    basically build upon each other. However, test_bar() is still run first,
    and test_foo() second.

    What I sometimes do is to simply number them, like test_1_foo() and
    test_2_bar(), but that seems ugly. I'm not even looking for a way to
    express complicated relations between tests, but is there a less ugly way
    to explicitly order them?
    You can pass unittest.main() a custom test loader which in turn can define a
    custom sortTestMethodsUsing() method. Here's a fancy example:

    import unittest

    def _toposort(data):
    # adapted from http://code.activestate.com/recipes/577413-topological-
    sort/
    for k, v in data.items():
    v.discard(k) # Ignore self dependencies
    extra_items_in_deps = reduce(set.union, data.values()) -
    set(data.keys())
    data.update((item, set()) for item in extra_items_in_deps)
    while True:
    ordered = set(item for item,dep in data.items() if not dep)
    if not ordered:
    break
    for item in sorted(ordered):
    yield item
    data = dict((item, (dep - ordered)) for item,dep in data.items()
    if item not in ordered)

    def _get_name(func):
    try:
    return func.__name__
    except AttributeError:
    pass
    assert isinstance(func, str)
    return func

    class Cmp(object):
    def __init__(self):
    self._deps = {}
    def depends(self, f, names):
    k = _get_name(f)
    assert k not in self._deps
    self._deps[k] = set(_get_name(n) for n in names)
    return f
    def make_loader(self):
    lookup = dict((name, index) for index, name
    in enumerate(_toposort(self._deps)))
    def compare(a, b):
    return cmp(lookup[a], lookup[b])
    class TestLoader(unittest.TestLoader):
    pass
    loader = TestLoader()
    loader.sortTestMethodsUsing = compare
    return loader
    def depends_on(self, *names):
    def d(f):
    return self.depends(f, names)
    return d


    c = Cmp()
    depends_on = c.depends_on

    class A(unittest.TestCase):
    @depends_on("test_beta")
    def test_alpha(self):
    pass
    @depends_on("test_gamma")
    def test_beta(self):
    pass
    def test_gamma(self):
    pass
    @depends_on(test_gamma)
    def test_delta(self):
    pass
    @depends_on(test_alpha, test_beta)
    def test_epsilon(self):
    pass

    if __name__ == "__main__":
    unittest.main(testLoader=c.make_loader())

    Peter
  • Ben Finney at Oct 7, 2010 at 9:35 pm

    Ulrich Eckhardt <eckhardt at satorlaser.com> writes:

    However, sometimes it doesn't make sense to run test_bar() if
    test_foo() already failed, because they basically build upon each
    other.
    That's a mistake. If the success of ?test_bar? depends on the result of
    ?test_foo?, then it's not an independent test and therefore isn't a unit
    test.

    Which is a good reason for re-ordering tests; it's even a good idea to
    randomise the order in which they're run, but AFAIK the ?unittest?
    library doesn't do that yet.

    If you have state needed by multiple tests, that's what the fixtures are
    for: define a ?setUp? (and, if useful, ?tearDown?) to handle common
    fixtures needed by tests in the test class, and ensure they're in a
    known state before each test.

    --
    \ ?Science is a way of trying not to fool yourself. The first |
    `\ principle is that you must not fool yourself, and you are the |
    _o__) easiest person to fool.? ?Richard P. Feynman, 1964 |
    Ben Finney
  • Steven D'Aprano at Oct 8, 2010 at 12:42 am

    On Fri, 08 Oct 2010 08:35:12 +1100, Ben Finney wrote:

    Ulrich Eckhardt <eckhardt at satorlaser.com> writes:
    However, sometimes it doesn't make sense to run test_bar() if
    test_foo() already failed, because they basically build upon each
    other.
    That's a mistake. If the success of ?test_bar? depends on the result of
    ?test_foo?, then it's not an independent test and therefore isn't a unit
    test.

    I don't think that is what Ulrich means to imply. I don't think he means
    that test_bar uses the result of test_foo, but only that if test_foo
    fails then test_bar has no hope of succeeding.

    To give an example, suppose you have:

    class MyClass:
    def __init__(self, cheese):
    self.cheese = cheese

    class MyTests(unittest.TestCase):
    def test_create(self):
    # Test that instances can be created correctly.
    self.assert_raises(TypeError, MyClass)
    x = MyClass("cheddar") # will fail if MyClass can't be created
    def test_cheese(self):
    # Test that instances have a cheese attribute.
    x = MyClass("swiss")
    self.assertEquals(x.cheese, "swiss")


    If test_create fails, then so will test_cheese, not because the later
    test is dependent on the first, but because creation is more fundamental
    than attribute access. I think the OP wants to skip test_cheese in the
    even that test_create fails.



    --
    Steven
  • Mike Kent at Oct 8, 2010 at 12:22 am
    But sometimes you just wanna do it the way you wanna do it. If you
    name your tests like 'test_01_yadda' and test_02_whatever', then they
    will be run in the order you want, as given by the numbers.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedOct 7, '10 at 5:05p
activeOct 8, '10 at 12:42a
posts5
users5
websitepython.org

People

Translate

site design / logo © 2022 Grokbase