Grokbase Groups Perl qa July 2007
FAQ
I've just noticed that the MediaWiki team have a test harness that
tracks the life of individual assertions over time:

* HTML nested bullet list, open tags (bug 5497) [Has never passed]
* HTML nested ordered list, open tags (bug 5497) [Has never passed]
* Fuzz testing: image with bogus manual thumbnail [Introduced
between 08-Apr-2007 07:15:22, 1.10alpha (r21099) and 25-Apr-2007
07:15:46, 1.10alpha

Nice eh?

With TAP that'd require either adding a persistent identifier to tests,
requiring that the test numbering doesn't change over time (new tests
added at the end) or that there's some heuristic that attempts to track
assertions that have been renumbered.

http://testanything.org/wiki/index.php/Assertions_are_named_dynamically

A partial solution may lie in the direction of Test Blocks[1]. If it
became normal practice for small clusters of related assertions to be
grouped in a block then the TAP document would gain structure that would
make it easier to automagically work out which assertions had been added
or removed. Test Blocks would also localise the impact on test numbering
when a new assertion is added; outside of the affected block the
numbering wouldn't change.

And I still like the Test Blocks proposal for other reasons...

[1] http://testanything.org/wiki/index.php/Test_Blocks

--
Andy Armstrong, Hexten

Search Discussions

  • Michael Peters at Jul 12, 2007 at 2:24 pm

    Andy Armstrong wrote:
    I've just noticed that the MediaWiki team have a test harness that
    tracks the life of individual assertions over time:

    * HTML nested bullet list, open tags (bug 5497) [Has never passed]
    * HTML nested ordered list, open tags (bug 5497) [Has never passed]
    * Fuzz testing: image with bogus manual thumbnail [Introduced
    between 08-Apr-2007 07:15:22, 1.10alpha (r21099) and 25-Apr-2007
    07:15:46, 1.10alpha

    Nice eh?

    With TAP that'd require either adding a persistent identifier to tests,
    requiring that the test numbering doesn't change over time (new tests
    added at the end) or that there's some heuristic that attempts to track
    assertions that have been renumbered.
    Why can't you just use test names?

    ok($is_ok, 'HTML nested bullet list, open tags (bug 5497)');

    If tracking a single test over time seems like something people would want, I
    think it could be added to Smolder without too much difficulty.

    --
    Michael Peters
    Developer
    Plus Three, LP
  • Andy Armstrong at Jul 12, 2007 at 3:25 pm

    Michael Peters wrote:
    Why can't you just use test names?

    ok($is_ok, 'HTML nested bullet list, open tags (bug 5497)');
    You certainly could - but I tend to think of them as descriptions rather
    than names and therefore not entirely ideal as the primary key. I don't
    really want to be forced not to change a description because I'll break
    the test system if I do.
    If tracking a single test over time seems like something people would want, I
    think it could be added to Smolder without too much difficulty.
    As far as I know the only demand is me thinking "isn't that neat" - but
    I do think it's neat :)

    --
    Andy Armstrong, Hexten
  • Smylers at Jul 13, 2007 at 5:52 am

    Andy Armstrong writes:

    Michael Peters wrote:
    Why can't you just use test names?

    ok($is_ok, 'HTML nested bullet list, open tags (bug 5497)');
    You certainly could - but I tend to think of them as descriptions
    rather than names and therefore not entirely ideal as the primary key.
    I don't really want to be forced not to change a description because
    I'll break the test system if I do.
    In the above surely "(bug 5497)" is sufficient as the primary key; the
    rest of the text could change.

    So this system could be made to work for anybody who has a bug database
    and consistently labels tests with them -- which obviously isn't
    everybody, but it might not be an excessive requirement for those who
    want the tracking feature.

    Actually you don't even need for the external database to exist, so long
    as you include the unique bug IDs somewhere in the description. You
    just need to pass to the tracker the pattern that matches your format of
    IDs from the descriptions; for the above example that'd be:

    qr/\(bug \d+\)/

    Smylers
  • Ian Malpass at Jul 12, 2007 at 5:03 pm

    Andy Armstrong wrote:

    With TAP that'd require either adding a persistent identifier to tests,
    requiring that the test numbering doesn't change over time (new tests
    added at the end) or that there's some heuristic that attempts to track
    assertions that have been renumbered.
    Requiring that test numbering doesn't change seems like an awfully large
    burden. It would mean you couldn't add extra values to a loop with a
    test in it, for example (unless it happened to be at the end of your
    test). It would be difficult to enforce, easy to forget/ignore/not know
    about, and failure would be awkward.

    A heuristic would be complicated and fragile, and if (when) it went awry
    your data would, again, go screwy.

    As such, I think I'd favour the persistent identifier approach, and add
    an extra argument to each test assertion:

    is ( $got, $exptected, $test_name, $id );

    You could then create and use a subclass of Test::More to care about the
    extra argument. Anyone just using plain Test::More wouldn't see any
    difference. It wouldn't be particularly taxing when writing tests, and
    would only impact those who felt that it would give them some benefit.

    I'm not sure how you'd deal with not using Test::More (or that use
    Test::More internally), or using things that do stuff like
    all_pod_files_ok(), but those seem like solvable problems.

    Ian
  • Andy Armstrong at Jul 12, 2007 at 5:24 pm

    Ian Malpass wrote:
    Requiring that test numbering doesn't change seems like an awfully large
    burden. It would mean you couldn't add extra values to a loop with a
    test in it, for example (unless it happened to be at the end of your
    test). It would be difficult to enforce, easy to forget/ignore/not know
    about, and failure would be awkward.
    Yeah, I wasn't advocating that :)
    A heuristic would be complicated and fragile, and if (when) it went awry
    your data would, again, go screwy.
    Yeah, possibly. I think it's more or less the same problem diff solves.
    Again, I'm not actually advocating it though.
    As such, I think I'd favour the persistent identifier approach, and add
    an extra argument to each test assertion:

    is ( $got, $exptected, $test_name, $id );

    You could then create and use a subclass of Test::More to care about the
    extra argument. Anyone just using plain Test::More wouldn't see any
    difference. It wouldn't be particularly taxing when writing tests, and
    would only impact those who felt that it would give them some benefit.

    I'm not sure how you'd deal with not using Test::More (or that use
    Test::More internally), or using things that do stuff like
    all_pod_files_ok(), but those seem like solvable problems.
    I wouldn't envisage unique IDs being mandatory.

    We currently have two proposals / work in progress that I think are
    relevant to this:

    TAP Diagnostic Syntax
    http://testanything.org/wiki/index.php/TAP_diagnostic_syntax

    That gives us somewhere to put a unique test id.

    TAP Blocks
    http://testanything.org/wiki/index.php/Test_Blocks

    Has one of its benefits localising the effects of any renumbering. A
    block is effectively a nested TAP document with its own plan. Any
    changes to the numbering inside the block don't propagate outside the
    block. That'd benefit tests that didn't use a unique ID.

    --
    Andy Armstrong, Hexten
  • Michael Peters at Jul 12, 2007 at 5:50 pm

    Andy Armstrong wrote:
    I've just noticed that the MediaWiki team have a test harness that
    tracks the life of individual assertions over time:

    * HTML nested bullet list, open tags (bug 5497) [Has never passed]
    * HTML nested ordered list, open tags (bug 5497) [Has never passed]
    * Fuzz testing: image with bogus manual thumbnail [Introduced
    between 08-Apr-2007 07:15:22, 1.10alpha (r21099) and 25-Apr-2007
    07:15:46, 1.10alpha
    By the way. Do you have a link to show this or is it just something that prints
    to the console when you run the tests?

    --
    Michael Peters
    Developer
    Plus Three, LP
  • Andy Armstrong at Jul 12, 2007 at 8:33 pm

    Michael Peters wrote:
    By the way. Do you have a link to show this or is it just something that prints
    to the console when you run the tests?
    They had an accident with the mail server that's obviously supposed to
    be relaying the tests and about 12 months worth spewed forth onto the
    list today:

    For example:
    http://lists.wikimedia.org/pipermail/wikitech-l/2007-July/032289.html

    After I'd seen a few hundred of them I decided to read one and was
    really impressed by the test history thing.

    --
    Andy Armstrong, Hexten

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupqa @
categoriesperl
postedJul 12, '07 at 2:17p
activeJul 13, '07 at 5:52a
posts8
users4
websiteqa.perl.org

People

Translate

site design / logo © 2021 Grokbase