FAQ
Test::Builder appends all Ok's to a structure it can return via
$tb->details. These Ok's have the fields 'ok' and 'actual_ok'. 'actual_ok'
is the true/false value passed to $tb->ok, 'ok' is adjusted to be true if
the test was run under TODO.

https://metacpan.org/source/EXODIST/Test-Simple-1.001014/lib/Test/Builder.pm#L845

The bug is in subtest handling. When you start a subtest you essentially
create a new Test::Builder singleton, and the TODO state is clear int he
new one, but it has a field 'parent_TODO' that gets set. This is used to
ensure that failures in a subtest when the parent has TODO set do not count
as actual failures.

The problem is that parent_todo is not considered when setting the OK's in
the details in subtests. That means if a subtest ok fails, and the parent
is in todo, the 'ok' and 'actual_ok' are both set to false.

The fix is easy, just change line 845 to look at parent_TODO. However this
breaks Test::ClassMoose because it uses details to verify its results.
https://metacpan.org/source/DROLSKY/Test-Class-Moose-0.67/t/basic.t#L55

I spoke with autarch about this test since he is the current maintainer. He
agrees with me that 'ok' should be true in that test, and that this is
probably a bug.

I have not yet figured out what else this may break on cpan, I will try to
do that tomorrow.

*What I am looking for from cpan-workers:*

1) Do you think this is actually a bug, or does it work as you think it
should?
2) Even if we all agree it is a bug, can we fix it, or do we need to
preserve it to avoid breaking things?

-Chad

Search Discussions

  • Karen Etheridge at Dec 22, 2015 at 1:11 am
    The problem is that parent_todo is not considered when setting the OK's
    in the details in subtests. That means if a subtest ok fails, and the
    parent is in todo, the 'ok' and 'actual_ok' are both set to false.

    Why is this a bug? After all, it's only the top level of TAP that ever
    matters when it comes to the overall test pass/failure, and a subtest's
    pass or fail status will be adjusted accordingly for the parent TODO at the
    *parent's* level. TODOs don't need to propagate downward to all contained
    subtests.

    Further, if we required that TODO state must propagate down to all
    contained subtests, that means that subtests must always be aware of that
    upper state -- which rules out any use of separate processes to to create
    the subtest results, which is conceptually possible (something else,
    thinking that it is a top-level test, can provide us the TAP, and then we
    nest it as a subtest into another test -- and this should be possible both
    with TAP strings and with an event streamer).

    On Mon, Dec 21, 2015 at 4:04 PM, Chad Granum wrote:

    Test::Builder appends all Ok's to a structure it can return via
    $tb->details. These Ok's have the fields 'ok' and 'actual_ok'. 'actual_ok'
    is the true/false value passed to $tb->ok, 'ok' is adjusted to be true if
    the test was run under TODO.


    https://metacpan.org/source/EXODIST/Test-Simple-1.001014/lib/Test/Builder.pm#L845

    The bug is in subtest handling. When you start a subtest you essentially
    create a new Test::Builder singleton, and the TODO state is clear int he
    new one, but it has a field 'parent_TODO' that gets set. This is used to
    ensure that failures in a subtest when the parent has TODO set do not count
    as actual failures.

    The problem is that parent_todo is not considered when setting the OK's in
    the details in subtests. That means if a subtest ok fails, and the parent
    is in todo, the 'ok' and 'actual_ok' are both set to false.

    The fix is easy, just change line 845 to look at parent_TODO. However this
    breaks Test::ClassMoose because it uses details to verify its results.
    https://metacpan.org/source/DROLSKY/Test-Class-Moose-0.67/t/basic.t#L55

    I spoke with autarch about this test since he is the current maintainer.
    He agrees with me that 'ok' should be true in that test, and that this is
    probably a bug.

    I have not yet figured out what else this may break on cpan, I will try to
    do that tomorrow.

    *What I am looking for from cpan-workers:*

    1) Do you think this is actually a bug, or does it work as you think it
    should?
    2) Even if we all agree it is a bug, can we fix it, or do we need to
    preserve it to avoid breaking things?

    -Chad



  • Chad Granum at Dec 22, 2015 at 4:59 am

    On Mon, Dec 21, 2015 at 5:10 PM, Karen Etheridge wrote:

    The problem is that parent_todo is not considered when setting the OK's
    in the details in subtests. That means if a subtest ok fails, and the
    parent is in todo, the 'ok' and 'actual_ok' are both set to false.

    Why is this a bug? After all, it's only the top level of TAP that ever
    matters when it comes to the overall test pass/failure, and a subtest's
    pass or fail status will be adjusted accordingly for the parent TODO at the
    *parent's* level. TODOs don't need to propagate downward to all contained
    subtests.
    This bug has nothing to do with TAP. '->details' is used to get a record of
    all tests that have passed through the Test::Builder instance. These
    details should accurately reflect if the test was a pass or fail, they do
    not. By not using parent_TODO here Test::Builder is at the very least being
    inconsistent with itself. When a failure occurs in a subtest where the
    parent is todo Test::Builder will send the diagnostics and errors to STDOUT
    instead of STDERR like it does with any TODO failure. This is the __ONLY__
    place where Test::Builder does not account for the parent_TODO on a
    subtest, and in that way it is being inconsistent if not buggy.

    Further, if we required that TODO state must propagate down to all
    contained subtests, that means that subtests must always be aware of that
    upper state -- which rules out any use of separate processes to to create
    the subtest results, which is conceptually possible (something else,
    thinking that it is a top-level test, can provide us the TAP, and then we
    nest it as a subtest into another test -- and this should be possible both
    with TAP strings and with an event streamer).
    I can't control TAP from other sources, but as I said, this bug has nothing
    to do with TAP. This has to do with Test::Builders internal understanding
    of tests __it__ produced. Outside TAP never has, nor will it ever effect
    this, they do not appear in the details structure.


    -Chad
  • Chad Granum at Dec 22, 2015 at 5:07 am
    Created https://github.com/Test-More/test-more/issues/616 to track this
    where more people can see it.
    On Mon, Dec 21, 2015 at 8:58 PM, Chad Granum wrote:


    On Mon, Dec 21, 2015 at 5:10 PM, Karen Etheridge wrote:

    The problem is that parent_todo is not considered when setting the OK's
    in the details in subtests. That means if a subtest ok fails, and the
    parent is in todo, the 'ok' and 'actual_ok' are both set to false.

    Why is this a bug? After all, it's only the top level of TAP that ever
    matters when it comes to the overall test pass/failure, and a subtest's
    pass or fail status will be adjusted accordingly for the parent TODO at the
    *parent's* level. TODOs don't need to propagate downward to all contained
    subtests.
    This bug has nothing to do with TAP. '->details' is used to get a record
    of all tests that have passed through the Test::Builder instance. These
    details should accurately reflect if the test was a pass or fail, they do
    not. By not using parent_TODO here Test::Builder is at the very least being
    inconsistent with itself. When a failure occurs in a subtest where the
    parent is todo Test::Builder will send the diagnostics and errors to STDOUT
    instead of STDERR like it does with any TODO failure. This is the __ONLY__
    place where Test::Builder does not account for the parent_TODO on a
    subtest, and in that way it is being inconsistent if not buggy.

    Further, if we required that TODO state must propagate down to all
    contained subtests, that means that subtests must always be aware of that
    upper state -- which rules out any use of separate processes to to create
    the subtest results, which is conceptually possible (something else,
    thinking that it is a top-level test, can provide us the TAP, and then we
    nest it as a subtest into another test -- and this should be possible both
    with TAP strings and with an event streamer).
    I can't control TAP from other sources, but as I said, this bug has
    nothing to do with TAP. This has to do with Test::Builders internal
    understanding of tests __it__ produced. Outside TAP never has, nor will it
    ever effect this, they do not appear in the details structure.


    -Chad

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcpan-workers @
categoriesperl
postedDec 22, '15 at 12:04a
activeDec 22, '15 at 5:07a
posts4
users2
websitecpan.org

2 users in discussion

Chad Granum: 3 posts Karen Etheridge: 1 post

People

Translate

site design / logo © 2018 Grokbase