Grokbase Groups Perl qa January 2007
FAQ
Pursuant to some discussion with BrowserUK at
http://perlmonks.org/?node_id=593087, I'm looking at this and seeing
some problems.

--

Failed Test Stat Wstat Total Fail List of Failed
-----------------------------------------------------------------
t/bar.t 4 1024 13 4 2 6-8
t/foo.t 1 256 10 1 5
(2 subtests UNEXPECTEDLY SUCCEEDED).
Failed 2/3 test scripts. 5/33 subtests failed.
Files=3, Tests=33, 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11
CPU)
Failed 2/3 test programs. 5/33 subtests failed.

--

How about this instead?

--

Failed Test | Total | Fail | List of Failed | TODO Passed
------------+-------+------+--------------------+----------------
t/bar.t | 13| 4|2, 6-8 |3-4
------------+-------+------+--------------------+----------------
t/foo.t | 10| 1|5 |

Time: 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 CPU)
Files=3. Failed 2/3 test programs. 5/33 subtests failed.

--

That will be annoying to put together, but it really seems a lot
cleaner, with no duplicate info (yes, I've already thought about the
"List of Failed" and "TODO Passed" spanning more than one line.)

Cheers,
Ovid

--

Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/

Search Discussions

  • Nicholas Clark at Jan 5, 2007 at 7:28 pm

    On Fri, Jan 05, 2007 at 11:11:25AM -0800, Ovid wrote:
    Pursuant to some discussion with BrowserUK at
    http://perlmonks.org/?node_id=593087, I'm looking at this and seeing
    some problems.

    --

    Failed Test Stat Wstat Total Fail List of Failed
    -----------------------------------------------------------------
    t/bar.t 4 1024 13 4 2 6-8
    t/foo.t 1 256 10 1 5
    (2 subtests UNEXPECTEDLY SUCCEEDED).
    Failed 2/3 test scripts. 5/33 subtests failed.
    Files=3, Tests=33, 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11
    CPU)
    Failed 2/3 test programs. 5/33 subtests failed.

    --

    How about this instead?

    --

    Failed Test | Total | Fail | List of Failed | TODO Passed
    ------------+-------+------+--------------------+----------------
    t/bar.t | 13| 4|2, 6-8 |3-4
    ------------+-------+------+--------------------+----------------
    t/foo.t | 10| 1|5 |
    I'd like Wstat, even if I don't have Stat. I like to know if tests coredumped.
    I may be in a minority here, but being able to optionally switch to that
    output is useful.

    I like the prominence of TODO passed

    I'm not sure if I like the lines making the table. I guess it's a bit of a
    bikeshed, but having horizontal lines between each will increase the amount
    of vertical space needed to convey the same information, which will mean
    fewer failures will be needed to exceed my screen's height.

    Nicholas Clark
  • Andy Lester at Jan 5, 2007 at 7:38 pm

    On Jan 5, 2007, at 1:28 PM, Nicholas Clark wrote:

    Failed Test | Total | Fail | List of Failed | TODO Passed
    ------------+-------+------+--------------------+----------------
    t/bar.t | 13| 4|2, 6-8 |3-4
    ------------+-------+------+--------------------+----------------
    t/foo.t | 10| 1|5 |
    The vertical lines are just noise. What Tufte calls "chartjunk".

    --
    Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
  • David Golden at Jan 5, 2007 at 7:44 pm

    On 1/5/07, Andy Lester wrote:
    On Jan 5, 2007, at 1:28 PM, Nicholas Clark wrote:

    Failed Test | Total | Fail | List of Failed | TODO Passed
    ------------+-------+------+--------------------+----------------
    t/bar.t | 13| 4|2, 6-8 |3-4
    ------------+-------+------+--------------------+----------------
    t/foo.t | 10| 1|5 |
    The vertical lines are just noise. What Tufte calls "chartjunk".
    Moreover, it looks really horrid with non-monospaced fonts.

    David
  • Ovid at Jan 5, 2007 at 7:49 pm

    --- David Golden wrote:
    On 1/5/07, Andy Lester wrote:
    On Jan 5, 2007, at 1:28 PM, Nicholas Clark wrote:

    Failed Test | Total | Fail | List of Failed | TODO Passed
    ------------+-------+------+--------------------+----------------
    t/bar.t | 13| 4|2, 6-8 |3-4
    ------------+-------+------+--------------------+----------------
    t/foo.t | 10| 1|5 |
    The vertical lines are just noise. What Tufte calls "chartjunk".
    Moreover, it looks really horrid with non-monospaced fonts.
    You use non-monospaced fonts in your terminal? :)

    Cheers,
    Ovid
  • Andy Lester at Jan 5, 2007 at 7:51 pm

    On Jan 5, 2007, at 1:49 PM, Ovid wrote:

    Moreover, it looks really horrid with non-monospaced fonts.
    You use non-monospaced fonts in your terminal? :)
    Results might get emailed to someone, or put into a web page that
    doesn't do monospaced fonts. SourceForge's forums are an excellent
    example of how to make things look like shit.

    --
    Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
  • David Golden at Jan 5, 2007 at 9:27 pm

    On 1/5/07, Ovid wrote:
    Moreover, it looks really horrid with non-monospaced fonts.
    You use non-monospaced fonts in your terminal? :)
    Thta's gmail for you.

    David
  • Ovid at Jan 5, 2007 at 7:46 pm

    --- Nicholas Clark wrote:

    I'd like Wstat, even if I don't have Stat. I like to know if tests
    coredumped.
    I may be in a minority here, but being able to optionally switch to
    that output is useful.
    Fair enough.
    I like the prominence of TODO passed
    Cool. I hated wedging it in there like that, but I couldn't think of
    another nice format.
    I'm not sure if I like the lines making the table. I guess it's a bit
    of a
    bikeshed, but having horizontal lines between each will increase the
    amount
    of vertical space needed to convey the same information, which will
    mean
    fewer failures will be needed to exceed my screen's height.
    So with optional Wstat, you're thinking something like this?

    Failed Test | Wstat | Total | Fail | List of Failed | TODO Passed
    ------------+-------+-------+------+--------------------+----------------
    t/bar.t | 1024| 13| 4|2, 6-8 |3-4
    t/foo.t | 256| 10| 1|5 |

    And Andy Lester wrote:
    The vertical lines are just noise. What Tufte calls "chartjunk".
    That's what I thought at first, too, but looks at this:

    Failed Test Total Fail List of Failed TODO Passed
    ------------+-------+------+---------------------+----------------
    t/bar.t 13 9 2, 6-8, 13, 17, 33-35 3-4
    t/foo.t 10 10 5, 19, 27, 37-38, 117 9-11

    In trying to pack in the 'TODO passed' information, I'm running out of
    room. That's becoming annoying to read (It would be even worse without
    those plus signs:

    Failed Test Total Fail List of Failed TODO Passed
    ------------------------------------------------------------------
    t/bar.t 13 9 2, 6-8, 13, 17, 33-35 3-4
    t/foo.t 10 10 5, 19, 27, 37-38, 117 9-11

    Did you mean that you didn't like the 'horizontal' lines?
  • Andy Lester at Jan 5, 2007 at 7:51 pm

    On Jan 5, 2007, at 1:46 PM, Ovid wrote:

    Failed Test Total Fail List of Failed TODO Passed
    ------------------------------------------------------------------
    t/bar.t 13 9 2, 6-8, 13, 17, 33-35 3-4
    t/foo.t 10 10 5, 19, 27, 37-38, 117 9-11

    Did you mean that you didn't like the 'horizontal' lines?
    Both.

    How about

    t/bar.t 9/13
    Failed: 2, 6-8, 13, 17, 33-35
    TODO passed: 3-4
    t/foo.t 10/10
    Failed: 5, 19, 27, 37-38, 117 9-11

    That should degrade nicely in non-monospaced.

    --
    Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
  • Paul Johnson at Jan 6, 2007 at 2:29 am

    On Fri, Jan 05, 2007 at 01:50:54PM -0600, Andy Lester wrote:
    On Jan 5, 2007, at 1:46 PM, Ovid wrote:

    Failed Test Total Fail List of Failed TODO Passed
    ------------------------------------------------------------------
    t/bar.t 13 9 2, 6-8, 13, 17, 33-35 3-4
    t/foo.t 10 10 5, 19, 27, 37-38, 117 9-11

    Did you mean that you didn't like the 'horizontal' lines?
    Both.

    How about

    t/bar.t 9/13
    Failed: 2, 6-8, 13, 17, 33-35
    TODO passed: 3-4
    t/foo.t 10/10
    Failed: 5, 19, 27, 37-38, 117 9-11
    FWIW, I like the tabular output. How about something like:

    Test Total Failed List of Failed TODO Passed
    -------+-------+------+----------------------+------------
    t/bar.t 13 9 2, 6-8, 13, 17, 33-35 3-4
    t/foo.t 10 10 5, 19, 27, 37-38, 117 9-11

    Possibly including wstat, just for Nicholas ;-)

    You'd have to be careful if there is a very large number of failed
    subtests (I can never remember what we decided those would be called)
    but, thinking about it, if more than a few of them have failed, I really
    don't care to see them all anyway. Just the first handful (perhaps the
    number that would fit in the column) is enough. If I really need to see
    them all I can get at that information some other way. The same applies
    for passed TODOs.
    That should degrade nicely in non-monospaced.
    And to be honest, I really don't see that as a big problem. How often
    do people need to look at results like this where they cannot control
    the font? If the worst comes to the worst, you can always copy and
    paste them into an editor.

    Just lobbing a few peanuts ...

    And if you squint at the table hard enough it starts to resemble a
    bikeshed. When you've all decided on something I didn't want, I'll
    spend ages creating my own output format and making it just the right
    shade of lilac. And rounding the corners.

    Actually, I won't, 'cos my tests never fail :-P

    --
    Paul Johnson - paul@pjcj.net
    http://www.pjcj.net
  • Geoffrey Young at Jan 10, 2007 at 4:34 pm
    sorry to chime in late...
    FWIW, I like the tabular output. How about something like:

    Test Total Failed List of Failed TODO Passed
    -------+-------+------+----------------------+------------
    t/bar.t 13 9 2, 6-8, 13, 17, 33-35 3-4
    t/foo.t 10 10 5, 19, 27, 37-38, 117 9-11
    something that consistently gets on my nerves is something like this

    Failed Test Stat Wstat Total Fail Failed List of
    Failed
    -------------------------------------------------------------------------------
    t/Search/Script/Model/Collection/ 5 1280 6 9 150.00% 2-6
    t/Search/Script/Model/Collection/ 7 1792 8 13 162.50% 2-8
    t/Search/Script/Model/Collection/ 5 1280 6 9 150.00% 2-6
    t/Search/Script/Model/Collection/ 7 1792 8 13 162.50% 2-8

    for tests with really long names, or where the test path is long, tests
    are truncated in such a way that the actual test name is hidden.
    annoying but not a biggie when running a single test, but when a slurry
    of tests from a large nightly run generate 10,000 lines of borked output
    and looking at the summary hides the actual failed test names I get a
    bit worked up.

    so, it would be nice to make sure the test name is recognizable. I
    guess the options here are no truncation at all or (for those who think
    lines over 72 characters are evil) at least starting from the right-hand
    side of the test name.

    --Geoff
  • Ovid at Jan 10, 2007 at 4:48 pm

    --- Geoffrey Young wrote:

    so, it would be nice to make sure the test name is recognizable. I
    guess the options here are no truncation at all or (for those who
    think lines over 72 characters are evil) at least starting from the
    right-hand side of the test name.
    I deliberately made it easy to subclass TAPx::Harness. While it's not
    perfect (lots more work to do), all you need to do is override the
    'summary' method to generate any sort of test summary you like. Then
    you can just use it:

    runtests --harness My::TAPx::Harness

    Cheers,
    Ovid
  • Aristotle Pagaltzis at Jan 6, 2007 at 1:41 pm

    * Ovid [2007-01-05 20:15]:
    How about this instead?

    --

    Failed Test | Total | Fail | List of Failed | TODO Passed
    ------------+-------+------+--------------------+----------------
    t/bar.t | 13| 4|2, 6-8 |3-4
    ------------+-------+------+--------------------+----------------
    t/foo.t | 10| 1|5 |

    Time: 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 CPU)
    Files=3. Failed 2/3 test programs. 5/33 subtests failed.
    Better, but it has flaws.

    • The horizontal lines are too noisy.

    • The column labels are too wide and conversely the lack of cell
    padding makes it unnecessarily uneven in blackness.

    • The “Files=3” bit, while a minor issue, is just ugly.

    • The failed tests and passing TODO tests columns can be
    empty, in which case they’re a huge waste of space.

    I think the following format reconciles the various points best
    (note: this is the same data as in your example, except that it
    assumes 4 test files in total):

    Test file | Failed | Bad tests
    ----------+---+----+-----------------------------------------
    t/bar.t : 4 / 13 : Fail: 2, 6-8; Pass TODO: 3-4
    t/foo.t : 1 / 10 : Fail: 5
    ... (2) : / 10
    Total (4) : 5 / 33 : Non-passing files: 2

    Time: 0 wallclock secs (0.10 cusr + 0.01 csys = 0.11 CPU)

    Note that the “Failed” header spans two columns.

    The “... (#)” ellipsis line for passed test files is there to
    supply the count of passed tests so the totals line is more
    logical. It has the side benefit of calling out explicitly how
    many files are omitted from the output; I like that. For polish,
    if the count of ellipsised test files is exactly 1, you could
    just show the one filename instead of ellipsising it.

    This format is much denser but also more evenly spaced than
    yours. There’s more noise than in Andy’s proposition simply
    because it’s a table but the non-variable bits are also more
    easily scannable. (In a longer example this would *really* show.)
    And though I picked meaningful characters as column separators to
    visually lighten the “chart junk,” it now occurs to me that this
    also means the format is minimally legible when rendered in a
    proportional font. It is also seems easier to implement to me
    than your format because there’s only a single column with really
    variable content width and it’s the trailing right column.

    Regards,
    --
    Aristotle Pagaltzis // <http://plasmasturm.org/>
  • Demerphq at Jan 6, 2007 at 6:05 pm

    On 1/6/07, A. Pagaltzis wrote:
    * Ovid [2007-01-05 20:15]:
    How about this instead?

    --

    Failed Test | Total | Fail | List of Failed | TODO Passed
    ------------+-------+------+--------------------+----------------
    t/bar.t | 13| 4|2, 6-8 |3-4
    ------------+-------+------+--------------------+----------------
    t/foo.t | 10| 1|5 |

    Time: 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 CPU)
    Files=3. Failed 2/3 test programs. 5/33 subtests failed.
    Better, but it has flaws.

    • The horizontal lines are too noisy.

    • The column labels are too wide and conversely the lack of cell
    padding makes it unnecessarily uneven in blackness.

    • The "Files=3" bit, while a minor issue, is just ugly.

    • The failed tests and passing TODO tests columns can be
    empty, in which case they're a huge waste of space.

    I think the following format reconciles the various points best
    (note: this is the same data as in your example, except that it
    assumes 4 test files in total):

    Test file | Failed | Bad tests
    ----------+---+----+-----------------------------------------
    t/bar.t : 4 / 13 : Fail: 2, 6-8; Pass TODO: 3-4
    t/foo.t : 1 / 10 : Fail: 5
    ... (2) : / 10
    Total (4) : 5 / 33 : Non-passing files: 2

    Time: 0 wallclock secs (0.10 cusr + 0.01 csys = 0.11 CPU)

    Note that the "Failed" header spans two columns.

    The "... (#)" ellipsis line for passed test files is there to
    supply the count of passed tests so the totals line is more
    logical. It has the side benefit of calling out explicitly how
    many files are omitted from the output; I like that. For polish,
    if the count of ellipsised test files is exactly 1, you could
    just show the one filename instead of ellipsising it.

    This format is much denser but also more evenly spaced than
    yours. There's more noise than in Andy's proposition simply
    because it's a table but the non-variable bits are also more
    easily scannable. (In a longer example this would *really* show.)
    And though I picked meaningful characters as column separators to
    visually lighten the "chart junk," it now occurs to me that this
    also means the format is minimally legible when rendered in a
    proportional font. It is also seems easier to implement to me
    than your format because there's only a single column with really
    variable content width and it's the trailing right column.
    FWIW, of the variants ive seen posted so far i'd vote for this one

    Yves


    --
    perl -Mre=debug -e "/just|another|perl|hacker/"

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupqa @
categoriesperl
postedJan 5, '07 at 7:11p
activeJan 10, '07 at 4:48p
posts14
users8
websiteqa.perl.org

People

Translate

site design / logo © 2021 Grokbase