Slaven Rezic has figured out that some test reports are exceeding the
400K maximum size that develooper.com will accept. It seems
reasonable to truncate the test output included. (Maybe long term
gzip and attach instead.)

400K seems like a lot. Ask has suggested it be smaller, but hasn't
specified a size.

What do you all think would be a reasonable maximum? 100K? 200K?

David

Search Discussions

  • David Cantrell at Oct 4, 2007 at 8:06 pm

    On Thu, Oct 04, 2007 at 02:09:01PM -0400, David Golden wrote:

    Slaven Rezic has figured out that some test reports are exceeding the
    400K maximum size that develooper.com will accept. It seems
    reasonable to truncate the test output included. (Maybe long term
    gzip and attach instead.)
    I've noticed it too, but not bothered mentioning it because I thought
    everyone knew :-)
    400K seems like a lot. Ask has suggested it be smaller, but hasn't
    specified a size.

    What do you all think would be a reasonable maximum? 100K? 200K?
    From a quick look over the reports I've sent this month, 40K looks like
    a good cutoff point. Add the introductory text, perl -V, and any
    comments I might add by hand, and it'll still be well under 50K.

    Can I also suggest - probably for the future as it would require more
    work - some way of automatically killing off a test that has done
    nothing but spew megabyte after megabyte after megabyte of errors and
    warnings to the console, and just make it a FAIL? Spewing a gazillion
    warnings is generally considered to be a Bad Thing even if they're
    harmless and the code actually works.

    --
    David Cantrell | http://www.cantrell.org.uk/david

    Irregular English:
    you have anecdotes; they have data; I have proof
  • David Landgren at Oct 5, 2007 at 12:13 pm

    David Cantrell wrote:
    On Thu, Oct 04, 2007 at 02:09:01PM -0400, David Golden wrote:
    From a quick look over the reports I've sent this month, 40K looks like
    a good cutoff point. Add the introductory text, perl -V, and any
    comments I might add by hand, and it'll still be well under 50K. Agreed.
    Can I also suggest - probably for the future as it would require more
    work - some way of automatically killing off a test that has done
    nothing but spew megabyte after megabyte after megabyte of errors and
    warnings to the console, and just make it a FAIL? Spewing a gazillion
    warnings is generally considered to be a Bad Thing even if they're
    harmless and the code actually works.
    Ditto. (Sorry to bet <aol> here, but there's not a lot more to be said
    than JFDI).
  • David Golden at Oct 5, 2007 at 12:46 pm

    On 10/5/07, David Landgren wrote:
    David Cantrell wrote:
    From a quick look over the reports I've sent this month, 40K looks like
    a good cutoff point. Add the introductory text, perl -V, and any
    comments I might add by hand, and it'll still be well under 50K.
    Agreed.
    I've set it at 100K in the devel version in the repo. (Engineering
    training instinctively says 'safety factor of 2'!)

    But I could be argued down. What's the basis for 40K? Is that a max?
    99th percentile? Twice max already?
    Can I also suggest - probably for the future as it would require more
    work - some way of automatically killing off a test that has done
    nothing but spew megabyte after megabyte after megabyte of errors and
    warnings to the console, and just make it a FAIL? Spewing a gazillion
    warnings is generally considered to be a Bad Thing even if they're
    harmless and the code actually works.
    Ditto. (Sorry to bet <aol> here, but there's not a lot more to be said
    than JFDI).
    That's actually really hard -- the output stream isn't being read
    interactively by a process. It's all batch, executed in a separate
    command line process and picked up from teed output later.

    I might be able to build that kind of a kill as a parameter to Tee --
    if the tee file exceeds a certain size, then kill the process. But
    even that might require some real work to be portable. Forks and
    timeouts are not the nicest stuff to work with on Win32.

    And arguably, CPAN Testers isn't supposed to be judging cleanliness of
    tests. Just whether tests work. If it's a test that's spewing
    "uninitialized variable" warnings, but the code works, then it should
    PASS, not FAIL. At most, the test could be aborted and the report
    discarded.

    David
  • Barbie at Oct 5, 2007 at 5:41 pm

    On Fri, Oct 05, 2007 at 08:46:13AM -0400, David Golden wrote:
    On 10/5/07, David Landgren wrote:
    David Cantrell wrote:
    From a quick look over the reports I've sent this month, 40K looks like
    a good cutoff point. Add the introductory text, perl -V, and any
    comments I might add by hand, and it'll still be well under 50K.
    Agreed.
    I've set it at 100K in the devel version in the repo. (Engineering
    training instinctively says 'safety factor of 2'!)

    But I could be argued down. What's the basis for 40K? Is that a max?
    99th percentile? Twice max already?
    It might be interesting to run a script that can do a check on that sort
    of thing. I should be able to a canabalised script from the current
    daily cpanstats script. If I get time over the weekend, I'll try and see
    what stats I get for so far this year.
    Can I also suggest - probably for the future as it would require more
    work - some way of automatically killing off a test that has done
    nothing but spew megabyte after megabyte after megabyte of errors and
    warnings to the console, and just make it a FAIL? Spewing a gazillion
    warnings is generally considered to be a Bad Thing even if they're
    harmless and the code actually works.
    That's actually really hard -- the output stream isn't being read
    interactively by a process. It's all batch, executed in a separate
    command line process and picked up from teed output later.

    I might be able to build that kind of a kill as a parameter to Tee --
    if the tee file exceeds a certain size, then kill the process. But
    even that might require some real work to be portable. Forks and
    timeouts are not the nicest stuff to work with on Win32.
    Don't go there, there be dragons!
    And arguably, CPAN Testers isn't supposed to be judging cleanliness of
    tests. Just whether tests work. If it's a test that's spewing
    "uninitialized variable" warnings, but the code works, then it should
    PASS, not FAIL. At most, the test could be aborted and the report
    discarded.
    This was another suggested report type I had several years ago. WARNING.
    Although the distribution passes, a report is produced and sent to the
    author, as per a FAIL report. It would still count as a PASS, but the
    author would get the benefit of being alerted to any warnings that they
    may not have been aware of.

    Unfortunately this would require a change to CPANPLUS (and probably
    CPAN/CPAN-Reporter) to capture the output rather than just create a
    simple PASS report.

    When I was testing I did get a few distributions that resulted in plenty
    of output, mostly from annoying distributions that insisted that you
    enter an value interactively, and thus got you into an endless loop
    waiting for me to come in in the morning to find I'd only tested a
    couple of distributions instead of the few hundred I'd expected! (sorry
    it's been a long day :)). But those are the more annoying, I don't
    expect you're going to get a decent cross-platform way of spotting those
    or stopping them. It was bad enough doing them manually on Win32 :(

    Barbie.
    --
    Birmingham Perl Mongers - http://birmingham.pm.org
    Miss Barbell Productions - http://www.missbarbell.co.uk
  • David Golden at Oct 5, 2007 at 7:05 pm

    On 10/5/07, Barbie wrote:
    Unfortunately this would require a change to CPANPLUS (and probably
    CPAN/CPAN-Reporter) to capture the output rather than just create a
    simple PASS report.
    It's worse -- you need to capture STDERR while trying to maintain the
    order of STDOUT and STDERR.
    it's been a long day :)). But those are the more annoying, I don't
    expect you're going to get a decent cross-platform way of spotting those
    or stopping them. It was bad enough doing them manually on Win32 :(
    CPAN.pm (and CPAN::Reporter) can timeout a PL file already.
    (interactivity_timeout) It would be fairly trivial to add
    "test_timeout" that times out the test process after a period of time.
    That would take care of those interactive tests.

    David
  • David Cantrell at Oct 6, 2007 at 7:17 pm

    Barbie wrote:
    On Fri, Oct 05, 2007 at 08:46:13AM -0400, David Golden wrote:
    I might be able to build that kind of a kill as a parameter to Tee --
    if the tee file exceeds a certain size, then kill the process. But
    even that might require some real work to be portable. Forks and
    timeouts are not the nicest stuff to work with on Win32.
    Don't go there, there be dragons!
    It wouldn't be the first time a module had had OS-specific features. In
    this case, "if you're on Unix you can ...".

    For extra super special fun, make it portable to VMS :-)
    This was another suggested report type I had several years ago. WARNING.
    Although the distribution passes, a report is produced and sent to the
    author, as per a FAIL report. It would still count as a PASS, but the
    author would get the benefit of being alerted to any warnings that they
    may not have been aware of.
    I like this a lot. We'd need to do a 'dry run' with WARNINGs *not*
    going to authors first though, to see how many false positives it
    creates. And yes, I'm sure it would need CPAN.pm changes *if you want
    to detect warnings*, but I suggest that anything sent to STDERR (which
    includes warnings) is Bad and should result in a WARNING email, because
    the ERR in STDERR means *error*. Using it correctly indicates that an
    error occured and the author should be told. If an author uses STDERR
    for anything that's not an error, that is itself an error, and the
    author should be told. That should simplify matters a little!
    When I was testing I did get a few distributions that resulted in plenty
    of output, mostly from annoying distributions that insisted that you
    enter an value interactively
    on which subject, tests which use Curses must die.

    --
    David Cantrell | Nth greatest programmer in the world

    Today's previously unreported paraphilia is tomorrow's Internet sensation
  • David Golden at Oct 6, 2007 at 8:59 pm

    On 10/6/07, David Cantrell wrote:
    It wouldn't be the first time a module had had OS-specific features. In
    this case, "if you're on Unix you can ...".
    CPAN::Reporter already uses Win32::Process when it needs to honor CPAN
    interactivity_timeout requests. (Which I do plan to extend to test
    timeouts, too.)
    to detect warnings*, but I suggest that anything sent to STDERR (which
    includes warnings) is Bad and should result in a WARNING email, because
    No, no, no, no!

    (a) to keep diagnostic output with tests in as close to the failing
    test, we want to merge STDOUT and STDERR for reports

    (b) plenty of modules test warnings by sending stuff to STDERR.
    (Probably shouldn't but they do.)

    David
  • David Cantrell at Oct 6, 2007 at 10:09 pm

    On Sat, Oct 06, 2007 at 04:59:21PM -0400, David Golden wrote:

    (b) plenty of modules test warnings by sending stuff to STDERR.
    (Probably shouldn't but they do.)
    They not only shouldn't, but simply printing to STDERR to test something
    to do with warnings is a bug. To test a warning, you need to warn() and
    capture it with $SIG{__WARN__}.

    --
    David Cantrell | A machine for turning tea into grumpiness

    I caught myself pulling grey hairs out of my beard.
    I'm definitely not going grey, but I am going vain.
  • David Golden at Oct 7, 2007 at 11:40 am

    On 10/6/07, David Cantrell wrote:
    On Sat, Oct 06, 2007 at 04:59:21PM -0400, David Golden wrote:
    (b) plenty of modules test warnings by sending stuff to STDERR.
    (Probably shouldn't but they do.)
    They not only shouldn't, but simply printing to STDERR to test something
    to do with warnings is a bug. To test a warning, you need to warn() and
    capture it with $SIG{__WARN__}.
    I phrased that poorly. Plenty of modules test behaviors that issue
    warnings or otherwise print to STDERR without capturing STDERR before
    it gets into the output stream of Test::Harness.

    This is just the nature of trying to work with TAP in-band with all
    other console output. Thus, the place for a new "grade" like this to
    be determined is with the TAPX/Test-Harness 3.0 crowd. E.g. a harness
    switch that detects warnings and causes a "RESULT:PASS/warnings
    detected" result that CPAN::Reporter could parse. I just don't think
    that the CPAN Testers stack should be adding any more heuristic test
    distinctions that aren't reflected in the underlying test framework.

    In my opinion, we're just trying to automate the capture of the output
    and results of "make test" -- the same way an end user would see it.
    I wouldn't want to introduce a bunch of process and IO handle
    manipulations that weaken or break that connection.

    David
  • David Cantrell at Oct 6, 2007 at 6:26 pm

    David Golden wrote:
    On 10/5/07, David Landgren wrote:
    David Cantrell wrote:
    From a quick look over the reports I've sent this month, 40K looks like
    a good cutoff point. Add the introductory text, perl -V, and any
    comments I might add by hand, and it'll still be well under 50K.
    Agreed.
    I've set it at 100K in the devel version in the repo. (Engineering
    training instinctively says 'safety factor of 2'!)

    But I could be argued down. What's the basis for 40K? Is that a max?
    99th percentile? Twice max already?
    Every single report that I sent since the beginning of last month that
    was over 40K included verbose errors (including warnings) of some kind
    or another, even the PASSes. I didn't look particularly hard at test
    reports below 40K.

    --
    David Cantrell | Godless Liberal Elitist

    There is no one true indentation style,
    But if there were K&R would be Its Prophets.
    Peace be upon Their Holy Beards.
  • David Golden at Oct 7, 2007 at 1:37 pm

    On 10/6/07, David Cantrell wrote:
    Every single report that I sent since the beginning of last month that
    was over 40K included verbose errors (including warnings) of some kind
    or another, even the PASSes. I didn't look particularly hard at test
    reports below 40K.
    I whipped up a script to mine the newsgroup. (Thank you,
    News::NNTPClient!) In hindsight, I should have cached the sizes on
    the articles scanned so it doesn't have to spend the time to do that
    again if I want to go further back in history.

    $ perl check-testers.pl 10000
    Found 487283 articles
    Based on the last 9753 reports:
    25-th percentile: 6 K
    50-th percentile: 6 K
    75-th percentile: 7 K
    90-th percentile: 10 K
    99-th percentile: 53 K

    The sizes are the size of the message, not just the test output. It
    looks like if I set the test output cutoff at 50K, we'll truncate < 1%
    of messages. That seems quite reasonable.

    David

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcpan-testers-discuss @
categoriesperl
postedOct 4, '07 at 6:09p
activeOct 7, '07 at 1:37p
posts12
users4
websitecpan.org

People

Translate

site design / logo © 2021 Grokbase