Grokbase Groups Perl qa July 2013
FAQ
I just submitted:

   https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/16

...to document two undocumented changes. First, the removal of "turn on -w by
default with no easy way to turn it off" and secondly the reversion of that.

These commits are, respectively:

   https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/18794a85

   https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/99159fc2

The first has a bit of discussion attached. I will only VERY briefly summarize
it as:

   schwern : We need to discuss this, and please see
              https://rt.cpan.org/Ticket/Display.html?id=58229
   ether : We should keep the original change.
   mst : We should keep the original change.
   miyagawa : We should keep the original change.
   leont : We should keep the original change.
   ewilhelm : We should keep the original change, but update EUMM to restore -w
   ribasushi: I don't see a reason to change the old behavior.

I will also add that I was in favor of the change, and presumably was Andy A.,
who made the commit in the first place.

I am all for getting this discussed publicly and brought to a consensus-based
conclusion.

--
rjbs

Search Discussions

  • Leon Timmermans at Jul 4, 2013 at 6:04 pm

    On Tue, Jul 2, 2013 at 8:51 PM, Ricardo Signes wrote:
    I just submitted:

    https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/16

    ...to document two undocumented changes. First, the removal of "turn on -w by
    default with no easy way to turn it off" and secondly the reversion of that.

    These commits are, respectively:

    https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/18794a85

    https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/99159fc2
    The change would have probably been better discussed in a less
    ephemeral medium than IRC, but people reverting it without a proper
    discussion is really only making that worse.

    We are doing a *terrible* job of communicating with each other here,
    and this thread getting warnocked is not making that better.
    ewilhelm : We should keep the original change, but update EUMM to restore -w
    I'm not sure that's what he meant.
    I am all for getting this discussed publicly and brought to a consensus-based
    conclusion.
    By what process? Define consensus? Given Andy is the official
    maintainer and Ovid is the effective maintainer, I don't think they
    need our consensus a priori.

    Leon
  • Ricardo Signes at Jul 4, 2013 at 8:45 pm
    * Leon Timmermans [2013-07-04T14:04:21]
    By what process? Define consensus? Given Andy is the official
    maintainer and Ovid is the effective maintainer, I don't think they
    need our consensus a priori.
    06perms.txt says:

       Test::Harness,ANDYA,m
       Test::Harness,MSCHWERN,c
       Test::Harness,OVID,c

    So presumably at least getting the three of them to agree on some kind of
    resolution process is a start.

    --
    rjbs
  • Ovid at Jul 7, 2013 at 9:45 am

    ----- Original Message -----

    From: Ricardo Signes <perl.qa@rjbs.manxome.org>

    * Leon Timmermans [2013-07-04T14:04:21]
    By what process? Define consensus? Given Andy is the official
    maintainer and Ovid is the effective maintainer, I don't think they
    need our consensus a priori.
    06perms.txt says:

    Test::Harness,ANDYA,m
    Test::Harness,MSCHWERN,c
    Test::Harness,OVID,c

    So presumably at least getting the three of them to agree on some kind of
    resolution process is a start.

    I'm winding up with astonishingly little bandwidth due to launching our company, so I was hoping to see a strong consensus from the users. I would also love to see examples of where the change or lack thereof is causing an issue. I am SWAMPED with so much email that receiving many opinions piecemeal makes it hard for me to follow along.

    Were I not so bandwidth-constrained, this would be less of an issue, but I'd like to see a good Wiki page or something with the pro/con arguments laid out. If this is too much, I should turn over maintainership to someone with more bandwidth to ensure I'm not a blocker.

    Cheers,
    Ovid
    --
    IT consulting, training, international recruiting
           http://www.allaroundtheworld.fr/.
    Buy my book! - http://bit.ly/beginning_perl
    Live and work overseas - http://www.overseas-exile.com/
  • Karen Etheridge at Jul 7, 2013 at 3:11 pm

    On Sun, Jul 07, 2013 at 02:45:22AM -0700, Ovid wrote:
    Were I not so bandwidth-constrained, this would be less of an issue, but I'd like to see a good Wiki page or something with the pro/con arguments laid out. If this is too much, I should turn over maintainership to someone with more bandwidth to ensure I'm not a blocker.
    wiki page created:
    https://github.com/Perl-Toolchain-Gang/Test-Harness/wiki/TAP::Harness-and--the-w-flag
  • Ovid at Jul 7, 2013 at 4:00 pm

    ----- Original Message -----

    From: Karen Etheridge <perl@froods.org>
    On Sun, Jul 07, 2013 at 02:45:22AM -0700, Ovid wrote:
    Were I not so bandwidth-constrained, this would be less of an issue, but
    I'd like to see a good Wiki page or something with the pro/con arguments
    laid out. If this is too much, I should turn over maintainership to someone with
    more bandwidth to ensure I'm not a blocker.

    wiki page created:
    https://github.com/Perl-Toolchain-Gang/Test-Harness/wiki/TAP::Harness-and--the-w-flag

    Karen,

    Thank you for putting that together. It made it much easier for me to follow this!

    The Problem:

    In Test::Harness, many people want to see -w no longer being enabled by default. Others object to this change because this alters the behavior and it we should tread carefully, particularly since this change effects everyone who installs modules. Backwards-incompatible changes  to toolchain modules should generally not be done lightly. What's worse, different people have different desired behaviors here (mainly ribasushi, as far as I can tell, but I don't know what the silent masses think).

    That being said, I tend to agree that -w should not be forced on those who do not want it.

    Solutions:

    1. Make the change.
    2. Don't make the change.
    3/4. Make this very easy to configure but keep/change the old behavior.

    We have a change that people want, but as Eric pointed out, it introduces an incompatibility in the Test::Harness backwards-compatibility layer. If I'm to be conservative on this, I have to say that the one place we *don't* want incompatible changes is in a backwards-compatibility later!

    I originally wrote TAP::Harness to be very configurable (we can argue later if I succeeded). Since Test::Harness is merely a compatibility layer on top of TAP::Harness, I like Eric Wilhelm's suggestion of software switching to TAP::Harness instead of Test::Harness. The basic change is simple. Instead of:

        use Test::Harness;
        runtests(@test_files);


    You do this:

        use TAP::Harness;
        my $harness = TAP::Harness->new( \%args );
        $harness->runtests(@tests);

    Of course, the devil is in the details and I imagine that many tools will be more seriously impacted.

    This allows us to maintain backwards-compatibility and gives uses a better interface, to boot!

    The Real Question:

    What toolchain software is being impacted by this and how hard would it be to make the switch? People have long known that TAP::Harness is a better alternative to Test::Harness, but the compatibility layer meant that people wouldn't have to make that switch right away. I suppose that some time in the past 5 1/2 years I should have been urging people to make the change. Perhaps now there is a reason to make that switch?

    Admittedly, the above sounds like a remarkably self-serving way of passing the buck (very handy right now), but the question is very real: what are the obstacles to those wanting a different behavior in making a switch?

    Cheers,
    Ovid

    --
    IT consulting, training, international recruiting
           http://www.allaroundtheworld.fr/.
    Buy my book! - http://bit.ly/beginning_perl
    Live and work overseas - http://www.overseas-exile.com/
  • Leon Timmermans at Jul 7, 2013 at 5:26 pm

    On Sun, Jul 7, 2013 at 6:00 PM, Ovid wrote:
    What toolchain software is being impacted by this and how hard would it be to make the switch?
    AFAIK only ExtUtils::MakeMaker and Module::Build use it. The latter
    can already use TAP::Harness, but there are some details we'd might
    want to fix first (such as environmental variables).
    Module::Build::Tiny is already using TAP::Harness exclusively.
    People have long known that TAP::Harness is a better alternative to Test::Harness, but the compatibility layer meant that people wouldn't have to make that switch right away.
    Given toolchain still has to support perl < 5.10.1, using TAP::Harness
    means an extra non-core dependency. I suspect that's the real reason.

    Possibly we could use TAP::Harness but fall back on Test::Harness if
    the former is not present. Disadvantage may be different test results
    depending on this because of the -w issue.

    Leon
  • Leon Timmermans at Sep 17, 2013 at 3:27 pm

    On Sun, Jul 7, 2013 at 11:45 AM, Ovid wrote:

    I'm winding up with astonishingly little bandwidth due to launching our
    company, so I was hoping to see a strong consensus from the users. I would
    also love to see examples of where the change or lack thereof is causing an
    issue. I am SWAMPED with so much email that receiving many opinions
    piecemeal makes it hard for me to follow along.

    Were I not so bandwidth-constrained, this would be less of an issue, but
    I'd like to see a good Wiki page or something with the pro/con arguments
    laid out. If this is too much, I should turn over maintainership to someone
    with more bandwidth to ensure I'm not a blocker.
    Just as I expected, "make it a wiki" means it gets warnocked again.

    Can we please make a decision, or if we must first come to an agreement on
    how to make it?

    Leon
  • Mark Stosberg at Sep 17, 2013 at 5:14 pm

    On 09/17/2013 11:26 AM, Leon Timmermans wrote:
    On Sun, Jul 7, 2013 at 11:45 AM, Ovid wrote:

    I'm winding up with astonishingly little bandwidth due to launching
    our company, so I was hoping to see a strong consensus from the
    users. I would also love to see examples of where the change or lack
    thereof is causing an issue. I am SWAMPED with so much email that
    receiving many opinions piecemeal makes it hard for me to follow along.

    Were I not so bandwidth-constrained, this would be less of an issue,
    but I'd like to see a good Wiki page or something with the pro/con
    arguments laid out. If this is too much, I should turn over
    maintainership to someone with more bandwidth to ensure I'm not a
    blocker.


    Just as I expected, "make it a wiki" means it gets warnocked again.

    Can we please make a decision, or if we must first come to an agreement
    on how to make it?
    I think a pro/con list is a reasonable request. I've read all the
    messages myself, am a regular user of Perl's testing tools, and the
    benefits and drawbacks are not clear to me either.

    An attempting-to-be-neutral summary would be very helpful.

         Mark
  • Chromatic at Sep 17, 2013 at 5:21 pm

    On Tuesday, September 17, 2013 01:13:26 PM Mark Stosberg wrote:

    An attempting-to-be-neutral summary would be very helpful.
    Pro of adding -w to test runs:
      - that's how Test::Harness has always worked, and people might be relying
    on that behavior

    Cons of adding -w to test runs:
      - you get warnings from dependencies (and their dependencies) because -w
    enables global action at a distance
      - using fatal warnings may cause your test suite to fail because of
    warnings in dependencies you don't directly control

    -- c
  • Mark Stosberg at Sep 17, 2013 at 5:40 pm

    On 09/17/2013 01:20 PM, chromatic wrote:
    On Tuesday, September 17, 2013 01:13:26 PM Mark Stosberg wrote:

    An attempting-to-be-neutral summary would be very helpful.
    Pro of adding -w to test runs:
    - that's how Test::Harness has always worked, and people might be relying
    on that behavior

    Cons of adding -w to test runs:
    - you get warnings from dependencies (and their dependencies) because -w
    enables global action at a distance
    - using fatal warnings may cause your test suite to fail because of
    warnings in dependencies you don't directly control
    Thank you.

    I would rather TAP::Harness not enable warnings by default.

    I would prefer to opt-in, rather than opt out.

         Mark
  • Buddy Burden at Sep 19, 2013 at 5:25 am

    Cons of adding -w to test runs:
    - you get warnings from dependencies (and their dependencies) because -w
    enables global action at a distance
    - using fatal warnings may cause your test suite to fail because of
    warnings in dependencies you don't directly control
             - you could get different output from running tests via `prove` than you
    do via running the test file directly, which could cause you to scrath your head
    and go "WTF??"

    I see Ovid mentioned in a later email that prove uses TAP::Harness instead of Test::Harness, implying that it doesn't have this problem, but I know I spent several hours a few months back trying to figure out just what the hell was going on, and it finally came down to _somebody_ flipping on -w unexpectedly.


       -- Buddy
  • Leon Timmermans at Sep 17, 2013 at 6:19 pm

    On Tue, Sep 17, 2013 at 7:13 PM, Mark Stosberg wrote:

    I think a pro/con list is a reasonable request. I've read all the
    messages myself, am a regular user of Perl's testing tools, and the
    benefits and drawbacks are not clear to me either.

    An attempting-to-be-neutral summary would be very helpful.

    I agree that's a reasonable request. However, the odds are no one except
    me, ether and Ovid have looked at that wikipage between the previous set of
    emails and my email 2 hours ago. IMO, it's the wrong medium.

    Leon
  • Eric Wilhelm at Sep 17, 2013 at 6:57 pm
    # from Leon Timmermans on Tuesday 17 September 2013:
    odds are no one ... looked at that wikipage ...
    IMO, it's the wrong medium.
    I agree. Also, Test::Harness is the wrong module for this discussion.

    # from chromatic on Tuesday 17 September 2013:
    Pro of adding -w to test runs:
    - that's how Test::Harness has always worked, ...
    This really isn't a question of pros/cons of -w. That should be an
    option for the user with sensible defaults. It already is an option
    with sensible defaults in TAP::Harness.

    I believe Test::Harness is a compatibility layer for the sake of
    compatibility (and the first sentence of its docs seems to confirm it.)
    If this is true, changing it in incompatible ways is counter to its
    reason to exist. So, it seems better to deprecate and then delete it
    rather than discuss changes.

    Thanks,
    Eric
    --
    ---------------------------------------------------
         http://scratchcomputing.com
    ---------------------------------------------------
  • Ovid at Sep 18, 2013 at 8:19 am
    As I said in my previous email on July 7th: backwards-incompatible changes to the backwards-compatibility layer (Test::Harness) are not a good idea. The proper response is to have people impacted by this issue switch to TAP::Harness, as was suggested several years ago when Test::Harness 3.0 was released. For example, the 'prove' utility calls App::Prove which calls TAP::Harness. If others are using Test::Harness directly, perhaps Eric is right and it should be deprecated? However, it's a core module and I don't know the implications of that.

    Cheers,
    Ovid
    --
    IT consulting, training, international recruiting
           http://www.allaroundtheworld.fr/.
    Buy my book! - http://bit.ly/beginning_perl
    Live and work overseas - http://www.overseas-exile.com/


    ________________________________
    From: Leon Timmermans <fawaka@gmail.com>
    To: Ovid <curtis_ovid_poe@yahoo.com>
    Cc: Ricardo Signes <perl.qa@rjbs.manxome.org>; "perl-qa@perl.org" <perl-qa@perl.org>
    Sent: Tuesday, 17 September 2013, 17:26
    Subject: Re: TAP::Harness and -w



    On Sun, Jul 7, 2013 at 11:45 AM, Ovid wrote:

    I'm winding up with astonishingly little bandwidth due to launching our company, so I was hoping to see a strong consensus from the users. I would also love to see examples of where the change or lack thereof is causing an issue. I am SWAMPED with so much email that receiving many opinions piecemeal makes it hard for me to follow along.
    Were I not so bandwidth-constrained, this would be less of an issue, but I'd like to see a good Wiki page or something with the pro/con arguments laid out. If this is too much, I should turn over maintainership to someone with more bandwidth to ensure I'm not a blocker.
    Just as I expected, "make it a wiki" means it gets warnocked again.


    Can we please make a decision, or if we must first come to an agreement on how to make it?

    Leon

  • Ricardo Signes at Sep 18, 2013 at 11:17 pm
    * Ovid [2013-09-18T04:19:28]
    As I said in my previous email on July 7th: backwards-incompatible changes to
    the backwards-compatibility layer (Test::Harness) are not a good idea. The
    proper response is to have people impacted by this issue switch to
    TAP::Harness, as was suggested several years ago when Test::Harness 3.0 was
    released.
    It looks like Module::Build already has use_tap_harness. So what's suggested
    is a patch to ExtUtils::MM_Any to switch from Test::Harness to TAP::Harness...?

    --
    rjbs
  • Leon Timmermans at Sep 18, 2013 at 11:22 pm

    On Thu, Sep 19, 2013 at 1:17 AM, Ricardo Signes wrote:

    It looks like Module::Build already has use_tap_harness. So what's
    suggested
    is a patch to ExtUtils::MM_Any to switch from Test::Harness to
    TAP::Harness...?
    Or both just setting $Test::Harness::switches = ''…

    Leon
  • Ricardo Signes at Sep 18, 2013 at 11:26 pm
    * Leon Timmermans [2013-09-18T19:21:44]
    Or both just setting $Test::Harness::switches = ''…
    Yes. I was cleaning out ~/tmp on the flight over here and I found this dzil
    plugin:

       package inc::STFUEUMM;
       use Moose;
       extends 'Dist::Zilla::Plugin::MakeMaker';
       use namespace::autoclean;

       after setup_installer => sub {
         my ($self) = @_;
         my ($file) = grep { $_->name eq 'Makefile.PL' } @{ $self->zilla->files };
         my $content = $file->content;

         $content .= <<'END_AWFUL_HACK';
       sub MY::test_via_harness {
           my($self, $perl, $tests) = @_;
           return qq{\t$perl "-MExtUtils::Command::MM" }.
                  qq{"-e" "require Test::Harness; undef *Test::Harness::Switches; test_harness(\$(TEST_VERBOSE), '\$(INST_LIB)', '\$(INST_ARCHLIB)')" $tests\n};
       }
       END_AWFUL_HACK

         $file->content($content);
       };


    I hadn't published that before because AndyA applied my patch to Test::Harness.
    Hmm...

    --
    rjbs
  • Ricardo Signes at Sep 21, 2013 at 4:19 am
    * Leon Timmermans [2013-09-17T11:26:40]
    Just as I expected, "make it a wiki" means it gets warnocked again.

    Can we please make a decision, or if we must first come to an agreement on
    how to make it?
    Allow me to offer this:

       https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/pull/72

    Rather than change Test::Harness, which would break the compatibility of the
    compatibility layer, we just fix the thing that *actually* bothers people: the
    behavior of `make test`.

    --
    rjbs
  • Leon Timmermans at Sep 21, 2013 at 9:31 am

    On Sat, Sep 21, 2013 at 6:19 AM, Ricardo Signes wrote:

    * Leon Timmermans [2013-09-17T11:26:40]
    Just as I expected, "make it a wiki" means it gets warnocked again.

    Can we please make a decision, or if we must first come to an agreement on
    how to make it?
    Allow me to offer this:

    https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/pull/72

    Rather than change Test::Harness, which would break the compatibility of
    the
    compatibility layer, we just fix the thing that *actually* bothers people:
    the
    behavior of `make test`.

    Given I've recently added a «use warnings;» to Test::Harness (not released
    yet), '' may be a better value for $Switches than undef.

    Leon
  • Eric Wilhelm at Jul 4, 2013 at 9:34 pm

    # from Leon Timmermans on Thursday 04 July 2013:
    ewilhelm : We should keep the original change, but update EUMM to
    restore -w
    I'm not sure that's what he meant.
    Yeah no. I said get rid of Test::Harness.

    TAP::Harness provides a cleaner interface and this default only exist in
    the old crufty compatibility interface.

    If you're going to keep the compatibility layer, don't change it. But
    if we're going to argue about changing it, I say we should burn it.

    " changing it in the Test::Harness reverse compatibility
    layer seems incompatible with reverse compatibility."
    "require TAP::Harness and use that
    interface (where it would be off by default anyway, right?)

    IIRC, Module::Build has --use-tap-harness or something, which should
    probably just be the default these days."

    --Eric
    --
    ---------------------------------------------------
         http://scratchcomputing.com
    ---------------------------------------------------
  • Ricardo Signes at Jul 4, 2013 at 10:11 pm
    * Eric Wilhelm [2013-07-04T17:33:45]
    # from Leon Timmermans on Thursday 04 July 2013:
    ewilhelm : We should keep the original change, but update EUMM to
    restore -w
    I'm not sure that's what he meant.
    Yeah no. I said get rid of Test::Harness.

    TAP::Harness provides a cleaner interface and this default only exist in
    the old crufty compatibility interface.
    Thanks for clarifying!

    --
    rjbs

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupqa @
categoriesperl
postedJul 2, '13 at 6:51p
activeSep 21, '13 at 9:31a
posts22
users8
websiteqa.perl.org

People

Translate

site design / logo © 2019 Grokbase