CPAN Testers AUTOMATED_TESTING config

CPAN Testers AUTOMATED_TESTING config

	Karen Etheridge
	May 31, 2013 at 10:34 pm

Hi Slaven,

recently I received a number of failing test reports from you, for
Plack-App-BeanstalkConsole-0.004:

http://www.cpantesters.org/cpan/report/27614552-c707-11e2-be51-ac14a65f1290
http://www.cpantesters.org/cpan/report/bc294a00-c706-11e2-b547-d1b9f05bb613
http://www.cpantesters.org/cpan/report/976876e6-c706-11e2-b45e-827567d01fb8
http://www.cpantesters.org/cpan/report/089d0ca6-c706-11e2-b644-eee7e73c5bf2
http://www.cpantesters.org/cpan/report/6dd3a9b2-c707-11e2-87dd-f08dfdc235a3

However, there should be *no testing at all* done for this distribution,
(save any reports that are sent manually via cpanm-reporter), because the
first line of Makefile.PL is:

exit 0 if $ENV{AUTOMATED_TESTING};

This is because this distribution requires the presence of a php
executable, and also the Makefile.PL downloads a large file from github --
so to respect github's servers, I'd rather this happen only when the module
is actually going to be used.

Automated CPAN testing is supposed to set the AUTOMATED_TESTING variable:

http://wiki.cpantesters.org/wiki/CPANAuthorNotes

     "Can I tell if my module is being tested by an automated client (a.k.a. 'smoker')?"

     Automated smoke testers should set $ENV{AUTOMATED_TESTING} to a true value. This allows authors to skip certain tests (or include certain tests) when the results are not being monitored by a human being.

     One could even go so far as to halt building and testing a distribution under automated testing by exiting with zero at the top of the Makefile.PL or Build.PL file:

     exit 0 if $ENV{AUTOMATED_TESTING};

Please could you set AUTOMATED_TESTING in your smokers? thank you!

Karen Etheridge
ether@cpan.org

Search Discussions

4 responses
Oldest
Nested

Slaven Rezic

at Jun 13, 2013 at 9:22 pm

⇧

Karen Etheridge writes:

Hi Slaven,

recently I received a number of failing test reports from you, for
Plack-App-BeanstalkConsole-0.004:

http://www.cpantesters.org/cpan/report/27614552-c707-11e2-be51-ac14a65f1290
http://www.cpantesters.org/cpan/report/bc294a00-c706-11e2-b547-d1b9f05bb613
http://www.cpantesters.org/cpan/report/976876e6-c706-11e2-b45e-827567d01fb8
http://www.cpantesters.org/cpan/report/089d0ca6-c706-11e2-b644-eee7e73c5bf2
http://www.cpantesters.org/cpan/report/6dd3a9b2-c707-11e2-87dd-f08dfdc235a3

However, there should be *no testing at all* done for this distribution,
(save any reports that are sent manually via cpanm-reporter), because the
first line of Makefile.PL is:

exit 0 if $ENV{AUTOMATED_TESTING};

This is because this distribution requires the presence of a php
executable, and also the Makefile.PL downloads a large file from github --
so to respect github's servers, I'd rather this happen only when the module
is actually going to be used.

Automated CPAN testing is supposed to set the AUTOMATED_TESTING variable:

http://wiki.cpantesters.org/wiki/CPANAuthorNotes

"Can I tell if my module is being tested by an automated client (a.k.a. 'smoker')?"

Automated smoke testers should set $ENV{AUTOMATED_TESTING} to a true value. This allows authors to skip certain tests (or include certain tests) when the results are not being monitored by a human being.

One could even go so far as to halt building and testing a distribution under automated testing by exiting with zero at the top of the Makefile.PL or Build.PL file:

exit 0 if $ENV{AUTOMATED_TESTING};

Please could you set AUTOMATED_TESTING in your smokers? thank you!

--
Slaven Rezic - slaven <at> rezic <dot> de
   BBBike - route planner for cyclists in Berlin
   WWW version: http://www.bbbike.de
   Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net

reply | permalink

David Golden

at Jun 13, 2013 at 11:13 pm

⇧

The Lancaster Consensus from this year's QA hackathon has updated the
desired semantics:
http://www.dagolden.com/index.php/2098/the-annotated-lancaster-consensus/

Specifically, AUTOMATED_TESTING is now intended to distinguish only
between things running for "smoke testing" versus things running as
part of module installation (or other, manually testing). It no
longer means "without recourse to a human operator" as it did after
the Oslo Consensus. That aspect is now covered by the new
"NONINTERACTIVE_TESTING" environment variable.

In your case -- automated testing with manual review of results --
both should be set, because when the tests run, it is both smoking and
non-interactive. :-)

Your manual review is awesome -- I wish more smoke testers did that --
but it doesn't change the environment we want to present to
distributions, so they can decide to opt-in or out of some or all
tests depending on circumstances surrounding the test.

David

On Thu, Jun 13, 2013 at 5:19 PM, Slaven Rezic wrote:
Karen Etheridge <perl@froods.org> writes:

Hi Slaven,

recently I received a number of failing test reports from you, for
Plack-App-BeanstalkConsole-0.004:

http://www.cpantesters.org/cpan/report/27614552-c707-11e2-be51-ac14a65f1290
http://www.cpantesters.org/cpan/report/bc294a00-c706-11e2-b547-d1b9f05bb613
http://www.cpantesters.org/cpan/report/976876e6-c706-11e2-b45e-827567d01fb8
http://www.cpantesters.org/cpan/report/089d0ca6-c706-11e2-b644-eee7e73c5bf2
http://www.cpantesters.org/cpan/report/6dd3a9b2-c707-11e2-87dd-f08dfdc235a3

However, there should be *no testing at all* done for this distribution,
(save any reports that are sent manually via cpanm-reporter), because the
first line of Makefile.PL is:

exit 0 if $ENV{AUTOMATED_TESTING};

This is because this distribution requires the presence of a php
executable, and also the Makefile.PL downloads a large file from github --
so to respect github's servers, I'd rather this happen only when the module
is actually going to be used.

Automated CPAN testing is supposed to set the AUTOMATED_TESTING variable:

http://wiki.cpantesters.org/wiki/CPANAuthorNotes

"Can I tell if my module is being tested by an automated client (a.k.a. 'smoker')?"

Automated smoke testers should set $ENV{AUTOMATED_TESTING} to a true value. This allows authors to skip certain tests (or include certain tests) when the results are not being monitored by a human being.

One could even go so far as to halt building and testing a distribution under automated testing by exiting with zero at the top of the Makefile.PL or Build.PL file:

exit 0 if $ENV{AUTOMATED_TESTING};

Please could you set AUTOMATED_TESTING in your smokers? thank you!

Actually, my smoker does not fail under this sentence: "... are not
being monitored by a human being". I look into every FAIL report before
sending. Really.

But I can setup a special rule in 01.DISABLED.yml for your distribution.

Regards,
Slaven

--
Slaven Rezic - slaven <at> rezic <dot> de
BBBike - route planner for cyclists in Berlin
WWW version: http://www.bbbike.de
Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net

--
David Golden <xdg@xdg.me>
Take back your inbox! → http://www.bunchmail.com/
Twitter/IRC: @xdg

reply | permalink

Bulk88

at Dec 22, 2015 at 3:49 pm

⇧

Slaven Rezic wrote:
Karen Etheridge <perl@froods.org> writes:

Hi Slaven,

recently I received a number of failing test reports from you, for
Plack-App-BeanstalkConsole-0.004:

http://www.cpantesters.org/cpan/report/27614552-c707-11e2-be51-ac14a65f1290
http://www.cpantesters.org/cpan/report/bc294a00-c706-11e2-b547-d1b9f05bb613
http://www.cpantesters.org/cpan/report/976876e6-c706-11e2-b45e-827567d01fb8
http://www.cpantesters.org/cpan/report/089d0ca6-c706-11e2-b644-eee7e73c5bf2
http://www.cpantesters.org/cpan/report/6dd3a9b2-c707-11e2-87dd-f08dfdc235a3

However, there should be *no testing at all* done for this distribution,
(save any reports that are sent manually via cpanm-reporter), because the
first line of Makefile.PL is:

exit 0 if $ENV{AUTOMATED_TESTING};

This is because this distribution requires the presence of a php
executable, and also the Makefile.PL downloads a large file from github --
so to respect github's servers, I'd rather this happen only when the module
is actually going to be used.

Automated CPAN testing is supposed to set the AUTOMATED_TESTING variable:

http://wiki.cpantesters.org/wiki/CPANAuthorNotes

"Can I tell if my module is being tested by an automated client (a.k.a. 'smoker')?"

Automated smoke testers should set $ENV{AUTOMATED_TESTING} to a true value. This allows authors to skip certain tests (or include certain tests) when the results are not being monitored by a human being.

One could even go so far as to halt building and testing a distribution under automated testing by exiting with zero at the top of the Makefile.PL or Build.PL file:

exit 0 if $ENV{AUTOMATED_TESTING};

Please could you set AUTOMATED_TESTING in your smokers? thank you!

As of 2015 your smoker has neither AUTOMATED_TESTING or
NONINTERACTIVE_TESTING set as described in
http://www.nntp.perl.org/group/perl.cpan.testers.discuss/2013/06/msg3159.html

See
http://www.cpantesters.org/cpan/report/8c7c2c36-a887-11e5-b1c7-2d09e1bfc7aa
http://www.cpantesters.org/cpan/report/8acca780-a887-11e5-b1c7-2d09e1bfc7aa
http://www.cpantesters.org/cpan/report/892a795c-a887-11e5-b1c7-2d09e1bfc7aa
http://www.cpantesters.org/cpan/report/8e0862cc-a887-11e5-b1c7-2d09e1bfc7aa

So there is no way for a module to know it is running on CPAN Testers on
your smoker, and no way to make a module's building/testing more verbose
for CPAN Testers and less verbose human watched installs, so I had to
make that module more verbose for everyone.

reply | permalink

Slaven Rezic

at Dec 22, 2015 at 7:47 pm

⇧

bulk88 writes:

Slaven Rezic wrote:

Karen Etheridge <perl@froods.org> writes:

Hi Slaven,

recently I received a number of failing test reports from you, for
Plack-App-BeanstalkConsole-0.004:

http://www.cpantesters.org/cpan/report/27614552-c707-11e2-be51-ac14a65f1290
http://www.cpantesters.org/cpan/report/bc294a00-c706-11e2-b547-d1b9f05bb613
http://www.cpantesters.org/cpan/report/976876e6-c706-11e2-b45e-827567d01fb8
http://www.cpantesters.org/cpan/report/089d0ca6-c706-11e2-b644-eee7e73c5bf2
http://www.cpantesters.org/cpan/report/6dd3a9b2-c707-11e2-87dd-f08dfdc235a3

However, there should be *no testing at all* done for this distribution,
(save any reports that are sent manually via cpanm-reporter), because the
first line of Makefile.PL is:

exit 0 if $ENV{AUTOMATED_TESTING};

This is because this distribution requires the presence of a php
executable, and also the Makefile.PL downloads a large file from github --
so to respect github's servers, I'd rather this happen only when the module
is actually going to be used.

Automated CPAN testing is supposed to set the AUTOMATED_TESTING variable:

http://wiki.cpantesters.org/wiki/CPANAuthorNotes

"Can I tell if my module is being tested by an automated client (a.k.a. 'smoker')?"

Automated smoke testers should set $ENV{AUTOMATED_TESTING} to
a true value. This allows authors to skip certain tests (or include
certain tests) when the results are not being monitored by a human
being.

One could even go so far as to halt building and testing a
distribution under automated testing by exiting with zero at the
top of the Makefile.PL or Build.PL file:

exit 0 if $ENV{AUTOMATED_TESTING};

Please could you set AUTOMATED_TESTING in your smokers? thank you!

As of 2015 your smoker has neither AUTOMATED_TESTING or
NONINTERACTIVE_TESTING set as described in
http://www.nntp.perl.org/group/perl.cpan.testers.discuss/2013/06/msg3159.html

See
http://www.cpantesters.org/cpan/report/8c7c2c36-a887-11e5-b1c7-2d09e1bfc7aa
http://www.cpantesters.org/cpan/report/8acca780-a887-11e5-b1c7-2d09e1bfc7aa
http://www.cpantesters.org/cpan/report/892a795c-a887-11e5-b1c7-2d09e1bfc7aa
http://www.cpantesters.org/cpan/report/8e0862cc-a887-11e5-b1c7-2d09e1bfc7aa

So there is no way for a module to know it is running on CPAN Testers
on your smoker, and no way to make a module's building/testing more
verbose for CPAN Testers and less verbose human watched installs, so I
had to make that module more verbose for everyone.

Hi Dragan,

I can do this, but again, it will be just a distroprefs file for your
distribution. I cannot see that any of the five documented environment
variables in David's blog article matches your request (more verbosity):
NONINTERACTIVE_TESTING is just about interactivity, not verbosity,
EXTENDED_TESTING probably does not apply (the amount of testing seems to
be the same), and AUTOMATED_TESTING does not apply for my wrapper script
around CPAN.pm --- in fact, the same script is used for my normal module
installs; so a "smoke" test+install and a "real" test+install is
indistinguishable (other than me using or not using the installed
module).

Regards,
Slaven

reply | permalink

Related Discussions

Discussion Navigation

view	thread \| post

Discussion Overview

group	cpan-testers-discuss @
categories	perl
posted	May 31, '13 at 10:34p
active	Dec 22, '15 at 7:47p
posts	5
users	4
website	cpan.org

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

CPAN Testers AUTOMATED_TESTING config

Search Discussions

Related Discussions

4 users in discussion