FAQ
If you grep CPAN you'll find a lot of ad-hoc ways of asking usespace if
network testing is permitted.

http://grep.cpan.me/?q=%28if|unless%29\s.%2BENV.*NET

It seems wide spread enough that we may as well establish a common way of
doing it, document it, and then enhance the available tooling to make it
make sense.

That way, instead of a user having to see a notice in their test output and
manually change ENV to suit ( and have to do this many different ways for
many different packages ), they'll just have to set the relevant option in
their installation stack.

And this would be a standard option for those running smoke testers, and
smoke testers who felt that network testing was something that *should*
work on their boxes could set the relevant configuration tunable, and we'll
get better CPAN testers coverage as a result.

Once we decide on an "official" and "blessed" ENV key, we can go about
getting support for it added in various testing tools, like:

https://metacpan.org/pod/Test::Is
https://metacpan.org/pod/Test::DescribeMe
and
https://metacpan.org/pod/Test::Settings

That way, Test authors only need to do:

use Test::Is 'networked' ;


Or similar instead of having to hand code a bunch of ENV conditions and
skip rules.

Though the exact syntax those modules should use can be decided at a later
date and we should simply decide on the ENV name to use.

Other thoughts:

Some tools have their distribution name prefixed in their ENV key, which
may be because those distributions consider network testing a bad idea,
even if it is otherwise indicated globally as ok, and requesting *explicit*
indication for itself to run. At least, that is why I imagine it is done.
It may just be out of paranoia of ENV key clutter causing unintentional
keys to be in scope and have people running tests without meaning to.

Technically we don't really need to solve this as a feature within the ENV
key, I hashed together something way more complicated that would allow
per-distribution granularity settings without any specific
testing/toolchain infrastructure, but it seems its not really necessary.

Ultimately if a smoke tester wanted that degree of granularity, the table
of rules could be established in the smoke testing toolchain itself, and
per-package ENV flags could be set up to allow network tests for some dists
and not for other dists using this simple approach.

I also considered using values other than "1" in this ENV flag( ie:
"Dist-Foo:Dist-Bar:Dist-Baz", but it all smells too much of featureitis and
things we don't need yet and overcomplicates things.

Essentially, the simplest thing that could work is just

  $ENV{NETWORK_TESTING}

This is informally used in a place or two already, mimics our existing set
of ENV vars, and its sensible enough that we may as well standardize on it.

Competition:

- A few places use NETWORK_TEST (singular)
( WWW::Offliberty, LBMA::Statistics , Net::Douban
- A few places use NETWORK_TESTS (plural)
( RDF::RDFa::Template }
- Quite a few places use SOMEPREFIX_NETWORK_TEST or SOMEPREFIX_NETWORK_TESTS
( Net::Social::Mapper, WorePAN, RDF::Query, RDF::Trine, Data::Feed )
- RUN_NETWORK_TESTS is seen in at least one place
( IO::Pipeley )
- Other Weird things exist:
   - TEST_YAR_NET_CONNECT in Net::YAR
   - AUTHOR_TEST_NET_ROUTE in Net::Route
   - HAS_INTERNET in WWW::OPG
   - TEST_INTERNET in WebService::CIA
   - PUGS_TEST_ALLOW_NETWORK in Perl6::Pugs
   - TEST_NET in MojoX::UserAgent
   - ONLINETESTS in HTTP::LoadGen
   - USE_NETWORK in Arch

Well, you get the picture. Its a complete mess.

I merely got bored with enumerating all the ways network testing could be
enabled, and I doubt anyone running a smoker would care enough to track
them all down like I just started to.

Better to say "If you want network tests to be smoke tested, USE THIS"



--
Kent
( ~KENTNL )

Search Discussions

  • David Golden at Jun 11, 2014 at 2:30 pm
    Kent,

    I'm not sold on this.

    The existing author/release/automated test environment variables
    define categories of tests that testers opt into.

    What you're describing is a variable to detect a capability. It's
    closer to NONINTERACTIVE_TESTING, but note which way the defaults go
    -- the assumption is that interactivity *is* available unless told
    otherwise.

    And the reason for the variable in the first place is that prompting
    can hang. Avoiding hangs is a higher priority for me than avoiding
    failure.

    I'd be more interested in seeing a standard way to probe the
    capability rather than rely on testers to declare capabilities.

         use Test::RequiresInternet; # or skip_all

    Or, if particular hosts/ports need to be reachable:

        use Test::RequiresInternet { "www.google.com" => 80 };

    David


    --
    David Golden <xdg@xdg.me> Twitter/IRC: @xdg
  • Kent Fredric at Jun 11, 2014 at 3:52 pm

    On 12 June 2014 02:29, David Golden wrote:

    And the reason for the variable in the first place is that prompting
    can hang. Avoiding hangs is a higher priority for me than avoiding
    failure.

    I'd be more interested in seeing a standard way to probe the
    capability rather than rely on testers to declare capabilities.

    use Test::RequiresInternet; # or skip_all

    Or, if particular hosts/ports need to be reachable:

    use Test::RequiresInternet { "www.google.com" => 80 };

    David
    Its not so much intended as a "Does the internet work", definitively
    speaking.

    But more "Has the user in question granted permission to use the internet".

    As such, Test:RequiresInternet with specific hosts and ports would be an
    orthogonal problem, a problem that could only be determined *after *permission
    is granted.

    That is, Test::RequiresInternet would require permission somehow to check
    that google was accessible.

    Granted one could bake support for that ENV flag into that module, and it
    would be practical to do so.

    But either way, we have to decide on *which* ENV flag to use, or *which*
    competing method a user may declare "Yes, please use my internet" on their
    box.

    After that is decided we can then probe exploring *availability* of
    resources, instead of simply being granted permission to *test* for
    availability.

    Yes, it would be nice to have these things, but I figured the ENV key would
    be a better initial stepping stone to get out of the way. :)

    --
    Kent
  • David Golden at Jun 11, 2014 at 5:09 pm

    On Wed, Jun 11, 2014 at 11:52 AM, Kent Fredric wrote:
    "Has the user in question granted permission to use the internet".
    I disagree with the premise that one needs to ask. Users are already
    running arbitrary Perl code and must take responsibility for anything
    that it might do and not run it if it does something they don't want
    it to do.

    The only thing that interests me is avoiding hassling people with
    failures in cases that have no possibility of success, e.g. connecting
    to a host on the Internet when the Internet is fire-walled.

    That's a question of capability, not permission.

    David
  • Jens Rehsack at Jun 11, 2014 at 5:37 pm

    Am 11.06.2014 um 19:08 schrieb David Golden <xdg@xdg.me>:
    On Wed, Jun 11, 2014 at 11:52 AM, Kent Fredric wrote:
    "Has the user in question granted permission to use the internet".
    I disagree with the premise that one needs to ask. Users are already
    running arbitrary Perl code and must take responsibility for anything
    that it might do and not run it if it does something they don't want
    it to do.

    The only thing that interests me is avoiding hassling people with
    failures in cases that have no possibility of success, e.g. connecting
    to a host on the Internet when the Internet is fire-walled.

    That's a question of capability, not permission.
    I think there're setups where smoke tests run in a private. not routed
    network segment. While this is uncommon, probably an envvar is the right
    approach, but it should be one like LOCALTESTS_ONLY instead of
    NETWORK_IS_AVAILABLE. This allows smokers to disable unwanted network
    tests.

    How does that sound? (Maybe I don't understand the concern correctly)

    Cheers
    --
    Jens Rehsack
    rehsack@gmail.com
  • David Golden at Jun 11, 2014 at 5:48 pm

    On Wed, Jun 11, 2014 at 1:37 PM, Jens Rehsack wrote:
    I think there're setups where smoke tests run in a private. not routed
    network segment. While this is uncommon, probably an envvar is the right
    approach, but it should be one like LOCALTESTS_ONLY instead of
    NETWORK_IS_AVAILABLE. This allows smokers to disable unwanted network
    tests.
    If it has to be done via env variable, phrasing this way, negatively
    like NONINTERACTIVE_TESTING, is better than an env var granting
    positive permissions.

    I still think it's better for tests to check rather than rely on
    testers to set such variables.


    --
    David Golden <xdg@xdg.me> Twitter/IRC: @xdg
  • Jens Rehsack at Jun 11, 2014 at 5:58 pm
    Am 11.06.2014 um 19:48 schrieb David Golden [...]
    I still think it's better for tests to check rather than rely on
    testers to set such variables.
    You never know whether a test fails because of failure or insufficient
    capabilities. So a restricting envvar isn't worse at all.

    Cheers
    --
    Jens Rehsack
    rehsack@gmail.com
  • Kent Fredric at Jun 11, 2014 at 6:24 pm

    On 12 June 2014 05:58, Jens Rehsack wrote:

    You never know whether a test fails because of failure or insufficient
    capabilities. So a restricting envvar isn't worse at all.
    I think he was more saying that he'd prefer:

         set NO_NETWORK_TESTING=1

    over

         set NETWORK_TESTING=1

    Where network testing should run by default and users on boxes where it
    *couldnt* work ( for whatever reason ) could disable it.

    That would be more helpful on an imaginary example environment that was
    sandboxed where calling network functions during 'make test' triggers a
    SIGKILL or something.

    And then with that proviso agreed upon, have a module that ascertains (
    using basic testing within the test itself ) if network behaviour is
    conducive to making the test pass, and if so, permit the test to run (
    guarding the test against actual network problems instead of relying on an
    ENV guard , and using the ENV guard only for users who have continued
    issues with the heuristic failing to fail properly )

    1. begin test
    2. load test networking module module
    3. is NO_NETWORK_TESTING? SKIP!
    4. can access specified resources? yes -> run tests
                                                           no -> SKIP!


    --
    Kent
  • Jens Rehsack at Jun 11, 2014 at 6:37 pm

    Am 11.06.2014 um 20:24 schrieb Kent Fredric <kentfredric@gmail.com>:


    On 12 June 2014 05:58, Jens Rehsack wrote:
    You never know whether a test fails because of failure or insufficient
    capabilities. So a restricting envvar isn't worse at all.

    I think he was more saying that he'd prefer:

    set NO_NETWORK_TESTING=1

    over

    set NETWORK_TESTING=1
    I would prefer LOCALTESTS_ONLY=1 over NO_NETWORK_TESTING=1
    Where network testing should run by default and users on boxes where it *couldnt* work ( for whatever reason ) could disable it.

    That would be more helpful on an imaginary example environment that was sandboxed where calling network functions during 'make test' triggers a SIGKILL or something.

    And then with that proviso agreed upon, have a module that ascertains ( using basic testing within the test itself ) if network behaviour is conducive to making the test pass, and if so, permit the test to run ( guarding the test against actual network problems instead of relying on an ENV guard , and using the ENV guard only for users who have continued issues with the heuristic failing to fail properly )

    1. begin test
    2. load test networking module module
    3. is NO_NETWORK_TESTING? SKIP!
    4. can access specified resources? yes -> run tests
    no -> SKIP!
    In that case I suggest to FAIL instead of skip. Maybe is that part of the error.

    Cheers
    --
    Jens Rehsack
    rehsack@gmail.com
  • Kent Fredric at Jun 12, 2014 at 7:29 am

    On 12 June 2014 06:37, Jens Rehsack wrote:

    1. begin test
    2. load test networking module module
    3. is NO_NETWORK_TESTING? SKIP!
    4. can access specified resources? yes -> run tests
    no -> SKIP!
    In that case I suggest to FAIL instead of skip. Maybe is that part of the
    error.


    Really whether or not that fails would be an installer choice.

    For instance, me, I would basically say "If you cant access the internet,
    don't bother doing tests that require the internet". After all, the running
    of the test suite is not to give me a "pass/fail: Your internet works", but
    to test the code in question functions as intended.

    The only time I'd want the tests to fail is if I was running the tests
    _AND_ want assurredness that the network tests in fact work as intended,
    and that the non-network tests are not sufficient for my personal installer
    satisfaction.

    Then those tests would fail every time they couldn't use the internet, and
    I'd know I have to fix my internet and run them again.

    But that case is very much not the usecase of a bulk smoke tester.

    If a tester box doesn't have internet visible during 'make test', I'd
    expect tests that require internet access to be skipped ( or exit N/A
    somehow ), not give the developer of the module spurious FAIL results that
    revolve around something the developer can do nothing about, FAIL reports
    that don't indicate the code is broken, merely that the person who ran the
    tests has broken internet.


    Based on some of the comments on #perl-qa , it seems there may also be a
    use for declaring 2 or 3 grades of internet access.

    1. Loopback only access: Basic network functions work, you can spawn a HTTP
    test server for something like PSGI, and connect to it from another Perl
    process and IPC between the two.

    2. LAN only access: There are other machines on a LAN, some semblance of
    DNS will probably work and resolve HOSTNAME to a network routable IP, but
    internet traffic is still expected to not work. ( Or there's a (broken?)
    firewall standing between LAN and WAN than can resolve global DNS addresses
    but not relay packets )

    3. Full internet


    Now we probably don't need to represent all of these, just they're scopes
    to take into consideration. You'd think case #1 would never need blocking
    off, however, the person who raised that case mentioned in the context of
    "Localhost is firewalled". And that would mess things up. ( And I doubt
    anyone is interested in going through a firewall just to allow open ports
    for some random Perl test )

    Most of those levels we could indeed check for with code, but having a
    "Don't even check, its not worth the cycles, you'll probably get it wrong,
    and trying may even cause an explosion because somebody attached C4 to a
    network device somewhere in the house" control would be still useful.

    ( And I guess the "weird networks are weird all over the place" problem is
    one reason why the default is to have network tests opt-in instead of
    opt-out. )

    --
    Kent

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcpan-workers @
categoriesperl
postedJun 11, '14 at 12:56p
activeJun 12, '14 at 7:29a
posts10
users3
websitecpan.org

People

Translate

site design / logo © 2018 Grokbase