Lancaster Consensus, deal with PUREPERL_ONLY=0

The only thing specified in the lancaster consensus is what must
happen if that command-line argument is true.

I think making a distinction between "0" and undefined will be
surprising to people and I would recommend against it.

Given this point - how can we give people an instrument to force XS
and fail if it's not available?
Cheers

As I mentioned before - you create a separate ::XS distribution, against
which the outliers declare dependencies. In general "forcing XS" when PP
is available is *always* *invairably* the wrong approach (which is why
they are called outliers above ;)

The user must always have a way to enforce or fail. And not every
distribution can be split into 2. So please forget the cases where
it's possible to split and let's come back to the question:

How can we enable the user/packager to make a clear choice?

Cheers

Let me rephrase: making available a "XS-only" choice, when both PP ans XS are
available is a mistake. Not just making the choice is a mistake,
*presenting it* is a mistake in its own right.

Unless you have a clear use case that you didn't mention before ;)

You should explain why that should be a mistake when presenting a "PP-only"
choice is not a mistake. That doesn't make any sense to me.
Is "The user makes the choice" not a clear use case? I rate this as _the_ use
case.

Cheers

I suspect vis-a-vis is reuired to clear this up. I'll refrain from
further additions to this thread until our next meet on the 3rd ;)

There exists installations that can run PP dists, but not XS dists.
There is no such thing as an installation that *cannot* run a PP
implementation.

The "give me only the PP version, even if XS is available and installable
on my system" choice is useful when preparing a fatpacked installation. I
do not see any gain in specifying "give me XS or give me death". Not every
dist is XS-based, so is not going to work out in most cases, anyway.

I'm wondering why it isn't always possible to split a dist into two
implementations, one PP and the other with XS optimizations. If the dist
simply cannot be implemented using pure Perl (Moose, for example), then
surely the right thing to is simply refuse to install on PP systems?

Can you clarify what the usecases are here? That is, /achieve?

I expected exactly that argument - "we do FatPacking, so it's important".
Not everyone does fat packing - there're a lot of use cases out there who
manage large build farms and package (with very strange toolchains here
and there) compiled distributions.

Others (eg. packagers) have always a compiler and want benefit from XS when
available and what is more important: if the "DEVELOPER MODE FLAGS IS ON",
they want to have a -Werror behavior even for "I enabled foo and it doesn't
appear".

In complex infrastructure (and CPAN is such a one), you'll never be able
to detect configuration/build issues otherwise.
Given, a user knows it's system can provide all infrastructure for XS.
And some modules always provide just PP, even if they shouldn't.
To help digging, bail out when XS check fails will help that type of user.

I'm sorry when I appear evangelizing - for me it feels last years that
most of p5 walks only in direction "very fast x86 linux farms for
horizontal scaling". This might be the loudest use-case - but not the
only one.

A lot of my customers tell me (during coffee break or similar): "We don't
write on public mailing lists or query for support for particular stuff.
When it doesn't work or requires to much effort, we use something else."

The primary question is: do we want large system management staff's
in big industry walking to Maven because we think "user shall be able to
control the fulfillment of his/her wish" is not a valid use-case or we
don't see why?

General (Unix) assumption is: the operator/administrator is god. They
know what they're doing and it's on the tools to provide the ability,
not to limit.

For me the limitation to either have pureperl or maybe more is a to
strong limitation. Query is now: do we (as perl community) want a
common way to remove that limitation or do I as I mean?

I'm fine with "Jens' distribution have a non-standard switch", but I
would prefer a "Toolchain community favors that knob to be used".

Cheers

In Lancaster, we considered having both a "give me PP" option *and* a
"give me XS" option and the group decided it was less confusing to
have just one.

Here's how I would do it:

(a) if the PP option is there, give them PP
(b) if the PP option is missing/false, check if there is a compiler
(c) if there is a compiler, give them XS or give them death
(d) if there isn't a compiler, give them PP

If for (c) that means running tests under some "XS-only" flag of your
own, that's fine with me.

David

http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html

It's the widest know of those flags - GNU autotools provide such a one, too.
xlc names it -qhalt=<severity> ...

Lot's of tools provide switches which exit when something doesn't do as
wanted (if a specific behavior is intended).
I didn't had the "user control > author mind" situation. I evolved by
implementing (both sides - provide CPAN distributions and use/package
them).
(b)-(d) shall die (or exit 0 in Makefile.PL) when XS-force flag is set.

I currently prove:
a) what is wanted (tri-state: PP-only, XS-only, check)
pp-b) do just pp
b) check for "sane xs" (perl.h, xsub.h, perl-api, ...)
xs-c) (b) or bail-out
check-c) (b) or goto (pp-b)
d) compile && pp

Is that somehow clear?

But ... that brings me to another point of view. Take Unix::Statgrab,
File::LibMagic or BSD::Resource as example - they won't work without XS
either. Shall they generally bail-out when PUREPERL_ONLY=1 is set?

Cheers

Dude! I know what the compiler flag does!

I don't understand if you're using "-Werror" as an analogy for how you
want cpan clients to behave or if you mean you want "-Werror" to halt
CPAN installation on error even when a PP fallback is available, or
something else entirely.

In my example, if not PUREPERL_ONLY, then if a compiler is present,
compilation works or it fails (possibly due to -Werror), and the
install is halted.
Yes and no. Because they don't offer a PP option, PUREPERL_ONLY is
irrelevant to them. They will fail in their "make" stage as usual.

A more user-friendly author could probe for a compiler and bail out in
Makefile.PL if one doesn't exist, possibly with one of the CPAN
Testers magic strings like "OS unsupported: need a compiler".

It's an incorrect and possibly misleading error message, but it will
result in a CPAN Testers report of "NA" instead of "FAIL". The other
option is "exit 0" which will just not send a report at all. E.g.
Devel::CheckLib

David

I mean: I want a behavior where a user (packager - the one who probably
indirectly invoke "perl Makefile.PL") has a tunable which allows clearly
to express whether
1) PUREPERL is wanted (to create a noarch package)
2) XS is wanted (to create an arch specific package, have the "fastest"
    possible installation - whatever)
3) decide on availability

(1) and (2) should really fail when they cannot support that switch
to avoid mistakes. (I'm a fan of the "die early" approach in case of
errors.)

Generally I fear the "*_ONLY" misleads - I would prefer "--enable-xs"
or "--disable-xs" - but that's a complete different story (and it's
finally not better when the switch itself is unknown).
For sure - but I want that the "perl Makefile.PL" dies with error when
PUREPERL_ONLY=0 (or XS_ONLY=1) is set and it's not possible to create
an XS module.
Well - currently there're PP modules which add a dependency for a
corresponding XS module when they find a compiler. Seeing the dependency
picture, it could make sense to fail for File::LibMagic when PP_ONLY
is enabled (and the requiring module falls back to File::MMagic).
That is what Config::AutoConf->with_xs() should provide. Currently
I have a private one in LMU (not yet pushed), but with an agreement
how to deal with PP_ONLY and how to provide a (however) XS_ONLY I
intend to provide such a check as a public API and every XS module
just should do "C::AC->with_xs()" for compiler and little xs-compile
checks.

Web-Search for "CPAN Testers magic strings" doesn't provide suitable
results - is there a list of those magic results?
If I would know what is expected, I will make Config::AutoConf behave.
Devel::CheckLib doesn't work at all - but I don't want to send pull
requests using C::AC until I'm sure that C::AC doesn't introduce
more pitfalls than it solves ;)

Cheers

Here's what I think is important to understand:

* PASS/FAIL reports are only supposed to happen if tests actually run
(e.g. "make test", etc.). These should not be sent if a cpan
reporting client can detect that perl prereqs are missing.

* UNKNOWN reports are supposed to happen when the dist never reaches
the test stage (e.g. "Makefile.PL" or "make" fails) or if there are no
test files to run. This means "I don't know if your tests pass or
fail because I never got that far"

* NA reports are supposed to happen when there is no possible way for
tests to ever succeed. Generally this is for platform/OS issues, like
Win32::Job will never succeed on Unix, or something that needs threads
will never run on a non-threaded perl. I've seen it used and used it
myself for situations where some configuration issue prevents success
-- e.g. you're running on OpenBSD and your perl is neither threaded or
single-threaded-but-linking-libpthead.

* "exit 0" from Makefile.PL has a coincidentally useful behavior that
CPAN Testers exploits. The CPAN clients happened to all check for
"Makefile" being created even if "Makefile.PL" returned exit code zero
and bailed out if it didn't exist. That allowed a way to let
Makefile.PL abort without triggering a CPAN Testers report message.
This turned out to be really useful for non-perl dependencies (e.g.
compiler, C libraries, external program, etc.)

David

This can help for Sys::Filesystem - good to know. I'll re-check how
that behaves against that description.
That's what I knew and how C::AC behaves. So everything is cool :D
Additionally - if one needs - there is a config.log containing all
made tests and for those who fail the complete scenario with call
and failure output.

Cheers

Hi Jens,

* Jens Rehsack [2014-06-02 13:30]:
how come? Is there a technical difficulty with that? What prevents it?

Regards,

This I find easy to see, at least if we are talking of the general case:

If I have my system set up to support XS then I want it to use XS, and
if for some reason it can’t, then there is a bug somewhere that needs to
be fixed, so I do *not* want it to silently give me the PP version and
carry on, but to fall down and die screaming so I’ll find out.

However, I am unsure of the context of the discussion.

The above reasoning is applicable and important mostly when it comes to
production deployments: I never want my servers trying to limp on with
a CPU- and/or memory-hungry setup that will only reveal itself as such
in monitoring data.

But maybe the discussion is about some other scenario in which the given
reasoning is inapplicable? Since I don’t know, the above use case may be
neither here nor there.

Regards,

I've been trying to think of how that could happen and so far all I
can come up with is this scenario:

* XS bits compile successfully, but the shared object croaks during
bootstrap, and the calling module traps the bootstrap error and falls
back to PP

David

There're several scenarios causing XS builds to fail:
1) compiler cannot create executables
    * invalid cflags configured (see http://www.cpantesters.org/cpan/report/a25a36da-ea1d-11e3-9cf0-a01f06268b4b for an example)
    * beta compiler test (sunpro 12.4 on linux fails loading libdwarf)
    * cc installed, but as missing
2) compiler works, but no perl headers
3) compiler does, but shared library building fails (that is fragile and minor modifications at the system causes that process to fail)
4) As you say: https://rt.cpan.org/Ticket/Display.html?id=75672

I think currently we don't cross-compile (handy market would make me want to change that), which allows tests for 4, but until 3 we can always test. 4 only when we don't cross-compile.

And what does "I prepared my system to support XS" mean? I installed a compiler or I installed precisely the same compiler/binutils used to build perl? I think, there should be a flag the user to be able to set to express the opinion the system is prepared for XS - not any magic detection. Users word counts.

Cheers

Oooh! Excellent example.

"make" succeeds and "make test" could fall back to PP when dynaloading fails.

I wonder if there's a way to be clever about this such that under
PUREPERL_ONLY, make never compiles the extension, and then the main
module doesn't just try bootstrap inside eval{} but actually checks to
see if the extension is in @INC. If so, bootstrap it or die; if not,
fall back to pure Perl.

David

talked yesterday with riba about that in deeper detail.
We decided to do as (probably) suggested by you and

sub legacy_exists { die "bogus legacy XS" if the extension is in @INC; return false }

autoconf->check_pp_wanted and return legacy_exists;
autoconf->have_compiler or return legacy_exists;
autoconf->check_perl_api or die;
if( autoconf->extensive_testing ) {
   autoconf->check_perl_extension_links or die;
   autoconf->check_perl_extension_loads or die;
}
return ok

http://www.cpantesters.org/cpan/report/a25a36da-ea1d-11e3-9cf0-a01f06268b4b is also an excellent example.

Regarding "if the extension is in @INC" (have separate installed XS module) needs a migration plan for LMU, P::U etc.
This is the final solution and we had several ideas yesterday for a migration plan - but wasn't complete.

Cheers

Here is the commit for that: https://github.com/rehsack/Config-AutoConf/commit/e8a14d079995ea519c3aebc698994ea5964bbf70

If there're no objections, I fire a release with that change (and the one which skips compile tests for bogus environments as Nigel Hone has ...)

Cheers

I simply detected that we (p5) don't cover that situation, we always
fallback to PP.
Production deployment is somehow similar to packaging - packaging is one
step in deployment (even tar xjf ... is packaging or rsync a prepared
directory). And then it comes - do the farm has the same arch and is
XS possible and wanted, does your package process gives you the ability
to distinguish between archs or is your process one blob to rule them
all - than is FatPacking your hammer and XS is unwanted.

But: Do the folks at CERN really FatPacking? AFAIK (I talk here and there
in train to guys having experiments there) no.
Depending on one's background, usecases sounds ridiculous. As pkgsrc maintainer
(16+ platforms 30+ architectures supported) I do not understand the
existence of fat packer at all. pbulk+pkgin is easier to deal with ...

But I accept the usecase - even when I don't understand it.

Cheers