One remaining question.

Should the context maintain a stack, where each time context() is called
(nested) it pushes the vars to the stack, and each call to release pops it,
or should it just store them on creation, do nothing on retrieval, and only
restore them on the final release?

Right now the version I have up on cpan just stores them on creation, and
restores them on final release. Nothing happens for nested calls to
context()/release(), all my downstream testing shows no breakages (not a
full smoke, but does include several modules sensitive to $! and $@

This question is not about backwards compatibility, but rather about what
test tool authors might/should expect the behavior to be. Both maintain
backwards compatibility equally. However the stack form will have more of a
performance impact, probably minor though, would need to benchmark/profile.

On Tue, Jan 12, 2016 at 6:12 PM, Sawyer X wrote:


Chad, thank you for the detailed response. I think I now understand
the scope of the problem and your solutions.

I think it makes sense to put this in the guts inside the construction
of a new context (or retrieval of current context) and in the release
of that context. Kent, am I right to believe this also addresses the
concerns you raised regarding testing of testing modules?

I'm sorry to say I'm quite ignorant [also] when it comes to the
testing underpinnings, so I'm not sure if there are additional
concerns this does not address, but otherwise it sounds very
reasonable to me.

Thanks again for taking the time to clarify in detail. :)
(It might be useful to start working on a document for the internals
for anyone who wants to hack on it. This should at least be in the
commit messages so it could be tracked down.)


On Tue, Jan 12, 2016 at 11:26 PM, Chad Granum wrote:
Yes, your understanding appears correct. And I can make it more clear.

This is a very simple test tool in Test::Builder (the chrome):
my $TB = Test::Builder->new; # Call to internals/guts (singleton)

sub ok($;$) {
my ($bool, $name) = @_;
$TB->ok($bool, $name); # Call to internals/guts
return $bool

Here it is again using Test2 instead of Test::Builder (the chrome):
sub ok($;$) {
my ($bool, $name) = @_;
my $ctx = context(); # Call to internals/guts (A)
$ctx->ok($bool, $name); # another one (B)
$ctx->release; # another one (C)
return $bool;

The lines marked with 'Call to internals/guts' kick off a series of things
that read/write from filehandles, possibly opens them, evals code/catches
exceptions, and any number of other things that can squash $! and $@. It
should be noted that 'ok' is not the only method ever called on either
Test::Builder or $ctx, this is just a very simple illustration.

Now for starters, Test::Builder uses a singleton, so it can do all its
initialization at load time, which allows it to leave several things
unprotected. The singleton was bad, so Test2 does not use one, which means
it has to be more protective of $! and $@ in more places to accomplish the
same end goal.

History, what Test::Builder does: It localizes $! and $@ in an eval wrapper
called _try() that it uses to wrap things it expects could squash $! and $@.
It also localizes $SIG{__DIE__} for various reasons. In some places where
$SIG{__DIE__} should not be localized it will instead use local
independently of _try(). There is also extra logic for subtests to ensure $?
from the end of the subtest is carried-over into the regular testing outside
of the subtest. Some places also need to be careful of $? because they run
in an end block where squashing $? unintentionally is bad. (Yeah, $? is
involved in all this as well, but much less so)

This results in a lot of places where things are localized, and several
places that run through an eval. People simply looking at the code may
overlook these things, and not know that the protection is happening. The
first time a new-dev will notice it is when tests start breaking because
they added an open/close/eval/etc in the wrong place. Thanfully there are
some tests for this, but not enough as I have found downstream things
(PerlIO::via::Timeout as an example) that break when $! is squashed in a way
Test::Builder never tests for.

Test::Builder does not localize $! and $@ in all its public method.
Realistically it cannot for 2 reasons:

Performance hit
Can mask real exceptions being thrown that are not caught by

In short, this is a significant maintenance burden, with insufficient
testing, and no catch-all solution.


Up until this email thread, Test2 was doing the same thing as
The problem is that Test2 does lots of other things differently for good
reasons, unfortunately it provides a lot more opportunity to squash $! and
$@. Like Test::Builder it is not reasonable to localize $! and $@ at every
entry point.

I decided to start this thread after a few downstream breakage was detected
due to $! being squashed. One because perl changes $! when you clone a
filehandle, even when no errors happen. Another because of a shared memory
read that was added in an optional concurrency driver used by
Test::SharedFork. I realized this would be an eternal whack-a-mole for all
future maintainers of both projects, and one that is hard to write
sufficient testing for.

The solution:
Go back to my second example. Notice there are 3 calls to the guts, marked
(A), (B), and (C). (A) and (C) are universal to all Test2 based tools, and
are also universally used in the Test::Builder dev releases when it calls
out to Test2. No tool will function properly if it does not use both of
those when it uses Test2. Calling context() should already always be done as
soon as possible. the call to release() should be called as late as

I realized I could make the call to context() store $! and $@ in the $ctx
object it returns. I could then also have release() restore them. This is
similar to localizing except it bypasses the flaws:

There is very little performance hit to this, in my 100k ok test, which
takes just under 2 seconds on my machine, it added ~100ms. That is peanuts.
There is more random variation in performance from run to run then the
increase itself.
It will not prevent $@/$! from being set by true uncaught exceptions, $ctx
does not restore the vars when it si destroyed, so if an exception occurs
release() is never called.
It is a lot less magic than scattering random locals throughout the codebase
It can be documented easily in one place
It is easy to test and maintain


Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 17 of 28 | next ›
Discussion Overview
groupcpan-workers @
postedJan 12, '16 at 12:53a
activeJan 18, '16 at 10:52p



site design / logo © 2021 Grokbase