Rafael Garcia-Suarez (rgarcias...@gmail.com)

2009/8/1 Bram <p5p@perl.wizbit.be>:
> While testing something I got a false positive in t/io/errno.t (
> http://perl5.git.perl.org/perl.git/commit/d7dfc388e04f41b8a0f5d8ef6e15ab3=
b79f483c8
> ) but only in the tests that use stdio, the test that use perlio work fin=
e.
>
> It can be reproduced by setting $! to a value before the call to readline=
().
> That is:
>
> diff --git a/t/io/errno.t b/t/io/errno.t
> index 6294ed7..3cd59c2 100644
> --- a/t/io/errno.t
> +++ b/t/io/errno.t
> @@ -9,7 +9,7 @@ require './test.pl';
>
> =A0plan( tests =3D> 16 );
>
> -my $test_prog =3D 'while(<>){print}; print $!';
> +my $test_prog =3D '$!=3D1;while(<>){print}; print $!';
> =A0my $saved_perlio;
>
> =A0BEGIN {
>
>
>
> Output of the test script:
>
> ./perl TEST -v io/errno.t
> t/io/errno....1..16
> ok 1 - Wrong errno, PERLIO=3Dperlio stdin=3D'test\n',
> ok 2 - Wrong errno, PERLIO=3Dperlio stdin=3D'test\n', $/=3Dundef
> ok 3 - Wrong errno, PERLIO=3Dperlio stdin=3D'test\n', $/=3D\2
> ok 4 - Wrong errno, PERLIO=3Dperlio stdin=3D'test\n', $/=3D\1024
> ok 5 - Wrong errno, PERLIO=3Dperlio stdin=3D'test',
> ok 6 - Wrong errno, PERLIO=3Dperlio stdin=3D'test', $/=3Dundef
> ok 7 - Wrong errno, PERLIO=3Dperlio stdin=3D'test', $/=3D\2
> ok 8 - Wrong errno, PERLIO=3Dperlio stdin=3D'test', $/=3D\1024
> # Failed at io/errno.t line 31
> # =A0 =A0 =A0got 'test
> # Operation not permitted'
> # expected 'test
> # '
> not ok 9 - Wrong errno, PERLIO=3Dstdio stdin=3D'test\n',
> [...]
>
> This shows that under perlio $! was changed from 1 to undef and that unde=
r
> stdio $! remained unchanged.
>
> Is this difference between perlio and stdio intentional? (read: can it be
> ignored?)

I don't think it's intentional. I also think that you can't rely on $!
if there was no error.

> Either way, I belive the test should be updated to:
>
> my $test_prog =3D 'undef $!;while(<>){print}; print $!';
>
> so a patch for this is attached.
> (In the current blead this creates no problem but in a future version it
> might. (I'm currently testing something and this test gave a false positi=
ve
> because of it))

Thanks, applied.

2009/11/15 karl williamson <public@khwilliamson.com>:
> I have been thinking somewhat of how to implement this. =A0The problem is=
we
> don't want to slow down things for this very rare (maybe non-existent) us=
e.
> =A0Since this feature is based on global subroutines, I would think one c=
ould
> store in a static variable if the existence of such a subroutine has been
> looked for or not. =A0Then it could be structured so that the normal flow=
had
> just one test added to check this static. =A0For example, initialize the
> static to 0 for not-looked at yet; set it to something else on the first
> time the case function is called, say 1 if found no routine; and anything
> else if found a routine. =A0There would have to be one variable for each =
of
> lc, uc, and ucfirst (since lcfirst uses the same case definitions as lc).
>
> Now, I'm not very familiar with Perl internals, so I'm asking. =A0I would
> think we would not want to normally use statics just because of lexical
> scoping and threads. =A0But in this case, there is no lexical scoping, an=
d the
> user-defined subroutines would apply to all threads. =A0So would a static
> variable be an ok thing to use here?

No : because of embedding. (Perl interpreter(s) might be embedded in
other programs, such as the vim editor for example)

The correct way to add a global variable is to add it to perlvars.h;
and to add a per-interpreter variable, you have to add it to
intrpvar.h. I think that in this case you want intrpvar.h.

2009/11/13 karl williamson <public@khwilliamson.com>:
> Attached is a patch that adds code to almost fix the case changing portio=
n
> of the "Unicode bug" (see
> http://rt.perl.org/rt3/Public/Bug/Display.html?id=3D58182). =A0This is a
> significant portion of perltodo's UTF-8 revamp.

Thanks, applied as 00f254e235ff10d6223aa9a402ad5b7a85689829.

> What this means is that characters whose ordinals on ASCII machines are i=
n
> the 128-255 range will have Unicode semantics as far as case changing goe=
s
> regardless of whether they are encoded in utf8 or not. =A0The reason I sa=
y it
> 'almost' fixes this is that any user-defined case mapping is still not
> called unless the scalar is in utf8. =A0See below for a discussion on tha=
t.
>
> I am leaving the legs that implement this behavior disabled by default un=
til
> the smokes show that the new stuff doesn't break anything; then I'll flip
> the bit to enable it. =A0If you want to play with it in the meantime, you=
can
> say 'no legacy "unicode8bit"';

Prudence is good. (so currently in blead the legacy-unicode8bit pragma
has the reverse meaning than the one it will have in 5.12)

> I've proposed changing the name of this; so that may happen, but it doesn=
't
> affect the heart of the code, so I'm delivering it now; it's easy to chan=
ge
> the name later.

I still like that name; the documentation might need improvements
though. When I read this line in the SYNOPSIS :

use legacy ':5.10'; # Keeps semantics the same as in perl 5.10

as a perl user, I'm not certain of how that works, because it's not
clear that the doc itself depends on a version strictly greater than
5.10. Maybe it's not such a good idea to use versioned bundles like
feature.pm does.

> I don't understand Perl magic. =A0So I have tried to avoid touching anyth=
ing
> around that. =A0But there are a couple comments from the old code that ma=
ke me
> think I really don't understand what's going on in regard to that. =A0I'm=
now
> thinking the comments are obsolete. =A0If someone would care to look at t=
hem,
> they are at lines 3963 and 4227 in pp.c and both say the same thing:
> "Overloaded values may have toggled the UTF-8 flag on source, so we need =
to
> check DO_UTF8 again here". =A0This doesn't make sense to me based on the =
code
> earlier in the functions they occur in, so I think they are wrong.

I'll look, but I don't promise to understand either.

> There are minor changes in several files: macros are created in headers t=
o
> access the bit and look up the case change mappings. =A0perl.h has two ta=
bles
> added with the mappings for all 256 Latin1 characters that give
> respectively, 1) their lowercased values; and 2) their upper and title ca=
sed
> values. =A0I removed trailing blank space in all the submitted files.
>
> There are significant changes in the casing functions in pp.c to accommod=
ate
> this new behavior. =A0Basically, if the bit is set, the case change is lo=
oked
> up via these tables instead of the existing mechanism. =A0If the bit is o=
ff or
> 'use locale' or 'use bytes' is in effect the existing mechanism is used.
>
> Complications arise because three characters in the latin1 range require
> special handling when title and uppercasing them. =A0Two of them have the=
case
> change out-of-range, which means that the result has to be converted to
> utf8, and the other expands to two in-range characters. The code in the
> ucfirst() function was rearranged to compute the case change first, so as=
to
> know if the length of the result changes. =A0If it doesn't change, and th=
e
> scalar isn't read-only etc, then only the first character needs to be
> touched. =A0Also, the uc() function has to cope with the possibility that=
in
> mid-stream it will have to decide to upgrade the result to utf8.
>
> I tried to make this efficient, and not slow things down from what they a=
re
> now. =A0It may actually be faster than the current implementation in gene=
ral
> because it does a table look up, which avoids some tests. =A0One test was
> added in the inner loop for upper casing, but then again the table lookup
> took out two tests
>
> There are two things I started to implement, but left #ifdef'd out
> currently:
>
> One of them implements the context sensitive casing that Unicode defines.
> =A0This is not implemented because I need more time to think about things=
; for
> one, Unicode has also recently revised their guidelines on this and I
> haven't looked at the new ones.
>
> The other would change the case of a utf8-encoded character in the Latin1
> range by using the built-in tables without having to go to the swash. =A0=
It is
> disabled because it would break the ability of user-defined case mappings
> overriding the default behavior for characters in that range.
>
> Which brings us to the topic of these user-defined case-changing mappings=
.
> =A0I just haven't gotten around to figuring out how to tell if such a fun=
ction
> is in existence or not. =A0Currently, they must be in main::. =A0Since th=
is is a
> very obscure corner of the language, I'm deferring fixing it until later.
> =A0Rafael gave me one hint about how to figure out if such a mapping func=
tion
> exists, but I haven't pursued it yet. =A0I understand that Zefram has bee=
n
> working on lexically-scoped subroutines, so that could affect this.

I agree with the deferring. Is anyone aware about some code using
user-defined case mappings, by the way ?

2009/10/30 karl williamson <public@khwilliamson.com>:
> I would like to start submitting patches to fix the "Unicode bug" wherein
> the semantics differs depending on the internal utf8-or-not representatio=
n
> of a scalar.
>
> I submitted the first steps for that quite a while back, and Rafael made =
a
> separate branch for that. =A0I'm not sure how to proceed. =A0I need that =
to go
> forward. =A0Do I do something, or what? =A0It is a tiny patch.
>
> At the time Rafael expressed some concern that I was using up a bit from
> PL_hints, and suggested using the newer hash instead. =A0I would prefer t=
o
> keep using PL_hints for several reasons:
> 1) The closely related bits for locale and "use bytes" are located here, =
and
> it seems to make sense to keep them all in the same structure. =A0And the=
re
> are several bits available even after this one is used up.
> 2) It's easier for me to just copy the existing paradigm for these than t=
o
> figure out how to do the other
> 3) I'm leery of the other scheme given all the problems there are with
> charnames in regcomp.c. =A0I'm pretty ignorant on this, and ignorance lea=
ds to
> paranoia, not bliss. =A0But I'm not sure we've gotten to the bottom of th=
e
> cause, and I'd just as soon not get involved with depending on something
> that may be flaky. =A0Now perhaps this is used with =A0full success elsew=
here,
> and my concerns are just paranoia. =A0But then I go back to reasons 1 and=
2.

I have now merged this branch in blead. Further patches welcome :)

That has been traditionnally the solution for all tests in core. The
standard test boilerplate makes sure that, when $ENV{PERL_CORE} is
set, tests will be only loaded from the core, not from the @INC of the
future perl that is  being built and tested, and not yet installed.

So this patch should be enough :

diff --git a/lib/CGI/t/fast.t b/lib/CGI/t/fast.t
index 45f8e12..4941c62 100644
--- a/lib/CGI/t/fast.t
+++ b/lib/CGI/t/fast.t
@@ -1,6 +1,14 @@
#!./perl -w

-use lib qw(t/lib);
+BEGIN {
+    if( $ENV{PERL_CORE} ) {
+        chdir 't';
+        @INC = '../lib';
+    }
+    else {
+        unshift @INC, 't/lib';
+    }
+}

# Due to a bug in older versions of MakeMaker & Test::Harness, we must
# ensure the blib's are in @INC, else we might use the core CGI.pm