Ismail Donmez (i...@pardus.org.tr)

Can confirm this here on 32bit x86 Linux.

Mirrored @ http://cekirdek.pardus.org.tr/~ismail/dist/perl-5.10.0.tar.gz

Regards,
ismail

Tuesday 18 December 2007 20:11:46 tarihinde Rafael Garcia-Suarez şunları
yazmıştı:
> Perl 5.10.0 is now out, the first in the 5.10.x major version series,
> after a five year long development process. It's currently being
> mirrored on CPAN.

Congratulations to whole perl5-porters team and to you Rafael!

So the solution would be adding hints that will affect lc/uc functions. so
something like

use Turkish;

would effect uc/lc but also there needs to a way to reset this, because one
can want to mix ascii-only uc/lc with Turkish uc/lc.

Any idea where to start implementing this? Or before that, does this sound
like a good plan?

Regards,
ismail

So we need to hint Perl about the language somehow.

Regards,
ismail

Thursday 13 December 2007 05:34:31 tarihinde Jarkko Hietaniemi şunları
yazmıştı:
> > Following test doesn't work as expected:
> >
> > use POSIX qw(locale_h);
> > use utf8;
> > binmode STDOUT, ':utf8';
> >
> > setlocale(LC_ALL, "tr_TR.UTF-8");
> > print uc("abğıiüşç");
> > print "\n";
> > print lc("ABĞIİÜŞÇ");
> > print "\n";
> >
> > it should print
> >
> > ABĞIİÜŞÇ
> > abğıiüiç
>
> I assume you meant
>
> ABĞIİÜŞÇ
> abğıiüşç
>
> (in the second one Åž -> ÅŸ, not Åž -> i)

Indeed, sorry for the typo.

> > So I guess a fix is needed in locale module?
>
> Apologies in advance but I'll be painfully verbose in my answer.

Thanks for the detailed reply, let me digest it and reply.

> No.
>
> Okay, let's be more verbose.
>
> Firstly: locales and Unicode have (almost) nothing in common. In Perl,
> and in general.  Thankfully.
>
> Locales were invented in ancient times before Unicode was invented.
> They handled things like (at C level) making isalpha() and toupper()
> understand that there might be more to this world than ASCII, and
> (more visible at user level) that there might be more to this world than
> English (like when displaying dates, or messages). Mostly this meant
> that locales spoke only in 8-bit character sets, such as ISO 8859-1,
> though there were multibyte locales, for example for Japanese. The big
> picture is that locales affected various library APIs to be a little bit
> more international.
>
> However, this approach had various problems, one major one being that in
> C the interfaces still thought characters are chars are characters, when
> in the real world things are more complicated, e.g. case changes being
> not 1:1, but M:N, in other words, string-to-string operations, instead
> of just character-to-character operations, and similar complications
> affect other character operations. Another evil feature was that
> locales were not really standardized: vendor A calling locale "X"
> and vendor B calling locale "X" was no guarantee that "X" meant the
> same thing in A and B, and conversely vendor C might call "Y" the exact
> same thing as A called "X". Getting errors fixed in locales was not
> easy, either.

I see.

> Secondly: as promised above, locales and Unicode do not have much in
> common in Perl. You can see this by dropping the 'use POSIX...' and
> setlocale() lines from the test case, the result will not change one
> little bit. The (attempted, not necessarily always successful) rule
> in Perl is: "if a scalar is in Unicode (UTF-8), the UTF-8 rules, such
> as casing rules, apply" (locale rules for casing are ignored, because of
> all the pain listed above, and more).
>
> Thirdly: in the above I said "(almost) nothing in common". Well, lately
> there have been UTF-8 locales, such as "tr_TR.UTF-8". This mainly only
> means that things like dates are encoded in UTF-8: it cannot change the
> fact that isupper() and toupper() in C still operate on single 8-bit
> characters (and even if abominations like iswupper() were used, they
> still are "single codepoint"). And that just won't work with Unicode.
>
> Fourthly: yes, I know Rafael managed to get the errors go away with
> removing "use locale" but I think this is just hiding some real
> problem, either in Pod::Html or in Perl's Unicode, because as I
> explain above, the two worlds of locales and Unicode should meet as
> little as possible. The old locales should be let to rot and die in
> peace. (There is a new development of locale-kind of data called CLDR,
> a spin-off from Unicode, which shows much promise. But this is of
> course something old systems will not magically install to themselves.)

Indeed removing use locale was just a workaround for now.

> Lastly, about this particular case (sorry it took me so long to get
> here, Ismail): Perl should know, to a limit, about the special cases
> of Turkish casing (pardon the pun). If you look at the file
> lib/unicore/SpecialCasing.txt in the source code, and after building
> at the files lib/unicore/To/Lower.pl and lib/unicore/To/Upper.pl,
> you can see that Perl builds up rules that should catch any special
> casing cas...situations. Now why this doesn't seem to happen right
> for the Turkish case, I have two theories:
>
> (1) something is broken
> (2) something is not yet implemented
>
> The (1) is easier in the sense that a Simple Matter Of Debugging session
> is needed to find out why the special casing rules are not kicking in
> when they should. The (2) is potentially harder: I see tricky parts
> in the SpecialCasing.txt, namely that the rules for casing can depend
> on the *context* of where there characters are. If that's the case
> here (since Turkish i seems to have such rules), that is something
> that needs to be implemented since I know anything like that has not
> been implemented, the major reason being that to apply the more complex
> rules we need to know in which *language* the string is.

Indeed this was the original problem, how can we know if the string is Turkish
anyway? We reported this as RT #36953 too. So my hacky solution does use

use Turkish 'uc';
use Turkish 'lc';

a neat hack if you ask me ;)

But as I mentioned we need a official solution maybe a use Turkish; that
changes how uc/lc behaves.

Ideas/Comments ?

Regards,
ismail

Wednesday 12 December 2007 07:54:11 tarihinde Eric Brine şunları yazmıştı:
> This should be addressed by using the appropriate locale via "use locale".
> See http://perldoc.perl.org/perllocale.html

Following test doesn't work as expected:

use POSIX qw(locale_h);
use utf8;
binmode STDOUT, ':utf8';

setlocale(LC_ALL, "tr_TR.UTF-8");
print uc("abğıiüşç");
print "\n";
print lc("ABĞIİÜŞÇ");
print "\n";


it should print

ABĞIİÜŞÇ
abğıiüiç

So I guess a fix is needed in locale module?

Tuesday 11 December 2007 13:09:16 tarihinde Rafael Garcia-Suarez þunlarý
yazmýþtý:
> However that could cause other problems. There's is probably a better
> way to do it, but I'm a bit short on time right now. Better patches
> welcome. (Tests do pass with this patch, though.)

Thanks for the patch I'll look at the code for a maybe better fix :-)

Regards,
ismail

Hi all,

As Rafael already noted in his last e-mail in Turkish handling of i/I is
tricky. The rules are simple,

uppercase of i is Ä° which I-with-a-dot-above
lowercase of I is ı which is i-dotless

Now Perl's uc/lc function doesn't take this into account. For Pardus Linux I
created a very hacky solution (attached to this mail) to create a Turkish
class and provide uc/lc functions which are Turkish aware.

What I want to ask is, how can I fix this correctly so it can be fixed in Perl
itself so that not only Pardus but all Perl distributors can use it.

Any ideas appreciated.

Regards,
ismail

Tuesday 11 December 2007 12:28:22 tarihinde Rafael Garcia-Suarez şunları
yazmıştı:
> On 10/12/2007, Ismail Dönmez <ismail@pardus.org.tr> wrote:
> > Hi all,
> >
> > Linux 2.6 system, glibc 2.7, gcc 4.3.0 trunk, I get
> >
> > 1 Embed Failure:
> >
> > ../lib/ExtUtils/t/Embed........................................../tmp/ccb
> >l8h9W.o: In function `main':
> > embed_test.c:(.text+0x2f): undefined reference to `Perl_sys_init3'
> > embed_test.c:(.text+0x14a): undefined reference to `Perl_sys_term'
> > collect2: ld returned 1 exit status
>
> I can't reproduce this. Something strange with your build set up ? can
> you run it verbosely, like this:
> cd t
> ./perl -MTestInit ../lib/ExtUtils/t/Embed.t
> and paste the output.

Updated to gcc trunk of today and can't reproduce this anymore, heisenbug
maybe not sure. Or it might be that tests are not ready for parallel build I
am doing make -j2

> > 3 POD failures:
>
> I can reproduce those if I use the locale tr_TR.utf-8. Looks like this
> code isn't prepared with locales that don't capitalize i as I. (Hint
> to bug hunters -- using a Turkish locale is a good way to test your
> code's assumptions.) This needs to be fixed, but I'm afraid it's a bit
> late to get the changes in 5.10.0 final -- the fix will go in the CPAN
> version though, so it will be possible to upgrade independently.

Would it be possible to get the fix for this seperately? Because for Pardus
Linux Turkish support is crucial so I would like to ship Perl without Turkish
problems and I really like to ship Perl 5.10.0 for Pardus 2008.

Regards,
ismail

Monday 10 December 2007 23:45:25 tarihinde Steve Peters þunlarý yazmýþtý:
> On Dec 10, 2007 9:00 AM, Ismail Dönmez <ismail@pardus.org.tr> wrote:
> > Monday 10 December 2007 16:46:27 tarihinde Nicholas Clark þunlarý
yazmýþtý:
> > > On Mon, Dec 10, 2007 at 03:09:13PM +0200, Ismail Dnmez wrote:
> > > > Hi all,
> > > >
> > > > Linux 2.6 system, glibc 2.7, gcc 4.3.0 trunk, I get
> > > >
> > > > 1 Embed Failure:
> > > >
> > > > ../lib/ExtUtils/t/Embed........................................../tmp
> > > >/ccb l8h9W.o: In function `main':
> > > > embed_test.c:(.text+0x2f): undefined reference to `Perl_sys_init3'
> > > > embed_test.c:(.text+0x14a): undefined reference to `Perl_sys_term'
> > > > collect2: ld returned 1 exit status
> > > >
> > > > 3 POD failures:
> > > >
> > > > ../lib/Pod/t/htmlescp............................................
> > > > #   Failed test 'html escape'
> > > > # at ../lib/Pod/t/pod2html-lib.pl line 40.
> > > > ../lib/Pod/t/htmlescp............................................NOK
> > > > 1/1# Looks like you failed 1 test of 1.
> > > > ../lib/Pod/t/htmlescp............................................dubi
> > > >ous Test returned status 1 (wstat 256, 0x100)
> > > > DIED. FAILED test 1
> > > > Failed 1/1 tests, 0.00% okay
> > > > ../lib/Pod/t/htmllink............................................
> > > > #   Failed test 'html links'
> > > > # at ../lib/Pod/t/pod2html-lib.pl line 40.
> > > > ../lib/Pod/t/htmllink............................................NOK
> > > > 1/1# Looks like you failed 1 test of 1.
> > > > ../lib/Pod/t/htmllink............................................dubi
> > > >ous Test returned status 1 (wstat 256, 0x100)
> > > > DIED. FAILED test 1
> > > > Failed 1/1 tests, 0.00% okay
> > > > ../lib/Pod/t/htmlview............................................
> > > > #   Failed test 'html rendering'
> > > > # at ../lib/Pod/t/pod2html-lib.pl line 40.
> > > > ../lib/Pod/t/htmlview............................................NOK
> > > > 1/1# Looks like you failed 1 test of 1.
> > > > ../lib/Pod/t/htmlview............................................dubi
> > > >ous Test returned status 1 (wstat 256, 0x100)
> > > > DIED. FAILED test 1
> > > > Failed 1/1 tests, 0.00% okay
> > > >
> > > > Any help/ideas appreciated.
> > >
> > > Please could you send the output from running ./perl -Ilib -V
> > > in your build directory. There may be clues in how your build of perl
> > > was configured.
> >
> > Here is the relevant output (thanks for the -I hint!) :
>
> Do you have another gcc on your system that you could try building
> with? If you do, can you please try it as well so we can see if the
> problem is related to Perl or gcc 4.3.0?

Same problem happens with gcc 3.4.6 , glibc 2.3.6 . So it seems its Perl's
fault.

Regards,
ismail

Here is the relevant output (thanks for the -I hint!) :

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.24-rc4, archname=i686-linux-thread-multi
    uname='linux ninjamobile 2.6.24-rc4 #6 smp sat dec 8 17:32:43 eet 2007 
i686 genuine Intel(r) cpu t2300 @ 1.66ghz gnulinux '
    
config_args='-des -Darchname=i686-linux -Dcccdlflags=-fPIC -Dccdlflags=-rdynamic -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr -Ulocincpth= -Duselargefiles -Dd_dosuid -Dusethreads -Duseithreads -Dd_semctl_semun -Dscriptdir=/usr/bin -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dlibperl=libperl.so.1.5.8 -Duseshrplib -Dman1ext=1 -Dman3ext=3pm -Dinc_version_list=5.8.0
5.8.0/i686-linux 5.8.2 5.8.2/i686-linux 5.8.4
5.8.4/i686-linux -Dcf_by=Pardus -Ud_csh -Di_ndbm -Di_gdbm -Di_db -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags 
='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe'
    ccversion='', gccversion='4.3.0 20071210 [trunk revision 130738]', 
gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', 
lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so.1.5.8
    gnulibc_version='2.7'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, 
ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.10.0/i686-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_ITHREADS
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
  Locally applied patches:
        RC2
  Built under linux
  Compiled at Dec 10 2007 14:13:19
  @INC:
    lib
    /usr/lib/perl5/5.10.0/i686-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/lib/perl5/site_perl/5.10.0/i686-linux-thread-multi
    /usr/lib/perl5/site_perl/5.10.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.10.0/i686-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    .

Hi all,

Linux 2.6 system, glibc 2.7, gcc 4.3.0 trunk, I get

1 Embed Failure:

../lib/ExtUtils/t/Embed........................................../tmp/ccbl8h9W.o:
In function `main':
embed_test.c:(.text+0x2f): undefined reference to `Perl_sys_init3'
embed_test.c:(.text+0x14a): undefined reference to `Perl_sys_term'
collect2: ld returned 1 exit status

3 POD failures:

../lib/Pod/t/htmlescp............................................
#   Failed test 'html escape'
#   at ../lib/Pod/t/pod2html-lib.pl line 40.
../lib/Pod/t/htmlescp............................................NOK 1/1#
Looks like you failed 1 test of 1.
../lib/Pod/t/htmlescp............................................dubious
        Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
        Failed 1/1 tests, 0.00% okay
../lib/Pod/t/htmllink............................................
#   Failed test 'html links'
#   at ../lib/Pod/t/pod2html-lib.pl line 40.
../lib/Pod/t/htmllink............................................NOK 1/1#
Looks like you failed 1 test of 1.
../lib/Pod/t/htmllink............................................dubious
        Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
        Failed 1/1 tests, 0.00% okay
../lib/Pod/t/htmlview............................................
#   Failed test 'html rendering'
#   at ../lib/Pod/t/pod2html-lib.pl line 40.
../lib/Pod/t/htmlview............................................NOK 1/1#
Looks like you failed 1 test of 1.
../lib/Pod/t/htmlview............................................dubious
        Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
        Failed 1/1 tests, 0.00% okay

Any help/ideas appreciated.

Regards,
ismail

Cuma 5 May 2006 12:51 tarihinde, [email protected: y...@anon.com] şunları yazmıştı:
> try this with Firefox 1.5.0.3
> �www.gavinsharp.com/tmp/ImageVuln.html

Gives "Unknown protocol (c)" here with Firefox 1.5.0.3

--