[perl #66834] allow space or tab before unquoted string terminators in addition to before quoted ones

# New Ticket Created by  David Nicol 
# Please include the string:  [perl #66834]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=66834 >



This is a bug report for perl from [email protected: davi...@cpan.org],
generated with the help of perlbug 1.35 running under perl v5.8.5.


-----------------------------------------------------------------
[Please enter your report here]

I should have fixed this a decade ago, but I fixed it tonight.

Perl's commitment to allowing white space most anywhere has been
broken in the case of here-document string terminators without quotes,
such as

   print << blah;
   test
   blah

which is a syntax error, while

   print << "blah";
   test
   blah

prints "test\n" as desired.

I think I reported this a long time ago and someone agreed that it was
a bug, but I couldn't find my  report in rt so here's a new ticket.

And a patch to duplicate the existing logic for skipping spaces and tabs
before the quoted terminators for bare terminators and backslash-prefaced
terminators, consisting of two new lines in toke.c (not dependent on my
other work there, but the comment in the earlier patch does show up in
this one) and two new tests, expected to go in heredoc.t which doesn't
exist in origin yet.

Please note the use of test.pl's C<fail> method and string eval as a
method for testing syntax which without repair simply doesn't parse. Is
what I have done the standard way to do that?

diff --git a/t/op/heredoc.t b/t/op/heredoc.t
index b1ab684..607b19f 100644
--- a/t/op/heredoc.t
+++ b/t/op/heredoc.t
@@ -7,7 +7,7 @@ BEGIN {
}

use strict;
-plan(tests => 6);
+plan(tests => 8);


# heredoc without newline (#65838)
@@ -65,3 +65,19 @@ HEREDOC
         "long terminator fails correctly"
     );
}
+
+# allow whitespace before the termstring
+is( << 'WITH', <<'WITHOUT', "allow SPACE_OR_TAB before quoted terminator");
+test
+WITH
+test
+WITHOUT
+
+eval <<\SYNTAX or fail( "allow SPACE_OR_TAB before terminator");
+is( << WITH, <<WITHOUT, "allow SPACE_OR_TAB before terminator");
+test
+WITH
+test
+WITHOUT
+1;
+SYNTAX
diff --git a/toke.c b/toke.c
index 24c28c5..5cccb61 100644
--- a/toke.c
+++ b/toke.c
@@ -11281,6 +11281,8 @@ S_scan_heredoc(pTHX_ register char *s)
      s++;
     }
     else {
+        if (*peek == '\\' || isALNUM_lazy_if(peek,UTF))
+            s = peek;
  if (*s == '\\')
             /* <<\FOO is equivalent to <<'FOO' */
      s++, term = '\'';

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=low
---
This perlbug was built using Perl v5.8.5 in the Red Hat build system.
It is being executed now by Perl v5.8.5 - Mon Jul 24 18:27:47 EDT 2006.

Site configuration information for perl v5.8.5:

Configured by Red Hat, Inc. at Mon Jul 24 18:27:47 EDT 2006.

Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
  Platform:
osname=linux, osvers=2.6.9-22.18.bz155725.el, archname=s390x-linux-thread-multi
uname='linux spark.z900.redhat.com 2.6.9-22.18.bz155725.el #1 smp thu nov 17 15:25:33 est 2005 s390x s390x s390x gnulinux '
config_args='-des -Doptimize=-O2 -g -pipe -m64 -Dversion=5.8.5 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dprivlib=/usr/lib/perl5/5.8.5 -Dsitelib=/usr/lib/perl5/site_perl/5.8.5 -Dvendorlib=/usr/lib/perl5/vendor_perl/5.8.5 -Darchlib=/usr/lib64/perl5/5.8.5/s390x-linux-thread-multi -Dsitearch=/usr/lib64/perl5/site_perl/5.8.5/s390x-linux-thread-multi -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.8.5/s390x-linux-thread-multi -Darchname=s390x-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dinc_version_list=5.8.4 5.8.3 5.8.2 5.8.1 5.8.0'
    hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=define uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -g -pipe -m64',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm'
ccversion='', gccversion='3.4.6 20060404 (Red Hat 3.4.6-2)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=87654321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =''
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.3.3.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.3.4'
  Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib64/perl5/5.8.5/s390x-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared'

Locally applied patches:
    

---
@INC for perl v5.8.5:
    /usr/lib64/perl5/5.8.5/s390x-linux-thread-multi
    /usr/lib/perl5/5.8.5
    /usr/lib64/perl5/site_perl/5.8.5/s390x-linux-thread-multi
    /usr/lib64/perl5/site_perl/5.8.4/s390x-linux-thread-multi
    /usr/lib64/perl5/site_perl/5.8.3/s390x-linux-thread-multi
    /usr/lib64/perl5/site_perl/5.8.2/s390x-linux-thread-multi
    /usr/lib64/perl5/site_perl/5.8.1/s390x-linux-thread-multi
    /usr/lib64/perl5/site_perl/5.8.0/s390x-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.5
    /usr/lib/perl5/site_perl/5.8.4
    /usr/lib/perl5/site_perl/5.8.3
    /usr/lib/perl5/site_perl/5.8.2
    /usr/lib/perl5/site_perl/5.8.1
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib64/perl5/vendor_perl/5.8.5/s390x-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.8.4/s390x-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.8.3/s390x-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.8.2/s390x-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.8.1/s390x-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.8.0/s390x-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.5
    /usr/lib/perl5/vendor_perl/5.8.4
    /usr/lib/perl5/vendor_perl/5.8.3
    /usr/lib/perl5/vendor_perl/5.8.2
    /usr/lib/perl5/vendor_perl/5.8.1
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    .

---
Environment for perl v5.8.5:
    HOME=/root
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:/root/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

That is actually the documented behavior.

perldoc perlop:

    <<EOF   

       ... There must be no space between the "<<" and the
       identifier, unless the identifier is quoted.  (If you put a
       space it will be treated as a null identifier, which is valid,
       and matches the first empty line.) ...

Ronald

On Thu, Jul 2, 2009 at 8:13 PM, Ronald J Kimball<rjk-perl-p5p@tamias.net> w=
rote:
> That is actually the documented behavior.
>
> perldoc perlop:
>
> =C2=A0 =C2=A0<<EOF
>
> =C2=A0 =C2=A0 =C2=A0 ... There must be no space between the "<<" and the
> =C2=A0 =C2=A0 =C2=A0 identifier, unless the identifier is quoted. =C2=A0(=
If you put a
> =C2=A0 =C2=A0 =C2=A0 space it will be treated as a null identifier, which=
is valid,
> =C2=A0 =C2=A0 =C2=A0 and matches the first empty line.) ...
>
> Ronald

Hmm.  No hint of deprecation of the unquoted null identifier in that paragr=
aph.

Before I go and include a touch to perlop.pod in an addendum to my Big
Heredoc Omnibus Patch, is there any situation where HEREDOC WHITESPACE
BAREWORD makes sense?  I have not been able to contrive one, the
"juxtaposition operator" having been rejected in
http://dev.perl.org/perl6/doc/design/apo/A03.html

until now:

   push @pieces,  split << while (@pieces < 20)
   padding padding

is that worth saving?

also there might be variables interpolated in the doc with
Tie::Function type side effects.

Are there any situations where HEREDOC WHITESPACE BAREWORD makes sense
and BAREWORD is not in the following short list?

   and eq if for foreach ge gt le lt ne or unless until while x

Everything else always binds rightward, and there is still no way to
define infix subroutines.

What I'd like to do is check the possible terminator against that list
and treat the situation like the implied null terminator (issue an
optional syntax warning that does not mention deprecation) when there
is a match, otherwise treat the unlisted bareword as the terminator.