FAQ
# New Ticket Created by Dan Collins
# Please include the string: [perl #128170]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=128170 >


Greetings Porters,

I have compiled bleadperl with the afl-gcc compiler using:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -Dusequadmath -des
AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

After reducing testcases using `afl-tmin` and performing additional minimization by hand, I have located the following testcase that triggers an assert fail in debugging builds of the perl interpreter. The testcase is the 6-character file:

/(?<=/

On normal builds, this exits with the expected error. On debug builds, this returns an assert fail.

dcollins@nightshade64:~$ perl/miniperl -e '/(?<=/'
Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE ?<=/ at -e line 1.
dcollins@nightshade64:~$ perldebug/miniperl -e '/(?<=/'
miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) < (pRExC_state->end)' failed.
Aborted

Debugging tool output is below. A bisect was performed and points to the following diff:

cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd is the first bad commit
commit cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd
Author: Karl Williamson <khw@cpan.org>
Date: Wed Feb 10 16:27:13 2016 -0700

     regcomp.c: Fix some parsing glitches

     I undertook a code review of how regcomp.c parses things in light of the
     tickets found by the fuzzer,
     https://rt.perl.org/Ticket/Display.html?id=126546. This commit is the
     result of my efforts so far. I was not planning to push it now, but the
     work found a couple of new error messages that should be raised, and
     this has to be done before the visible changes code freeze coming up all
     too soon. I will add test cases after that freeze, including if to see
     that these changes fix all the observed issues.

     The audit was tedious, and may have missed some things. Several issues
     occurred in multiple places. One is to not advance the parse by
     UTF8SKIP appropriately; another is to subtract one byte from the parse
     and assume that that is pointing to the beginning of the previous
     character (which under UTF-8 it may not). Another is to assume that
     that the pattern is a C string, that there are no interior NULs in it.
     I also found unnecessary tests, given that an SV always has a
     terminating NUL.

:040000 040000 a58a2d3154c5e346f9dfab14f7ca9897cbe06cfb 41b83ee5e555d3ddc87935c3a3b30db821741214 M pod
:100644 100644 c00fdffd93dd9de69326e9b567d912dd9a1bee3f aa06bc6e8da12440f5d5b649cd5faa24a8853cc7 M regcomp.c
bisect run success



**GDB**

dcollins@nightshade64:~$ gdb --args perldebug/miniperl -e '/(?<=/'
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from perldebug/miniperl...done.
(gdb) run
Starting program: /home/dcollins/perldebug/miniperl -e /\(\?\<=/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) < (pRExC_state->end)' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff6cfa8fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6cf23a7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff6cf2452 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000006efe09 in S_reg (pRExC_state=0x7fffffffdcf0, paren=60, flagp=0x7fffffffd8bc, depth=5) at regcomp.c:10621
#5 0x00000000007186ed in S_regatom (pRExC_state=pRExC_state@entry=0x7fffffffdcf0, flagp=flagp@entry=0x7fffffffda34, depth=depth@entry=4) at regcomp.c:12354
#6 0x0000000000725d0d in S_regpiece (depth=<optimized out>, flagp=<synthetic pointer>, pRExC_state=0x7fffffffdcf0) at regcomp.c:11421
#7 S_regbranch (pRExC_state=pRExC_state@entry=0x7fffffffdcf0, flagp=flagp@entry=0x7fffffffdacc, first=first@entry=1, depth=depth@entry=2) at regcomp.c:11346
#8 0x0000000000745097 in S_reg (pRExC_state=pRExC_state@entry=0x7fffffffdcf0, flagp=flagp@entry=0x7fffffffdbbc, depth=1, paren=0) at regcomp.c:11084
#9 0x000000000077b231 in Perl_re_op_compile (patternp=<optimized out>, pat_count=<optimized out>, expr=<optimized out>, eng=0xf33160 <PL_core_reg_engine>, old_re=0x0,
     is_bare_re=<optimized out>, orig_rx_flags=0, pm_flags=0) at regcomp.c:7027
#10 0x00000000004c4033 in Perl_pmruntime (o=0x1202718, expr=0x12026d8, repl=0x0, isreg=isreg@entry=true, floor=<optimized out>) at op.c:5632
#11 0x000000000064b5cd in Perl_yyparse (gramtype=gramtype@entry=258) at perly.y:1028
#12 0x0000000000512111 in S_parse_body (xsinit=0xec1b00 <xs_init>, env=0x0) at perl.c:2365
#13 perl_parse (my_perl=<optimized out>, xsinit=xsinit@entry=0xec1b00 <xs_init>, argc=<optimized out>, argv=<optimized out>, env=env@entry=0x0) at perl.c:1681
#14 0x000000000040f838 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at miniperlmain.c:120
(gdb) f 4
#4 0x00000000006efe09 in S_reg (pRExC_state=0x7fffffffdcf0, paren=60, flagp=0x7fffffffd8bc, depth=5) at regcomp.c:10621
10621 assert(RExC_parse < RExC_end);
(gdb) info locals
is_logical = false
seqstart = <optimized out>
endptr = 0x0
has_intervening_patws = false
ret = 0x0
br = <optimized out>
lastbr = <optimized out>
ender = 0x0
parno = 0
flags = 0
oregflags = 0
have_branch = false
is_open = false
freeze_paren = 0
after_freeze = 0
num = <optimized out>
parse_start = 0x1202861 "?<="
oregcomp_parse = 0x1202861 "?<="
re_debug_flags = 0
(gdb) l
10616 goto capturing_parens;
10617 }
10618 RExC_seen |= REG_LOOKBEHIND_SEEN;
10619 RExC_in_lookbehind++;
10620 RExC_parse++;
10621 assert(RExC_parse < RExC_end);
10622 /* FALLTHROUGH */
10623 case '=': /* (?=...) */
10624 RExC_seen_zerolen++;
10625 break;
(gdb)


**PERL -V**

dcollins@nightshade64:~/perl$ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 25 subversion 1) configuration:
   Commit id: 8255316ac82c0fbd7ae49f2f9a217ce063d7bbfb
   Platform:
     osname=linux, osvers=4.5.0-2-amd64, archname=x86_64-linux-ld
     uname='linux nightshade64 4.5.0-2-amd64 #1 smp debian 4.5.3-2 (2016-05-08) x86_64 gnulinux '
     config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache afl-gcc -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -des'
     hint=recommended, useposix=true, d_sigaction=define
     useithreads=undef, usemultiplicity=undef
     use64bitint=define, use64bitall=define, uselongdouble=define
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='ccache afl-gcc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
     optimize='-g',
     cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
     ccversion='', gccversion='6.1.0', gccosandvers=''
     intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
     ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
     alignbytes=16, prototype=define
   Linker and Libraries:
     ld='ccache afl-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
     libpth=/usr/local/lib /usr/local/lib/gcc/x86_64-pc-linux-gnu/6.1.0/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
     libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
     perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
     libc=libc-2.22.so, so=so, useshrplib=false, libperl=libperl.a
     gnulibc_version='2.22'
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
     cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl):
   Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
                         PERL_DONT_CREATE_GVSV
                         PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
                         PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL
                         USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES
                         USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
                         USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
                         USE_PERLIO USE_PERL_ATOF
   Built under linux
   Compiled at May 16 2016 07:45:27
   @INC:
     lib
     /usr/local/perl-afl/lib/site_perl/5.25.1/x86_64-linux-ld
     /usr/local/perl-afl/lib/site_perl/5.25.1
     /usr/local/perl-afl/lib/5.25.1/x86_64-linux-ld
     /usr/local/perl-afl/lib/5.25.1
     /usr/local/perl-afl/lib/site_perl/5.24.0
     /usr/local/perl-afl/lib/site_perl
     .

Search Discussions

  • Demerphq at May 21, 2016 at 4:32 am

    On 17 May 2016 3:04 p.m., "Dan Collins" wrote:
    # New Ticket Created by Dan Collins
    # Please include the string: [perl #128170]
    # in the subject line of all future correspondence about this issue.
    # <URL: https://rt.perl.org/Ticket/Display.html?id=128170 >


    Greetings Porters,

    I have compiled bleadperl with the afl-gcc compiler using:

    ./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache
    afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly
    -Uman1dir -Uman3dir -Dusequadmath -des
    AFL_HARDEN=1 make && make test

    And then fuzzed the resulting binary using:

    AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

    After reducing testcases using `afl-tmin` and performing additional
    minimization by hand, I have located the following testcase that triggers
    an assert fail in debugging builds of the perl interpreter. The testcase is
    the 6-character file:
    /(?<=/

    On normal builds, this exits with the expected error. On debug builds,
    this returns an assert fail.
    dcollins@nightshade64:~$ perl/miniperl -e '/(?<=/'
    Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE ?<=/ at -e line 1.
    dcollins@nightshade64:~$ perldebug/miniperl -e '/(?<=/'
    miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) <
    (pRExC_state->end)' failed.
    Aborted

    Debugging tool output is below. A bisect was performed and points to the
    following diff:
    cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd is the first bad commit
    commit cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd
    Author: Karl Williamson <khw@cpan.org>
    Date: Wed Feb 10 16:27:13 2016 -0700

    regcomp.c: Fix some parsing glitches

    I undertook a code review of how regcomp.c parses things in light of the
    tickets found by the fuzzer,
    https://rt.perl.org/Ticket/Display.html?id=126546. This commit is the
    result of my efforts so far. I was not planning to push it now, but the
    work found a couple of new error messages that should be raised, and
    this has to be done before the visible changes code freeze coming up all
    too soon. I will add test cases after that freeze, including if to see
    that these changes fix all the observed issues.

    The audit was tedious, and may have missed some things. Several issues
    occurred in multiple places. One is to not advance the parse by
    UTF8SKIP appropriately; another is to subtract one byte from the parse
    and assume that that is pointing to the beginning of the previous
    character (which under UTF-8 it may not). Another is to assume that
    that the pattern is a C string, that there are no interior NULs in it.
    I also found unnecessary tests, given that an SV always has a
    terminating NUL.
    This is not correct. Xs code can and in practice does create sv's with no
    trailing nulls.
    :040000 040000 a58a2d3154c5e346f9dfab14f7ca9897cbe06cfb
    41b83ee5e555d3ddc87935c3a3b30db821741214 M pod
    :100644 100644 c00fdffd93dd9de69326e9b567d912dd9a1bee3f
    aa06bc6e8da12440f5d5b649cd5faa24a8853cc7 M regcomp.c
    bisect run success



    **GDB**

    dcollins@nightshade64:~$ gdb --args perldebug/miniperl -e '/(?<=/'
    GNU gdb (GDB) 7.10
    Copyright (C) 2015 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <
    http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-unknown-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from perldebug/miniperl...done.
    (gdb) run
    Starting program: /home/dcollins/perldebug/miniperl -e /\(\?\<=/
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) <
    (pRExC_state->end)' failed.
    Program received signal SIGABRT, Aborted.
    0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    (gdb) bt
    #0 0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    #1 0x00007ffff6cfa8fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
    #2 0x00007ffff6cf23a7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    #3 0x00007ffff6cf2452 in __assert_fail () from
    /lib/x86_64-linux-gnu/libc.so.6
    #4 0x00000000006efe09 in S_reg (pRExC_state=0x7fffffffdcf0, paren=60,
    flagp=0x7fffffffd8bc, depth=5) at regcomp.c:10621
    #5 0x00000000007186ed in S_regatom (pRExC_state=pRExC_state@entry=0x7fffffffdcf0,
    flagp=flagp@entry=0x7fffffffda34, depth=depth@entry=4) at regcomp.c:12354
    #6 0x0000000000725d0d in S_regpiece (depth=<optimized out>,
    flagp=<synthetic pointer>, pRExC_state=0x7fffffffdcf0) at regcomp.c:11421
    #7 S_regbranch (pRExC_state=pRExC_state@entry=0x7fffffffdcf0,
    flagp=flagp@entry=0x7fffffffdacc, first=first@entry=1, depth=depth@entry=2)
    at regcomp.c:11346
    #8 0x0000000000745097 in S_reg (pRExC_state=pRExC_state@entry=0x7fffffffdcf0,
    flagp=flagp@entry=0x7fffffffdbbc, depth=1, paren=0) at regcomp.c:11084
    #9 0x000000000077b231 in Perl_re_op_compile (patternp=<optimized out>,
    pat_count=<optimized out>, expr=<optimized out>, eng=0xf33160
    <PL_core_reg_engine>, old_re=0x0,
    is_bare_re=<optimized out>, orig_rx_flags=0, pm_flags=0) at
    regcomp.c:7027
    #10 0x00000000004c4033 in Perl_pmruntime (o=0x1202718, expr=0x12026d8,
    repl=0x0, isreg=isreg@entry=true, floor=<optimized out>) at op.c:5632
    #11 0x000000000064b5cd in Perl_yyparse (gramtype=gramtype@entry=258) at
    perly.y:1028
    #12 0x0000000000512111 in S_parse_body (xsinit=0xec1b00 <xs_init>,
    env=0x0) at perl.c:2365
    #13 perl_parse (my_perl=<optimized out>, xsinit=xsinit@entry=0xec1b00
    <xs_init>, argc=<optimized out>, argv=<optimized out>, env=env@entry=0x0)
    at perl.c:1681
    #14 0x000000000040f838 in main (argc=<optimized out>, argv=<optimized
    out>, env=<optimized out>) at miniperlmain.c:120
    (gdb) f 4
    #4 0x00000000006efe09 in S_reg (pRExC_state=0x7fffffffdcf0, paren=60,
    flagp=0x7fffffffd8bc, depth=5) at regcomp.c:10621
    10621 assert(RExC_parse < RExC_end);
    (gdb) info locals
    is_logical = false
    seqstart = <optimized out>
    endptr = 0x0
    has_intervening_patws = false
    ret = 0x0
    br = <optimized out>
    lastbr = <optimized out>
    ender = 0x0
    parno = 0
    flags = 0
    oregflags = 0
    have_branch = false
    is_open = false
    freeze_paren = 0
    after_freeze = 0
    num = <optimized out>
    parse_start = 0x1202861 "?<="
    oregcomp_parse = 0x1202861 "?<="
    re_debug_flags = 0
    (gdb) l
    10616 goto capturing_parens;
    10617 }
    10618 RExC_seen |= REG_LOOKBEHIND_SEEN;
    10619 RExC_in_lookbehind++;
    10620 RExC_parse++;
    10621 assert(RExC_parse < RExC_end);
    10622 /* FALLTHROUGH */
    10623 case '=': /* (?=...) */
    10624 RExC_seen_zerolen++;
    10625 break;
    (gdb)


    **PERL -V**

    dcollins@nightshade64:~/perl$ ./perl -Ilib -V
    Summary of my perl5 (revision 5 version 25 subversion 1) configuration:
    Commit id: 8255316ac82c0fbd7ae49f2f9a217ce063d7bbfb
    Platform:
    osname=linux, osvers=4.5.0-2-amd64, archname=x86_64-linux-ld
    uname='linux nightshade64 4.5.0-2-amd64 #1 smp debian 4.5.3-2
    (2016-05-08) x86_64 gnulinux '
    config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache
    afl-gcc -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly -Uman1dir
    -Uman3dir -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    use64bitint=define, use64bitall=define, uselongdouble=define
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='ccache afl-gcc', ccflags ='-fwrapv -fno-strict-aliasing -pipe
    -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-g',
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong
    -I/usr/local/include'
    ccversion='', gccversion='6.1.0', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678,
    doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16,
    longdblkind=3
    ivtype='long', ivsize=8, nvtype='long double', nvsize=16,
    Off_t='off_t', lseeksize=8
    alignbytes=16, prototype=define
    Linker and Libraries:
    ld='ccache afl-gcc', ldflags =' -fstack-protector-strong
    -L/usr/local/lib'
    libpth=/usr/local/lib
    /usr/local/lib/gcc/x86_64-pc-linux-gnu/6.1.0/include-fixed
    /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib
    /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.22'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib
    -fstack-protector-strong'

    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
    PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL
    USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES
    USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
    USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at May 16 2016 07:45:27
    @INC:
    lib
    /usr/local/perl-afl/lib/site_perl/5.25.1/x86_64-linux-ld
    /usr/local/perl-afl/lib/site_perl/5.25.1
    /usr/local/perl-afl/lib/5.25.1/x86_64-linux-ld
    /usr/local/perl-afl/lib/5.25.1
    /usr/local/perl-afl/lib/site_perl/5.24.0
    /usr/local/perl-afl/lib/site_perl
    .
  • Karl Williamson at May 23, 2016 at 8:07 pm

    On 05/20/2016 10:32 PM, demerphq wrote:
    On 17 May 2016 3:04 p.m., "Dan Collins" wrote:

    # New Ticket Created by Dan Collins
    # Please include the string: [perl #128170]
    # in the subject line of all future correspondence about this issue.
    # <URL: https://rt.perl.org/Ticket/Display.html?id=128170 >


    Greetings Porters,

    I have compiled bleadperl with the afl-gcc compiler using:

    ./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache
    afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly
    -Uman1dir -Uman3dir -Dusequadmath -des
    AFL_HARDEN=1 make && make test

    And then fuzzed the resulting binary using:

    AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

    After reducing testcases using `afl-tmin` and performing additional
    minimization by hand, I have located the following testcase that
    triggers an assert fail in debugging builds of the perl interpreter. The
    testcase is the 6-character file:
    /(?<=/

    On normal builds, this exits with the expected error. On debug
    builds, this returns an assert fail.
    dcollins@nightshade64:~$ perl/miniperl -e '/(?<=/'
    Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE ?<=/ at -e line 1.
    dcollins@nightshade64:~$ perldebug/miniperl -e '/(?<=/'
    miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) <
    (pRExC_state->end)' failed.
    Aborted

    Debugging tool output is below. A bisect was performed and points to
    the following diff:
    cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd is the first bad commit
    commit cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd
    Author: Karl Williamson <khw@cpan.org > Date: Wed Feb 10 16:27:13 2016 -0700

    regcomp.c: Fix some parsing glitches

    I undertook a code review of how regcomp.c parses things in light of the
    tickets found by the fuzzer,
    https://rt.perl.org/Ticket/Display.html?id=126546. This commit is the
    result of my efforts so far. I was not planning to push it now, but the
    work found a couple of new error messages that should be raised, and
    this has to be done before the visible changes code freeze coming up all
    too soon. I will add test cases after that freeze, including if to see
    that these changes fix all the observed issues.

    The audit was tedious, and may have missed some things. Several issues
    occurred in multiple places. One is to not advance the parse by
    UTF8SKIP appropriately; another is to subtract one byte from the parse
    and assume that that is pointing to the beginning of the previous
    character (which under UTF-8 it may not). Another is to assume that
    that the pattern is a C string, that there are no interior NULs in it.
    I also found unnecessary tests, given that an SV always has a
    terminating NUL.
    This is not correct. Xs code can and in practice does create sv's with
    no trailing nulls.
    Hmm. I've written a bunch of code with that assumption, partly because
    a bunch of code that was already in place has that assumption.

    I did add assertions to this effect in the regex code at the same time
    as this commit was done.

    Can you give an example of how XS could do this? All I can think of is
    if it arbitrarily turns on a POK flag.
    :040000 040000 a58a2d3154c5e346f9dfab14f7ca9897cbe06cfb
    41b83ee5e555d3ddc87935c3a3b30db821741214 M pod
    :100644 100644 c00fdffd93dd9de69326e9b567d912dd9a1bee3f
    aa06bc6e8da12440f5d5b649cd5faa24a8853cc7 M regcomp.c
    bisect run success



    **GDB**

    dcollins@nightshade64:~$ gdb --args perldebug/miniperl -e '/(?<=/'
    GNU gdb (GDB) 7.10
    Copyright (C) 2015 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later
    <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-unknown-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from perldebug/miniperl...done.
    (gdb) run
    Starting program: /home/dcollins/perldebug/miniperl -e /\(\?\<=/
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library
    "/lib/x86_64-linux-gnu/libthread_db.so.1".
    miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) <
    (pRExC_state->end)' failed.
    Program received signal SIGABRT, Aborted.
    0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    (gdb) bt
    #0 0x00007ffff6cf9478 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    #1 0x00007ffff6cfa8fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
    #2 0x00007ffff6cf23a7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
    #3 0x00007ffff6cf2452 in __assert_fail () from
    /lib/x86_64-linux-gnu/libc.so.6
    #4 0x00000000006efe09 in S_reg (pRExC_state=0x7fffffffdcf0,
    paren=60, flagp=0x7fffffffd8bc, depth=5) at regcomp.c:10621
    #5 0x00000000007186ed in S_regatom
    (pRExC_state=pRExC_state@entry=0x7fffffffdcf0,
    flagp=flagp@entry=0x7fffffffda34, depth=depth@entry=4) at regcomp.c:12354
    #6 0x0000000000725d0d in S_regpiece (depth=<optimized out>,
    flagp=<synthetic pointer>, pRExC_state=0x7fffffffdcf0) at regcomp.c:11421
    #7 S_regbranch (pRExC_state=pRExC_state@entry=0x7fffffffdcf0,
    flagp=flagp@entry=0x7fffffffdacc, first=first@entry=1,
    depth=depth@entry=2) at regcomp.c:11346
    #8 0x0000000000745097 in S_reg
    (pRExC_state=pRExC_state@entry=0x7fffffffdcf0,
    flagp=flagp@entry=0x7fffffffdbbc, depth=1, paren=0) at regcomp.c:11084
    #9 0x000000000077b231 in Perl_re_op_compile (patternp=<optimized
    out>, pat_count=<optimized out>, expr=<optimized out>, eng=0xf33160
    <PL_core_reg_engine>, old_re=0x0,
    is_bare_re=<optimized out>, orig_rx_flags=0, pm_flags=0) at
    regcomp.c:7027
    #10 0x00000000004c4033 in Perl_pmruntime (o=0x1202718,
    expr=0x12026d8, repl=0x0, isreg=isreg@entry=true, floor=<optimized out>)
    at op.c:5632
    #11 0x000000000064b5cd in Perl_yyparse (gramtype=gramtype@entry=258)
    at perly.y:1028
    #12 0x0000000000512111 in S_parse_body (xsinit=0xec1b00 <xs_init>,
    env=0x0) at perl.c:2365
    #13 perl_parse (my_perl=<optimized out>, xsinit=xsinit@entry=0xec1b00
    <xs_init>, argc=<optimized out>, argv=<optimized out>,
    env=env@entry=0x0) at perl.c:1681
    #14 0x000000000040f838 in main (argc=<optimized out>, argv=<optimized
    out>, env=<optimized out>) at miniperlmain.c:120
    (gdb) f 4
    #4 0x00000000006efe09 in S_reg (pRExC_state=0x7fffffffdcf0,
    paren=60, flagp=0x7fffffffd8bc, depth=5) at regcomp.c:10621
    10621 assert(RExC_parse < RExC_end);
    (gdb) info locals
    is_logical = false
    seqstart = <optimized out>
    endptr = 0x0
    has_intervening_patws = false
    ret = 0x0
    br = <optimized out>
    lastbr = <optimized out>
    ender = 0x0
    parno = 0
    flags = 0
    oregflags = 0
    have_branch = false
    is_open = false
    freeze_paren = 0
    after_freeze = 0
    num = <optimized out>
    parse_start = 0x1202861 "?<="
    oregcomp_parse = 0x1202861 "?<="
    re_debug_flags = 0
    (gdb) l
    10616 goto capturing_parens;
    10617 }
    10618 RExC_seen |= REG_LOOKBEHIND_SEEN;
    10619 RExC_in_lookbehind++;
    10620 RExC_parse++;
    10621 assert(RExC_parse < RExC_end);
    10622 /* FALLTHROUGH */
    10623 case '=': /* (?=...) */
    10624 RExC_seen_zerolen++;
    10625 break;
    (gdb)


    **PERL -V**

    dcollins@nightshade64:~/perl$ ./perl -Ilib -V
    Summary of my perl5 (revision 5 version 25 subversion 1) configuration:
    Commit id: 8255316ac82c0fbd7ae49f2f9a217ce063d7bbfb
    Platform:
    osname=linux, osvers=4.5.0-2-amd64, archname=x86_64-linux-ld
    uname='linux nightshade64 4.5.0-2-amd64 #1 smp debian 4.5.3-2
    (2016-05-08) x86_64 gnulinux '
    config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=ccache
    afl-gcc -Duselongdouble -Duse64bitall -Doptimize=-g -Uversiononly
    -Uman1dir -Uman3dir -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    use64bitint=define, use64bitall=define, uselongdouble=define
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='ccache afl-gcc', ccflags ='-fwrapv -fno-strict-aliasing -pipe
    -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-g',
    cppflags='-fwrapv -fno-strict-aliasing -pipe
    -fstack-protector-strong -I/usr/local/include'
    ccversion='', gccversion='6.1.0', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8,
    byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define,
    longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='long double', nvsize=16,
    Off_t='off_t', lseeksize=8
    alignbytes=16, prototype=define
    Linker and Libraries:
    ld='ccache afl-gcc', ldflags =' -fstack-protector-strong
    -L/usr/local/lib'
    libpth=/usr/local/lib
    /usr/local/lib/gcc/x86_64-pc-linux-gnu/6.1.0/include-fixed
    /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib
    /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so <http://libc-2.22.so>, so=so, useshrplib=false,
    libperl=libperl.a
    gnulibc_version='2.22'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -g -L/usr/local/lib
    -fstack-protector-strong'

    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
    PERL_MALLOC_WRAP
    PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL
    USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES
    USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_LOCALE_TIME
    USE_LONG_DOUBLE
    USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at May 16 2016 07:45:27
    @INC:
    lib
    /usr/local/perl-afl/lib/site_perl/5.25.1/x86_64-linux-ld
    /usr/local/perl-afl/lib/site_perl/5.25.1
    /usr/local/perl-afl/lib/5.25.1/x86_64-linux-ld
    /usr/local/perl-afl/lib/5.25.1
    /usr/local/perl-afl/lib/site_perl/5.24.0
    /usr/local/perl-afl/lib/site_perl
    .
  • Demerphq at May 23, 2016 at 9:16 pm

    On 23 May 2016 16:06, "Karl Williamson" wrote:
    On 05/20/2016 10:32 PM, demerphq wrote:


    On 17 May 2016 3:04 p.m., "Dan Collins" <perlbug-followup@perl.org
    wrote:
    # New Ticket Created by Dan Collins
    # Please include the string: [perl #128170]
    # in the subject line of all future correspondence about this issue.
    # <URL: https://rt.perl.org/Ticket/Display.html?id=128170 >


    Greetings Porters,

    I have compiled bleadperl with the afl-gcc compiler using:

    ./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache
    afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly
    -Uman1dir -Uman3dir -Dusequadmath -des
    AFL_HARDEN=1 make && make test

    And then fuzzed the resulting binary using:

    AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

    After reducing testcases using `afl-tmin` and performing additional
    minimization by hand, I have located the following testcase that
    triggers an assert fail in debugging builds of the perl interpreter. The
    testcase is the 6-character file:
    /(?<=/

    On normal builds, this exits with the expected error. On debug
    builds, this returns an assert fail.
    dcollins@nightshade64:~$ perl/miniperl -e '/(?<=/'
    Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE ?<=/ at -e line 1.
    dcollins@nightshade64:~$ perldebug/miniperl -e '/(?<=/'
    miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) <
    (pRExC_state->end)' failed.
    Aborted

    Debugging tool output is below. A bisect was performed and points to
    the following diff:
    cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd is the first bad commit
    commit cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd
    Author: Karl Williamson <khw@cpan.org >
    Date: Wed Feb 10 16:27:13 2016 -0700

    regcomp.c: Fix some parsing glitches

    I undertook a code review of how regcomp.c parses things in light of the
    tickets found by the fuzzer,
    https://rt.perl.org/Ticket/Display.html?id=126546. This commit is the
    result of my efforts so far. I was not planning to push it now, but the
    work found a couple of new error messages that should be raised,
    and
    this has to be done before the visible changes code freeze coming up all
    too soon. I will add test cases after that freeze, including if to see
    that these changes fix all the observed issues.

    The audit was tedious, and may have missed some things. Several issues
    occurred in multiple places. One is to not advance the parse by
    UTF8SKIP appropriately; another is to subtract one byte from the parse
    and assume that that is pointing to the beginning of the previous
    character (which under UTF-8 it may not). Another is to assume
    that
    that the pattern is a C string, that there are no interior NULs in it.
    I also found unnecessary tests, given that an SV always has a
    terminating NUL.
    This is not correct. Xs code can and in practice does create sv's with
    no trailing nulls.

    Hmm. I've written a bunch of code with that assumption, partly because a
    bunch of code that was already in place has that assumption.
    I did add assertions to this effect in the regex code at the same time as
    this commit was done.
    Can you give an example of how XS could do this? All I can think of is
    if it arbitrarily turns on a POK flag.

    Yeah, exactly. But in xs that is pretty standard. I wouldnt rely on our
    strings always having a trailing null.

    Yves
  • Father Chrysostomos via RT at May 23, 2016 at 10:46 pm

    On Mon May 23 14:16:31 2016, demerphq wrote:
    Yeah, exactly. But in xs that is pretty standard. I wouldnt rely on our
    strings always having a trailing null.
    In fact, I would like to remove that requirement (to allow substr to avoid copying the string), but someone else has pointed out that it is also common in XS to pass SvPV to system functions that expect a trailing null.

    --

    Father Chrysostomos


    ---
    via perlbug: queue: perl5 status: open
    https://rt.perl.org/Ticket/Display.html?id=128170
  • Demerphq at May 23, 2016 at 11:06 pm

    On 23 May 2016 6:46 p.m., "Father Chrysostomos via RT" wrote:
    On Mon May 23 14:16:31 2016, demerphq wrote:
    Yeah, exactly. But in xs that is pretty standard. I wouldnt rely on our
    strings always having a trailing null.
    In fact, I would like to remove that requirement (to allow substr to
    avoid copying the string), but someone else has pointed out that it is also
    common in XS to pass SvPV to system functions that expect a trailing null.

    Maybe we should have a SvPV_null macro that DTRT for such cases and abandon
    any requirements that pv's are null terminated.

    Yves
  • Father Chrysostomos via RT at May 23, 2016 at 11:52 pm

    On Mon May 23 16:07:03 2016, demerphq wrote:
    On 23 May 2016 6:46 p.m., "Father Chrysostomos via RT" <
    perlbug-followup@perl.org> wrote:
    On Mon May 23 14:16:31 2016, demerphq wrote:
    Yeah, exactly. But in xs that is pretty standard. I wouldnt rely on our
    strings always having a trailing null.
    In fact, I would like to remove that requirement (to allow substr to
    avoid copying the string), but someone else has pointed out that it is also
    common in XS to pass SvPV to system functions that expect a trailing null.

    Maybe we should have a SvPV_null macro that DTRT for such cases and abandon
    any requirements that pv's are null terminated.
    It might have to be the other way round (SvPV guarantees a null; a new variant returns the actual string), for backward-compatibility. I have opened a separate ticket for this (#128226), since it is not directly related to the topic of this ticket.


    --

    Father Chrysostomos


    ---
    via perlbug: queue: perl5 status: open
    https://rt.perl.org/Ticket/Display.html?id=128170
  • Karl Williamson at May 24, 2016 at 4:17 am

    On 05/23/2016 03:15 PM, demerphq wrote:
    On 23 May 2016 16:06, "Karl Williamson" wrote:
    On 05/20/2016 10:32 PM, demerphq wrote:


    On 17 May 2016 3:04 p.m., "Dan Collins" <perlbug-followup@perl.org
    wrote:
    # New Ticket Created by Dan Collins
    # Please include the string: [perl #128170]
    # in the subject line of all future correspondence about this issue.
    # <URL: https://rt.perl.org/Ticket/Display.html?id=128170 >


    Greetings Porters,

    I have compiled bleadperl with the afl-gcc compiler using:

    ./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc='ccache
    afl-gcc' -Uuselongdouble -Duse64bitall -Doptimize=-g -Uversiononly
    -Uman1dir -Uman3dir -Dusequadmath -des
    AFL_HARDEN=1 make && make test

    And then fuzzed the resulting binary using:

    AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @@

    After reducing testcases using `afl-tmin` and performing additional
    minimization by hand, I have located the following testcase that
    triggers an assert fail in debugging builds of the perl interpreter. The
    testcase is the 6-character file:
    /(?<=/

    On normal builds, this exits with the expected error. On debug
    builds, this returns an assert fail.
    dcollins@nightshade64:~$ perl/miniperl -e '/(?<=/'
    Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE ?<=/ at -e line 1.
    dcollins@nightshade64:~$ perldebug/miniperl -e '/(?<=/'
    miniperl: regcomp.c:10621: S_reg: Assertion `(pRExC_state->parse) <
    (pRExC_state->end)' failed.
    Aborted

    Debugging tool output is below. A bisect was performed and points to
    the following diff:
    cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd is the first bad commit
    commit cfbef7dc3bfb89e4ed2c00ea9c9e3bcfd0b170fd
    Author: Karl Williamson <khw@cpan.org <mailto:khw@cpan.org >>
    Date: Wed Feb 10 16:27:13 2016 -0700

    regcomp.c: Fix some parsing glitches

    I undertook a code review of how regcomp.c parses things in light of the
    tickets found by the fuzzer,
    https://rt.perl.org/Ticket/Display.html?id=126546. This commit
    is the
    result of my efforts so far. I was not planning to push it now, but the
    work found a couple of new error messages that should be
    raised, and
    this has to be done before the visible changes code freeze coming up all
    too soon. I will add test cases after that freeze, including if to see
    that these changes fix all the observed issues.

    The audit was tedious, and may have missed some things. Several issues
    occurred in multiple places. One is to not advance the parse by
    UTF8SKIP appropriately; another is to subtract one byte from the parse
    and assume that that is pointing to the beginning of the previous
    character (which under UTF-8 it may not). Another is to
    assume that
    that the pattern is a C string, that there are no interior NULs in it.
    I also found unnecessary tests, given that an SV always has a
    terminating NUL.
    This is not correct. Xs code can and in practice does create sv's with
    no trailing nulls.

    Hmm. I've written a bunch of code with that assumption, partly
    because a bunch of code that was already in place has that assumption.
    I did add assertions to this effect in the regex code at the same
    time as this commit was done.
    Can you give an example of how XS could do this? All I can think of
    is if it arbitrarily turns on a POK flag.

    Yeah, exactly. But in xs that is pretty standard. I wouldnt rely on our
    strings always having a trailing null.

    Yves
    I'm having trouble understanding how one would set such a thing up. At
    some point one would have to populate a sequence of bytes. I don't see
    how to populate anything but a short sequence unless the SV is already a
    PV. Could you outline roughly how it would be done?
  • Father Chrysostomos via RT at May 24, 2016 at 5:13 am

    On Mon May 23 21:18:16 2016, public@khwilliamson.com wrote:
    I'm having trouble understanding how one would set such a thing up.
    At
    some point one would have to populate a sequence of bytes. I don't
    see
    how to populate anything but a short sequence unless the SV is already
    a
    PV. Could you outline roughly how it would be done?
    http://grep.cpan.me/?q=SvPVX.*+%3D[+]

    --

    Father Chrysostomos


    ---
    via perlbug: queue: perl5 status: open
    https://rt.perl.org/Ticket/Display.html?id=128170

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupperl5-porters @
categoriesperl
postedMay 17, '16 at 10:03p
activeMay 24, '16 at 5:13a
posts9
users3
websiteperl.org

People

Translate

site design / logo © 2022 Grokbase