FAQ
Hi,

I work for a project that has been using mod_perl for quite a while.
Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl
2.0.7 (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since
then, we've been unable to start Apache server (segmentation faults)
with same httpd.conf which worked previously.

As far as I can tell, the segfault seems to be triggered while trying to
load any Perl module using either PerlModule directive or in <Perl>
block in httpd.conf that directly or indirectly use Carp, or even just
try to load Carp itself. Strace shows that the process would segfault
while loading (or right after loading) Carp.pm. The server can still
successfully start and work when carefully removing such modules in
httpd.conf.

Is anyone seeing this same issue or have any ideas how it can be fixed
other than trying to hack through all the Perl modules to make sure Carp
is not called?


perl -V
Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

Platform:
osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp
tue jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dprefix=/ops/tools -des
-Accflags=-DPERL_RELOCATABLE_INCPUSH'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
/lib64 /usr/lib64 /usr/local/lib64
libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.12'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
-fstack-protector'


Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
Built under linux
Compiled at Aug 1 2012 16:14:39
%ENV:
PERL5LIB="/ops/tools/lib"
@INC:
/ops/tools/lib
/ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
/ops/tools/lib/perl5/site_perl/5.16.0
/ops/tools/lib/perl5/5.16.0/x86_64-linux
/ops/tools/lib/perl5/5.16.0
.

Note that we compiled Perl with relocation flag
(-DPERL_RELOCATABLE_INCPUSH) turned on, yet same issue can also be found
without the flag in our test.

Any help/advice would be appreciated, thank you in advance.


Best,
Jason

Search Discussions

  • Fred Moyer at Aug 14, 2012 at 6:18 pm
    Can you try the latest version from source?

    http://perl.apache.org/download/source.html

    It looks like you are using perl 5.16, some recent fixes were
    committed there. The subversion tip will likely be the 2.0.8 RC in a
    week or so.
    On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin wrote:
    Hi,

    I work for a project that has been using mod_perl for quite a while.
    Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
    (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then, we've
    been unable to start Apache server (segmentation faults) with same
    httpd.conf which worked previously.

    As far as I can tell, the segfault seems to be triggered while trying to
    load any Perl module using either PerlModule directive or in <Perl> block in
    httpd.conf that directly or indirectly use Carp, or even just try to load
    Carp itself. Strace shows that the process would segfault while loading (or
    right after loading) Carp.pm. The server can still successfully start and
    work when carefully removing such modules in httpd.conf.

    Is anyone seeing this same issue or have any ideas how it can be fixed other
    than trying to hack through all the Perl modules to make sure Carp is not
    called?


    perl -V
    Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

    Platform:
    osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
    uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
    jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dprefix=/ops/tools -des
    -Accflags=-DPERL_RELOCATABLE_INCPUSH'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
    gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
    lseeksize=8
    alignbytes=8, prototype=define
    Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
    /lib64 /usr/lib64 /usr/local/lib64
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.12'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
    -fstack-protector'


    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
    PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
    USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
    USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at Aug 1 2012 16:14:39
    %ENV:
    PERL5LIB="/ops/tools/lib"
    @INC:
    /ops/tools/lib
    /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/site_perl/5.16.0
    /ops/tools/lib/perl5/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/5.16.0
    .

    Note that we compiled Perl with relocation flag (-DPERL_RELOCATABLE_INCPUSH)
    turned on, yet same issue can also be found without the flag in our test.

    Any help/advice would be appreciated, thank you in advance.


    Best,
    Jason
  • Jason Lin at Aug 14, 2012 at 9:58 pm
    Hello Fred,

    Thank you for pointing out the upcoming release. Unfortunately, after
    checking out and installing the latest version from SVN, I'm still
    seeing the same issue. Apache still segfault while trying to load any
    Perl module that directly or indirectly use Carp. According to "SVN info
    -r HEAD", the reversion I'm using is 1373130.

    One thing I noticed from strace is that the module seems to be loaded
    successfully the first time ( according to mod_perl2 doc, Apache 2 will
    perform a restart right after initial start. so every module will be
    loaded twice... ), and only when it is to be loaded the second time the
    segfault occurs. Perhaps there are some differences between the first
    and second module load during apache/mod_perl initialization that's
    causing the segfault?


    Best,
    Jason
    On 08/14/2012 12:17 PM, Fred Moyer wrote:
    Can you try the latest version from source?

    http://perl.apache.org/download/source.html

    It looks like you are using perl 5.16, some recent fixes were
    committed there. The subversion tip will likely be the 2.0.8 RC in a
    week or so.
    On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin wrote:
    Hi,

    I work for a project that has been using mod_perl for quite a while.
    Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
    (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then, we've
    been unable to start Apache server (segmentation faults) with same
    httpd.conf which worked previously.

    As far as I can tell, the segfault seems to be triggered while trying to
    load any Perl module using either PerlModule directive or in <Perl> block in
    httpd.conf that directly or indirectly use Carp, or even just try to load
    Carp itself. Strace shows that the process would segfault while loading (or
    right after loading) Carp.pm. The server can still successfully start and
    work when carefully removing such modules in httpd.conf.

    Is anyone seeing this same issue or have any ideas how it can be fixed other
    than trying to hack through all the Perl modules to make sure Carp is not
    called?


    perl -V
    Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

    Platform:
    osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
    uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
    jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dprefix=/ops/tools -des
    -Accflags=-DPERL_RELOCATABLE_INCPUSH'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
    gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
    lseeksize=8
    alignbytes=8, prototype=define
    Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
    /lib64 /usr/lib64 /usr/local/lib64
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.12'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
    -fstack-protector'


    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
    PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
    USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
    USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at Aug 1 2012 16:14:39
    %ENV:
    PERL5LIB="/ops/tools/lib"
    @INC:
    /ops/tools/lib
    /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/site_perl/5.16.0
    /ops/tools/lib/perl5/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/5.16.0
    .

    Note that we compiled Perl with relocation flag (-DPERL_RELOCATABLE_INCPUSH)
    turned on, yet same issue can also be found without the flag in our test.

    Any help/advice would be appreciated, thank you in advance.


    Best,
    Jason
  • Doug Hunt at Aug 20, 2012 at 5:00 pm
    Hi Fred, modperl folks:

    We've narrowed down this bug a bit more. We are currently running perl
    5.16.1 (latest), the latest mod_perl from source and apache version
    2.2.22.

    If we run a simple httpd.conf with this line:

    PerlModule Carp

    Then single-process startup will fail on the second perl startup.
    (perl starts up twice when apache is started, as explained in
    http://perl.apache.org/docs/2.0/user/handlers/server.html).

    httpd -X
    (segmentation fault)

    We can tell that apache starts up perl successfully the first time by
    running strace. The segfault can be traced to one line in Carp.pm:

    --------------------%<----------------------------------------------------
    # Transform an argument to a function into a string.
    sub format_arg {

    ...

    # The following handling of "control chars" is direct from
    # the original code - it is broken on Unicode though.
    # Suggestions?
    is_utf8($arg) or $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
    return $arg;
    }
    --------------------%<-------------------------------------------------------

    If we comment out the line which starts 'is_utf8...' then the segfault is
    prevented. It strikes me as odd that just *compiling* that line (which is
    legal perl) should cause a segfault, but that is what we repeatedly
    observe.

    Any ideas or thoughts welcome!

    Regards,

    Doug Hunt

    dhunt@ucar.edu
    Software Engineer
    UCAR - COSMIC, Tel. (303) 497-2611
    On Tue, 14 Aug 2012, Fred Moyer wrote:

    Can you try the latest version from source?

    http://perl.apache.org/download/source.html

    It looks like you are using perl 5.16, some recent fixes were
    committed there. The subversion tip will likely be the 2.0.8 RC in a
    week or so.
    On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin wrote:
    Hi,

    I work for a project that has been using mod_perl for quite a while.
    Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
    (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then, we've
    been unable to start Apache server (segmentation faults) with same
    httpd.conf which worked previously.

    As far as I can tell, the segfault seems to be triggered while trying to
    load any Perl module using either PerlModule directive or in <Perl> block in
    httpd.conf that directly or indirectly use Carp, or even just try to load
    Carp itself. Strace shows that the process would segfault while loading (or
    right after loading) Carp.pm. The server can still successfully start and
    work when carefully removing such modules in httpd.conf.

    Is anyone seeing this same issue or have any ideas how it can be fixed other
    than trying to hack through all the Perl modules to make sure Carp is not
    called?


    perl -V
    Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

    Platform:
    osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
    uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
    jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dprefix=/ops/tools -des
    -Accflags=-DPERL_RELOCATABLE_INCPUSH'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
    gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
    lseeksize=8
    alignbytes=8, prototype=define
    Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
    /lib64 /usr/lib64 /usr/local/lib64
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.12'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
    -fstack-protector'


    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
    PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
    USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
    USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at Aug 1 2012 16:14:39
    %ENV:
    PERL5LIB="/ops/tools/lib"
    @INC:
    /ops/tools/lib
    /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/site_perl/5.16.0
    /ops/tools/lib/perl5/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/5.16.0
    .

    Note that we compiled Perl with relocation flag (-DPERL_RELOCATABLE_INCPUSH)
    turned on, yet same issue can also be found without the flag in our test.

    Any help/advice would be appreciated, thank you in advance.


    Best,
    Jason
  • Doug Hunt at Aug 20, 2012 at 8:48 pm
    Hi once more:

    I've installed a mod_perl compiled with MP_DEBUG=1 and obtained a core
    file, from which I get this stack trace when loading Carp.pm. It seems
    it segfaults when compiling the regular expression from this line
    in Carp.pm:

    is_utf8($arg) or $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;

    Does this help at all?

    Regards,

    Doug Hunt

    -------------------%<--------------------------------------------
    (gdb) where
    #0 0x0000000000513346 in Perl__invlist_union_maybe_complement_2nd ()
    #1 0x000000000051642d in S_regclass ()
    #2 0x0000000000518e38 in S_regpiece ()
    #3 0x000000000051ab30 in S_regbranch ()
    #4 0x000000000051ae2d in S_reg ()
    #5 0x0000000000518dae in S_regpiece ()
    #6 0x000000000051ab30 in S_regbranch ()
    #7 0x000000000051ae2d in S_reg ()
    #8 0x000000000051dadc in Perl_re_compile ()
    #9 0x00000000004d2d24 in Perl_pmruntime ()
    #10 0x0000000000506404 in Perl_yyparse ()
    #11 0x00000000005721b0 in S_doeval ()
    #12 0x0000000000573ea9 in Perl_pp_require ()
    #13 0x0000000000489ee1 in modperl_pp_require ()
    #14 0x000000000053fc93 in Perl_runops_standard ()
    #15 0x00000000004dd0c3 in Perl_eval_sv ()
    #16 0x000000000047c3de in modperl_require_module (pv=0x1b8a498 "Carp",
    logfailure=1)
    #17 0x0000000000475e6d in modperl_config_apply_PerlModule (s=0x1b68950,
    scfg=0x1b6ca00, perl=0x1beeb10, p=0x1b5f138)
    #18 0x000000000047322f in modperl_startup (s=0x1b68950, p=0x1b5f138)
    #19 0x0000000000472ee2 in modperl_startup (s=0x1b68950, p=0x1b5f138)
    #20 0x000000000047415b in modperl_init (base_server=0x1b68950,
    p=0x1b5f138)
    #21 0x0000000000474484 in modperl_hook_init (pconf=0x1b5f138,
    plog=0x1ba5368, ptemp=0x1b69188, s=0x1b68950)
    #22 0x0000000000464701 in ap_run_open_logs ()
    #23 0x000000000044abed in main ()
    ----------------------%<-------------------------------------------------



    dhunt@ucar.edu
    Software Engineer
    UCAR - COSMIC, Tel. (303) 497-2611
    On Mon, 20 Aug 2012, Doug Hunt wrote:

    Hi Fred, modperl folks:

    We've narrowed down this bug a bit more. We are currently running perl
    5.16.1 (latest), the latest mod_perl from source and apache version 2.2.22.

    If we run a simple httpd.conf with this line:

    PerlModule Carp

    Then single-process startup will fail on the second perl startup.
    (perl starts up twice when apache is started, as explained in
    http://perl.apache.org/docs/2.0/user/handlers/server.html).

    httpd -X
    (segmentation fault)

    We can tell that apache starts up perl successfully the first time by running
    strace. The segfault can be traced to one line in Carp.pm:

    --------------------%<----------------------------------------------------
    # Transform an argument to a function into a string.
    sub format_arg {

    ...

    # The following handling of "control chars" is direct from
    # the original code - it is broken on Unicode though.
    # Suggestions?
    is_utf8($arg) or $arg =~
    s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
    return $arg;
    }
    --------------------%<-------------------------------------------------------

    If we comment out the line which starts 'is_utf8...' then the segfault is
    prevented. It strikes me as odd that just *compiling* that line (which is
    legal perl) should cause a segfault, but that is what we repeatedly observe.

    Any ideas or thoughts welcome!

    Regards,

    Doug Hunt

    dhunt@ucar.edu
    Software Engineer
    UCAR - COSMIC, Tel. (303) 497-2611
    On Tue, 14 Aug 2012, Fred Moyer wrote:

    Can you try the latest version from source?

    http://perl.apache.org/download/source.html

    It looks like you are using perl 5.16, some recent fixes were
    committed there. The subversion tip will likely be the 2.0.8 RC in a
    week or so.
    On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin wrote:
    Hi,

    I work for a project that has been using mod_perl for quite a while.
    Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
    (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then,
    we've
    been unable to start Apache server (segmentation faults) with same
    httpd.conf which worked previously.

    As far as I can tell, the segfault seems to be triggered while trying to
    load any Perl module using either PerlModule directive or in <Perl> block
    in
    httpd.conf that directly or indirectly use Carp, or even just try to load
    Carp itself. Strace shows that the process would segfault while loading
    (or
    right after loading) Carp.pm. The server can still successfully start and
    work when carefully removing such modules in httpd.conf.

    Is anyone seeing this same issue or have any ideas how it can be fixed
    other
    than trying to hack through all the Perl modules to make sure Carp is not
    called?


    perl -V
    Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

    Platform:
    osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
    uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
    jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dprefix=/ops/tools -des
    -Accflags=-DPERL_RELOCATABLE_INCPUSH'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing
    -pipe
    -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
    gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
    lseeksize=8
    alignbytes=8, prototype=define
    Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
    /lib64 /usr/lib64 /usr/local/lib64
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.12'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
    -fstack-protector'


    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
    PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
    USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
    USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at Aug 1 2012 16:14:39
    %ENV:
    PERL5LIB="/ops/tools/lib"
    @INC:
    /ops/tools/lib
    /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/site_perl/5.16.0
    /ops/tools/lib/perl5/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/5.16.0
    .

    Note that we compiled Perl with relocation flag
    (-DPERL_RELOCATABLE_INCPUSH)
    turned on, yet same issue can also be found without the flag in our test.

    Any help/advice would be appreciated, thank you in advance.


    Best,
    Jason
  • Doug Hunt at Aug 21, 2012 at 5:03 pm
    Hi again: Tracking this down further. I noticed that when we build this
    apache/mod_perl we had turned off the 'make test' step. Attached is a
    formal bug report with debugging enabled for both mod_perl and perl. The
    test suite does not even start, as the server segfaults before the tests
    can be run.

    It still seems to be failing near Perl__invlist_union_maybe_complement_2nd
    when compiling a regular expression.

    Thanks much,

    Doug Hunt

    dhunt@ucar.edu
    Software Engineer
    UCAR - COSMIC, Tel. (303) 497-2611
    On Mon, 20 Aug 2012, Doug Hunt wrote:

    Hi once more:

    I've installed a mod_perl compiled with MP_DEBUG=1 and obtained a core file,
    from which I get this stack trace when loading Carp.pm. It seems
    it segfaults when compiling the regular expression from this line in Carp.pm:

    is_utf8($arg) or $arg =~
    s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;

    Does this help at all?

    Regards,

    Doug Hunt

    -------------------%<--------------------------------------------
    (gdb) where
    #0 0x0000000000513346 in Perl__invlist_union_maybe_complement_2nd ()
    #1 0x000000000051642d in S_regclass ()
    #2 0x0000000000518e38 in S_regpiece ()
    #3 0x000000000051ab30 in S_regbranch ()
    #4 0x000000000051ae2d in S_reg ()
    #5 0x0000000000518dae in S_regpiece ()
    #6 0x000000000051ab30 in S_regbranch ()
    #7 0x000000000051ae2d in S_reg ()
    #8 0x000000000051dadc in Perl_re_compile ()
    #9 0x00000000004d2d24 in Perl_pmruntime ()
    #10 0x0000000000506404 in Perl_yyparse ()
    #11 0x00000000005721b0 in S_doeval ()
    #12 0x0000000000573ea9 in Perl_pp_require ()
    #13 0x0000000000489ee1 in modperl_pp_require ()
    #14 0x000000000053fc93 in Perl_runops_standard ()
    #15 0x00000000004dd0c3 in Perl_eval_sv ()
    #16 0x000000000047c3de in modperl_require_module (pv=0x1b8a498 "Carp",
    logfailure=1)
    #17 0x0000000000475e6d in modperl_config_apply_PerlModule (s=0x1b68950,
    scfg=0x1b6ca00, perl=0x1beeb10, p=0x1b5f138)
    #18 0x000000000047322f in modperl_startup (s=0x1b68950, p=0x1b5f138)
    #19 0x0000000000472ee2 in modperl_startup (s=0x1b68950, p=0x1b5f138)
    #20 0x000000000047415b in modperl_init (base_server=0x1b68950, p=0x1b5f138)
    #21 0x0000000000474484 in modperl_hook_init (pconf=0x1b5f138, plog=0x1ba5368,
    ptemp=0x1b69188, s=0x1b68950)
    #22 0x0000000000464701 in ap_run_open_logs ()
    #23 0x000000000044abed in main ()
    ----------------------%<-------------------------------------------------



    dhunt@ucar.edu
    Software Engineer
    UCAR - COSMIC, Tel. (303) 497-2611
    On Mon, 20 Aug 2012, Doug Hunt wrote:

    Hi Fred, modperl folks:

    We've narrowed down this bug a bit more. We are currently running perl
    5.16.1 (latest), the latest mod_perl from source and apache version 2.2.22.

    If we run a simple httpd.conf with this line:

    PerlModule Carp

    Then single-process startup will fail on the second perl startup.
    (perl starts up twice when apache is started, as explained in
    http://perl.apache.org/docs/2.0/user/handlers/server.html).

    httpd -X
    (segmentation fault)

    We can tell that apache starts up perl successfully the first time by
    running strace. The segfault can be traced to one line in Carp.pm:

    --------------------%<----------------------------------------------------
    # Transform an argument to a function into a string.
    sub format_arg {

    ...

    # The following handling of "control chars" is direct from
    # the original code - it is broken on Unicode though.
    # Suggestions?
    is_utf8($arg) or $arg =~
    s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
    return $arg;
    }

    --------------------%<-------------------------------------------------------

    If we comment out the line which starts 'is_utf8...' then the segfault is
    prevented. It strikes me as odd that just *compiling* that line (which is
    legal perl) should cause a segfault, but that is what we repeatedly
    observe.

    Any ideas or thoughts welcome!

    Regards,

    Doug Hunt

    dhunt@ucar.edu
    Software Engineer
    UCAR - COSMIC, Tel. (303) 497-2611
    On Tue, 14 Aug 2012, Fred Moyer wrote:

    Can you try the latest version from source?

    http://perl.apache.org/download/source.html

    It looks like you are using perl 5.16, some recent fixes were
    committed there. The subversion tip will likely be the 2.0.8 RC in a
    week or so.
    On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin wrote:
    Hi,

    I work for a project that has been using mod_perl for quite a while.
    Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl
    2.0.7
    (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then,
    we've
    been unable to start Apache server (segmentation faults) with same
    httpd.conf which worked previously.

    As far as I can tell, the segfault seems to be triggered while trying to
    load any Perl module using either PerlModule directive or in <Perl> block
    in
    httpd.conf that directly or indirectly use Carp, or even just try to load
    Carp itself. Strace shows that the process would segfault while loading
    (or
    right after loading) Carp.pm. The server can still successfully start and
    work when carefully removing such modules in httpd.conf.

    Is anyone seeing this same issue or have any ideas how it can be fixed
    other
    than trying to hack through all the Perl modules to make sure Carp is not
    called?


    perl -V
    Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

    Platform:
    osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
    uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
    jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dprefix=/ops/tools -des
    -Accflags=-DPERL_RELOCATABLE_INCPUSH'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing
    -pipe
    -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
    -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
    -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
    gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
    lseeksize=8
    alignbytes=8, prototype=define
    Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
    /lib64 /usr/lib64 /usr/local/lib64
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.12'
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
    -fstack-protector'


    Characteristics of this binary (from libperl):
    Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
    PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
    PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
    USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
    USE_LOCALE_COLLATE USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
    Built under linux
    Compiled at Aug 1 2012 16:14:39
    %ENV:
    PERL5LIB="/ops/tools/lib"
    @INC:
    /ops/tools/lib
    /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/site_perl/5.16.0
    /ops/tools/lib/perl5/5.16.0/x86_64-linux
    /ops/tools/lib/perl5/5.16.0
    .

    Note that we compiled Perl with relocation flag
    (-DPERL_RELOCATABLE_INCPUSH)
    turned on, yet same issue can also be found without the flag in our test.

    Any help/advice would be appreciated, thank you in advance.


    Best,
    Jason

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmodperl @
categoriesmodperl, perl
postedAug 13, '12 at 10:01p
activeAug 21, '12 at 5:03p
posts6
users3
websiteperl.apache.org

People

Translate

site design / logo © 2021 Grokbase