FAQ

[P5P] Patch 5.14.1: Solaris /proc/self/exe and relocatable libperl.so

Search Discussions

10 responses

  • Paul Johnson at Sep 9, 2011 at 12:26 am

    On Thu, Sep 08, 2011 at 11:28:47PM +0000, Johann 'Myrkraverk' Oskarsson wrote:
    Hi all,

    Here are two patches that enhance relocatable Perl 5.14.1 on Solaris.
    I've not checked if there are already equivalents in the latest
    development version so apologies if these are duplicate efforts.

    The first patch adds support for /proc/self/path/a.out which is
    equivalent to /proc/self/exe on Linux. This patch is complete and can
    be applied.
    Thanks very much for looking into this. Do you know whether the problems I
    noted in
    http://www.nntp.perl.org/group/perl.perl5.porters/2009/03/msg145214.html are
    still a concern?

    --
    Paul Johnson - paul@pjcj.net
    http://www.pjcj.net
  • Johann 'Myrkraverk' Oskarsson at Sep 9, 2011 at 12:59 am

    Paul Johnson writes:

    On Thu, Sep 08, 2011 at 11:28:47PM +0000, Johann 'Myrkraverk'
    Oskarsson wrote:
    The first patch adds support for /proc/self/path/a.out which is
    equivalent to /proc/self/exe on Linux. This patch is complete and can
    be applied.
    Thanks very much for looking into this. Do you know whether the
    problems I noted in
    http://www.nntp.perl.org/group/perl.perl5.porters/2009/03/msg145214.html
    are still a concern?
    I'll look into it.

    1. Without the configpm patch, why do the tests pass on linux (and
    presumably BSD) but not on Solaris?

    This might beyond my expertise. I'll run the tests on Linux and
    Solaris and see what I'll get.

    Can you tell me what the configpm part of the patch is doing? I may be
    building Perl but I don't use it myself.

    2. Should the configpm patch have been a part of the original patch
    (1d230ada), and something additional needs to be done to install in
    the correct location?

    It seemed to me that the Configure script starts to use ... as a
    directory name on subsequent runs. This and other reasons is why I
    kept nuking the build directory while getting the escape for $ORIGIN
    right. Do you remember if you started the build that installed in
    ... from a clean directory or not?

    3. Does anyone have any time and appropriate OSes to look further into
    this? I'm afraid my tuits have run out for now.

    I can test on Solaris and Linux, not FreeBSD (or other BSDs).


    --
    Johann Oskarsson http://www.2ndquadrant.com/ |[]
    PostgreSQL Development, 24x7 Support, Training and Services --+--
    Blog: http://my.opera.com/myrkraverk/blog/
  • Paul Johnson at Sep 10, 2011 at 6:15 pm

    On Fri, Sep 09, 2011 at 12:59:32AM +0000, Johann 'Myrkraverk' Oskarsson wrote:
    It seemed to me that the Configure script starts to use ... as a
    directory name on subsequent runs. This and other reasons is why I
    kept nuking the build directory while getting the escape for $ORIGIN
    right. Do you remember if you started the build that installed in
    ... from a clean directory or not?
    I'm afraid I don't, but if I were to guess I would say tht it probably wasn't.

    I don't have access to a Solaris machine anymore, but I'm glad you're looking
    into this and just wanted to point you to where I had previously got stuck.

    Thanks again,

    --
    Paul Johnson - paul@pjcj.net
    http://www.pjcj.net
  • Nicholas Clark at Sep 12, 2011 at 9:12 pm

    On Fri, Sep 09, 2011 at 12:59:32AM +0000, Johann 'Myrkraverk' Oskarsson wrote:
    Paul Johnson <paul@pjcj.net> writes:
    On Thu, Sep 08, 2011 at 11:28:47PM +0000, Johann 'Myrkraverk'
    Oskarsson wrote:
    The first patch adds support for /proc/self/path/a.out which is
    equivalent to /proc/self/exe on Linux. This patch is complete and can
    be applied.
    Thanks very much for looking into this. Do you know whether the
    problems I noted in
    http://www.nntp.perl.org/group/perl.perl5.porters/2009/03/msg145214.html
    are still a concern?
    I'll look into it.

    1. Without the configpm patch, why do the tests pass on linux (and
    presumably BSD) but not on Solaris?

    This might beyond my expertise. I'll run the tests on Linux and
    Solaris and see what I'll get.
    That they pass in one place and not the other doesn't make sense to me.

    I don't have access to Solaris. Without it, I really can't dig further.
    Can you tell me what the configpm part of the patch is doing? I may be
    building Perl but I don't use it myself.
    It's a list of which keys in %Config could have ... in them.
    As installbin can have ... in it, it should be in the list.

    It's used to generate the list of keys that actually have ... in them, to
    be used in lib/Config_heavy.pl

    If you look at lib/Config_heavy.pl you'll see that it has (roughly) config.sh
    embedded in it, with entries that need relocating starting '...'

    The list is used to generate the code to relocate them:

    foreach my $what (qw(prefixexp archlibexp man1direxp man3direxp privlibexp scriptdirexp sitearchexp sitebinexp sitelibexp siteman1direxp siteman3direxp sitescriptexp siteprefixexp sitelib_stem installarchlib installman1dir installman3dir installprefixexp installprivlib installscript installsitearch installsitebin installsitelib installsiteman1dir installsiteman3dir installsitescript)) {
    s/^($what=)(['"])(.*?)\2/$1 . $2 . relocate_inc($3) . $2/me;
    }

    where relocate_inc() is defined in lib/Config.pm as

    sub relocate_inc {
    my $libdir = shift;
    return $libdir unless $libdir =~ s!^\.\.\./!!;
    my $prefix = $^X;
    if ($prefix =~ s!/[^/]*$!!) {
    while ($libdir =~ m!^\.\./!) {
    # Loop while $libdir starts "../" and $prefix still has a trailing
    # directory
    last unless $prefix =~ s!/([^/]+)$!!;
    # but bail out if the directory we picked off the end of $prefix is .
    # or ..
    if ($1 eq '.' or $1 eq '..') {
    # Undo! This should be rare, hence code it this way rather than a
    # check each time before the s!!! above.
    $prefix = "$prefix/$1";
    last;
    }
    # Remove that leading ../ and loop again
    substr ($libdir, 0, 3, ''); } $libdir = "$prefix/$libdir"; } $libdir;
    }

    3. Does anyone have any time and appropriate OSes to look further into
    this? I'm afraid my tuits have run out for now.

    I can test on Solaris and Linux, not FreeBSD (or other BSDs).
    I can make time, but only have access to Linux, FreeBSD and 1 slowish OpenBSD
    machine.

    Nicholas Clark
  • H.Merijn Brand at Sep 9, 2011 at 6:21 am

    On Thu, 08 Sep 2011 23:28:47 +0000, "Johann 'Myrkraverk' Oskarsson" wrote:

    Hi all,

    Here are two patches that enhance relocatable Perl 5.14.1 on Solaris.
    I've not checked if there are already equivalents in the latest
    development version so apologies if these are duplicate efforts.

    The first patch adds support for /proc/self/path/a.out which is
    equivalent to /proc/self/exe on Linux. This patch is complete and can
    be applied.
    Looks safe. I'll wait a bit for more comments before I can do the magic

    My first question however is: How much of this can be done in hints and
    how much is really Solaris specific? If something actually is
    solaris-only, I'd prefer to delegate to hints/solaris.sh if possible
    Currently there is only a hints/solaris_2.sh, but you will see that
    there are many os's already that have hints for specific os releases
    (e.g. aix,irix, isc, next, sco, sunos, and svr)
    The next patch is more of a crude hack to get the Perl executables to
    use $ORIGIN in the runpath so that libperl.so can be both dynamic and
    relocatable.
    That part bothers me a bit in that the first hunk makes me think that
    freebsd and linux will also refer to $ORIGIN, but I don't see that
    happen in the second hunk, but this might be hidden in below remarks :)
    There are several issues to resolve before it can be applied:

    1. It is Solaris only at the moment, while both Linux and FreeBSD
    and maybe other Unixen have the same feature.

    2. Binaries embedding Perl will not have the right path to
    libperl.so unless they are installed in the same directory,
    or equivalent depth as the perl binaries themselves.

    3. Users of ExtUtils::Embed will need to take this into account
    and possibly add their own custom runpath.

    4. The patch as written escapes the $ sign in $ORIGIN suitable
    for Makefiles. If ExtUtils::Embed users use some other build
    tool it may fail. This brings us back to number 3.

    5. The code will need to be refactored to better suit the Configure
    script. Suitable question and appropriate warnings are missing.

    I do hope $ORIGIN support can be added and I'm not against doing the
    gruntwork for it to happen.
    --
    H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
    using 5.00307 through 5.14 and porting perl5.15.x on HP-UX 10.20, 11.00,
    11.11, 11.23 and 11.31, OpenSuSE 10.1, 11.0 .. 11.4 and AIX 5.2 and 5.3.
    http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
    http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
  • Johann 'Myrkraverk' Oskarsson at Sep 10, 2011 at 3:07 pm

    "H.Merijn Brand" <h.m.brand@xs4all.nl> writes:

    The first patch adds support for /proc/self/path/a.out which is
    equivalent to /proc/self/exe on Linux. This patch is complete and
    can be applied.
    Looks safe. I'll wait a bit for more comments before I can do the
    magic

    My first question however is: How much of this can be done in hints
    and how much is really Solaris specific? If something actually is
    solaris-only, I'd prefer to delegate to hints/solaris.sh if possible
    Currently there is only a hints/solaris_2.sh, but you will see that
    there are many os's already that have hints for specific os releases
    (e.g. aix,irix, isc, next, sco, sunos, and svr)
    In theory any system can implement /proc/self/path/a.out, just as
    /proc/self/exe. ISTM that the question here is: Do we move this to
    the Solaris hints file and risk missing it on other systems? If yes,
    I'm willing to work on a patch for that.
    The next patch is more of a crude hack to get the Perl executables
    to use $ORIGIN in the runpath so that libperl.so can be both
    dynamic and relocatable.
    That part bothers me a bit in that the first hunk makes me think
    that freebsd and linux will also refer to $ORIGIN, but I don't see
    that happen in the second hunk, but this might be hidden in below
    remarks :)
    Yes, it was hidden in the later remarks. I have found mention of
    $ORIGIN as early as 2.7 in the ld.so.1 man page. Are older Solaris
    releases still actively supported? On the other hand it should be
    easy enough to condition it on not SunOS 5.[0-6].

    I have no idea when it cropped up in FreeBSD and Linux.

    Another thing, instead of testing for userelocatableinc inside the
    system specific code of setting the runpath switch it is better to set
    shrpdir or archlibexp to $ORIGIN first. This will make that part of
    the patch system-independent.

    Another variable useorigininshrpdir (or more aptly named) can then be
    added to the appropriate hints files.

    Documenting this with the appriopriate warnings to users of
    ExtUtils::Embed is also needed. I'm not sure I'm up to that without
    some help.

    Also, ExtUtils::Embed might need the "feature" to calculate the delta
    between $ORIGIN and libperl.so since not all executables will end up
    in the same place perl will. Or always give an absolute path unless
    explicitly requested for $ORIGIN. Here is my output.

    % perl -MExtUtils::Embed -e ldopts
    -R\$$ORIGIN/../../lib/5.14.1/i86pc-solaris-thread-multi-64/CORE
    ...

    which is appropriate for Solaris Make.

    ldopts :sh= perl -MExtUtils::Embed -e ldopts

    origin:
    @echo ${ldopts}

    However, this particular quoting (\$$) does not seem to work in Gnu
    Make. Or maybe I'm just using it wrong.

    Once I have a good idea what to do about ExtUtils::Embed I'm willing
    to work on patches for it.


    --
    Johann Oskarsson http://www.2ndquadrant.com/ |[]
    PostgreSQL Development, 24x7 Support, Training and Services --+--
    Blog: http://my.opera.com/myrkraverk/blog/
  • Nicholas Clark at Sep 12, 2011 at 9:16 pm

    On Fri, Sep 09, 2011 at 08:21:10AM +0200, H.Merijn Brand wrote:

    My first question however is: How much of this can be done in hints and
    how much is really Solaris specific? If something actually is
    solaris-only, I'd prefer to delegate to hints/solaris.sh if possible
    Currently there is only a hints/solaris_2.sh, but you will see that
    there are many os's already that have hints for specific os releases
    (e.g. aix,irix, isc, next, sco, sunos, and svr)
    Currently Configure has:

    if $issymlink /proc/self/exe ; then
    $ls -l /proc/self/exe > reflect
    if $contains /`basename $ls` reflect >/dev/null 2>&1; then
    echo "You have Linux-like /proc/self/exe."
    procselfexe='"/proc/self/exe"'
    val="$define"
    fi
    fi
    if $issymlink /proc/curproc/file ; then
    $ls -l /proc/curproc/file > reflect
    if $contains /`basename $ls` reflect >/dev/null 2>&1; then
    echo "You have BSD-like /proc/curproc/file."
    procselfexe='"/proc/curproc/file"'
    val="$define"
    fi
    fi

    The patch seems to add a third copy of the same thing.

    Can't this become a loop, testing 3 possible names for the symlink?
    [Possibly it should have been a loop already - whilst it's a nice detail
    if it comes for free, I don't think that it's that useful to know that it's
    Linux-like, BSD-like or Solaris-like. And copy-paste isn't free.

    Nicholas Clark
  • Johann 'Myrkraverk' Oskarsson at Sep 13, 2011 at 10:00 pm
  • Nicholas Clark at Sep 12, 2011 at 9:29 pm

    On Thu, Sep 08, 2011 at 11:28:47PM +0000, Johann 'Myrkraverk' Oskarsson wrote:

    The next patch is more of a crude hack to get the Perl executables to
    use $ORIGIN in the runpath so that libperl.so can be both dynamic and
    relocatable.
    There are several issues to resolve before it can be applied:
    2. Binaries embedding Perl will not have the right path to
    libperl.so unless they are installed in the same directory, or
    equivalent depth as the perl binaries themselves.

    3. Users of ExtUtils::Embed will need to take this into account and
    possibly add their own custom runpath.

    4. The patch as written escapes the $ sign in $ORIGIN suitable for
    Makefiles. If ExtUtils::Embed users use some other build tool it may
    fail. This brings us back to number 3.

    5. The code will need to be refactored to better suit the Configure
    script. Suitable question and appropriate warnings are missing.

    I do hope $ORIGIN support can be added and I'm not against doing the
    gruntwork for it to happen.
    Capable volunteers are always welcome.

    What I'm missing is the use case for this.

    The intent of -Duserelocatableinc was to provide a way to make a
    self-contained perl installation that would run and find its libraries
    correctly without needing any (hacky) fixup. This allows an "application"
    to package itself up with a perl interpreter, and run wherever its unpacked.

    I'm having trouble seeing how "relocatable" (by which I'm thinking "might be
    moved") is useful at the same time as "provides a libperl.so for other
    things to use", given that the other things have to know how to find
    libperl.so.

    ie, why is a "relocatable" libperl.so useful to other programs?
    What does it give them over just linking to a "relocatable" libperl.a?

    Nicholas Clark
  • Johann 'Myrkraverk' Oskarsson at Sep 13, 2011 at 1:36 am

    Nicholas Clark writes:

    On Thu, Sep 08, 2011 at 11:28:47PM +0000, Johann 'Myrkraverk'
    Oskarsson wrote:
    I do hope $ORIGIN support can be added and I'm not against doing
    the gruntwork for it to happen.
    Capable volunteers are always welcome.

    What I'm missing is the use case for this.
    This originally came up as I was packaging a relocatable libperl for
    PostgreSQL's PL/Perl.

    A blog on making Postgres relocatable:

    http://bit.ly/p9T2aV

    Another entry on embedding a relocatable Perl with it might be coming
    up soon.

    [Further remarks snipped]

    The merit of a dynamic libperl for applications embedding it is of
    course debatable. I believe the support should be there so
    builders/packagers can make the choice.


    --
    Johann Oskarsson http://www.2ndquadrant.com/ |[]
    PostgreSQL Development, 24x7 Support, Training and Services --+--
    Blog: http://my.opera.com/myrkraverk/blog/

Related Discussions

Discussion Navigation
viewthread | post