FAQ
In what order does python import modules on a Linux system? I have a
package that is both installed in /usr/lib64/python2.5/site-packages,
and a newer version of the same module in a working directory.

I want to import the version from the working directory, but when I
print module.__file__ in the interpreter after importing the module,
I get the version that's in site-packages.

I've played with the PYTHONPATH environmental variable by setting it
to just the path of the working directory, but when I import the module
I still pick up the version in site-packages.

/usr/lib64 is in my PATH variable, but doesn't appear anywhere else. I
don't want to remove /usr/lib64 from my PATH because that will break
a lot of stuff.

Can I force python to import from my PYTHONPATH first, before looking
in the system directory?

Catherine

Search Discussions

  • Dan Stromberg at Jan 12, 2011 at 1:20 am

    On Tue, Jan 11, 2011 at 4:30 PM, Catherine Moroney wrote:
    In what order does python import modules on a Linux system? ?I have a
    package that is both installed in /usr/lib64/python2.5/site-packages,
    and a newer version of the same module in a working directory.

    I want to import the version from the working directory, but when I
    print module.__file__ in the interpreter after importing the module,
    I get the version that's in site-packages.

    I've played with the PYTHONPATH environmental variable by setting it
    to just the path of the working directory, but when I import the module
    I still pick up the version in site-packages.

    /usr/lib64 is in my PATH variable, but doesn't appear anywhere else. ?I
    don't want to remove /usr/lib64 from my PATH because that will break
    a lot of stuff.

    Can I force python to import from my PYTHONPATH first, before looking
    in the system directory?

    Catherine
    --
    http://mail.python.org/mailman/listinfo/python-list
    Please import sys and inspect sys.path; this defines the search path
    for imports.

    By looking at sys.path, you can see where in the search order your
    $PYTHONPATH is going.

    It might actually be better to give your script a command line option
    that says "Throw the following directory at the beginning of
    sys.path".
  • Ben Finney at Jan 12, 2011 at 1:26 am

    Catherine Moroney <Catherine.M.Moroney at jpl.nasa.gov> writes:

    In what order does python import modules on a Linux system?
    An import caused by a statement is done when that statement is executed.
    So the answer to that question is: in the order the statements occur in
    the execution flow.
    I have a package that is both installed in
    /usr/lib64/python2.5/site-packages, and a newer version of the same
    module in a working directory.
    Ah, you're asking about the search path for importing modules.

    You would do well to work through the entire Python tutorial
    <URL:http://docs.python.org/tutorial/> to cover the basics like that.
    Don't merely read it; actually *do* it, exploring each example and
    experimenting to understand before proceeding.

    The module import search path is part of that coverage
    <URL:http://docs.python.org/tutorial/modules.html#the-module-search-path>.
    I want to import the version from the working directory, but when I
    print module.__file__ in the interpreter after importing the module,
    I get the version that's in site-packages.
    You will be interested in the features enabled by PEP 328
    <URL:http://www.python.org/dev/peps/pep-0328/>, specifically the feature
    of relative imports.

    --
    \ ?When I turned two I was really anxious, because I'd doubled my |
    `\ age in a year. I thought, if this keeps up, by the time I'm six |
    _o__) I'll be ninety.? ?Steven Wright |
    Ben Finney
  • Catherine Moroney at Jan 12, 2011 at 7:07 pm
    I've looked at my sys.path variable and I see that it has
    a whole bunch of site-package directories, followed by the
    contents of my $PYTHONPATH variable, followed by a list of
    misc site-package variables (see below).

    I've verified that if I manually reverse the order of sys.path
    I can then import the proper version of the module that I want.
    But this is not a permanent solution for me as this will mess up
    other people are who working with the same code.

    But, I'm curious as to where the first bunch of 'site-package'
    entries come from. The
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    is not present in any of my environmental variables yet it shows up
    as one of the first entries in sys.path.

    A colleague of mine is running on the same system as I am, and he
    does not have the problem of the
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    variable showing up as one of the first entries in sys.path.

    Thanks for the education,

    Catherine

    Dan Stromberg wrote:
    On Tue, Jan 11, 2011 at 4:30 PM, Catherine Moroney
    wrote:
    In what order does python import modules on a Linux system? I have a
    package that is both installed in /usr/lib64/python2.5/site-packages,
    and a newer version of the same module in a working directory.

    I want to import the version from the working directory, but when I
    print module.__file__ in the interpreter after importing the module,
    I get the version that's in site-packages.

    I've played with the PYTHONPATH environmental variable by setting it
    to just the path of the working directory, but when I import the module
    I still pick up the version in site-packages.

    /usr/lib64 is in my PATH variable, but doesn't appear anywhere else. I
    don't want to remove /usr/lib64 from my PATH because that will break
    a lot of stuff.

    Can I force python to import from my PYTHONPATH first, before looking
    in the system directory?

    Catherine
    --
    http://mail.python.org/mailman/listinfo/python-list
    Please import sys and inspect sys.path; this defines the search path
    for imports.

    By looking at sys.path, you can see where in the search order your
    $PYTHONPATH is going.

    It might actually be better to give your script a command line option
    that says "Throw the following directory at the beginning of
    sys.path".
  • Chris Rebert at Jan 12, 2011 at 9:31 pm

    Dan Stromberg wrote:
    On Tue, Jan 11, 2011 at 4:30 PM, Catherine Moroney
    wrote:
    In what order does python import modules on a Linux system? ?I have a
    package that is both installed in /usr/lib64/python2.5/site-packages,
    and a newer version of the same module in a working directory.

    I want to import the version from the working directory, but when I
    print module.__file__ in the interpreter after importing the module,
    I get the version that's in site-packages.

    I've played with the PYTHONPATH environmental variable by setting it
    to just the path of the working directory, but when I import the module
    I still pick up the version in site-packages.

    /usr/lib64 is in my PATH variable, but doesn't appear anywhere else. ?I
    don't want to remove /usr/lib64 from my PATH because that will break
    a lot of stuff.

    Can I force python to import from my PYTHONPATH first, before looking
    in the system directory?
    Please import sys and inspect sys.path; this defines the search path
    for imports.

    By looking at sys.path, you can see where in the search order your
    $PYTHONPATH is going.
    On Wed, Jan 12, 2011 at 11:07 AM, Catherine Moroney
    wrote:
    I've looked at my sys.path variable and I see that it has
    a whole bunch of site-package directories, followed by the
    contents of my $PYTHONPATH variable, followed by a list of
    misc site-package variables (see below). <snip>
    But, I'm curious as to where the first bunch of 'site-package'
    entries come from. ?The
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    is not present in any of my environmental variables yet it shows up
    as one of the first entries in sys.path.
    You probably have a .pth file somewhere that adds it (since it's an
    egg, probably site-packages/easy-install.pth).
    See http://docs.python.org/install/index.html#modifying-python-s-search-path

    Cheers,
    Chris
  • Dan Stromberg at Jan 13, 2011 at 2:27 am
    I don't know where the site-packages directories are coming from -
    maybe a site.py or sitecustomize.py? Sometimes you can strace with a
    very large -s to see where something like this is coming from. -o is
    your friend for saving the output to a file in, EG, /tmp.

    What I usually do is to put specific versions of modules I need, that
    are different from what the OS is providing, in the same CWD as my
    script during testing. Then in my experience, CPython imports them
    instead. When it comes time for a production run, you can drop the
    special modules into the OS directories somewhere.

    However, another option is to add a --test command line argument to
    your script, and make the option just do something like:

    if '--test' in sys.argv:
    sys.path.insert(0, os.path.expanduser('~/magic-modules'))

    ...for example (and that in sys.argv test could probably be improved
    to fit your program's specifics).

    On Wed, Jan 12, 2011 at 11:07 AM, Catherine Moroney
    wrote:
    I've looked at my sys.path variable and I see that it has
    a whole bunch of site-package directories, followed by the
    contents of my $PYTHONPATH variable, followed by a list of
    misc site-package variables (see below).

    I've verified that if I manually reverse the order of sys.path
    I can then import the proper version of the module that I want.
    But this is not a permanent solution for me as this will mess up
    other people are who working with the same code.

    But, I'm curious as to where the first bunch of 'site-package'
    entries come from. ?The
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    is not present in any of my environmental variables yet it shows up
    as one of the first entries in sys.path.

    A colleague of mine is running on the same system as I am, and he
    does not have the problem of the
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    variable showing up as one of the first entries in sys.path.

    Thanks for the education,

    Catherine

    Dan Stromberg wrote:
    On Tue, Jan 11, 2011 at 4:30 PM, Catherine Moroney
    wrote:
    In what order does python import modules on a Linux system? ?I have a
    package that is both installed in /usr/lib64/python2.5/site-packages,
    and a newer version of the same module in a working directory.

    I want to import the version from the working directory, but when I
    print module.__file__ in the interpreter after importing the module,
    I get the version that's in site-packages.

    I've played with the PYTHONPATH environmental variable by setting it
    to just the path of the working directory, but when I import the module
    I still pick up the version in site-packages.

    /usr/lib64 is in my PATH variable, but doesn't appear anywhere else. ?I
    don't want to remove /usr/lib64 from my PATH because that will break
    a lot of stuff.

    Can I force python to import from my PYTHONPATH first, before looking
    in the system directory?

    Catherine
    --
    http://mail.python.org/mailman/listinfo/python-list
    Please import sys and inspect sys.path; this defines the search path
    for imports.

    By looking at sys.path, you can see where in the search order your
    $PYTHONPATH is going.

    It might actually be better to give your script a command line option
    that says "Throw the following directory at the beginning of
    sys.path".
    --
    http://mail.python.org/mailman/listinfo/python-list
  • Ben Finney at Jan 13, 2011 at 2:47 am
    (Please don't top-post replies. Instead, reply in-line with the quoted
    material, removing irrelevant material. An example is this message.)

    Catherine Moroney <Catherine.M.Moroney at jpl.nasa.gov> writes:
    I've looked at my sys.path variable and I see that it has a whole
    bunch of site-package directories, followed by the contents of my
    $PYTHONPATH variable, followed by a list of misc site-package
    variables (see below).
    Can you show the full list here? What does ?sys.path? actually contain
    when your Python interpreter is running?
    I've verified that if I manually reverse the order of sys.path I can
    then import the proper version of the module that I want.
    I think the PEP 328 relative import feature is likely to be what you
    want: you can specify that a specific import should search for the file
    relative to the current module's file, instead of from ?sys.path?.

    As described earlier, by working through the Python tutorial
    <URL:http://docs.python.org/tutorial/> you will learn about the
    different styles of import statement, including relative versus absolute
    <URL:http://docs.python.org/tutorial/modules.html>.
    But, I'm curious as to where the first bunch of 'site-package' entries
    come from. The
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    is not present in any of my environmental variables yet it shows up as
    one of the first entries in sys.path.
    That's probably there because you have used the Setuptools system (or
    something related) to install a third-party package. That is achieved by
    meddling with the start-up module search path, to make the third-party
    package available for import. The result is often a messy search path.

    Python's system for installing third-party components is far from ideal;
    it has seen a lot of improvement recently, but on the other hand is
    carrying a lot of baggage from its earlier design. This is one of the
    symptoms.
    Thanks for the education,
    I hope that helps!

    --
    \ ?To me, boxing is like a ballet, except there's no music, no |
    `\ choreography, and the dancers hit each other.? ?Jack Handey |
    _o__) |
    Ben Finney
  • Catherine Moroney at Jan 12, 2011 at 7:38 pm
    I've looked at my sys.path variable and I see that it has
    a whole bunch of site-package directories, followed by the
    contents of my $PYTHONPATH variable, followed by a list of
    misc site-package variables (see below).

    I've verified that if I manually reverse the order of sys.path
    I can then import the proper version of the module that I want.
    But this is not a permanent solution for me as this will mess up
    other people are who working with the same code.

    But, I'm curious as to where the first bunch of 'site-package'
    entries come from. The
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    is not present in any of my environmental variables yet it shows up
    as one of the first entries in sys.path.

    A colleague of mine is running on the same system as I am, and he
    does not have the problem of the
    /usr/lib64/python2.5/site-packages/pyhdfeos-1.0_r57_58-py2.5-linux-x86_64.egg
    variable showing up as one of the first entries in sys.path.

    Thanks for the education,

    Catherine

    Dan Stromberg wrote:
    On Tue, Jan 11, 2011 at 4:30 PM, Catherine Moroney
    wrote:
    In what order does python import modules on a Linux system? I have a
    package that is both installed in /usr/lib64/python2.5/site-packages,
    and a newer version of the same module in a working directory.

    I want to import the version from the working directory, but when I
    print module.__file__ in the interpreter after importing the module,
    I get the version that's in site-packages.

    I've played with the PYTHONPATH environmental variable by setting it
    to just the path of the working directory, but when I import the module
    I still pick up the version in site-packages.

    /usr/lib64 is in my PATH variable, but doesn't appear anywhere else. I
    don't want to remove /usr/lib64 from my PATH because that will break
    a lot of stuff.

    Can I force python to import from my PYTHONPATH first, before looking
    in the system directory?

    Catherine
    --
    http://mail.python.org/mailman/listinfo/python-list
    Please import sys and inspect sys.path; this defines the search path
    for imports.

    By looking at sys.path, you can see where in the search order your
    $PYTHONPATH is going.

    It might actually be better to give your script a command line option
    that says "Throw the following directory at the beginning of
    sys.path".

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJan 12, '11 at 12:30a
activeJan 13, '11 at 2:47a
posts8
users4
websitepython.org

People

Translate

site design / logo © 2022 Grokbase