FAQ
Here's a toy example. I just want to make a function that efficiently
uses (here just returns) something from a module. Compare:
def foo():
... from math import pi
... return pi
...
import dis
dis.dis(foo)
0 SET_LINENO 1

3 SET_LINENO 2
6 LOAD_CONST 1 (('pi',))
9 IMPORT_NAME 0 (math)
12 IMPORT_FROM 1 (pi)
15 STORE_FAST 0 (pi)
18 POP_TOP

19 SET_LINENO 3
22 LOAD_FAST 0 (pi)
25 RETURN_VALUE
26 LOAD_CONST 0 (None)
29 RETURN_VALUE

and the result of this:
def mkfoo():
... from math import pi
... def foo():
... return pi
... return foo
...
foo2 = mkfoo()
dis.dis(foo2)
0 SET_LINENO 3

3 SET_LINENO 4
6 LOAD_DEREF 0 (pi)
9 RETURN_VALUE
10 LOAD_CONST 0 (None)
13 RETURN_VALUE

It seems foo2 is doing a lot less work, unless LOAD_DEREF is very bad, which I wouldn't think.
Is/should-there-be a syntax for accomplishing this more naturally?

What would be best Pythonic practice for getting a foo2? It seems like if there were
a way to say "do this part once at compile time" we could get this kind of thing more directly.
e.g.,

def foo3():
if __compiling__: # not real code, obviously
from math import pi
return pi

Using a module imported to global space is easy but not as efficient,
and it clutters the global name space needlessly:
import math
def bar():
... return math.pi
...
dis.dis(bar)
0 SET_LINENO 1

3 SET_LINENO 2
6 LOAD_GLOBAL 0 (math)
9 LOAD_ATTR 1 (pi)
12 RETURN_VALUE
13 LOAD_CONST 0 (None)
16 RETURN_VALUE

Regards,
Bengt Richter

Search Discussions

  • Greg Ewing at Nov 13, 2002 at 1:32 am

    Bengt Richter wrote:

    def mkfoo():
    ... from math import pi
    ... def foo():
    ... return pi
    ... return foo
    ...

    What would be best Pythonic practice for getting a foo2? It seems like if there were
    a way to say "do this part once at compile time"

    It's not really compile-time that we want to do it,
    but module load time, or perhaps function definition
    time.

    One way would be to use a default-argument hack:

    import math

    def foo(m = math):
    return m.pi

    but this has all the usual drawbacks of the default-argument
    hack.

    This suggests that perhaps there should be a way
    of getting something that is just like a default
    argument, except that it's not part of the argument
    list. Maybe

    def foo():
    const m = math
    return m.pi

    The "const" statement would be evaluated when
    the "def" was executed, and "m" would appear in the
    local namespace when the function was called.

    Although "const" might not be the best name for
    it, since it's not really a constant, more like
    a local with an initial value.

    --
    Greg Ewing, Computer Science Dept,
    University of Canterbury,
    Christchurch, New Zealand
    http://www.cosc.canterbury.ac.nz/~greg
  • Neal Norwitz at Nov 13, 2002 at 3:03 am

    On Tue, 12 Nov 2002 20:32:02 -0500, Greg Ewing wrote:

    Bengt Richter wrote:
    def mkfoo():
    ... from math import pi
    ... def foo():
    ... return pi
    ... return foo
    ...

    What would be best Pythonic practice for getting a foo2? It seems like
    if there were a way to say "do this part once at compile time"

    It's not really compile-time that we want to do it, but module load
    time, or perhaps function definition time.

    One way would be to use a default-argument hack:

    import math

    def foo(m = math):
    return m.pi

    but this has all the usual drawbacks of the default-argument hack.

    This suggests that perhaps there should be a way of getting something
    that is just like a default argument, except that it's not part of the
    argument list. Maybe

    def foo():
    const m = math
    return m.pi

    The "const" statement would be evaluated when the "def" was executed,
    and "m" would appear in the local namespace when the function was
    called.

    Although "const" might not be the best name for it, since it's not
    really a constant, more like a local with an initial value.
    You can *kinda* do this with 2.1 (at least) and up:
    import math
    def foo(): return foo.pi
    ...
    foo.pi = math.pi
    foo()
    3.1415926535897931

    I realize this isn't exactly what you are asking for,
    but it is a way that works today.

    Neal
  • Bengt Richter at Nov 13, 2002 at 5:18 pm

    On Tue, 12 Nov 2002 22:03:18 -0500, Neal Norwitz wrote:
    On Tue, 12 Nov 2002 20:32:02 -0500, Greg Ewing wrote:

    Bengt Richter wrote:
    def mkfoo():
    ... from math import pi
    ... def foo():
    ... return pi
    ... return foo
    ...

    What would be best Pythonic practice for getting a foo2? It seems like
    if there were a way to say "do this part once at compile time"

    It's not really compile-time that we want to do it, but module load
    time, or perhaps function definition time.

    One way would be to use a default-argument hack:

    import math

    def foo(m = math):
    return m.pi

    but this has all the usual drawbacks of the default-argument hack.

    This suggests that perhaps there should be a way of getting something
    that is just like a default argument, except that it's not part of the
    argument list. Maybe

    def foo():
    const m = math
    return m.pi

    The "const" statement would be evaluated when the "def" was executed,
    and "m" would appear in the local namespace when the function was
    called.

    Although "const" might not be the best name for it, since it's not
    really a constant, more like a local with an initial value.
    You can *kinda* do this with 2.1 (at least) and up:
    import math
    def foo(): return foo.pi
    ...
    foo.pi = math.pi
    foo()
    3.1415926535897931

    I realize this isn't exactly what you are asking for,
    but it is a way that works today.
    Thanks. It does answer part of the requirement. I.e., you can release the math
    module with del math, but ... it's dependent on a persistent global name
    binding to foo. I.e.,
    bar = foo
    del foo
    bar()
    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 1, in foo
    NameError: global name 'foo' is not defined
    dis.dis(bar)
    0 SET_LINENO 1

    3 SET_LINENO 1
    6 LOAD_GLOBAL 0 (foo)
    9 LOAD_ATTR 1 (pi)
    12 RETURN_VALUE
    13 LOAD_CONST 0 (None)
    16 RETURN_VALUE

    plus the global lookup plus attribute lookup is not optimally fast.

    Regards,
    Bengt Richter
  • David Eppstein at Nov 13, 2002 at 5:32 pm
    In article <aqu1hd$tjk$0 at 216.39.172.122>, bokr at oz.net (Bengt Richter)
    wrote:
    import math
    def foo(): return foo.pi
    ...
    foo.pi = math.pi
    foo()
    3.1415926535897931

    I realize this isn't exactly what you are asking for,
    but it is a way that works today.
    Thanks. It does answer part of the requirement. I.e., you can release the math
    module with del math, but ... it's dependent on a persistent global name
    binding to foo.
    How about:
    import math
    def foo(pi = math.pi): return pi
    bar = foo
    del math
    del foo
    bar()
    3.1415926535897931

    --
    David Eppstein UC Irvine Dept. of Information & Computer Science
    eppstein at ics.uci.edu http://www.ics.uci.edu/~eppstein/
  • Greg Ewing at Nov 14, 2002 at 12:00 am

    Neal Norwitz wrote:

    You can *kinda* do this with 2.1 (at least) and up:
    import math
    def foo(): return foo.pi
    ...
    foo.pi = math.pi
    foo()
    3.1415926535897931

    But that defeats the purpose, which presumably was to
    avoid a global name lookup!

    --
    Greg Ewing, Computer Science Dept,
    University of Canterbury,
    Christchurch, New Zealand
    http://www.cosc.canterbury.ac.nz/~greg
  • Lexy Zhitenev at Nov 13, 2002 at 9:20 am

    "Greg Ewing" wrote:
    ... Maybe

    def foo():
    const m = math
    return m.pi

    The "const" statement would be evaluated when
    the "def" was executed, and "m" would appear in the
    local namespace when the function was called.

    Although "const" might not be the best name for
    it, since it's not really a constant, more like
    a local with an initial value.
    What is this 'const' name? I haven't ever heard of this in Python.
  • Gerhard Häring at Nov 13, 2002 at 9:33 am

    Lexy Zhitenev wrote in comp.lang.python:
    "Greg Ewing" wrote:
    ... Maybe

    def foo():
    const m = math
    return m.pi

    The "const" statement would be evaluated when
    the "def" was executed, and "m" would appear in the
    local namespace when the function was called.

    Although "const" might not be the best name for
    it, since it's not really a constant, more like
    a local with an initial value.
    What is this 'const' name? I haven't ever heard of this in Python.
    It doesn't exist. Greg was talking about a possible language
    extension.

    -- Gerhard
  • Erik Max Francis at Nov 13, 2002 at 9:54 am

    Lexy Zhitenev wrote:

    What is this 'const' name? I haven't ever heard of this in Python.
    It isn't anything; he's proposing a new feature.

    --
    Erik Max Francis / max at alcyone.com / http://www.alcyone.com/max/
    __ San Jose, CA, USA / 37 20 N 121 53 W / &tSftDotIotE
    / \ The woman's movement is no longer a cause but a symptom.
    \__/ Joan Didion
    Bosskey.net: Aliens vs. Predator 2 / http://www.bosskey.net/avp2/
    A personal guide to Aliens vs. Predator 2.
  • Bengt Richter at Nov 13, 2002 at 5:02 pm

    On Wed, 13 Nov 2002 14:32:02 +1300, Greg Ewing wrote:
    Bengt Richter wrote:
    def mkfoo():
    ... from math import pi
    ... def foo():
    ... return pi
    ... return foo
    ...

    What would be best Pythonic practice for getting a foo2? It seems like if there were
    a way to say "do this part once at compile time"

    It's not really compile-time that we want to do it,
    but module load time, or perhaps function definition
    time.
    Well, ok, I was speaking loosely of compile-time as the time between reading source
    and when final executable byte codes are laid down, but of course it's more complex
    than that (especially with Python), as mkfoo incidentally demonstrates.

    But the general idea is to find points in the sequence where one could intercede with
    some meta-operations to modify code that gets executed at the end of the cascade.

    You mention load time and function definition time. (Class definition time would seem
    to similar, with metaclasses aready providing much control). Also, execfile time has
    some things in common with module load time. Is there a unified "-time" concept re both?
    Are there different possibly useful intercession points in loading .py vs .pyc?

    What other "-times" can be clearly identified? If we had macros, I'm sure there would
    be a "macro-interpretation time" (read-time?) associated with that. (BTW, how about times
    associated with encoding transformations of source and i/o?)

    But, getting back to foo/mkfoo, the intent was to set up values for efficient access
    and to release unneeded references as soon as possible, compiling (in a narrower sense)
    the source effectively partitioned into code and metacode for execution at appropriate
    separate times to yield a clean foo function.
    One way would be to use a default-argument hack:
    Which would be a definition-time binding, I guess you would say?
    import math

    def foo(m = math):
    return m.pi

    but this has all the usual drawbacks of the default-argument
    hack.
    Plus that version retains a reference to math, where I'd like to think that
    I would wind up with just a reference to a double with the 3.14.. value. I.e.,
    def foo(pi = math.pi):
    return pi

    (not that I want to pursue the default-argument hack, just to indicate that
    whatever the methodology, I don't think the generated code should be forcing persistence
    of anything it doesn't really need).
    This suggests that perhaps there should be a way
    of getting something that is just like a default
    argument, except that it's not part of the argument
    list. Maybe

    def foo():
    const m = math
    return m.pi

    The "const" statement would be evaluated when
    the "def" was executed, and "m" would appear in the
    local namespace when the function was called.

    Although "const" might not be the best name for
    it, since it's not really a constant, more like
    a local with an initial value.
    Yes. Also I think I would like a keyword that governs
    a block scope instead of just the scope of an assignment. E.g.,
    (resisting the temptation of discovering a general mechanism here ;-)

    def foo():
    predef:
    import math
    pi = math.pi
    del math
    seven = 7
    return pi, seven

    meaning the predef block would be done once at define-time, and you
    could do whatever you wanted to predefine the local environment that
    would be seen at the first call (execution-time). You could also delete
    unneeded bindings as above. This kind of flexibility would be hard if not
    impossible to get with 'const' & such. I'm not sure whether a postdef:
    section would be of use, but if so these things should probably be
    subsumed under a single keyword (e.g. "meta" for "meta predef: and "meta postdef" ?)
    especially if one anticipated other contexts of use as well.

    The predef block looks like a kind of meta-code, and I would think the
    compiler could generate a code block for that separate from the code of
    the function body that defines what is called at execution time (foo), loosely
    corresponding to mkfoo and foo code in the example, not a single body with
    a special conditional part and references hanging on to things beyond their
    intial and only use.

    Regards,
    Bengt Richter
  • Greg Ewing at Nov 14, 2002 at 12:09 am

    Bengt Richter wrote:

    Plus that version retains a reference to math, where I'd like to think that
    I would wind up with just a reference to a double with the 3.14.. value. I.e.,
    def foo(pi = math.pi):
    return pi

    Yes, that's probably what I should have written.
    Also I think I would like a keyword that governs
    a block scope instead of just the scope of an assignment. E.g.,

    def foo():
    predef:
    import math
    pi = math.pi
    del math
    seven = 7
    return pi, seven

    Too complicated. You could get the same effect by putting the
    code outside the def:

    from math import math_pi

    def foo():
    const pi = math_pi
    const seven = 7
    ...

    del math_pi

    --
    Greg Ewing, Computer Science Dept,
    University of Canterbury,
    Christchurch, New Zealand
    http://www.cosc.canterbury.ac.nz/~greg
  • Bengt Richter at Nov 14, 2002 at 9:29 am

    On Thu, 14 Nov 2002 13:09:04 +1300, Greg Ewing wrote:
    Bengt Richter wrote:
    Plus that version retains a reference to math, where I'd like to think that
    I would wind up with just a reference to a double with the 3.14.. value. I.e.,
    def foo(pi = math.pi):
    return pi

    Yes, that's probably what I should have written.
    Also I think I would like a keyword that governs
    a block scope instead of just the scope of an assignment. E.g.,

    def foo():
    predef:
    import math
    pi = math.pi
    del math
    seven = 7
    return pi, seven

    Too complicated. You could get the same effect by putting the
    code outside the def:

    from math import math_pi
    did you mean
    from math import pi as math_pi
    ?
    def foo():
    const pi = math_pi
    const seven = 7
    ...

    del math_pi
    Not quite the same effect: you are introducing temporary global bindings.
    And ISTM your version really only looks less complicated because you moved
    two statements out of the local scope.

    To avoid the global temp, I suppose you could write

    def foo():
    const pi = [__import__('math')][0].pi
    const seven = 7

    but down that road lies much abuse, trying to cram block scope intentions
    into expression scope. I don't think it's possible to anticipate what
    people will want to do, except that people will tend to work around limitations
    in ugly ways if there's no clean way to express their intent. So I'd rather
    provide a freer "scope" ;-)

    BTW, I'd suggest "preset" as closer to the semantic mark than "const" if your
    proposal gets adopted. Maybe my block introducer should similarly be "presets:".
    Then for minimal one-liner usage, it would look concise and similar

    def foo():
    presets: seven = 7
    ...
    vs.

    def foo():
    const seven = 7
    ...

    or
    def foo():
    preset seven = 7
    ...

    I like the option of simple multiple values on a single line too.

    def foo():
    presets: seven = 7; eight = 2**3
    ...

    Regards,
    Bengt Richter
  • Bengt Richter at Nov 14, 2002 at 11:08 am
    On 14 Nov 2002 09:29:21 GMT, bokr at oz.net (Bengt Richter) wrote:
    [...]
    To avoid the global temp, I suppose you could write

    def foo():
    const pi = [__import__('math')][0].pi
    or, less ridiculously,
    const pi = __import__('math').pi
    ;-/
    const seven = 7
    Regards,
    Bengt Richter
  • Aahz at Nov 16, 2002 at 7:18 am

    In article <aqrig4$d2h$0 at 216.39.172.122>, Bengt Richter wrote:
    Here's a toy example. I just want to make a function that efficiently
    uses (here just returns) something from a module.
    Can you explain a bit more clearly what the purpose of this?
    --
    Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/

    A: No.
    Q: Is top-posting okay?
  • Jeff Epler at Nov 16, 2002 at 2:29 pm

    In article <aqrig4$d2h$0 at 216.39.172.122>, Bengt Richter wrote:
    Here's a toy example. I just want to make a function that efficiently
    uses (here just returns) something from a module.
    On Sat, Nov 16, 2002 at 02:18:41AM -0500, Aahz wrote:
    Can you explain a bit more clearly what the purpose of this?
    Speed, I assume.

    f0 0.945003986359 # empty loop
    f1 2.456807971 # math.pi
    f2 1.20684301853 # (local) pi

    LOAD_FAST is bound to be faster than LOAD_GLOBAL + LOAD_ATTR.

    Jeff


    import math, time

    R = range(1000000)
    def f0():
    for i in R: pass

    def f1():
    for i in R: math.pi

    def f2(pi=math.pi):
    for i in R: pi

    for f in (f0, f1, f2):
    t0 = time.time()
    f()
    t1 = time.time()
    print f.__name__, t1-t0
  • Bengt Richter at Nov 16, 2002 at 4:19 pm

    On 16 Nov 2002 02:18:41 -0500, aahz at pythoncraft.com (Aahz) wrote:
    In article <aqrig4$d2h$0 at 216.39.172.122>, Bengt Richter wrote:
    Here's a toy example. I just want to make a function that efficiently
    uses (here just returns) something from a module.
    Can you explain a bit more clearly what the purpose of this?
    In a word, optimization.

    Both speed and space.

    The reason I focused on an item from a module is that typical usage
    is to import the relevant module to make a global binding (e.g., globalmodulename)
    and then to refer to the desired item as globalmodulename.desireditem.

    This is a fine mechanism when you want dynamic access to the current value of
    something that may be getting updated, but if you want the value that was bound at the
    time you are executing def myfunction (possibly with a view that the entire presence
    of the module was temporary and would not be available for future access via a global
    because it was not desired to hold it with a reference anywhere[1], but more likely
    because it's just a constant, and you are avoiding magic numbers by referring to an
    official definition module that may contain much else you don't need), then you want
    to do the dereferencing at define time.

    And for speed you don't want to execute the byte codes to do it more than once either,
    nor do you want to execute byte codes for conditional setup/use of a cached value.
    You want the first and all executions to see a preset binding generated at define time.

    mkfoo was one way to do that now, but a simple presets: block would be clearer and handier.

    [1] I assumed that it is possible to release all references to an imported module, so that
    the garbage collector can eliminate it from memory, but that may have been a bad assumption.
    sys.modules seem to get an entry whether you use

    import math
    pi = math.pi
    del math

    or

    pi = __import__('math').pi

    or even

    privatedir = {}
    exec 'pi = __import__("math").pi' in privatedir
    pi = privatedir['pi']
    del privatedir

    Plus, deleting an entry in sys.modules seems risky, since you don't know who else
    might be depending on it, even if you checked just before your import.

    What does one have to do to "un-import" a module? (I'm talking about a simple case, not an
    import with tricky side effects). I can think of a way to launder out references by a
    two-pass python execution from the outside using temp files, but yuck.

    As an example goal again, how would one do

    pi = __import__('math').pi

    and wind up with *exactly* the same net effect as

    pi = 3.1415926535897931

    with no math module stuff held by any reference (assume there wasn't any before the __import__).

    BTW, the system seems to keep other references than sys.modules:

    [ 6:35] C:\pywk\cal>python -v
    # D:\python22\lib\site.pyc matches D:\python22\lib\site.py
    import site # precompiled from D:\python22\lib\site.pyc
    # D:\python22\lib\os.pyc matches D:\python22\lib\os.py
    import os # precompiled from D:\python22\lib\os.pyc
    import nt # builtin
    # D:\python22\lib\ntpath.pyc matches D:\python22\lib\ntpath.py
    import ntpath # precompiled from D:\python22\lib\ntpath.pyc
    # D:\python22\lib\stat.pyc matches D:\python22\lib\stat.py
    import stat # precompiled from D:\python22\lib\stat.pyc
    # D:\python22\lib\UserDict.pyc matches D:\python22\lib\UserDict.py
    import UserDict # precompiled from D:\python22\lib\UserDict.pyc
    # D:\python22\lib\copy_reg.pyc matches D:\python22\lib\copy_reg.py
    import copy_reg # precompiled from D:\python22\lib\copy_reg.pyc
    # D:\python22\lib\types.pyc matches D:\python22\lib\types.py
    import types # precompiled from D:\python22\lib\types.pyc
    # D:\python22\lib\__future__.pyc matches D:\python22\lib\__future__.py
    import __future__ # precompiled from D:\python22\lib\__future__.pyc
    Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
    import sys
    sys.modules.get('math')
    pi = __import__('math').pi
    import math # builtin
    sys.modules.get('math')
    <module 'math' (built-in)>
    del sys.modules['math']
    sys.modules.get('math')
    import math
    import math # previously loaded (math)
    ^^^^^^^^^^^^^^^^^^^^^^^^

    Is there no way to do a private throwaway import?

    Regards,
    Bengt Richter
  • Anton Vredegoor at Nov 16, 2002 at 9:39 pm

    On 16 Nov 2002 16:19:17 GMT, bokr at oz.net (Bengt Richter) wrote:
    Is there no way to do a private throwaway import?
    You spared me some time by answering the "homework" issue, I thought I
    couldn't let that go, because in the internet culture we live in now
    using newsgroups to get questions answered or even complete programs
    written or found is given some people the same future shock as the
    introduction of the calculator did in times long ago.

    Speaking of the future, I have some code for you that's more on topic
    for this thread, I am practically sure it has the same problems as the
    other solutions proposed, but it might give you some more handles to
    work with. At the moment I am not inclined to check this all out for
    you (after all its *your* homework :-) but maybe it has some use, if
    not just pretend it was never there.

    Regards,
    Anton.

    from __future__ import generators

    def const():
    import math
    pi = math.pi
    del math

    def skippi(x):
    return x-int(x/pi)*pi

    d = {"skippi" : skippi}

    while 1:
    yield d


    def test():
    c = const()
    constants = c.next()
    skippi = constants["skippi"]
    print skippi(3.2)


    if __name__=='__main__':
    test()
  • Greg Ewing at Nov 18, 2002 at 2:49 am

    Bengt Richter wrote:

    I assumed that it is possible to release all references to an imported module, so that
    the garbage collector can eliminate it from memory

    That won't happen -- the module remains in the global
    list of modules maintained by the interpreter, even if
    all other references to it go away.

    --
    Greg Ewing, Computer Science Dept,
    University of Canterbury,
    Christchurch, New Zealand
    http://www.cosc.canterbury.ac.nz/~greg
  • Bengt Richter at Nov 18, 2002 at 4:03 am

    On Mon, 18 Nov 2002 15:49:10 +1300, Greg Ewing wrote:
    Bengt Richter wrote:
    I assumed that it is possible to release all references to an imported module, so that
    the garbage collector can eliminate it from memory

    That won't happen -- the module remains in the global
    list of modules maintained by the interpreter, even if
    all other references to it go away.
    You clipped my footnote section and reference, which I thought
    indicated that I saw that that is the way it ordinarily works:

    """
    >>>
    import sys
    sys.modules.get('math')
    pi = __import__('math').pi
    import math # builtin
    sys.modules.get('math')
    <module 'math' (built-in)>
    del sys.modules['math']
    sys.modules.get('math')
    import math
    import math # previously loaded (math)
    ^^^^^^^^^^^^^^^^^^^^^^^^

    Is there no way to do a private throwaway import?
    """

    So, on to the final question, which I'd still be interested in an answer to.
    BTW, to me, irreversibly tangled loading operations smack of problems that
    you "solve" by rebooting windows.

    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.

    After all, it's basically just instantiating an instance of a special class,
    isn't it? I would think if I dig in the import code there should be a place
    where an isolated instance is created, before it's irreversibly (?) hooked into
    the system.

    Regards,
    Bengt Richter
  • Brian Quinlan at Nov 18, 2002 at 9:04 am

    Bengt wrote:
    IOW, IMO there ought to be a way to do a temporary
    global-side-effect-free
    import and delete all traces of its having been done -- without
    restarting
    the interpreter or invoking a second interpreter instance.
    Does that involve unloading a dynamically imported library? If so, you
    will have to figure out how to do that in a cross-platform manner and
    without breaking existing Python extensions.

    Here are some other issues:
    1. What happens if two "throwaway" imports of the same module happen at
    once?
    2. What happens if the extension module modifies global state?
    3. What happens if the extension module causes external entities to
    hold references to its objects?

    Cheers,
    Brian
  • Jeff Epler at Nov 18, 2002 at 12:47 pm

    On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.
    Let us know when you've implemented it.

    Some of the problems:

    Modules need not be side-effect-free. For instance, the following is a
    (very harmful) legal Python module:
    from os import system
    system("rm -rf /")

    Modules may refer to other modules, so you'll need to make any other
    "import"s that happen during that time special.

    Unloading C modules may not work on all platforms, and may never be
    possible when there are references to objects in the module.

    If any nontrivial object from the module is kept, the module will be around
    anyway, even if it's not listed in sys.modules. Functions hold a reference
    to the module's globals, for instance. Any non-trivial class has methods,
    and any interesting object is an instance.

    I think that the presence of modules in sys.modules is used by the system
    to break cycles like that of functions. At cleanup, a module's dictionary
    is cleared which breaks the module->function->module cycles and allows
    collection.

    It's still not exactly clear to me what all these complications would gain
    me anyway...

    Jeff
  • Holger krekel at Nov 18, 2002 at 1:20 pm

    Jeff Epler wrote:
    On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.
    Let us know when you've implemented it.
    yes bengt, go for it :-)
    Some of the problems:

    Modules need not be side-effect-free. For instance, the following is a
    (very harmful) legal Python module:
    from os import system
    system("rm -rf /")
    sure, removing modules can only be as side effect free as the modules
    itself is. You can't even reverse print 'hello world'.

    I am sure there are uses for removing module objects. I'd recommend
    going for an 'try_remove_module' function which tries to achieve
    module unloading. It's probably neccessary to assert that there
    are no references to any objects in that to-be-removed module.
    So you might be able to say:

    import somemodule

    # do stuff with somemodule but don't permanently bind to any
    # of its objects

    try_remove_module(somemodule)
    del somemodule # or do a hack inside try_remove_module

    # somemodule should now be removed from memory, unless it's
    # a C-module or other special cases to be determined :-)

    On a side note, yesterday i talked to a perl hacker and he said
    that mod_perl suffers a lot from interpreter bloat because it's
    very hard to unload something plus perl has memory allocation
    problems with its mutable strings (memory usage always stays at
    the maximum size that string ever had). python doesn't have the latter
    problem because its strings are immutable. Don't know about
    how lists and dicts are handled, though.

    So I think that e.g. apps with mod_python could benefit from
    beeing able to unload some modules which are only temporarily used
    as helpers.

    regards,

    holger
  • Bengt Richter at Nov 18, 2002 at 5:33 pm

    On Mon, 18 Nov 2002 06:47:54 -0600, Jeff Epler wrote:
    On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.
    Let us know when you've implemented it.

    Some of the problems:

    Modules need not be side-effect-free. For instance, the following is a
    That's not a problem, that's a feature ;-)
    I didn't mean that all modules should be free of side effects or bindings
    that force their persistence. Most of the time you want some side effects,
    but you may want only the side effect to persist, not the module.

    <eg>
    E.g., if you had a program that needs a lot of memory and does processing
    in distinct phases. Obviously you can split the job into separate programs
    that use temp files. But what if you want use memory to communicate between
    phases? E.g., load a module that retrieves data from the net, another that
    parses it and builds convenient data structures, another that requires an
    interactive GUI to classify visually presented data, another to create a backup
    of clean data and editing log, another to generate pdf reports, etc. etc.
    The succeeding phases are going to suffer from memory squeeze if you can't
    unload modules that you no longer need.

    If you load a module that needs another, of course reference counting should
    work as it does, but when you unload a module explicitly, IMO it ought to be
    a bit like clearing a dict. A lot of references will get decremented, and
    some will go to zero. Why not a module's reference count, to let it disappear
    when no longer needed (or have it held in an LRU cache to be bumped as necessary)?
    </eg>
    (very harmful) legal Python module:
    from os import system
    system("rm -rf /")

    Modules may refer to other modules, so you'll need to make any other
    "import"s that happen during that time special. Yes.
    Unloading C modules may not work on all platforms, and may never be
    possible when there are references to objects in the module.
    Sure, there are all kinds of circumstances when objects legitimately keep
    others alive, and that would have to include modules. But I don't see that
    as a problem for cases where you program not to do that.
    If any nontrivial object from the module is kept, the module will be around
    anyway, even if it's not listed in sys.modules. Functions hold a reference
    to the module's globals, for instance. Any non-trivial class has methods,
    and any interesting object is an instance.
    Why would functions *always* hold a reference to the module's global name space,
    if it made no use of it? In anticipation of an exec that might? I'm willing for
    that to raise an exception if I've unloaded the module. Likewise I'll accept '?'
    for various inspection trails that I've cut off. (Maybe some e.g. inspection-supporting
    stuff would have to get weak references for private imports, or soemthing
    like that).
    I think that the presence of modules in sys.modules is used by the system
    to break cycles like that of functions. At cleanup, a module's dictionary
    is cleared which breaks the module->function->module cycles and allows
    collection.

    It's still not exactly clear to me what all these complications would gain
    me anyway...
    Mainly recovering memory space for re-use.
    See <eg></eg> above, and I'm sure you can
    think of other scenarios. I'm not saying there
    are no workarounds, but I do think there are cases
    where a clean unload could be useful.

    Regards,
    Bengt Richter
  • Aahz at Nov 29, 2002 at 1:05 pm

    In article <arb8a0$rfm$0 at 216.39.172.122>, Bengt Richter wrote:
    E.g., if you had a program that needs a lot of memory and does
    processing in distinct phases. Obviously you can split the job into
    separate programs that use temp files. But what if you want use memory
    to communicate between phases? E.g., load a module that retrieves
    data from the net, another that parses it and builds convenient data
    structures, another that requires an interactive GUI to classify
    visually presented data, another to create a backup of clean data and
    editing log, another to generate pdf reports, etc. etc. The succeeding
    phases are going to suffer from memory squeeze if you can't unload
    modules that you no longer need.

    If you load a module that needs another, of course reference counting
    should work as it does, but when you unload a module explicitly, IMO it
    ought to be a bit like clearing a dict. A lot of references will get
    decremented, and some will go to zero. Why not a module's reference
    count, to let it disappear when no longer needed (or have it held in an
    LRU cache to be bumped as necessary)?
    With the exception of C libraries, I'm not aware of any *modules* that
    are such a memory drain, although they may store data that is a memory
    drain. And C libraries are going to be much harder to unload, to the
    point that I'd bet Guido would quickly veto any such proposal.
    --
    Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/

    "If you don't know what your program is supposed to do, you'd better not
    start writing it." --Dijkstra
  • Michael Hudson at Nov 19, 2002 at 11:47 am

    bokr at oz.net (Bengt Richter) writes:

    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.
    I don't think this is so hard:

    def load_file(file, modname):
    import new
    mod = new.module(modname)
    exec open(file).read() in mod.__dict__
    return mod

    Then mod should only have the one reference:
    import new
    sys.getrefcount(new.module('temp'))
    1

    Also see the imp module; some of the hooks there don't insert in
    sys.modules (but I can't remember which ones now).

    Cheers,
    M.

    --
    Two decades later, well-known hacker Henry Spencer described the
    Perl scripting language as a "Swiss-Army chainsaw", intending to
    convey his evaluation of the language as exceedingly powerful but
    ugly and noisy and prone to belch noxious fumes. -- the jargon file
  • Bengt Richter at Nov 19, 2002 at 4:22 pm

    On Tue, 19 Nov 2002 11:47:57 GMT, Michael Hudson wrote:
    bokr at oz.net (Bengt Richter) writes:
    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.
    I don't think this is so hard:

    def load_file(file, modname):
    ^^^^- tch, tch ;-)
    import new
    mod = new.module(modname)
    exec open(file).read() in mod.__dict__
    return mod
    But as Jeff pointed out, if there are any imports in your imported file,
    you have to encapsulate them too. So you need a temporary sandbox
    import that captures all the import action into a private container/space.

    IIRC I saw some docs on sandboxing import. That's probably the trail to follow.
    Then mod should only have the one reference:
    import new
    sys.getrefcount(new.module('temp'))
    1
    Also see the imp module; some of the hooks there don't insert in
    sys.modules (but I can't remember which ones now).
    Thanks for the reminders. BTW, is there a reason you preferred

    exec open(file).read() in mod.__dict__
    over
    execfile(filepath, mod.__dict__)
    above?

    Regards,
    Bengt Richter
  • Michael Hudson at Nov 19, 2002 at 5:25 pm

    bokr at oz.net (Bengt Richter) writes:
    On Tue, 19 Nov 2002 11:47:57 GMT, Michael Hudson wrote:

    bokr at oz.net (Bengt Richter) writes:
    IOW, IMO there ought to be a way to do a temporary global-side-effect-free
    import and delete all traces of its having been done -- without restarting
    the interpreter or invoking a second interpreter instance.
    I don't think this is so hard:

    def load_file(file, modname):
    ^^^^- tch, tch ;-)
    It was fname at one point, but I thought that obscure...
    import new
    mod = new.module(modname)
    exec open(file).read() in mod.__dict__
    return mod
    But as Jeff pointed out, if there are any imports in your imported file,
    you have to encapsulate them too. So you need a temporary sandbox
    import that captures all the import action into a private container/space.
    Hmm. Yes.
    IIRC I saw some docs on sandboxing import. That's probably the trail
    to follow.
    Yeah, just don't touch sys.modules in your __import__, I guess.

    I wonder if

    import new, __builtin__
    mod = new.module(modname)
    d = __builtin__.__dict__.copy()
    d['__import__'] = my_hook
    mod.__dict__['__builtins__'] = d
    exec open(file).read() in mod.__dict__
    return mod

    would be a good start?
    Thanks for the reminders. BTW, is there a reason you preferred

    exec open(file).read() in mod.__dict__
    over
    execfile(filepath, mod.__dict__)
    above?
    Nope.

    Cheers,
    M.

    --
    Exam invigilation - it doesn't come much harder than that, esp if
    the book you're reading turns out to be worse than expected.
    -- Dirk Bruere, sci.physics.research
  • Bengt Richter at Nov 18, 2002 at 6:58 pm

    On Mon, 18 Nov 2002 01:04:43 -0800, Brian Quinlan wrote:
    Bengt wrote:
    IOW, IMO there ought to be a way to do a temporary
    global-side-effect-free
    import and delete all traces of its having been done -- without
    restarting
    the interpreter or invoking a second interpreter instance.
    Does that involve unloading a dynamically imported library? If so, you
    will have to figure out how to do that in a cross-platform manner and
    without breaking existing Python extensions. Ideally.
    Here are some other issues:
    1. What happens if two "throwaway" imports of the same module happen at
    once?
    IWT something analogous to execfile. I.e., depending on what name spaces you chose.
    2. What happens if the extension module modifies global state?
    Depends on which global state and how. I.e., (re)binding a name in globals()
    so that it refers to the module directly or indirectly would naturally prevent unloading.
    But binding a name anywhere outside the module to something independent of the module,
    like immutables or dynamically generated data, etc. would be typical usage, not a problem.
    3. What happens if the extension module causes external entities to
    hold references to its objects?
    Sure, that would be a problem for objects that are not separable, but my hope
    was that e.g. constants and pure functions and such in .py modules could be
    referenced and held while the module and anything unreferenced in it was deleted.
    For C modules I would expect limitations.

    Hm. Which raises the question of what you get if you write

    pi = __import('math').pi

    vs
    pi = __import('math').pi*1.0

    I.e., is the first pi a reference to an embedded double within the C module's memory space,
    which is therefore not separable? Or is it a true heap-borne python object constructed at load
    time? I'd expect the second pi to be free of any math module attachment, but is the first?
    I imagine you could write it to work either way.

    The same principle could apply to other attributes of a C module, IWT.

    Regards,
    Bengt Richter

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedNov 12, '02 at 6:49p
activeNov 29, '02 at 1:05p
posts28
users13
websitepython.org

People

Translate

site design / logo © 2022 Grokbase