FAQ
Here's a toy example. I just want to make a function that efficiently
uses (here just returns) something from a module. Compare:
def foo():
... from math import pi
... return pi
...
import dis
dis.dis(foo)
0 SET_LINENO 1

3 SET_LINENO 2
9 IMPORT_NAME 0 (math)
12 IMPORT_FROM 1 (pi)
15 STORE_FAST 0 (pi)
18 POP_TOP

19 SET_LINENO 3
25 RETURN_VALUE
29 RETURN_VALUE

and the result of this:
def mkfoo():
... from math import pi
... def foo():
... return pi
... return foo
...
foo2 = mkfoo()
dis.dis(foo2)
0 SET_LINENO 3

3 SET_LINENO 4
9 RETURN_VALUE
13 RETURN_VALUE

It seems foo2 is doing a lot less work, unless LOAD_DEREF is very bad, which I wouldn't think.
Is/should-there-be a syntax for accomplishing this more naturally?

What would be best Pythonic practice for getting a foo2? It seems like if there were
a way to say "do this part once at compile time" we could get this kind of thing more directly.
e.g.,

def foo3():
if __compiling__: # not real code, obviously
from math import pi
return pi

Using a module imported to global space is easy but not as efficient,
and it clutters the global name space needlessly:
import math
def bar():
... return math.pi
...
dis.dis(bar)
0 SET_LINENO 1

3 SET_LINENO 2
12 RETURN_VALUE
16 RETURN_VALUE

Regards,
Bengt Richter

## Search Discussions

•  at Nov 13, 2002 at 1:32 am ⇧

Bengt Richter wrote:

def mkfoo():
... from math import pi
... def foo():
... return pi
... return foo
...

What would be best Pythonic practice for getting a foo2? It seems like if there were
a way to say "do this part once at compile time"

It's not really compile-time that we want to do it,
but module load time, or perhaps function definition
time.

One way would be to use a default-argument hack:

import math

def foo(m = math):
return m.pi

but this has all the usual drawbacks of the default-argument
hack.

This suggests that perhaps there should be a way
of getting something that is just like a default
argument, except that it's not part of the argument
list. Maybe

def foo():
const m = math
return m.pi

The "const" statement would be evaluated when
the "def" was executed, and "m" would appear in the
local namespace when the function was called.

Although "const" might not be the best name for
it, since it's not really a constant, more like
a local with an initial value.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
•  at Nov 13, 2002 at 3:03 am ⇧

On Tue, 12 Nov 2002 20:32:02 -0500, Greg Ewing wrote:

Bengt Richter wrote:
def mkfoo():
... from math import pi
... def foo():
... return pi
... return foo
...

What would be best Pythonic practice for getting a foo2? It seems like
if there were a way to say "do this part once at compile time"

It's not really compile-time that we want to do it, but module load
time, or perhaps function definition time.

One way would be to use a default-argument hack:

import math

def foo(m = math):
return m.pi

but this has all the usual drawbacks of the default-argument hack.

This suggests that perhaps there should be a way of getting something
that is just like a default argument, except that it's not part of the
argument list. Maybe

def foo():
const m = math
return m.pi

The "const" statement would be evaluated when the "def" was executed,
and "m" would appear in the local namespace when the function was
called.

Although "const" might not be the best name for it, since it's not
really a constant, more like a local with an initial value.
You can *kinda* do this with 2.1 (at least) and up:
import math
def foo(): return foo.pi
...
foo.pi = math.pi
foo()
3.1415926535897931

I realize this isn't exactly what you are asking for,
but it is a way that works today.

Neal
•  at Nov 13, 2002 at 5:18 pm ⇧

On Tue, 12 Nov 2002 22:03:18 -0500, Neal Norwitz wrote:
On Tue, 12 Nov 2002 20:32:02 -0500, Greg Ewing wrote:

Bengt Richter wrote:
def mkfoo():
... from math import pi
... def foo():
... return pi
... return foo
...

What would be best Pythonic practice for getting a foo2? It seems like
if there were a way to say "do this part once at compile time"

It's not really compile-time that we want to do it, but module load
time, or perhaps function definition time.

One way would be to use a default-argument hack:

import math

def foo(m = math):
return m.pi

but this has all the usual drawbacks of the default-argument hack.

This suggests that perhaps there should be a way of getting something
that is just like a default argument, except that it's not part of the
argument list. Maybe

def foo():
const m = math
return m.pi

The "const" statement would be evaluated when the "def" was executed,
and "m" would appear in the local namespace when the function was
called.

Although "const" might not be the best name for it, since it's not
really a constant, more like a local with an initial value.
You can *kinda* do this with 2.1 (at least) and up:
import math
def foo(): return foo.pi
...
foo.pi = math.pi
foo()
3.1415926535897931

I realize this isn't exactly what you are asking for,
but it is a way that works today.
Thanks. It does answer part of the requirement. I.e., you can release the math
module with del math, but ... it's dependent on a persistent global name
binding to foo. I.e.,
bar = foo
del foo
bar()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 1, in foo
NameError: global name 'foo' is not defined
dis.dis(bar)
0 SET_LINENO 1

3 SET_LINENO 1
12 RETURN_VALUE
16 RETURN_VALUE

plus the global lookup plus attribute lookup is not optimally fast.

Regards,
Bengt Richter
•  at Nov 13, 2002 at 5:32 pm ⇧
In article <aqu1hd$tjk$0 at 216.39.172.122>, bokr at oz.net (Bengt Richter)
wrote:
import math
def foo(): return foo.pi
...
foo.pi = math.pi
foo()
3.1415926535897931

I realize this isn't exactly what you are asking for,
but it is a way that works today.
Thanks. It does answer part of the requirement. I.e., you can release the math
module with del math, but ... it's dependent on a persistent global name
binding to foo.
import math
def foo(pi = math.pi): return pi
bar = foo
del math
del foo
bar()
3.1415926535897931

--
David Eppstein UC Irvine Dept. of Information & Computer Science
eppstein at ics.uci.edu http://www.ics.uci.edu/~eppstein/
•  at Nov 14, 2002 at 12:00 am ⇧

Neal Norwitz wrote:

You can *kinda* do this with 2.1 (at least) and up:
import math
def foo(): return foo.pi
...
foo.pi = math.pi
foo()
3.1415926535897931

But that defeats the purpose, which presumably was to
avoid a global name lookup!

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
•  at Nov 13, 2002 at 9:20 am ⇧

"Greg Ewing" wrote:
... Maybe

def foo():
const m = math
return m.pi

The "const" statement would be evaluated when
the "def" was executed, and "m" would appear in the
local namespace when the function was called.

Although "const" might not be the best name for
it, since it's not really a constant, more like
a local with an initial value.
What is this 'const' name? I haven't ever heard of this in Python.
•  at Nov 13, 2002 at 9:33 am ⇧

Lexy Zhitenev wrote in comp.lang.python:
"Greg Ewing" wrote:
... Maybe

def foo():
const m = math
return m.pi

The "const" statement would be evaluated when
the "def" was executed, and "m" would appear in the
local namespace when the function was called.

Although "const" might not be the best name for
it, since it's not really a constant, more like
a local with an initial value.
What is this 'const' name? I haven't ever heard of this in Python.
It doesn't exist. Greg was talking about a possible language
extension.

-- Gerhard
•  at Nov 13, 2002 at 9:54 am ⇧

Lexy Zhitenev wrote:

What is this 'const' name? I haven't ever heard of this in Python.
It isn't anything; he's proposing a new feature.

--
Erik Max Francis / max at alcyone.com / http://www.alcyone.com/max/
__ San Jose, CA, USA / 37 20 N 121 53 W / &tSftDotIotE
/ \ The woman's movement is no longer a cause but a symptom.
\__/ Joan Didion
Bosskey.net: Aliens vs. Predator 2 / http://www.bosskey.net/avp2/
A personal guide to Aliens vs. Predator 2.
•  at Nov 13, 2002 at 5:02 pm ⇧

On Wed, 13 Nov 2002 14:32:02 +1300, Greg Ewing wrote:
Bengt Richter wrote:
def mkfoo():
... from math import pi
... def foo():
... return pi
... return foo
...

What would be best Pythonic practice for getting a foo2? It seems like if there were
a way to say "do this part once at compile time"

It's not really compile-time that we want to do it,
but module load time, or perhaps function definition
time.
Well, ok, I was speaking loosely of compile-time as the time between reading source
and when final executable byte codes are laid down, but of course it's more complex
than that (especially with Python), as mkfoo incidentally demonstrates.

But the general idea is to find points in the sequence where one could intercede with
some meta-operations to modify code that gets executed at the end of the cascade.

You mention load time and function definition time. (Class definition time would seem
to similar, with metaclasses aready providing much control). Also, execfile time has
some things in common with module load time. Is there a unified "-time" concept re both?
Are there different possibly useful intercession points in loading .py vs .pyc?

What other "-times" can be clearly identified? If we had macros, I'm sure there would
be a "macro-interpretation time" (read-time?) associated with that. (BTW, how about times
associated with encoding transformations of source and i/o?)

But, getting back to foo/mkfoo, the intent was to set up values for efficient access
and to release unneeded references as soon as possible, compiling (in a narrower sense)
the source effectively partitioned into code and metacode for execution at appropriate
separate times to yield a clean foo function.
One way would be to use a default-argument hack:
Which would be a definition-time binding, I guess you would say?
import math

def foo(m = math):
return m.pi

but this has all the usual drawbacks of the default-argument
hack.
Plus that version retains a reference to math, where I'd like to think that
I would wind up with just a reference to a double with the 3.14.. value. I.e.,
def foo(pi = math.pi):
return pi

(not that I want to pursue the default-argument hack, just to indicate that
whatever the methodology, I don't think the generated code should be forcing persistence
of anything it doesn't really need).
This suggests that perhaps there should be a way
of getting something that is just like a default
argument, except that it's not part of the argument
list. Maybe

def foo():
const m = math
return m.pi

The "const" statement would be evaluated when
the "def" was executed, and "m" would appear in the
local namespace when the function was called.

Although "const" might not be the best name for
it, since it's not really a constant, more like
a local with an initial value.
Yes. Also I think I would like a keyword that governs
a block scope instead of just the scope of an assignment. E.g.,
(resisting the temptation of discovering a general mechanism here ;-)

def foo():
predef:
import math
pi = math.pi
del math
seven = 7
return pi, seven

meaning the predef block would be done once at define-time, and you
could do whatever you wanted to predefine the local environment that
would be seen at the first call (execution-time). You could also delete
unneeded bindings as above. This kind of flexibility would be hard if not
impossible to get with 'const' & such. I'm not sure whether a postdef:
section would be of use, but if so these things should probably be
subsumed under a single keyword (e.g. "meta" for "meta predef: and "meta postdef" ?)
especially if one anticipated other contexts of use as well.

The predef block looks like a kind of meta-code, and I would think the
compiler could generate a code block for that separate from the code of
the function body that defines what is called at execution time (foo), loosely
corresponding to mkfoo and foo code in the example, not a single body with
a special conditional part and references hanging on to things beyond their
intial and only use.

Regards,
Bengt Richter
•  at Nov 14, 2002 at 12:09 am ⇧

Bengt Richter wrote:

Plus that version retains a reference to math, where I'd like to think that
I would wind up with just a reference to a double with the 3.14.. value. I.e.,
def foo(pi = math.pi):
return pi

Yes, that's probably what I should have written.
Also I think I would like a keyword that governs
a block scope instead of just the scope of an assignment. E.g.,

def foo():
predef:
import math
pi = math.pi
del math
seven = 7
return pi, seven

Too complicated. You could get the same effect by putting the
code outside the def:

from math import math_pi

def foo():
const pi = math_pi
const seven = 7
...

del math_pi

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
•  at Nov 14, 2002 at 9:29 am ⇧

On Thu, 14 Nov 2002 13:09:04 +1300, Greg Ewing wrote:
Bengt Richter wrote:
Plus that version retains a reference to math, where I'd like to think that
I would wind up with just a reference to a double with the 3.14.. value. I.e.,
def foo(pi = math.pi):
return pi

Yes, that's probably what I should have written.
Also I think I would like a keyword that governs
a block scope instead of just the scope of an assignment. E.g.,

def foo():
predef:
import math
pi = math.pi
del math
seven = 7
return pi, seven

Too complicated. You could get the same effect by putting the
code outside the def:

from math import math_pi
did you mean
from math import pi as math_pi
?
def foo():
const pi = math_pi
const seven = 7
...

del math_pi
Not quite the same effect: you are introducing temporary global bindings.
And ISTM your version really only looks less complicated because you moved
two statements out of the local scope.

To avoid the global temp, I suppose you could write

def foo():
const pi = [__import__('math')][0].pi
const seven = 7

but down that road lies much abuse, trying to cram block scope intentions
into expression scope. I don't think it's possible to anticipate what
people will want to do, except that people will tend to work around limitations
in ugly ways if there's no clean way to express their intent. So I'd rather
provide a freer "scope" ;-)

BTW, I'd suggest "preset" as closer to the semantic mark than "const" if your
proposal gets adopted. Maybe my block introducer should similarly be "presets:".
Then for minimal one-liner usage, it would look concise and similar

def foo():
presets: seven = 7
...
vs.

def foo():
const seven = 7
...

or
def foo():
preset seven = 7
...

I like the option of simple multiple values on a single line too.

def foo():
presets: seven = 7; eight = 2**3
...

Regards,
Bengt Richter
•  at Nov 14, 2002 at 11:08 am ⇧
On 14 Nov 2002 09:29:21 GMT, bokr at oz.net (Bengt Richter) wrote:
[...]
To avoid the global temp, I suppose you could write

def foo():
const pi = [__import__('math')][0].pi
or, less ridiculously,
const pi = __import__('math').pi
;-/
const seven = 7
Regards,
Bengt Richter
•  at Nov 16, 2002 at 7:18 am ⇧

In article <aqrig4$d2h$0 at 216.39.172.122>, Bengt Richter wrote:
Here's a toy example. I just want to make a function that efficiently
uses (here just returns) something from a module.
Can you explain a bit more clearly what the purpose of this?
--
Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/

A: No.
Q: Is top-posting okay?
•  at Nov 16, 2002 at 2:29 pm ⇧

In article <aqrig4$d2h$0 at 216.39.172.122>, Bengt Richter wrote:
Here's a toy example. I just want to make a function that efficiently
uses (here just returns) something from a module.
On Sat, Nov 16, 2002 at 02:18:41AM -0500, Aahz wrote:
Can you explain a bit more clearly what the purpose of this?
Speed, I assume.

f0 0.945003986359 # empty loop
f1 2.456807971 # math.pi
f2 1.20684301853 # (local) pi

Jeff

import math, time

R = range(1000000)
def f0():
for i in R: pass

def f1():
for i in R: math.pi

def f2(pi=math.pi):
for i in R: pi

for f in (f0, f1, f2):
t0 = time.time()
f()
t1 = time.time()
print f.__name__, t1-t0
•  at Nov 16, 2002 at 4:19 pm ⇧

On 16 Nov 2002 02:18:41 -0500, aahz at pythoncraft.com (Aahz) wrote:
In article <aqrig4$d2h$0 at 216.39.172.122>, Bengt Richter wrote:
Here's a toy example. I just want to make a function that efficiently
uses (here just returns) something from a module.
Can you explain a bit more clearly what the purpose of this?
In a word, optimization.

Both speed and space.

The reason I focused on an item from a module is that typical usage
is to import the relevant module to make a global binding (e.g., globalmodulename)
and then to refer to the desired item as globalmodulename.desireditem.

This is a fine mechanism when you want dynamic access to the current value of
something that may be getting updated, but if you want the value that was bound at the
time you are executing def myfunction (possibly with a view that the entire presence
of the module was temporary and would not be available for future access via a global
because it was not desired to hold it with a reference anywhere[1], but more likely
because it's just a constant, and you are avoiding magic numbers by referring to an
official definition module that may contain much else you don't need), then you want
to do the dereferencing at define time.

And for speed you don't want to execute the byte codes to do it more than once either,
nor do you want to execute byte codes for conditional setup/use of a cached value.
You want the first and all executions to see a preset binding generated at define time.

mkfoo was one way to do that now, but a simple presets: block would be clearer and handier.

[1] I assumed that it is possible to release all references to an imported module, so that
the garbage collector can eliminate it from memory, but that may have been a bad assumption.
sys.modules seem to get an entry whether you use

import math
pi = math.pi
del math

or

pi = __import__('math').pi

or even

privatedir = {}
exec 'pi = __import__("math").pi' in privatedir
pi = privatedir['pi']
del privatedir

Plus, deleting an entry in sys.modules seems risky, since you don't know who else
might be depending on it, even if you checked just before your import.

What does one have to do to "un-import" a module? (I'm talking about a simple case, not an
import with tricky side effects). I can think of a way to launder out references by a
two-pass python execution from the outside using temp files, but yuck.

As an example goal again, how would one do

pi = __import__('math').pi

and wind up with *exactly* the same net effect as

pi = 3.1415926535897931

with no math module stuff held by any reference (assume there wasn't any before the __import__).

BTW, the system seems to keep other references than sys.modules:

[ 6:35] C:\pywk\cal>python -v
# D:\python22\lib\site.pyc matches D:\python22\lib\site.py
import site # precompiled from D:\python22\lib\site.pyc
# D:\python22\lib\os.pyc matches D:\python22\lib\os.py
import os # precompiled from D:\python22\lib\os.pyc
import nt # builtin
# D:\python22\lib\ntpath.pyc matches D:\python22\lib\ntpath.py
import ntpath # precompiled from D:\python22\lib\ntpath.pyc
# D:\python22\lib\stat.pyc matches D:\python22\lib\stat.py
import stat # precompiled from D:\python22\lib\stat.pyc
# D:\python22\lib\UserDict.pyc matches D:\python22\lib\UserDict.py
import UserDict # precompiled from D:\python22\lib\UserDict.pyc
# D:\python22\lib\copy_reg.pyc matches D:\python22\lib\copy_reg.py
import copy_reg # precompiled from D:\python22\lib\copy_reg.pyc
# D:\python22\lib\types.pyc matches D:\python22\lib\types.py
import types # precompiled from D:\python22\lib\types.pyc
# D:\python22\lib\__future__.pyc matches D:\python22\lib\__future__.py
import __future__ # precompiled from D:\python22\lib\__future__.pyc
Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
>>>
import sys
sys.modules.get('math')
pi = __import__('math').pi
import math # builtin
sys.modules.get('math')
<module 'math' (built-in)>
del sys.modules['math']
sys.modules.get('math')
import math
import math # previously loaded (math)
^^^^^^^^^^^^^^^^^^^^^^^^

Is there no way to do a private throwaway import?

Regards,
Bengt Richter
•  at Nov 16, 2002 at 9:39 pm ⇧

On 16 Nov 2002 16:19:17 GMT, bokr at oz.net (Bengt Richter) wrote:
Is there no way to do a private throwaway import?
You spared me some time by answering the "homework" issue, I thought I
couldn't let that go, because in the internet culture we live in now
using newsgroups to get questions answered or even complete programs
written or found is given some people the same future shock as the
introduction of the calculator did in times long ago.

Speaking of the future, I have some code for you that's more on topic
for this thread, I am practically sure it has the same problems as the
other solutions proposed, but it might give you some more handles to
work with. At the moment I am not inclined to check this all out for
you (after all its *your* homework :-) but maybe it has some use, if
not just pretend it was never there.

Regards,
Anton.

from __future__ import generators

def const():
import math
pi = math.pi
del math

def skippi(x):
return x-int(x/pi)*pi

d = {"skippi" : skippi}

while 1:
yield d

def test():
c = const()
constants = c.next()
skippi = constants["skippi"]
print skippi(3.2)

if __name__=='__main__':
test()
•  at Nov 18, 2002 at 2:49 am ⇧

Bengt Richter wrote:

I assumed that it is possible to release all references to an imported module, so that
the garbage collector can eliminate it from memory

That won't happen -- the module remains in the global
list of modules maintained by the interpreter, even if
all other references to it go away.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
•  at Nov 18, 2002 at 4:03 am ⇧

On Mon, 18 Nov 2002 15:49:10 +1300, Greg Ewing wrote:
Bengt Richter wrote:
I assumed that it is possible to release all references to an imported module, so that
the garbage collector can eliminate it from memory

That won't happen -- the module remains in the global
list of modules maintained by the interpreter, even if
all other references to it go away.
You clipped my footnote section and reference, which I thought
indicated that I saw that that is the way it ordinarily works:

"""
>>>
import sys
sys.modules.get('math')
pi = __import__('math').pi
import math # builtin
sys.modules.get('math')
<module 'math' (built-in)>
del sys.modules['math']
sys.modules.get('math')
import math
import math # previously loaded (math)
^^^^^^^^^^^^^^^^^^^^^^^^

Is there no way to do a private throwaway import?
"""

So, on to the final question, which I'd still be interested in an answer to.
you "solve" by rebooting windows.

IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.

After all, it's basically just instantiating an instance of a special class,
isn't it? I would think if I dig in the import code there should be a place
where an isolated instance is created, before it's irreversibly (?) hooked into
the system.

Regards,
Bengt Richter
•  at Nov 18, 2002 at 9:04 am ⇧

Bengt wrote:
IOW, IMO there ought to be a way to do a temporary
global-side-effect-free
import and delete all traces of its having been done -- without
restarting
the interpreter or invoking a second interpreter instance.
will have to figure out how to do that in a cross-platform manner and
without breaking existing Python extensions.

Here are some other issues:
1. What happens if two "throwaway" imports of the same module happen at
once?
2. What happens if the extension module modifies global state?
3. What happens if the extension module causes external entities to
hold references to its objects?

Cheers,
Brian
•  at Nov 18, 2002 at 12:47 pm ⇧

On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.
Let us know when you've implemented it.

Some of the problems:

Modules need not be side-effect-free. For instance, the following is a
(very harmful) legal Python module:
from os import system
system("rm -rf /")

Modules may refer to other modules, so you'll need to make any other
"import"s that happen during that time special.

Unloading C modules may not work on all platforms, and may never be
possible when there are references to objects in the module.

If any nontrivial object from the module is kept, the module will be around
anyway, even if it's not listed in sys.modules. Functions hold a reference
to the module's globals, for instance. Any non-trivial class has methods,
and any interesting object is an instance.

I think that the presence of modules in sys.modules is used by the system
to break cycles like that of functions. At cleanup, a module's dictionary
is cleared which breaks the module->function->module cycles and allows
collection.

It's still not exactly clear to me what all these complications would gain
me anyway...

Jeff
•  at Nov 18, 2002 at 1:20 pm ⇧

Jeff Epler wrote:
On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.
Let us know when you've implemented it.
yes bengt, go for it :-)
Some of the problems:

Modules need not be side-effect-free. For instance, the following is a
(very harmful) legal Python module:
from os import system
system("rm -rf /")
sure, removing modules can only be as side effect free as the modules
itself is. You can't even reverse print 'hello world'.

I am sure there are uses for removing module objects. I'd recommend
going for an 'try_remove_module' function which tries to achieve
are no references to any objects in that to-be-removed module.
So you might be able to say:

import somemodule

# do stuff with somemodule but don't permanently bind to any
# of its objects

try_remove_module(somemodule)
del somemodule # or do a hack inside try_remove_module

# somemodule should now be removed from memory, unless it's
# a C-module or other special cases to be determined :-)

On a side note, yesterday i talked to a perl hacker and he said
that mod_perl suffers a lot from interpreter bloat because it's
very hard to unload something plus perl has memory allocation
problems with its mutable strings (memory usage always stays at
the maximum size that string ever had). python doesn't have the latter
problem because its strings are immutable. Don't know about
how lists and dicts are handled, though.

So I think that e.g. apps with mod_python could benefit from
beeing able to unload some modules which are only temporarily used
as helpers.

regards,

holger
•  at Nov 18, 2002 at 5:33 pm ⇧

On Mon, 18 Nov 2002 06:47:54 -0600, Jeff Epler wrote:
On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.
Let us know when you've implemented it.

Some of the problems:

Modules need not be side-effect-free. For instance, the following is a
That's not a problem, that's a feature ;-)
I didn't mean that all modules should be free of side effects or bindings
that force their persistence. Most of the time you want some side effects,
but you may want only the side effect to persist, not the module.

<eg>
E.g., if you had a program that needs a lot of memory and does processing
in distinct phases. Obviously you can split the job into separate programs
that use temp files. But what if you want use memory to communicate between
phases? E.g., load a module that retrieves data from the net, another that
parses it and builds convenient data structures, another that requires an
interactive GUI to classify visually presented data, another to create a backup
of clean data and editing log, another to generate pdf reports, etc. etc.
The succeeding phases are going to suffer from memory squeeze if you can't
unload modules that you no longer need.

If you load a module that needs another, of course reference counting should
work as it does, but when you unload a module explicitly, IMO it ought to be
a bit like clearing a dict. A lot of references will get decremented, and
some will go to zero. Why not a module's reference count, to let it disappear
when no longer needed (or have it held in an LRU cache to be bumped as necessary)?
</eg>
(very harmful) legal Python module:
from os import system
system("rm -rf /")

Modules may refer to other modules, so you'll need to make any other
"import"s that happen during that time special. Yes.
Unloading C modules may not work on all platforms, and may never be
possible when there are references to objects in the module.
Sure, there are all kinds of circumstances when objects legitimately keep
others alive, and that would have to include modules. But I don't see that
as a problem for cases where you program not to do that.
If any nontrivial object from the module is kept, the module will be around
anyway, even if it's not listed in sys.modules. Functions hold a reference
to the module's globals, for instance. Any non-trivial class has methods,
and any interesting object is an instance.
Why would functions *always* hold a reference to the module's global name space,
if it made no use of it? In anticipation of an exec that might? I'm willing for
that to raise an exception if I've unloaded the module. Likewise I'll accept '?'
for various inspection trails that I've cut off. (Maybe some e.g. inspection-supporting
stuff would have to get weak references for private imports, or soemthing
like that).
I think that the presence of modules in sys.modules is used by the system
to break cycles like that of functions. At cleanup, a module's dictionary
is cleared which breaks the module->function->module cycles and allows
collection.

It's still not exactly clear to me what all these complications would gain
me anyway...
Mainly recovering memory space for re-use.
See <eg></eg> above, and I'm sure you can
think of other scenarios. I'm not saying there
are no workarounds, but I do think there are cases
where a clean unload could be useful.

Regards,
Bengt Richter
•  at Nov 29, 2002 at 1:05 pm ⇧

In article <arb8a0$rfm$0 at 216.39.172.122>, Bengt Richter wrote:
E.g., if you had a program that needs a lot of memory and does
processing in distinct phases. Obviously you can split the job into
separate programs that use temp files. But what if you want use memory
to communicate between phases? E.g., load a module that retrieves
data from the net, another that parses it and builds convenient data
structures, another that requires an interactive GUI to classify
visually presented data, another to create a backup of clean data and
editing log, another to generate pdf reports, etc. etc. The succeeding
phases are going to suffer from memory squeeze if you can't unload
modules that you no longer need.

If you load a module that needs another, of course reference counting
should work as it does, but when you unload a module explicitly, IMO it
ought to be a bit like clearing a dict. A lot of references will get
decremented, and some will go to zero. Why not a module's reference
count, to let it disappear when no longer needed (or have it held in an
LRU cache to be bumped as necessary)?
With the exception of C libraries, I'm not aware of any *modules* that
are such a memory drain, although they may store data that is a memory
drain. And C libraries are going to be much harder to unload, to the
point that I'd bet Guido would quickly veto any such proposal.
--
Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/

"If you don't know what your program is supposed to do, you'd better not
start writing it." --Dijkstra
•  at Nov 19, 2002 at 11:47 am ⇧

bokr at oz.net (Bengt Richter) writes:

IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.
I don't think this is so hard:

import new
mod = new.module(modname)
return mod

Then mod should only have the one reference:
import new
sys.getrefcount(new.module('temp'))
1

Also see the imp module; some of the hooks there don't insert in
sys.modules (but I can't remember which ones now).

Cheers,
M.

--
Two decades later, well-known hacker Henry Spencer described the
Perl scripting language as a "Swiss-Army chainsaw", intending to
convey his evaluation of the language as exceedingly powerful but
ugly and noisy and prone to belch noxious fumes. -- the jargon file
•  at Nov 19, 2002 at 4:22 pm ⇧

On Tue, 19 Nov 2002 11:47:57 GMT, Michael Hudson wrote:
bokr at oz.net (Bengt Richter) writes:
IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.
I don't think this is so hard:

^^^^- tch, tch ;-)
import new
mod = new.module(modname)
return mod
But as Jeff pointed out, if there are any imports in your imported file,
you have to encapsulate them too. So you need a temporary sandbox
import that captures all the import action into a private container/space.

IIRC I saw some docs on sandboxing import. That's probably the trail to follow.
Then mod should only have the one reference:
import new
sys.getrefcount(new.module('temp'))
1
Also see the imp module; some of the hooks there don't insert in
sys.modules (but I can't remember which ones now).
Thanks for the reminders. BTW, is there a reason you preferred

over
execfile(filepath, mod.__dict__)
above?

Regards,
Bengt Richter
•  at Nov 19, 2002 at 5:25 pm ⇧

bokr at oz.net (Bengt Richter) writes:
On Tue, 19 Nov 2002 11:47:57 GMT, Michael Hudson wrote:

bokr at oz.net (Bengt Richter) writes:
IOW, IMO there ought to be a way to do a temporary global-side-effect-free
import and delete all traces of its having been done -- without restarting
the interpreter or invoking a second interpreter instance.
I don't think this is so hard:

^^^^- tch, tch ;-)
It was fname at one point, but I thought that obscure...
import new
mod = new.module(modname)
return mod
But as Jeff pointed out, if there are any imports in your imported file,
you have to encapsulate them too. So you need a temporary sandbox
import that captures all the import action into a private container/space.
Hmm. Yes.
IIRC I saw some docs on sandboxing import. That's probably the trail
to follow.
Yeah, just don't touch sys.modules in your __import__, I guess.

I wonder if

import new, __builtin__
mod = new.module(modname)
d = __builtin__.__dict__.copy()
d['__import__'] = my_hook
mod.__dict__['__builtins__'] = d
return mod

would be a good start?
Thanks for the reminders. BTW, is there a reason you preferred

over
execfile(filepath, mod.__dict__)
above?
Nope.

Cheers,
M.

--
Exam invigilation - it doesn't come much harder than that, esp if
the book you're reading turns out to be worse than expected.
-- Dirk Bruere, sci.physics.research
•  at Nov 18, 2002 at 6:58 pm ⇧

On Mon, 18 Nov 2002 01:04:43 -0800, Brian Quinlan wrote:
Bengt wrote:
IOW, IMO there ought to be a way to do a temporary
global-side-effect-free
import and delete all traces of its having been done -- without
restarting
the interpreter or invoking a second interpreter instance.
will have to figure out how to do that in a cross-platform manner and
without breaking existing Python extensions. Ideally.
Here are some other issues:
1. What happens if two "throwaway" imports of the same module happen at
once?
IWT something analogous to execfile. I.e., depending on what name spaces you chose.
2. What happens if the extension module modifies global state?
Depends on which global state and how. I.e., (re)binding a name in globals()
so that it refers to the module directly or indirectly would naturally prevent unloading.
But binding a name anywhere outside the module to something independent of the module,
like immutables or dynamically generated data, etc. would be typical usage, not a problem.
3. What happens if the extension module causes external entities to
hold references to its objects?
Sure, that would be a problem for objects that are not separable, but my hope
was that e.g. constants and pure functions and such in .py modules could be
referenced and held while the module and anything unreferenced in it was deleted.
For C modules I would expect limitations.

Hm. Which raises the question of what you get if you write

pi = __import('math').pi

vs
pi = __import('math').pi*1.0

I.e., is the first pi a reference to an embedded double within the C module's memory space,
which is therefore not separable? Or is it a true heap-borne python object constructed at load
time? I'd expect the second pi to be free of any math module attachment, but is the first?
I imagine you could write it to work either way.

The same principle could apply to other attributes of a C module, IWT.

Regards,
Bengt Richter

## Related Discussions

Discussion Overview
 group python-list categories python posted Nov 12, '02 at 6:49p active Nov 29, '02 at 1:05p posts 28 users 13 website python.org

### 13 users in discussion

Content

People

Support

Translate

site design / logo © 2022 Grokbase