[Python] Automatic reloading, metaclasses, and pickle

This happens because unpickling doesn't recreate your object by
calling its type. MetaInstanceTracker registers an instance only
when it is created by calling a class.

You can solve this by moving the instance registration to
AutoReloader.__new__ and using pickle protocol version 2, but the
best solution is to avoid both pickle (old pickles break if you
change your code) and autoreloading (it's meant to be used in
interactive console and entertaining ircbots, not in normal code).

Ziga

Thanks Ziga. I use pickle protocol 2 and binary file types with the
command: "cPickle.dump(obj, file, 2)"

I did your suggestion, i commented out the "__call__" function of
MetaInstanceTracker and copied the text to the __new__ function of
AutoReloader (code appended). I got a crazy recursive error message
(also appended below). In my code, I am creating a new instance,
rather than using the pickled object (it needs to work in both modes).

Thanks very much for helping me get through this. With my development
approach, finding a solution to this problem is really important to
me.

...
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
RuntimeError: maximum recursion depth exceeded

<code>
import weakref, inspect

class MetaInstanceTracker(type):
def __new__(cls, name, bases, ns):
t = super(MetaInstanceTracker, cls).__new__(cls, name, bases,
ns)
t.__instance_refs__ = []
return t
def __instances__(self):
instances = [(r, r()) for r in self.__instance_refs__]
instances = filter(lambda (x,y): y is not None, instances)
self.__instance_refs__ = [r for (r, o) in instances]
return [o for (r, o) in instances]
## def __call__(self, *args, **kw):
## instance = super(MetaInstanceTracker, self).__call__(*args,
**kw)
## self.__instance_refs__.append(weakref.ref(instance))
## return instance

class InstanceTracker:
__metaclass__ = MetaInstanceTracker

class MetaAutoReloader(MetaInstanceTracker):
def __new__(cls, name, bases, ns):
new_class = super(MetaAutoReloader, cls).__new__(
cls, name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if d.has_key(name):
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(new_class)
new_class.__instance_refs__.append(
weakref.ref(instance))
# this section only works in 2.3
for subcls in old_class.__subclasses__():
newbases = ()
for base in subcls.__bases__:
if base is old_class:
newbases += (new_class,)
else:
newbases += (base,)
subcls.__bases__ = newbases
break
return new_class

class AutoReloader:
__metaclass__ = MetaAutoReloader
def change_class(self, new_class):
self.__class__ = new_class
def __new__(self, *args, **kw):
instance = super(MetaInstanceTracker, self).__call__(*args,
**kw)
self.__instance_refs__.append(weakref.ref(instance))
return instance

Thanks,
Andrew Felch

Here is a version that should work. It should work with all protocols,
see InstanceTracker.__reduce_ex__. Note that all subclasses of
InstanceTracker and AutoReloader should be cautious when overriding
the
__new__ method. They must call their base class' __new__ method,
preferably by using super(), or the tracking won't work.


# adapted from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

import weakref, inspect


class MetaInstanceTracker(type):

def __init__(cls, name, bases, ns):
super(MetaInstanceTracker, cls).__init__(name, bases, ns)
cls.__instance_refs__ = []

def __instances__(cls):
instances = []
validrefs = []
for ref in cls.__instance_refs__:
instance = ref()
if instance is not None:
instances.append(instance)
validrefs.append(ref)
cls.__instance_refs__ = validrefs
return instances


class InstanceTracker(object):

__metaclass__ = MetaInstanceTracker

def __new__(*args, **kwargs):
cls = args[0]
self = super(InstanceTracker, cls).__new__(*args, **kwargs)
cls.__instance_refs__.append(weakref.ref(self))
return self

def __reduce_ex__(self, proto):
return super(InstanceTracker, self).__reduce_ex__(2)


class MetaAutoReloader(MetaInstanceTracker):

def __init__(cls, name, bases, ns):
super(MetaAutoReloader, cls).__init__(name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if name in d:
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(cls)

cls.__instance_refs__.append(weakref.ref(instance))
for subcls in old_class.__subclasses__():
newbases = []
for base in subcls.__bases__:
if base is old_class:
newbases.append(cls)
else:
newbases.append(base)
subcls.__bases__ = tuple(newbases)
break


class AutoReloader(InstanceTracker):

__metaclass__ = MetaAutoReloader

def change_class(self, new_class):
self.__class__ = new_class


Ziga

I pasted the code into mine and replaced the old. It seems not to
work for either unpickled objects or new objects. I add methods to a
class that inherits from AutoReloader and reload the module, but the
new methods are not callable on the old objects. Man! It seems we're
so close, it will be huge if this ends up working. This stuff is so
over my head, I wish I could help in some additional way.

-Andrew

Did you copy and paste the entire module? I fiddled almost all parts
of
the original code. I did some base testing and it worked for me. Could
you post the traceback? Note that Google Groups messed up indentation;
in MetaAutoReloader.__init__, the line starting with
cls.__instance_refs__ should be at the same level as previous line.

Did you restart Python? InstanceTracker, MetaInstanceTracker and
MetaAutoReloader are not auto reloaded :).

Ziga

Thanks for checking. I think I narrowed the problem down to
inheritance. I inherit from list or some other container first:

class PointList( list, AutoReloader ):
def PrintHi1(self):
print "Hi2"

class MyPrintingClass( AutoReloader ):
def PrintHi2(self):
print "Hi2v2"

Automatic reloading works for MyPrintingClass but not for PointList.
Any ideas?

-Andrew

Ah yes, this is the problem of list.__new__ not calling the next
class in MRO. Try to switch the bases, so that AutoReloader's
__new__ method will be called first.

Ziga

That did it! Thanks so much!

This really seems to extend Python to be a much better rapid
prototyping langauge. It certainly allows me to now use classes where
I avoided them like the plague before. Perhaps I'll be able to make
programs > 1500 lines now! :-)

-Andrew

So there is another bug crippling Python (see
https://sourceforge.net/tracker/?funcÞtail&atid5470&aidb8925&group_idT70
). Pickle and cPickle get confused after reloading, and won't let you
dump the objects to file using protocol 2. The bug occurred for me
when an instance of an AutoReloader class had a member not derived
from AutoReloader (error text: "it's not the same object" ). The
workaround is to use text mode, which I assume means not using
protocol 2. After that, you can reload them in text mode, and then
dump them using protocol 2. This is not so bad since you only do it
when you were going to dump objects to file anyway, so it really only
takes 3x as long on those occassions. The metaclass adaptation is
still incredibly useful because it avoids having to do pickle dump-
loads when you wouldn't otherwise have had to (approaching an infinite
speedup in those cases, thanks Ziga! :-).

Another caveat is that pickle dumping doesn't work if there is a
pointer to a global function (that changes) that is a member of an
instance of a class derived from AutoLoader. The error message has
subtext: "it's not the same object ". That doesn't seem so bad to me,
since you don't have to do that type of thing (function pointers) so
much now that you can use classes!

The best solution would be to make pickle protocol 2 have the same
behavior as the text protocol (i.e. actually work in this case).

Regarding the function pointers, anybody (Ziga :-) know why it is that
pickle chokes so hard on this? Is there some adaptation of the
metaclass or a less beautiful hack that could fix it?

Thanks to those readers following me down the rabbit hole. Ziga's
metaclass adaptation has saved Python in my mind, and these nits seem
to be worth discussing.

- Andrew Felch