FAQ
Hello all,

I'm using the metaclass trick for automatic reloading of class member
functions, found at: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

My problem is that if I
1) pickle an object that inherits from "AutoReloader"
2) unpickle the object
3) modify one of the pickled' object's derived class methods
4) reload the module holding the class

... then the changes don't affect the unpickled object. If I unpickle
the object again, of course the changes take effect.

My friend that loves Smalltalk is laughing at me. I thought I had the
upperhand when I discovered the metaclasses but now I am not sure what
to do. I really don't want to have to unpickle again, I'm processing
video and it can take a long time.

By the way, I used to avoid all of these problems by never making
classes, and always building complex structures of lists,
dictionaries, and tuples with global functions. It's going to take me
a while to kick those horrible habits (during my transition, I'm
deriving from list, dict, etc. hehe), perhaps a link to the metaclass
trick is in order in the tutorial's comments on reload?

Any help that avoids having to unpickle again is appreciated!

Thanks,
Andrew Felch

Search Discussions

  • Ziga Seilnacht at Feb 27, 2007 at 8:50 pm

    Andrew Felch wrote:
    Hello all,

    I'm using the metaclass trick for automatic reloading of class member
    functions, found at:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

    My problem is that if I
    1) pickle an object that inherits from "AutoReloader"
    2) unpickle the object
    3) modify one of the pickled' object's derived class methods
    4) reload the module holding the class

    ... then the changes don't affect the unpickled object. If I unpickle
    the object again, of course the changes take effect.

    My friend that loves Smalltalk is laughing at me. I thought I had the
    upperhand when I discovered the metaclasses but now I am not sure what
    to do. I really don't want to have to unpickle again, I'm processing
    video and it can take a long time.

    By the way, I used to avoid all of these problems by never making
    classes, and always building complex structures of lists,
    dictionaries, and tuples with global functions. It's going to take me
    a while to kick those horrible habits (during my transition, I'm
    deriving from list, dict, etc. hehe), perhaps a link to the metaclass
    trick is in order in the tutorial's comments on reload?

    Any help that avoids having to unpickle again is appreciated!

    Thanks,
    Andrew Felch
    This happens because unpickling doesn't recreate your object by
    calling its type. MetaInstanceTracker registers an instance only
    when it is created by calling a class.

    You can solve this by moving the instance registration to
    AutoReloader.__new__ and using pickle protocol version 2, but the
    best solution is to avoid both pickle (old pickles break if you
    change your code) and autoreloading (it's meant to be used in
    interactive console and entertaining ircbots, not in normal code).

    Ziga
  • Andrewfelch at Feb 27, 2007 at 10:02 pm

    On Feb 27, 3:50 pm, "Ziga Seilnacht" wrote:
    Andrew Felch wrote:
    Hello all,
    I'm using the metaclass trick for automatic reloading of class member
    functions, found at:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164
    My problem is that if I
    1) pickle an object that inherits from "AutoReloader"
    2) unpickle the object
    3) modify one of the pickled' object's derived class methods
    4) reload the module holding the class
    ... then the changes don't affect the unpickled object. If I unpickle
    the object again, of course the changes take effect.
    My friend that loves Smalltalk is laughing at me. I thought I had the
    upperhand when I discovered the metaclasses but now I am not sure what
    to do. I really don't want to have to unpickle again, I'm processing
    video and it can take a long time.
    By the way, I used to avoid all of these problems by never making
    classes, and always building complex structures of lists,
    dictionaries, and tuples with global functions. It's going to take me
    a while to kick those horrible habits (during my transition, I'm
    deriving from list, dict, etc. hehe), perhaps a link to the metaclass
    trick is in order in the tutorial's comments on reload?
    Any help that avoids having to unpickle again is appreciated!
    Thanks,
    Andrew Felch
    This happens because unpickling doesn't recreate your object by
    calling its type. MetaInstanceTracker registers an instance only
    when it is created by calling a class.

    You can solve this by moving the instance registration to
    AutoReloader.__new__ and using pickle protocol version 2, but the
    best solution is to avoid both pickle (old pickles break if you
    change your code) and autoreloading (it's meant to be used in
    interactive console and entertaining ircbots, not in normal code).

    Ziga- Hide quoted text -

    - Show quoted text -
    Thanks Ziga. I use pickle protocol 2 and binary file types with the
    command: "cPickle.dump(obj, file, 2)"

    I did your suggestion, i commented out the "__call__" function of
    MetaInstanceTracker and copied the text to the __new__ function of
    AutoReloader (code appended). I got a crazy recursive error message
    (also appended below). In my code, I am creating a new instance,
    rather than using the pickled object (it needs to work in both modes).

    Thanks very much for helping me get through this. With my development
    approach, finding a solution to this problem is really important to
    me.

    ...
    File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
    instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
    File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
    instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
    File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
    instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
    File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
    instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
    File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
    instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
    File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
    instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
    RuntimeError: maximum recursion depth exceeded

    <code>
    import weakref, inspect

    class MetaInstanceTracker(type):
    def __new__(cls, name, bases, ns):
    t = super(MetaInstanceTracker, cls).__new__(cls, name, bases,
    ns)
    t.__instance_refs__ = []
    return t
    def __instances__(self):
    instances = [(r, r()) for r in self.__instance_refs__]
    instances = filter(lambda (x,y): y is not None, instances)
    self.__instance_refs__ = [r for (r, o) in instances]
    return [o for (r, o) in instances]
    ## def __call__(self, *args, **kw):
    ## instance = super(MetaInstanceTracker, self).__call__(*args,
    **kw)
    ## self.__instance_refs__.append(weakref.ref(instance))
    ## return instance

    class InstanceTracker:
    __metaclass__ = MetaInstanceTracker

    class MetaAutoReloader(MetaInstanceTracker):
    def __new__(cls, name, bases, ns):
    new_class = super(MetaAutoReloader, cls).__new__(
    cls, name, bases, ns)
    f = inspect.currentframe().f_back
    for d in [f.f_locals, f.f_globals]:
    if d.has_key(name):
    old_class = d[name]
    for instance in old_class.__instances__():
    instance.change_class(new_class)
    new_class.__instance_refs__.append(
    weakref.ref(instance))
    # this section only works in 2.3
    for subcls in old_class.__subclasses__():
    newbases = ()
    for base in subcls.__bases__:
    if base is old_class:
    newbases += (new_class,)
    else:
    newbases += (base,)
    subcls.__bases__ = newbases
    break
    return new_class

    class AutoReloader:
    __metaclass__ = MetaAutoReloader
    def change_class(self, new_class):
    self.__class__ = new_class
    def __new__(self, *args, **kw):
    instance = super(MetaInstanceTracker, self).__call__(*args,
    **kw)
    self.__instance_refs__.append(weakref.ref(instance))
    return instance

    Thanks,
    Andrew Felch
  • Ziga Seilnacht at Feb 27, 2007 at 10:30 pm

    Andrew Felch wrote:
    Thanks Ziga. I use pickle protocol 2 and binary file types with the
    command: "cPickle.dump(obj, file, 2)"

    I did your suggestion, i commented out the "__call__" function of
    MetaInstanceTracker and copied the text to the __new__ function of
    AutoReloader (code appended). I got a crazy recursive error message
    (also appended below). In my code, I am creating a new instance,
    rather than using the pickled object (it needs to work in both modes).

    Thanks very much for helping me get through this. With my development
    approach, finding a solution to this problem is really important to
    me.
    Here is a version that should work. It should work with all protocols,
    see InstanceTracker.__reduce_ex__. Note that all subclasses of
    InstanceTracker and AutoReloader should be cautious when overriding
    the
    __new__ method. They must call their base class' __new__ method,
    preferably by using super(), or the tracking won't work.


    # adapted from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

    import weakref, inspect


    class MetaInstanceTracker(type):

    def __init__(cls, name, bases, ns):
    super(MetaInstanceTracker, cls).__init__(name, bases, ns)
    cls.__instance_refs__ = []

    def __instances__(cls):
    instances = []
    validrefs = []
    for ref in cls.__instance_refs__:
    instance = ref()
    if instance is not None:
    instances.append(instance)
    validrefs.append(ref)
    cls.__instance_refs__ = validrefs
    return instances


    class InstanceTracker(object):

    __metaclass__ = MetaInstanceTracker

    def __new__(*args, **kwargs):
    cls = args[0]
    self = super(InstanceTracker, cls).__new__(*args, **kwargs)
    cls.__instance_refs__.append(weakref.ref(self))
    return self

    def __reduce_ex__(self, proto):
    return super(InstanceTracker, self).__reduce_ex__(2)


    class MetaAutoReloader(MetaInstanceTracker):

    def __init__(cls, name, bases, ns):
    super(MetaAutoReloader, cls).__init__(name, bases, ns)
    f = inspect.currentframe().f_back
    for d in [f.f_locals, f.f_globals]:
    if name in d:
    old_class = d[name]
    for instance in old_class.__instances__():
    instance.change_class(cls)

    cls.__instance_refs__.append(weakref.ref(instance))
    for subcls in old_class.__subclasses__():
    newbases = []
    for base in subcls.__bases__:
    if base is old_class:
    newbases.append(cls)
    else:
    newbases.append(base)
    subcls.__bases__ = tuple(newbases)
    break


    class AutoReloader(InstanceTracker):

    __metaclass__ = MetaAutoReloader

    def change_class(self, new_class):
    self.__class__ = new_class


    Ziga
  • Andrewfelch at Feb 27, 2007 at 10:54 pm

    On Feb 27, 5:30 pm, "Ziga Seilnacht" wrote:
    Andrew Felch wrote:
    Thanks Ziga. I use pickle protocol 2 and binary file types with the
    command: "cPickle.dump(obj, file, 2)"
    I did your suggestion, i commented out the "__call__" function of
    MetaInstanceTracker and copied the text to the __new__ function of
    AutoReloader (code appended). I got a crazy recursive error message
    (also appended below). In my code, I am creating a new instance,
    rather than using the pickled object (it needs to work in both modes).
    Thanks very much for helping me get through this. With my development
    approach, finding a solution to this problem is really important to
    me.
    Here is a version that should work. It should work with all protocols,
    see InstanceTracker.__reduce_ex__. Note that all subclasses of
    InstanceTracker and AutoReloader should be cautious when overriding
    the
    __new__ method. They must call their base class' __new__ method,
    preferably by using super(), or the tracking won't work.

    # adapted fromhttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

    import weakref, inspect

    class MetaInstanceTracker(type):

    def __init__(cls, name, bases, ns):
    super(MetaInstanceTracker, cls).__init__(name, bases, ns)
    cls.__instance_refs__ = []

    def __instances__(cls):
    instances = []
    validrefs = []
    for ref in cls.__instance_refs__:
    instance = ref()
    if instance is not None:
    instances.append(instance)
    validrefs.append(ref)
    cls.__instance_refs__ = validrefs
    return instances

    class InstanceTracker(object):

    __metaclass__ = MetaInstanceTracker

    def __new__(*args, **kwargs):
    cls = args[0]
    self = super(InstanceTracker, cls).__new__(*args, **kwargs)
    cls.__instance_refs__.append(weakref.ref(self))
    return self

    def __reduce_ex__(self, proto):
    return super(InstanceTracker, self).__reduce_ex__(2)

    class MetaAutoReloader(MetaInstanceTracker):

    def __init__(cls, name, bases, ns):
    super(MetaAutoReloader, cls).__init__(name, bases, ns)
    f = inspect.currentframe().f_back
    for d in [f.f_locals, f.f_globals]:
    if name in d:
    old_class = d[name]
    for instance in old_class.__instances__():
    instance.change_class(cls)

    cls.__instance_refs__.append(weakref.ref(instance))
    for subcls in old_class.__subclasses__():
    newbases = []
    for base in subcls.__bases__:
    if base is old_class:
    newbases.append(cls)
    else:
    newbases.append(base)
    subcls.__bases__ = tuple(newbases)
    break

    class AutoReloader(InstanceTracker):

    __metaclass__ = MetaAutoReloader

    def change_class(self, new_class):
    self.__class__ = new_class

    Ziga
    I pasted the code into mine and replaced the old. It seems not to
    work for either unpickled objects or new objects. I add methods to a
    class that inherits from AutoReloader and reload the module, but the
    new methods are not callable on the old objects. Man! It seems we're
    so close, it will be huge if this ends up working. This stuff is so
    over my head, I wish I could help in some additional way.

    -Andrew
  • Ziga Seilnacht at Feb 27, 2007 at 11:23 pm

    Andrew Felch wrote:

    I pasted the code into mine and replaced the old. It seems not to
    work for either unpickled objects or new objects. I add methods to a
    class that inherits from AutoReloader and reload the module, but the
    new methods are not callable on the old objects. Man! It seems we're
    so close, it will be huge if this ends up working. This stuff is so
    over my head, I wish I could help in some additional way.

    -Andrew
    Did you copy and paste the entire module? I fiddled almost all parts
    of
    the original code. I did some base testing and it worked for me. Could
    you post the traceback? Note that Google Groups messed up indentation;
    in MetaAutoReloader.__init__, the line starting with
    cls.__instance_refs__ should be at the same level as previous line.

    Did you restart Python? InstanceTracker, MetaInstanceTracker and
    MetaAutoReloader are not auto reloaded :).

    Ziga
  • Andrewfelch at Feb 27, 2007 at 11:36 pm

    On Feb 27, 3:23 pm, "Ziga Seilnacht" wrote:
    Andrew Felch wrote:
    I pasted the code into mine and replaced the old. It seems not to
    work for either unpickled objects or new objects. I add methods to a
    class that inherits from AutoReloader and reload the module, but the
    new methods are not callable on the old objects. Man! It seems we're
    so close, it will be huge if this ends up working. This stuff is so
    over my head, I wish I could help in some additional way.
    -Andrew
    Did you copy and paste the entire module? I fiddled almost all parts
    of
    the original code. I did some base testing and it worked for me. Could
    you post the traceback? Note that Google Groups messed up indentation;
    in MetaAutoReloader.__init__, the line starting with
    cls.__instance_refs__ should be at the same level as previous line.

    Did you restart Python? InstanceTracker, MetaInstanceTracker and
    MetaAutoReloader are not auto reloaded :).

    Ziga
    Thanks for checking. I think I narrowed the problem down to
    inheritance. I inherit from list or some other container first:

    class PointList( list, AutoReloader ):
    def PrintHi1(self):
    print "Hi2"

    class MyPrintingClass( AutoReloader ):
    def PrintHi2(self):
    print "Hi2v2"

    Automatic reloading works for MyPrintingClass but not for PointList.
    Any ideas?

    -Andrew
  • Ziga Seilnacht at Feb 27, 2007 at 11:47 pm

    Andrew Felch wrote:

    Thanks for checking. I think I narrowed the problem down to
    inheritance. I inherit from list or some other container first:

    class PointList( list, AutoReloader ):
    def PrintHi1(self):
    print "Hi2"

    class MyPrintingClass( AutoReloader ):
    def PrintHi2(self):
    print "Hi2v2"

    Automatic reloading works for MyPrintingClass but not for PointList.
    Any ideas?

    -Andrew
    Ah yes, this is the problem of list.__new__ not calling the next
    class in MRO. Try to switch the bases, so that AutoReloader's
    __new__ method will be called first.

    Ziga
  • Andrewfelch at Feb 28, 2007 at 12:49 am

    On Feb 27, 3:47 pm, "Ziga Seilnacht" wrote:
    Andrew Felch wrote:
    Thanks for checking. I think I narrowed the problem down to
    inheritance. I inherit from list or some other container first:
    class PointList( list, AutoReloader ):
    def PrintHi1(self):
    print "Hi2"
    class MyPrintingClass( AutoReloader ):
    def PrintHi2(self):
    print "Hi2v2"
    Automatic reloading works for MyPrintingClass but not for PointList.
    Any ideas?
    -Andrew
    Ah yes, this is the problem of list.__new__ not calling the next
    class in MRO. Try to switch the bases, so that AutoReloader's
    __new__ method will be called first.

    Ziga
    That did it! Thanks so much!

    This really seems to extend Python to be a much better rapid
    prototyping langauge. It certainly allows me to now use classes where
    I avoided them like the plague before. Perhaps I'll be able to make
    programs > 1500 lines now! :-)

    -Andrew
  • Andrewfelch at Feb 28, 2007 at 5:49 pm

    On Feb 27, 6:47 pm, "Ziga Seilnacht" wrote:
    Andrew Felch wrote:
    Thanks for checking. I think I narrowed the problem down to
    inheritance. I inherit from list or some other container first:
    class PointList( list, AutoReloader ):
    def PrintHi1(self):
    print "Hi2"
    class MyPrintingClass( AutoReloader ):
    def PrintHi2(self):
    print "Hi2v2"
    Automatic reloading works for MyPrintingClass but not for PointList.
    Any ideas?
    -Andrew
    Ah yes, this is the problem of list.__new__ not calling the next
    class in MRO. Try to switch the bases, so that AutoReloader's
    __new__ method will be called first.

    Ziga- Hide quoted text -

    - Show quoted text -
    So there is another bug crippling Python (see
    https://sourceforge.net/tracker/?funcÞtail&atid5470&aidb8925&group_idT70
    ). Pickle and cPickle get confused after reloading, and won't let you
    dump the objects to file using protocol 2. The bug occurred for me
    when an instance of an AutoReloader class had a member not derived
    from AutoReloader (error text: "it's not the same object" ). The
    workaround is to use text mode, which I assume means not using
    protocol 2. After that, you can reload them in text mode, and then
    dump them using protocol 2. This is not so bad since you only do it
    when you were going to dump objects to file anyway, so it really only
    takes 3x as long on those occassions. The metaclass adaptation is
    still incredibly useful because it avoids having to do pickle dump-
    loads when you wouldn't otherwise have had to (approaching an infinite
    speedup in those cases, thanks Ziga! :-).

    Another caveat is that pickle dumping doesn't work if there is a
    pointer to a global function (that changes) that is a member of an
    instance of a class derived from AutoLoader. The error message has
    subtext: "it's not the same object ". That doesn't seem so bad to me,
    since you don't have to do that type of thing (function pointers) so
    much now that you can use classes!

    The best solution would be to make pickle protocol 2 have the same
    behavior as the text protocol (i.e. actually work in this case).

    Regarding the function pointers, anybody (Ziga :-) know why it is that
    pickle chokes so hard on this? Is there some adaptation of the
    metaclass or a less beautiful hack that could fix it?

    Thanks to those readers following me down the rabbit hole. Ziga's
    metaclass adaptation has saved Python in my mind, and these nits seem
    to be worth discussing.

    - Andrew Felch

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedFeb 27, '07 at 7:41p
activeFeb 28, '07 at 5:49p
posts10
users2
websitepython.org

2 users in discussion

Andrewfelch: 6 posts Ziga Seilnacht: 4 posts

People

Translate

site design / logo © 2022 Grokbase