FAQ
Hi,

I'm having a rather obscure problem with my custom __reduce__ function. I can't use
__getstate__ to customize the pickling of my class because I want to change the actual
type that is put into the pickle stream. So I started experimenting with __reduce__, but
am running into some trouble.

I've pasted my test code below. It works fine if 'substitute' is True, but as soon as it
is set to False, it is supposed to call the original __reduce__ method of the base
class. However, that seems to crash because of infinite recursion on Jython and
IronPython and I don't know why. It works fine in CPython and Pypy.

I wonder if my understanding of __reduce__ is wrong, or that I've hit a bug in
IronPython and Jython? Do I need to do something with __reduce_ex__ as well?

Any help is very much appreciated.

Irmen de Jong



# ironpython / jython __reduce__ recursion problem test program

import pickle

class Substitute(object):
def __init__(self, name):
self.name=name
def getname(self):
return self.name

class TestClass(object):
def __init__(self, name):
self.name=name
self.substitute=True
def getname(self):
return self.name
def __reduce__(self):
if self.substitute:
return Substitute, ("SUBSTITUTED:"+self.name,)
else:
# call the original __reduce__ from the base class
return super(TestClass, self).__reduce__() # crashes on ironpython/jython

obj=TestClass("janet")

s=pickle.dumps(obj)
d=pickle.loads(s)
print(d)
print(d.getname())

# now disable the substitution and try again
obj.substitute=False
s=pickle.dumps(obj)
d=pickle.loads(s)
print(d)
print(d.getname())

Search Discussions

  • Chris Torek at Jun 14, 2011 at 12:40 am
    In article <4df669ea$0$49182$e4fe514c at news.xs4all.nl>
    Irmen de Jong wrote:
    I've pasted my test code below. It works fine if 'substitute' is True,
    but as soon as it is set to False, it is supposed to call the original
    __reduce__ method of the base class. However, that seems to crash
    because of infinite recursion on Jython and IronPython and I don't
    know why. It works fine in CPython and Pypy.
    In this particular case (no fancy inheritance going on), the base
    __reduce__ method would be object.__reduce__. Perhaps in those
    implementations, object.__reduce__ goes back to TestClass.__reduce__,
    rather than being appropriately magic.
    I wonder if my understanding of __reduce__ is wrong, or that I've
    hit a bug in IronPython and Jython? Do I need to do something with
    __reduce_ex__ as well?
    You should not *need* to; __reduce_ex__ is just there so that you
    can do something different for different versions of the pickle
    protocol (I believe).

    Nonetheless, there is something at least slightly suspicious here:
    import pickle

    class Substitute(object):
    def __init__(self, name):
    self.name=name
    def getname(self):
    return self.name

    class TestClass(object):
    def __init__(self, name):
    self.name=name
    self.substitute=True
    def getname(self):
    return self.name
    def __reduce__(self):
    if self.substitute:
    return Substitute, ("SUBSTITUTED:"+self.name,)
    else:
    # call the original __reduce__ from the base class
    return super(TestClass, self).__reduce__() # crashes on
    ironpython/jython
    [snip]

    In general, the way __reduce__ is written in other class implementations
    (as distributed with Python2.5 at least) boils down to the very
    simple:

    def __reduce__(self):
    return self.__class__, (arg, um, ents)

    For instance, consider a class with a piece that looks like this:

    def __init__(self, name, value):
    self.name = name
    self.value = value
    self.giant_cached_state = None

    def make_parrot_move(self):
    if self.giant_cached_state is None:
    self._do_lots_of_computation()
    return self._quickstuff_using_cache()

    Here, the Full Internal State is fairly long but the part that
    needs to be saved (or, for copy operations, copied -- but you can
    override this with __copy__ and __deepcopy__ members, if copying
    the cached state is a good idea) is quite short. Pickled instances
    need only save the name and value, not any of the computed cached
    stuff (if present). So:

    def __reduce__(self):
    return self.__class__, (name, value)

    If you define this (and no __copy__ and no __deepcopy__), the
    pickler will save the name and value and call __init__ with the
    name and value arguments. The copy.copy and copy.deepcopy operations
    will also call __init__ with these arguments (unless you add
    __copy__(self) and __deepcopy__(self) functions).

    So, it seems like in this case, you would want:

    def __reduce__(self):
    if self.substitute:
    return Substitute, ("SUBSTITUTED:"+self.name,)
    else:
    return self.__class__, (self.name,)

    or if you want to be paranoid and only do a Substitute if
    self.__class__ is your own class:

    if type(self) == TestClass and self.substitute:
    return Substitute, ("SUBSTITUTED:"+self.name,)
    else:
    return self.__class__, (self.name,)

    In CPython, if I import your code (saved in foo.py):
    x = foo.TestClass("janet")
    x
    <foo.TestClass object at 0x66290>
    x.name
    'janet'
    x.__reduce__()
    (<class 'foo.Substitute'>, ('SUBSTITUTED:janet',))
    x.substitute=False
    x.__reduce__()
    (<function _reconstructor at 0x70bf0>, (<class 'foo.TestClass'>, <type 'object'>, None), {'name': 'janet', 'substitute': False})

    which is of course the same as:
    object.__reduce__(x)
    (<function _reconstructor at 0x70bf0>, (<class 'foo.TestClass'>, <type 'object'>, None), {'name': 'janet', 'substitute': False})

    which means that CPython's object.__reduce__() uses a "smart" fallback
    reconstructor. Presumably IronPython and Jython lack this.
    --
    In-Real-Life: Chris Torek, Wind River Systems
    Salt Lake City, UT, USA (40?39.22'N, 111?50.29'W) +1 801 277 2603
    email: gmail (figure it out) http://web.torek.net/torek/index.html
  • Irmen de Jong at Jun 14, 2011 at 6:17 pm

    On 14-6-2011 2:40, Chris Torek wrote:
    Nonetheless, there is something at least slightly suspicious here:
    [... snip explanations...]

    Many thanks Chris, for the extensive reply. There's some useful knowledge in it.

    My idea to call the base class reduce as the default fallback causes the problems:
    return return super(TestClass, self).__reduce__()
    If, following your suggestion, I replace that with:
    return self.__class__, (self.name,)
    it works fine.

    By the way, in the meantime I've played around with copyreg.pickle, and that code worked
    in all Python implementations I've tried it in. This code feels better too, because it
    is possible to simply use
    return self.__reduce__()
    as a fallback in it, because we're not touching the object's own __reduce__ in any way.

    Anyway thanks again

    Irmen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 13, '11 at 7:50p
activeJun 14, '11 at 6:17p
posts3
users2
websitepython.org

2 users in discussion

Irmen de Jong: 2 posts Chris Torek: 1 post

People

Translate

site design / logo © 2022 Grokbase