FAQ
Hello!

I have been thinking about how write exception safe constructors in
Python. By exception safe I mean a constructor that does not leak
resources when an exception is raised within it. The following is an
example of one possible way to do it:

class Foo(object):
def __init__(self, name, fail=False):
self.name = name
if not fail:
print '%s.__init__(%s)' % (self.__class__.__name__,
self.name)
else:
print '%s.__init__(%s), FAIL' % (self.__class__.__name__,
self.name)
raise Exception()

def close(self):
print '%s.close(%s)' % (self.__class__.__name__, self.name)

class Bar(object):
def __init__(self):
try:
self.a = Foo('a')
self.b = Foo('b', fail=True)
except:
self.close()


def close(self):
if hasattr(self, 'a'):
self.a.close()
if hasattr(self, 'b'):
self.b.close()

bar = Bar()

As you can see this is less than straight forward. Is there some kind
of best practice that I'm not aware of?

:.:: mattias

Search Discussions

  • Scott David Daniels at Jul 6, 2009 at 9:15 pm

    brasse wrote:
    I have been thinking about how write exception safe constructors in
    Python. By exception safe I mean a constructor that does not leak
    resources when an exception is raised within it. ...
    As you can see this is less than straight forward. Is there some kind
    of best practice that I'm not aware of?
    Not so tough. Something like this tweaked version of your example:

    class Foo(object):
    def __init__(self, name, fail=False):
    self.name = name
    if not fail:
    print '%s.__init__(%s)' % (type(self).__name__, name)
    else:
    print '%s.__init__(%s), FAIL' % (type(self).__name__, name)
    raise ValueError('Asked to fail: %r' % fail)

    def close(self):
    print '%s.close(%s)' % (type(self).__name__, self.name)


    class Bar(object):
    def __init__(self):
    unwind = []
    try:
    self.a = Foo('a')
    unwind.append(a)
    self.b = Foo('b', fail=True)
    unwind.append(b)
    ...
    except Exception, why:
    while unwind):
    unwind.pop().close()
    raise

    bar = Bar()

    --Scott David Daniels
    Scott.Daniels at Acm.Org
  • Gabriel Genellina at Jul 6, 2009 at 11:49 pm
    En Mon, 06 Jul 2009 18:15:44 -0300, Scott David Daniels
    <Scott.Daniels at acm.org> escribi?:
    brasse wrote:
    I have been thinking about how write exception safe constructors in
    Python. By exception safe I mean a constructor that does not leak
    resources when an exception is raised within it. ...
    As you can see this is less than straight forward. Is there some kind
    of best practice that I'm not aware of?
    Not so tough. Something like this tweaked version of your example:
    Another variant: Presumably, there is already a method in Bar responsible
    for "normal" cleanup; just make sure it gets called (and write it in a
    robust way):
    class Foo(object):
    def __init__(self, name, fail=False):
    self.name = name
    if not fail:
    print '%s.__init__(%s)' % (type(self).__name__, name)
    else:
    print '%s.__init__(%s), FAIL' % (type(self).__name__, name)
    raise ValueError('Asked to fail: %r' % fail)

    def close(self):
    print '%s.close(%s)' % (type(self).__name__, self.name)
    class Bar(object):
    a = None # default values
    b = None

    def __init__(self):
    try:
    self.a = Foo('a')
    self.b = Foo('b', fail=True)
    except Exception, why:
    self.cleanup()
    raise

    def cleanup(self):
    if self.a is not None:
    self.a.close()
    if self.b is not None:
    self.b.close()

    bar = Bar()

    --
    Gabriel Genellina
  • Mattias Brändström at Jul 8, 2009 at 10:43 pm

    On Jul 6, 11:15?pm, Scott David Daniels wrote:
    brasse wrote:
    I have been thinking about how write exception safe constructors in
    Python. By exception safe I mean a constructor that does not leak
    resources when an exception is raised within it.
    ...
    ?> As you can see this is less than straight forward. Is there some kind
    ?> of best practice that I'm not aware of?

    Not so tough. ?Something like this tweaked version of your example:

    class Foo(object):
    ? ? ?def __init__(self, name, fail=False):
    ? ? ? ? ?self.name = name
    ? ? ? ? ?if not fail:
    ? ? ? ? ? ? ?print '%s.__init__(%s)' % (type(self).__name__, name)
    ? ? ? ? ?else:
    ? ? ? ? ? ? ?print '%s.__init__(%s), FAIL' % (type(self).__name__, name)
    ? ? ? ? ? ? ?raise ValueError('Asked to fail: %r' % fail)

    ? ? ?def close(self):
    ? ? ? ? ?print '%s.close(%s)' % (type(self).__name__, self.name)

    class Bar(object):
    ? ? ?def __init__(self):
    ? ? ? ? ?unwind = []
    ? ? ? ? ?try:
    ? ? ? ? ? ? ?self.a = Foo('a')
    ? ? ? ? ? ? ?unwind.append(a)
    ? ? ? ? ? ? ?self.b = Foo('b', fail=True)
    ? ? ? ? ? ? ?unwind.append(b)
    ? ? ? ? ? ? ?...
    ? ? ? ? ?except Exception, why:
    ? ? ? ? ? ? ?while unwind):
    ? ? ? ? ? ? ? ? ?unwind.pop().close()
    ? ? ? ? ? ? ?raise

    bar = Bar()
    OK. That's another way. I have been thinking about this some more and
    right now I am experimenting with a decorator that could help me do
    these kinds of things. This is what I have right now:

    import functools

    class Foo(object):
    def __init__(self, name, fail=False):
    self.name = name
    self.closed = False
    if not fail:
    print '%s.__init__(%s)' % (self.__class__.__name__,
    self.name)
    else:
    print '%s.__init__(%s), FAIL' % (self.__class__.__name__,
    self.name)
    raise Exception()

    def close(self):
    print '%s.close(%s)' % (self.__class__.__name__, self.name)

    def safe(f):
    @functools.wraps(f)
    def wrapper(self, *args, **kwargs):
    variables = []
    def recorder(self, name, value):
    if name != '__class__':
    variables.append(name)
    object.__setattr__(self, name, value)

    class Customized(object): pass
    setattr(Customized, '__setattr__', recorder)
    old_class = self.__class__
    self.__class__ = Customized

    try:
    f(self, *args, **kwargs)
    self.__class__ = old_class
    except:
    # clean up objects created in __init__ (i.e. the call to f
    ())
    for name in reversed(variables):
    o = getattr(self, name)
    if hasattr(o, 'close'):
    o.close()
    raise
    return wrapper

    class Bar(object):
    @safe
    def __init__(self):
    self.a = Foo('a')
    self.b = Foo('b')
    self.c = Foo('c', fail=True)

    bar = Bar()

    The safe decorator will record all attributes created on self. If an
    exception is raised during the execution of the decorated constructor
    it will call close on all the recorded objects. I realize that this
    decorator still needs a lot of work before it is usable, but I think
    it illustrates the concept. It might even be possible for the
    decorator to create a close/cleanup method dynamically.

    To have a decorator like this that actually worked would be great
    since it would remove the need to write error prone cleanup code. Any
    thoughts?

    :.:: mattias
    --Scott David Daniels
    Scott.Dani... at Acm.Org
  • Lie Ryan at Jul 9, 2009 at 5:30 pm

    brasse wrote:
    Hello!
    I have been thinking about how write exception safe constructors in
    Python. By exception safe I mean a constructor that does not leak
    resources when an exception is raised within it. The following is an
    example of one possible way to do it:
    First, your automatic cleaning Bar() silently pass an unitialized Bar()
    into the calling code. When a code fails, it should fail as loud as
    possible. Bar() should raises an Exception and the calling code should
    turn into something like:

    try:
    bar = Bar()
    except Exception, e:
    print 'bar failed to construct'

    And about the cleaning up, how about:

    class CleanWrap(object):
    def __init__(self, cls, cleanup):
    self.cls = cls
    self.cleanup = cleanup
    def __call__(self, *args, **kargs):
    try:
    return self.cls(*args, **kargs)
    except:
    self.cleanup()
    raise

    class Bar(object):
    def __init__(self):
    CleanWrappedFoo = CleanWrap(Foo, self.close)
    self.a = CleanWrappedFoo('a')
    self.b = CleanWrappedFoo('b', fail=True)
    def close(self):
    if hasattr(self, 'a'):
    self.a.close()
    if hasattr(self, 'b'):
    self.b.close()

    try:
    bar = Bar()
    except Exception, e:
    print 'Bar() failed to construct'

    ==========
    My next attempt is pretty neat, using a totally different approach, see
    the docstring for details:

    class Foo1(object):
    def __init__(self, name, fail=False):
    self.name = name
    cls_name = self.__class__.__name__
    if not fail:
    print '%s.__init__(%s)' % (cls_name, self.name)
    else:
    print '%s.__init__(%s), FAIL' % (cls_name, self.name)
    raise Exception()

    def close(self):
    print '%s.close(%s)' % (self.__class__.__name__, self.name)

    class Foo2(object):
    def __init__(self, name, fail=False):
    self.name = name
    cls_name = self.__class__.__name__
    if not fail:
    print '%s.__init__(%s)' % (cls_name, self.name)
    else:
    print '%s.__init__(%s), FAIL' % (cls_name, self.name)
    raise Exception()

    def close(self):
    print '%s.close(%s)' % (self.__class__.__name__, self.name)


    class CleanList(object):
    ''' Each CleanList() instance is rigged so that if exceptions happen
    in the same CleanList() instance's wrapped class, all objects
    created from the wrapped classes in the same CleanList()
    instance will be cleaned (see "Usage" for much better
    explanation).

    Usage:
    cleaner = CleanList()
    othercleaner = CleanList()
    F = cleaner(F, F.close)
    G = cleaner(G, G.close)
    H = othercleaner(H, H.close)
    a = F()
    b = F()
    c = G()
    d = H()
    cleaner.cleanall()
    cleaner.cleanall() will clean a, b, and c but not d
    exceptions in (F|G|H).__init__ will trigger
    cleaner.cleanall()

    Can be subclassed if you want to override the conditions
    that determines the triggering of cleanups
    '''
    def wrap(self, cls):
    ''' Wrapper factory that customizes Wrapper's subclass
    '''
    class Wrapper(cls):
    ''' Wraps the class' __init__ with cleanup guard.
    Subclasses cls to simulate cls as close as possible
    '''

    # change __class__ to make printing prettier
    # e.g. Foo1.__init__ instead of Wrapper.__init__
    # probably should be removed
    # I'm not sure of the side effects of changing __class__
    __class__ = cls

    def __init__(self_in, *args, **kargs):
    try:
    sup = super(Wrapper, self_in)
    ret = sup.__init__(*args, **kargs)
    except:
    self.cleanall()
    raise
    else:
    self.add_to_list(cls, self_in)
    return ret
    return Wrapper

    def __init__(self):
    self.cleaners = {}

    def __call__(self, cls, cleanup):
    ''' wraps the class constructor '''
    # cleanup, []:
    # cleanup is the function called to clean
    # [] is the object list for that `cleanup` function
    # may not be the best data structure, but it works...
    self.cleaners[cls] = cleanup, []
    return self.wrap(cls)

    def cleanall(self):
    ''' clean all objects '''
    for cleaner, insts in self.cleaners.values():
    for inst in insts:
    cleaner(inst)

    def add_to_list(self, cls, inst):
    ''' add objects to the cleanup list '''
    self.cleaners[cls][1].append(inst)

    class Bar(object):
    def __init__(self):
    clist = CleanList()
    otherclist = CleanList()
    CleanFoo1 = clist(Foo1, Foo1.close)
    CleanFoo2 = clist(Foo2, Foo2.close)
    OtherCleanFoo1 = otherclist(Foo1, Foo1.close)
    self.a = CleanFoo1('a')
    self.b = CleanFoo2('b')
    # self.c should not be close()ed
    self.c = OtherCleanFoo1('c')
    self.d = CleanFoo1('d', fail=True)
    self.e = CleanFoo2('e')

    class Car(object):
    def __init__(self):
    Clean = CleanList()
    CleanFoo1 = Clean(Foo1, Foo1.close)
    CleanFoo2 = Clean(Foo2, Foo2.close)
    self.a = CleanFoo1('a')
    self.b = CleanFoo2('b')
    self.c = CleanFoo1('c')
    self.d = CleanFoo2('d')

    try:
    bar = Car()
    except Exception, e:
    print e
    print 'Car() failed to construct'
    print
    try:
    bar = Bar()
    except Exception, e:
    print e
    print 'Bar() failed to construct'
  • Mattias Brändström at Jul 9, 2009 at 9:10 pm

    On Jul 9, 7:30?pm, Lie Ryan wrote:
    brasse wrote:
    Hello!
    I have been thinking about how write exception safe constructors in
    Python. By exception safe I mean a constructor that does not leak
    resources when an exception is raised within it. The following is an
    example of one possible way to do it:
    First, your automatic cleaning Bar() silently pass an unitialized Bar()
    into the calling code. When a code fails, it should fail as loud as
    possible. Bar() should raises an Exception and the calling code should
    turn into something like:

    try:
    ? ? bar = Bar()
    except Exception, e:
    ? ? print 'bar failed to construct'
    I know, I missed that little detail. :-)
    And about the cleaning up, how about:

    class CleanWrap(object):
    ? ? def __init__(self, cls, cleanup):
    ? ? ? ? self.cls = cls
    ? ? ? ? self.cleanup = cleanup
    ? ? def __call__(self, *args, **kargs):
    ? ? ? ? try:
    ? ? ? ? ? ? return self.cls(*args, **kargs)
    ? ? ? ? except:
    ? ? ? ? ? ? self.cleanup()
    ? ? ? ? ? ? raise

    class Bar(object):
    ? ? def __init__(self):
    ? ? ? ? CleanWrappedFoo = CleanWrap(Foo, self.close)
    ? ? ? ? self.a = CleanWrappedFoo('a')
    ? ? ? ? self.b = CleanWrappedFoo('b', fail=True)
    ? ? def close(self):
    ? ? ? ? if hasattr(self, 'a'):
    ? ? ? ? ? ? self.a.close()
    ? ? ? ? if hasattr(self, 'b'):
    ? ? ? ? ? ? self.b.close()
    I think this example adds about as much overhead as my original
    example with the try block. Ideally I would like to be able to write
    my classes in the most straight forward manner possible. In my mind
    that would be something like a naive constructor and a close/dispose
    method:

    def __init__(self):
    self.a = Foo()
    self.b = Bar()

    def close(self):
    self.a.close()
    self.b.close()

    See my other post (the one with the decorator named safe) in this
    thread for an experiment I thought was promising for a while. However
    there are some some use cases it has to cover to be useful:

    (1) References to objects owned by someone else
    def __init__(self, some_resource):
    self.not_my_responsibility = some_resource
    self.a = Foo()
    # If Foo.__init__ raises an exception self.not_my responsibility
    should not be closed.

    (2) Local objects
    def __init__(self):
    x = Bar()
    self.a = Foo(x)
    # If Foo.__init__ raises an exception x should also be closed.

    So far I have not been able to find any good solutions for (at least)
    case 1. I'll give it a few more days, but I'm not that hopeful. I
    think that I will end up writing my classes very much like Bar in my
    original post. If someone has any ideas for how to solve my two use
    cases above with my decorator based approach I would really like to
    hear them!

    :.:: mattias
  • Shai at Jul 15, 2009 at 10:45 pm
    Since nobody else mentioned this...

    Python classes have a magic method called __del__ which is usually
    called just before an object is garbage-collected. Further, Python
    uses reference-counting to tell when an object is no longer
    accessible. This means that if your resource classes define __del__
    methods, these will be called properly when the object using them is
    destroyed, and you don't need to write an explicit close() method for
    this.

    class Resource(object):
    def __init__(self, param):
    # acquire resource

    def __del__(self):
    # release resource

    not_my_responsibility = Resource(1)

    class Foo(object):
    def __init__(self):
    self.ref = not_my_responsibility # self.ref.__del__() will not be
    called as long as the module exists
    local = Resource(2) # local.__del__() will be called as soon as
    __init__ is exited
    self.held = Resource(3) # self.held.__del__() will be called when
    the object dies
    z = 1/0 # The exception in the initializer will kill the object,
    triggering some Resource.__del__() calls

    There are two caveats:

    1) __del__ methods prevent instances of your class from being
    collected when they are involved in cyclical structures; this means if
    your structures start to get complex (sometimes a doubly-linked list
    is complex enough), you may find yourself leaking memory.

    2) The bit about reference counting, which allows predictable
    destruction, holds for CPython, but not for Jython, and IIRC also not
    for IronPython (I don't know about PyPy or other implementations). It
    is a feature of the reference implementation, not the language
    definition.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJul 6, '09 at 8:36p
activeJul 15, '09 at 10:45p
posts7
users5
websitepython.org

People

Translate

site design / logo © 2022 Grokbase