FAQ
Given this iterator:

class SomeIterableObject(object):
....
....

def __iter__(self):
ukeys = self.updates.keys()
for key in ukeys:
if self.updates.has_key(key):
yield self.updates[key]
for rec in self.inserts:
yield rec
....
....

How can I get this exception:

RuntimeError: dictionary changed size during iteration


It is true that self.updates is being changed during the iteration. But
I have created the "ukeys" variable solely to prevent this kind of
error. Here is a proof of correctness:
d = {1:1,2:2}
k = d.keys()
del d[1]
k
[1, 2]
k is d.keys()
False

So what is wrong with this iterator? Why am I getting this error message?

Thanks,

Laszlo

Search Discussions

  • Peter Otten at Apr 20, 2011 at 1:33 pm

    Laszlo Nagy wrote:

    Given this iterator:

    class SomeIterableObject(object):
    ....
    ....

    def __iter__(self):
    ukeys = self.updates.keys()
    for key in ukeys:
    if self.updates.has_key(key):
    yield self.updates[key]
    for rec in self.inserts:
    yield rec
    ....
    ....

    How can I get this exception:

    RuntimeError: dictionary changed size during iteration


    It is true that self.updates is being changed during the iteration. But
    I have created the "ukeys" variable solely to prevent this kind of
    error. Here is a proof of correctness:
    d = {1:1,2:2}
    k = d.keys()
    del d[1]
    k
    [1, 2]
    k is d.keys()
    False

    So what is wrong with this iterator? Why am I getting this error message?
    The keys() method which used to return a list in 2.x was changed in 3.x to
    return a view object and to become more or less the equivalent of the old
    dict.iterkeys():
    d = dict(a=1)
    keys = d.keys()
    keys
    dict_keys(['a'])
    for k in keys:
    ... d["b"] = 42
    ...
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration
    keys
    dict_keys(['a', 'b'])

    You now have to create the list explicitly to avoid the error:
    d = dict(a=1)
    keys = list(d.keys())
    for k in keys:
    ... d["b"] = 42
    ...
    d
    {'a': 1, 'b': 42}
    keys
    ['a']
  • Peter Otten at Apr 20, 2011 at 1:40 pm

    Peter Otten wrote:

    Laszlo Nagy wrote:
    Given this iterator:

    class SomeIterableObject(object):
    ....
    ....

    def __iter__(self):
    ukeys = self.updates.keys()
    for key in ukeys:
    if self.updates.has_key(key):
    Hm, I see you are using has_key() which is 2.x-only and invalidates my
    explanation :(
    yield self.updates[key]
    for rec in self.inserts:
    yield rec
    ....
    ....

    How can I get this exception:

    RuntimeError: dictionary changed size during iteration


    It is true that self.updates is being changed during the iteration. But
    I have created the "ukeys" variable solely to prevent this kind of
    error. Here is a proof of correctness:
    d = {1:1,2:2}
    k = d.keys()
    del d[1]
    k
    [1, 2]
    k is d.keys()
    False

    So what is wrong with this iterator? Why am I getting this error message?
    The keys() method which used to return a list in 2.x was changed in 3.x to
    return a view object and to become more or less the equivalent of the old
    dict.iterkeys():
    d = dict(a=1)
    keys = d.keys()
    keys
    dict_keys(['a'])
    for k in keys:
    ... d["b"] = 42
    ...
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration
    keys
    dict_keys(['a', 'b'])

    You now have to create the list explicitly to avoid the error:
    d = dict(a=1)
    keys = list(d.keys())
    for k in keys:
    ... d["b"] = 42
    ...
    d
    {'a': 1, 'b': 42}
    keys
    ['a']
  • Mel at Apr 20, 2011 at 2:18 pm

    Laszlo Nagy wrote:

    Given this iterator:

    class SomeIterableObject(object):
    ....
    ....

    def __iter__(self):
    ukeys = self.updates.keys()
    for key in ukeys:
    if self.updates.has_key(key):
    yield self.updates[key]
    for rec in self.inserts:
    yield rec
    ....
    ....

    How can I get this exception:

    RuntimeError: dictionary changed size during iteration


    It is true that self.updates is being changed during the iteration. But
    I have created the "ukeys" variable solely to prevent this kind of
    error. Here is a proof of correctness:
    d = {1:1,2:2}
    k = d.keys()
    del d[1]
    k
    [1, 2]
    k is d.keys()
    False

    So what is wrong with this iterator? Why am I getting this error message?
    `ukeys` isn't a different dictionary from `self.updates.keys` I'ts merely
    another name referring to the same dict object. I think

    ukeys = dict (self.updates.keys)

    would do what you want.

    Mel.
  • Mel at Apr 20, 2011 at 2:22 pm

    Mel wrote:
    Laszlo Nagy wrote:
    `ukeys` isn't a different dictionary from `self.updates.keys` I'ts merely
    another name referring to the same dict object. I think

    ukeys = dict (self.updates.keys)

    would do what you want.
    Sorry. Belay that. Thought I'd had enough coffee.

    Mel.
  • John Nagle at Apr 22, 2011 at 6:22 am

    On 4/20/2011 5:52 AM, Laszlo Nagy wrote:
    Given this iterator:

    class SomeIterableObject(object):
    ....
    ....

    def __iter__(self):
    ukeys = self.updates.keys()
    for key in ukeys:
    if self.updates.has_key(key):
    yield self.updates[key]
    for rec in self.inserts:
    yield rec
    ....
    ....

    How can I get this exception:

    RuntimeError: dictionary changed size during iteration


    It is true that self.updates is being changed during the iteration. But
    I have created the "ukeys" variable solely to prevent this kind of
    error. Here is a proof of correctness:
    I think we need to see some more code here.

    Also, what version of Python are you running?

    http://docs.python.org/library/stdtypes.html

    says "keys()
    Return a copy of the dictionary?s list of keys."

    So "ukeys" should be a unique list, not some part of
    self.updates or a generator.

    At least in Python 2.6, keys() does return a unique list. Each
    call to "keys()" returns a new list object unconnected to the
    dictionary from which the keys were extracted. But someone may
    have decided in a later version to return a generator, as
    an optimization. Did that happen?

    John Nagle
  • Mark Niemczyk at Apr 22, 2011 at 12:39 pm
    As of Python 3.x (which I suspect you are running):

    "The objects returned by dict.keys(), dict.values() and dict.items() are view objects. They provide a dynamic view on the dictionary?s entries, which means that when the dictionary changes, the view reflects these changes.", and

    "Iterating views while adding or deleting entries in the dictionary may raise a RuntimeError or fail to iterate over all entries."

    see: http://docs.python.org/release/3.1.3/library/stdtypes.html#dict-views

    On my system:
    ActivePython 3.2.0.0 (ActiveState Software Inc.) based on
    Python 3.2 (r32:88445, Feb 21 2011, 11:25:33)
    [GCC 4.2.1 (Apple Inc. build 5664)] on darwin
    Type "copyright", "credits" or "license()" for more information.
    ukeys = {}
    type(ukeys.keys())
    <class 'dict_keys'>
    >>>

    So, to achieve your objective, one solution could be to change the ukeys assignment statement to:

    ukeys = list(self.updates.keys())



    Hope this helps,

    Mark N.
  • Roy Smith at Apr 22, 2011 at 1:15 pm
    In article <mailman.644.1303306435.9059.python-list at python.org>,
    Peter Otten wrote:
    You now have to create the list explicitly to avoid the error:
    d = dict(a=1)
    keys = list(d.keys())
    for k in keys:
    ... d["b"] = 42
    ...
    That works, but if d is large, it won't be very efficient because it has
    to generate a large list.

    If d is large, and the number of keys to be mutated is relatively small,
    a better solution may be to do it in two passes. The first loop
    traverses the iterator and builds a list of things to be changed. The
    second loop changes them.

    changes = [ ]
    for key in d.iterkeys():
    if is_bad(key):
    changes.append(key)
    for key in changes:
    d[key] = "I'm not dead yet"

    Both solutions are O(n), but the second may run significantly faster and
    use less memory.
  • Laszlo Nagy at May 6, 2011 at 4:21 pm

    ...
    That works, but if d is large, it won't be very efficient because it has
    to generate a large list.
    It is not large. But I'm using Python 2.6 , not Python 3.

    I did not get this error again in the last two days. I'll post a new
    reply if I encounter it again. (It happened just a few times out of many
    thousand program invocations)
  • Paul Rubin at May 7, 2011 at 9:07 pm

    Roy Smith <roy at panix.com> writes:
    changes = [ ]
    for key in d.iterkeys():
    if is_bad(key):
    changes.append(key)
    changes = list(k for k in d if is_bad(k))

    is a little bit more direct.
  • Roy Smith at May 7, 2011 at 10:12 pm
    In article <7xd3jukyn9.fsf at ruckus.brouhaha.com>,
    Paul Rubin wrote:
    Roy Smith <roy at panix.com> writes:
    changes = [ ]
    for key in d.iterkeys():
    if is_bad(key):
    changes.append(key)
    changes = list(k for k in d if is_bad(k))

    is a little bit more direct.
    This is true. I still file list comprehensions under "new fangled
    toys". While I use them, and appreciate their value, I admit they're
    not always the first thing that comes to my mind.

    OBTW,
    changes = [k for k in d if is_bad(k)]
    is even more direct :-)
  • Hans Mulder at May 8, 2011 at 7:32 pm

    On 08/05/2011 00:12, Roy Smith wrote:
    In article<7xd3jukyn9.fsf at ruckus.brouhaha.com>,
    Paul Rubinwrote:
    Roy Smith<roy at panix.com> writes:
    changes = [ ]
    for key in d.iterkeys():
    if is_bad(key):
    changes.append(key)
    changes = list(k for k in d if is_bad(k))

    is a little bit more direct.
    This is true. I still file list comprehensions under "new fangled
    toys". While I use them, and appreciate their value, I admit they're
    not always the first thing that comes to my mind.

    OBTW,
    changes = [k for k in d if is_bad(k)]
    is even more direct :-)
    How about:

    changes = filter(is_bad, d)

    Or would that be too compact?

    -- HansM
  • Paul Rubin at May 8, 2011 at 7:42 pm

    Hans Mulder <hansmu at xs4all.nl> writes:
    How about:
    changes = filter(is_bad, d)
    Or would that be too compact?
    I thought of writing something like that but filter in python 3 creates
    an iterator that would have the same issue of walking the dictionary
    while the dictionary is mutating.

    changes = list(filter(is_bad, d))

    should work.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedApr 20, '11 at 12:52p
activeMay 8, '11 at 7:42p
posts13
users8
websitepython.org

People

Translate

site design / logo © 2022 Grokbase