FAQ
Context: Embedded Python interpreter, version 2.6.6

I have a list of dictionaries, where each dictionary has a "type"
element which is a string. I want to reduce the list to just the
dictionaries which have the same "type" as the first one.

lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]

I'm seeing a weird difference between two otherwise-equivalent-looking
ways of doing the job.

type=lst[0]["type"].lower()

lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one type

lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one type

If I use the filter() method, the resulting list is completely empty.
If I use the list comprehension, it works perfectly. Oddly, either
version works in the stand-alone interpreter.

I have no idea where to start looking for the problem. Hints, please!

Chris Angelico

Search Discussions

  • Chris Angelico at Apr 20, 2011 at 3:20 am

    On Wed, Apr 20, 2011 at 1:10 PM, Chris Angelico wrote:
    type=lst[0]["type"].lower()

    lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one type
    After posting, I realised that "type" is a built-in identifier, and
    did a quick variable name change to "posttype" which shouldn't have a
    problem. Now, the filter/lambda line bombs with NameError: global name
    'posttype' is not defined. Could it be that the lambda is ignoring
    local variables? If that's so, then it would have been comparing the
    dictionary entry against the built-in type <type>, which will of
    course never match. That could also explain why running the same code
    as a standalone .py file works (variables defined in the body of a .py
    file are global by default, if I have this correct). But why would
    lambda bind to global variables but not local ones?

    The Python code is being called from C as:

    PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

    where py_globals and locals are dictionaries, code is a const char *
    with the code.

    I have Lucia di Lammermoor singing in the background, and I think I'll
    be going as mad as she soon! This is so weird...

    Chris Angelico
  • Chris Rebert at Apr 20, 2011 at 3:22 am
    On Tue, Apr 19, 2011 at 8:10 PM, Chris Angelico wrote:
    <snip>
    type=lst[0]["type"].lower()
    Tangent: Don't call it "type"; you're shadowing the built-in class of
    the same name.

    Cheers,
    Chris
  • Chris Angelico at Apr 20, 2011 at 3:23 am

    On Wed, Apr 20, 2011 at 1:22 PM, Chris Rebert wrote:
    On Tue, Apr 19, 2011 at 8:10 PM, Chris Angelico wrote:
    <snip>
    type=lst[0]["type"].lower()
    Tangent: Don't call it "type"; you're shadowing the built-in class of
    the same name.
    By "shadowing" you mean that the global variable still exists, right?
    I'm creating a local variable with the same name? That's how I'm
    interpreting the results of changing the variable name.

    ChrisA
  • Chris Rebert at Apr 20, 2011 at 3:45 am

    On Tue, Apr 19, 2011 at 8:23 PM, Chris Angelico wrote:
    On Wed, Apr 20, 2011 at 1:22 PM, Chris Rebert wrote:
    On Tue, Apr 19, 2011 at 8:10 PM, Chris Angelico wrote:
    <snip>
    type=lst[0]["type"].lower()
    Tangent: Don't call it "type"; you're shadowing the built-in class of
    the same name.
    By "shadowing" you mean that the global variable still exists, right?
    I'm creating a local variable with the same name? That's how I'm
    interpreting the results of changing the variable name.
    Built-ins aren't quite the same as globals, but essentially yes:
    $ python
    Python 2.6.6 (r266:84292, Jan 12 2011, 13:35:00)
    len
    <built-in function len>
    len = 5
    len
    5
    del len
    len
    <built-in function len>
    del len
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    NameError: name 'len' is not defined
    len
    <built-in function len>

    Cheers,
    Chris
  • Chris Angelico at Apr 20, 2011 at 3:59 am

    On Wed, Apr 20, 2011 at 1:45 PM, Chris Rebert wrote:
    Built-ins aren't quite the same as globals, but essentially yes:
    Sure. That might explain some of the weirdness, but it doesn't explain
    why things were still weird with the variable named posttype. However,
    since the list comp appears to work, I'm going with that... would love
    to know what's going on, but at least things aren't utterly b0rked
    like they were two hours ago!

    Chris Angelico
  • Ian Kelly at Apr 20, 2011 at 6:08 am

    On Tue, Apr 19, 2011 at 9:59 PM, Chris Angelico wrote:
    On Wed, Apr 20, 2011 at 1:45 PM, Chris Rebert wrote:
    Built-ins aren't quite the same as globals, but essentially yes:
    Sure. That might explain some of the weirdness, but it doesn't explain
    why things were still weird with the variable named posttype. However,
    since the list comp appears to work, I'm going with that... would love
    to know what's going on, but at least things aren't utterly b0rked
    like they were two hours ago!
    What is the scope the code is running in? If this is part of a class
    definition, that could explain why the lambda is not seeing the type /
    posttype closure: because there isn't one.

    Cheers,
    Ian
  • Chris Angelico at Apr 20, 2011 at 6:22 am

    On Wed, Apr 20, 2011 at 4:08 PM, Ian Kelly wrote:
    What is the scope the code is running in? ?If this is part of a class
    definition, that could explain why the lambda is not seeing the type /
    posttype closure: because there isn't one.
    It's inside an if, but that's all. The body of the code has:

    if len(lst):
    posttype=...
    lst=...

    No other structural elements to get in the way - unless calling the
    code as described is creating one.

    Chris Angelico
  • Tim Roberts at Apr 20, 2011 at 7:16 am

    Chris Angelico wrote:
    On Wed, Apr 20, 2011 at 1:45 PM, Chris Rebert wrote:
    Built-ins aren't quite the same as globals, but essentially yes:
    Sure. That might explain some of the weirdness, but it doesn't explain
    why things were still weird with the variable named posttype.
    It's because, unlike some other languages (like Pascal), Python doesn't
    have infinitely recursive nested namespaces. Glossing over details, there
    is a global namespace, and there is a local namespace. A new function gets
    a new local namespace. "posttype" is part of the local namespace of the
    outer function, but it's not part of the local namespace of the lambda.

    You can solve this through the common lamba idiom of a closure:

    lst=filter(lambda x,posttype=posttype: x["type"].lower()==posttype,lst)
    --
    Tim Roberts, timr at probo.com
    Providenza & Boekelheide, Inc.
  • Chris Angelico at Apr 20, 2011 at 8:36 am

    On Wed, Apr 20, 2011 at 5:16 PM, Tim Roberts wrote:
    It's because, unlike some other languages (like Pascal), Python doesn't
    have infinitely recursive nested namespaces. ?Glossing over details, there
    is a global namespace, and there is a local namespace. ?A new function gets
    a new local namespace. ?"posttype" is part of the local namespace of the
    outer function, but it's not part of the local namespace of the lambda.
    Okay, well that saves me from going insane anyhow! (Although it may be
    too late to save me from that.) I skimmed your email and then tried to
    explain it all to my boss, who has some programming experience but
    only rudimentary Python, and after ten or fifteen minutes we concluded
    that I needed to put some comments in the code.
    You can solve this through the common lamba idiom of a closure:

    lst=filter(lambda x,posttype=posttype: x["type"].lower()==posttype,lst)
    Seems a little odd, but sure. I guess this means that a function's
    default arguments are evaluated in the parent context, but the body is
    evaluated in its own context?
    From what I understand, though, from __future__ import nested_scopes
    became a permanent part of the language long before 2.6.6, and PEP 227
    describes this exact situation of lambda functions:
    http://www.python.org/dev/peps/pep-0227/ gives an example involving
    Tkinter. Unfortunately googling for 'python nested scope' mainly turns
    up old information from 2.1 and thereabouts; has anything changed
    since then?

    Infinitely-nested scopes is definitely how I, as a programmer, tend to
    think about my code. Any place where the computer's interpretation of
    the code differs from mine is a place where either the computer needs
    to change its thinking, or I do. And it's usually easier to change me,
    except that I use so many languages. :)

    Chris Angelico
  • Mel at Apr 20, 2011 at 2:12 pm

    Chris Angelico wrote:
    On Wed, Apr 20, 2011 at 5:16 PM, Tim Roberts wrote:
    You can solve this through the common lamba idiom of a closure:

    lst=filter(lambda x,posttype=posttype: x["type"].lower()==posttype,lst)
    Seems a little odd, but sure. I guess this means that a function's
    default arguments are evaluated in the parent context, but the body is
    evaluated in its own context?
    The operation of calling a function has to evaluate arguments provided in
    the caller's namespace and assign them to variables in the called function's
    namespace. Yes.

    Mel.
  • Steven D'Aprano at Apr 20, 2011 at 10:16 am

    On Wed, 20 Apr 2011 13:10:21 +1000, Chris Angelico wrote:

    Context: Embedded Python interpreter, version 2.6.6

    I have a list of dictionaries, where each dictionary has a "type"
    element which is a string. I want to reduce the list to just the
    dictionaries which have the same "type" as the first one.
    [snip discussion and code]

    It should, and does, work as expected, both in the global scope:

    lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"},
    ... {42: None, "type": "CALC"}]
    t = lst[0]["type"].lower()

    filter(lambda x: x["type"].lower() == t, lst)
    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]
    [i for i in lst if i["type"].lower() == t]
    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]

    and in a function:
    def test():
    ... lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"},
    ... {42: None, "type": "CALC"}]
    ... t = lst[0]["type"].lower()
    ... print filter(lambda x: x["type"].lower() == t, lst)
    ... print [i for i in lst if i["type"].lower() == t]
    ...
    test()
    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]
    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]


    [...]
    If I use the filter() method, the resulting list is completely empty. If
    I use the list comprehension, it works perfectly. Oddly, either version
    works in the stand-alone interpreter.
    Let me guess... you're using an IDE?

    There's your problem. IDEs often play silly buggers with the environment
    in order to be "clever". You've probably found a bug in whatever IDE
    you're using.

    And this is why I won't touch the buggers with a 30 ft pole, at least not
    for anything serious.



    --
    Steven
  • Chris Angelico at Apr 20, 2011 at 11:25 am

    On Wed, Apr 20, 2011 at 8:16 PM, Steven D'Aprano wrote:
    There's your problem. IDEs often play silly buggers with the environment
    in order to be "clever". You've probably found a bug in whatever IDE
    you're using.

    And this is why I won't touch the buggers with a 30 ft pole, at least not
    for anything serious.
    Not an IDE, but it's running in an embedded interpreter. Which is why
    I looked there for issues, but I can't find any. (Checking sys.version
    and friends shows that it's the same version of Python either way.) I
    described the embed environment above, in the hopes that someone would
    spot an "obvious error", and if your instant suspicion is an IDE, then
    that's probably it.

    Chris Angelico
  • Peter Otten at Apr 20, 2011 at 10:41 am

    Chris Angelico wrote:

    Context: Embedded Python interpreter, version 2.6.6

    I have a list of dictionaries, where each dictionary has a "type"
    element which is a string. I want to reduce the list to just the
    dictionaries which have the same "type" as the first one.

    lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]

    I'm seeing a weird difference between two otherwise-equivalent-looking
    ways of doing the job.

    type=lst[0]["type"].lower()

    lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one
    type

    lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one
    type

    If I use the filter() method, the resulting list is completely empty.
    If I use the list comprehension, it works perfectly. Oddly, either
    version works in the stand-alone interpreter.

    I have no idea where to start looking for the problem. Hints, please!

    Chris Angelico
    The assignment writes to the local namespace, the lambda function reads from
    the global namespace; this will only work as expected if the two namespaces
    are the same:
    exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {}
    []
    ns = {}
    exec """type = 42; print filter(lambda x: x == type, [42])""" in ns
    [42]

    The list comprehension doesn't introduce another namespace, at least in 2.x:

    $ python2.7 -c 'exec("type = 42; print([x for x in [42] if x == type])", {},
    {})'
    [42]
    $ python3.2 -c 'exec("type = 42; print([x for x in [42] if x == type])", {},
    {})'
    []
  • Ian Kelly at Apr 20, 2011 at 2:44 pm

    On Wed, Apr 20, 2011 at 4:41 AM, Peter Otten wrote:
    The assignment writes to the local namespace, the lambda function reads from
    the global namespace; this will only work as expected if the two namespaces
    are the same:
    exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {}
    []
    ns = {}
    exec """type = 42; print filter(lambda x: x == type, [42])""" in ns
    [42]
    That must be a quirk of exec, because it works just fine without using
    exec, both in and out of functions, either from the interactive
    interpreter or from a script:

    Python 2.7.1 (r271:86832, Apr 2 2011, 19:44:19)
    [GCC 4.4.5] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    def f():
    ... t = 42
    ... print filter(lambda x: x == t, [42])
    ...
    f()
    [42]
    t = 43
    print filter(lambda x: x == t, [43])
    [43]

    So, the question for the OP: Is this file being run with execfile?

    Cheers,
    Ian
  • Chris Angelico at Apr 20, 2011 at 6:03 pm

    On Thu, Apr 21, 2011 at 12:44 AM, Ian Kelly wrote:
    So, the question for the OP: ?Is this file being run with execfile?
    Not execfile per se; the code is fetched from the database and then
    executed with:

    PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

    Is Py_file_input the problem?

    Chris Angelico
  • Ian Kelly at Apr 20, 2011 at 6:35 pm

    On Wed, Apr 20, 2011 at 12:03 PM, Chris Angelico wrote:
    On Thu, Apr 21, 2011 at 12:44 AM, Ian Kelly wrote:
    So, the question for the OP: ?Is this file being run with execfile?
    Not execfile per se; the code is fetched from the database and then
    executed with:

    PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

    Is Py_file_input the problem?
    Not specifically. The problem is that you're execing the code, and
    the locals and globals are two different namespaces, as Peter
    suggested. Since the locals namespace is not associated with a
    function, the compiler evidently doesn't generate closures for it, and
    thus the lambda (which has its own locals namespace) can't see it.

    That's my take, anyway. Somebody else may have better insight than me.

    Cheers,
    Ian

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedApr 20, '11 at 3:10a
activeApr 20, '11 at 6:35p
posts17
users7
websitepython.org

People

Translate

site design / logo © 2022 Grokbase