FAQ
Hello all

I am testing my code with list comprehensions against for loops.

the loop:

dipList=[float(val[1]) for val in datalist]
dip1=[]
for dp in dipList:
if dp == 90:
dip1.append(dp - 0.01)
else:
dip1.append(dp)

listcomp:

dipList=[float(val[1]) for val in datalist]
dip1=[(dp, dp-0.01)[dp=�.0] for dp in dipList]


Tenting the time spent by each approach (using time.clock()), with a
file with about 100,000 entries, I get 0.03s for the loop and 0.05s
for the listcomp.

thoughts?

TIA
Carlos

Search Discussions

  • Alf P. Steinbach at Dec 17, 2009 at 5:42 pm

    * Carlos Grohmann:
    Hello all

    I am testing my code with list comprehensions against for loops.

    the loop:

    dipList=[float(val[1]) for val in datalist]
    dip1=[]
    for dp in dipList:
    if dp == 90:
    dip1.append(dp - 0.01)
    else:
    dip1.append(dp)

    listcomp:

    dipList=[float(val[1]) for val in datalist]
    dip1=[(dp, dp-0.01)[dp=�.0] for dp in dipList]


    Tenting the time spent by each approach (using time.clock()), with a
    file with about 100,000 entries, I get 0.03s for the loop and 0.05s
    for the listcomp.

    thoughts?
    In the list comprehension you're constructing n tuples that you're not
    constructing in the loop.

    Have you tried this with

    dip1 = [dp - 0.01 if dp == 90 else dp for dp in dipList]

    ?


    Cheers & hth.,

    - Alf
  • Carlos Grohmann at Dec 17, 2009 at 6:04 pm

    Have you tried this with

    ? ?dip1 = [dp - 0.01 if dp == 90 else dp for dp in dipList]
    Yes that is better! many thanks!
  • Sturlamolden at Dec 18, 2009 at 6:49 pm

    On 17 Des, 18:42, "Alf P. Steinbach" wrote:

    Have you tried this with

    ? ?dip1 = [dp - 0.01 if dp == 90 else dp for dp in dipList]
    And for comparison with map:

    map(lambda dp: dp - 0.01 if dp == 90 else dp, dipList)
  • Sturlamolden at Dec 18, 2009 at 6:44 pm

    On 17 Des, 18:37, Carlos Grohmann wrote:

    Tenting the time spent by each approach (using time.clock()), with a
    file with about 100,000 entries, I get 0.03s for the loop and 0.05s
    for the listcomp.

    thoughts?
    Anything else being equal, list comprehensions will be the faster
    becuase they incur fewer name and attribute lookups. It will be the
    same as the difference between a for loop and a call to map. A list
    comprehension is basically an enhancement of map.
  • Ryan Kelly at Dec 19, 2009 at 1:28 am

    Tenting the time spent by each approach (using time.clock()), with a
    file with about 100,000 entries, I get 0.03s for the loop and 0.05s
    for the listcomp.
    Anything else being equal, list comprehensions will be the faster
    becuase they incur fewer name and attribute lookups. It will be the
    same as the difference between a for loop and a call to map. A list
    comprehension is basically an enhancement of map.
    Not so. If you use the "dis" module to peek at the bytecode generated
    for a list comprehension, you'll see it's very similar to that generated
    for an explicit for-loop. The byte-code for a call to map is very
    different.

    Basically: both a for-loop and a list-comp do the looping in python
    bytecode, while a call to map will do the actual looping in C.
    def comper():
    ... return [i*2 for i in xrange(10)]
    ...
    dis.dis(comper)
    2 0 BUILD_LIST 0
    3 DUP_TOP
    4 STORE_FAST 0 (_[1])
    7 LOAD_GLOBAL 0 (xrange)
    10 LOAD_CONST 1 (10)
    13 CALL_FUNCTION 1
    16 GET_ITER
    17 FOR_ITER 17 (to 37)
    20 STORE_FAST 1 (i)
    23 LOAD_FAST 0 (_[1])
    26 LOAD_FAST 1 (i)
    29 LOAD_CONST 2 (2)
    32 BINARY_MULTIPLY
    33 LIST_APPEND
    34 JUMP_ABSOLUTE 17
    37 DELETE_FAST 0 (_[1])
    40 RETURN_VALUE


    def maper():
    ... return map(lambda i: i*2,xrange(10))
    ...
    dis.dis(maper)
    2 0 LOAD_GLOBAL 0 (map)
    3 LOAD_CONST 1 (<code object ...)
    6 MAKE_FUNCTION 0
    9 LOAD_GLOBAL 1 (xrange)
    12 LOAD_CONST 2 (10)
    15 CALL_FUNCTION 1
    18 CALL_FUNCTION 2
    21 RETURN_VALUE
    >>>



    Cheers,

    Ryan

    --
    Ryan Kelly
    http://www.rfk.id.au | This message is digitally signed. Please visit
    ryan at rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details

    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: not available
    Type: application/pgp-signature
    Size: 197 bytes
    Desc: This is a digitally signed message part
    URL: <http://mail.python.org/pipermail/python-list/attachments/20091219/32b992f8/attachment-0001.pgp>
  • Gregory Ewing at Dec 19, 2009 at 5:49 am

    Ryan Kelly wrote:
    Someone else wrote:
    It will be the
    same as the difference between a for loop and a call to map.
    Not so. If you use the "dis" module to peek at the bytecode generated
    for a list comprehension, you'll see it's very similar to that generated
    for an explicit for-loop.
    The usual advice is that if you have a built-in function that
    does what you want done for each element, then using map() is
    probably the fastest way.

    However, if you need to create a Python function to pass to
    map(), the list comprehension may well be faster, because it
    avoids the cost of a Python function call per element.

    --
    Greg
  • Steven D'Aprano at Dec 19, 2009 at 8:04 am

    On Sat, 19 Dec 2009 12:28:32 +1100, Ryan Kelly wrote:

    Anything else being equal, list comprehensions will be the faster
    becuase they incur fewer name and attribute lookups. It will be the
    same as the difference between a for loop and a call to map. A list
    comprehension is basically an enhancement of map.
    Not so. If you use the "dis" module to peek at the bytecode generated
    for a list comprehension, you'll see it's very similar to that generated
    for an explicit for-loop. The byte-code for a call to map is very
    different.
    "Very similar" and "very different" byte-code mean very little regarding
    speed.

    Basically: both a for-loop and a list-comp do the looping in python
    bytecode, while a call to map will do the actual looping in C.
    This is a classic example of the confirmation fallacy -- if you say that
    for-loops and list-comps are very similar, you need to actually check the
    byte-code of both. You don't. You need to compare the byte-code of all
    three operations, not just two of them, e.g.:

    dis.dis(compile("map(f, seq)", '', 'exec'))
    dis.dis(compile("[f(x) for x in seq]", '', 'exec'))
    dis.dis(compile("L = []\nfor x in seq: L.append(f(x))", '', 'exec'))

    But in fact just looking at the byte-code isn't helpful, because it tells
    you nothing about the relative speed of each operation. You need to
    actually time the operations.
    from timeit import Timer
    t1 = Timer("map(len, 'abcdefgh')", setup='')
    t2 = Timer("[len(c) for c in 'abcdefgh']", setup='')
    t3 = Timer("""L = []
    ... for c in 'abcdefgh':
    ... L.append(len(c))
    ... """, setup='')
    min(t1.repeat())
    3.9076540470123291
    min(t2.repeat())
    4.5931642055511475
    min(t3.repeat())
    7.4744069576263428


    So, on my PC, with Python 2.5, with this example, a for-loop is about 60%
    slower than a list comp and about 90% slower than map; the list comp is
    about 20% slower than map.

    But that only holds for *that* example. Here's another one:

    def f(x):
    ... return 1+2*x+3*x**2
    ...
    values = [1,2,3,4,5,6]
    t1 = Timer("map(f, values)", setup='from __main__ import f, values')
    t2 = Timer("[f(x) for x in values]",
    ... setup='from __main__ import f, values')
    t3 = Timer("""L = []
    ... for x in values:
    ... L.append(f(x))
    ... """, setup='from __main__ import f, values')
    min(t1.repeat())
    7.0339860916137695
    min(t2.repeat())
    6.8053178787231445
    min(t3.repeat())
    9.1957418918609619


    For this example map and the list comp are nearly the same speed, with
    map slightly slower; but the for-loop is still significantly worse.

    Of course, none of these timing tests are terribly significant. The
    actual difference in time is of the order of a millionth of a second per
    call to map compared to the list comp or the for-loop, for these small
    examples. Most of the time you shouldn't care about time differences of
    that magnitude, and write whatever is easiest.


    --
    Steven
  • Sturlamolden at Dec 19, 2009 at 2:18 pm

    On 19 Des, 02:28, Ryan Kelly wrote:

    Not so. ?If you use the "dis" module to peek at the bytecode generated
    for a list comprehension, you'll see it's very similar to that generated
    for an explicit for-loop. ?The byte-code for a call to map is very
    different.
    First, you failed to realize that the bytecode is different because
    map is doing the work in C.

    Second, you did not provide bytecode for the for-loop.
  • Carl Banks at Dec 18, 2009 at 6:50 pm

    On Dec 17, 9:37?am, Carlos Grohmann wrote:
    Tenting the time spent by each approach (using time.clock()), with a
    file with about 100,000 entries, I get 0.03s for the loop and 0.05s
    for the listcomp.

    thoughts?
    You shouldn't trust your intuition in things like this. Some features
    were added to Python to make writing easier, not to make it run
    faster. This time your intuition was correct. Next time, who knows?


    Carl Banks
  • Sturlamolden at Dec 18, 2009 at 6:55 pm

    On 17 Des, 18:37, Carlos Grohmann wrote:

    Tenting the time spent by each approach (using time.clock()), with a
    file with about 100,000 entries, I get 0.03s for the loop and 0.05s
    for the listcomp.

    thoughts?
    Let me ask a retoric question:

    - How much do you really value 20 ms of CPU time?
  • Brian J Mingus at Dec 18, 2009 at 8:44 pm

    On Fri, Dec 18, 2009 at 11:55 AM, sturlamolden wrote:
    On 17 Des, 18:37, Carlos Grohmann wrote:

    Tenting the time spent by each approach (using time.clock()), with a
    file with about 100,000 entries, I get 0.03s for the loop and 0.05s
    for the listcomp.

    thoughts?
    Let me ask a retoric question:

    - How much do you really value 20 ms of CPU time?
    If it takes 1 nanosecond to execute a single instruction then 20
    milliseconds represents 20 million instructions. Therefore I value 20ms of
    CPU time very much indeed.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-list/attachments/20091218/137c15ce/attachment-0001.htm>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedDec 17, '09 at 5:37p
activeDec 19, '09 at 2:18p
posts12
users8
websitepython.org

People

Translate

site design / logo © 2022 Grokbase