Grokbase › Groups › Python › python-list › April 2007
FAQ
Hi,
I am using Python Thread library for my parallel processing course
project. I am doing matrix convolution on a multi-processor machine
running Solaris. I just found out that no speed-up is obtained with
threading. It is probably because of something called GIL in Python.
How can I get around
that GIL and get speed-up?
Thanks in advance.
Daniel

Search Discussions

  • Fuzzyman at Apr 21, 2007 at 11:18 pm ⇧

    On Apr 21, 10:53 pm, Dennis Lee Bieber wrote:
    On 21 Apr 2007 14:02:12 -0700, danfan1... at yahoo.com declaimed the
    following in comp.lang.python:
    Hi,
    I am using Python Thread library for my parallel processing course
    project. I am doing matrix convolution on a multi-processor machine
    running Solaris. I just found out that no speed-up is obtained with
    threading. It is probably because of something called GIL in Python.
    How can I get around
    that GIL and get speed-up?
    Threading in Python is optimized for I/O bound processing, wherein
    the threads spend most of their lives sleeping (blocked waiting for some
    I/O to complete, or some lock/event/condition to change state). It is
    not optimized for parallel number crunching.

    Options:

    Don't use the common "CPython" (eg, the Python built from C-language
    source using the C-runtime library). Jython (a version that runs on the
    JVM, using Java libraries) may not be afflicted with the GIL.
    IronPython is *definitely* not restricted by the GIL.

    Fuzzyman
    http://www.voidspace.org.uk/ironpython/index.shtml
  • Neil Hodgson at Apr 22, 2007 at 12:03 am ⇧

    Fuzzyman:

    IronPython is *definitely* not restricted by the GIL.
    IronPython is currently mostly slower than CPython although the
    particular problem should be tested to see if IronPython helps.

    Some recent benchmarks between IronPython and CPython:
    http://sparcs.kaist.ac.kr/~tinuviel/pybench/

    Neil
  • Fuzzyman at Apr 22, 2007 at 12:14 am ⇧

    On Apr 22, 1:03 am, Neil Hodgson wrote:
    Fuzzyman:
    IronPython is *definitely* not restricted by the GIL.
    IronPython is currently mostly slower than CPython although the
    particular problem should be tested to see if IronPython helps.

    Some recent benchmarks between IronPython and CPython:http://sparcs.kaist.ac.kr/~tinuviel/pybench/
    Yep, I've seen that. :-)

    http://www.voidspace.org.uk/python/weblog/arch_d7_2007_04_21.shtml#e687

    It's not entirely slower, but mainly slower.

    OTOH, if you split a job into two threads on a twin-core machine,
    IronPython performance will increase dramatically whilst CPython will
    go down...

    Additionally, extending IronPython from C# is orders of magnitude
    easier than extending CPython from C.

    Fuzzyman
    http://www.voidspace.org.uk/ironpython/index.shtml
    Neil
  • Danfan1981 at Apr 22, 2007 at 2:34 pm ⇧
    Thanks guys. But I think IronPython only works on Windows machine, but
    I am using a Sun machine. I was suggested to use Jython, which can run
    on Sun. But I need to use Numpy for matrix operations, which is only
    available to CPython.
  • Stefan Behnel at Apr 22, 2007 at 3:22 pm ⇧

    danfan1981 at yahoo.com schrieb:
    Thanks guys. But I think IronPython only works on Windows machine, but
    I am using a Sun machine.
    Isn't there a mono port for Sun?

    Stefan
  • Klaas at Apr 23, 2007 at 8:52 pm ⇧

    On Apr 21, 5:14 pm, Fuzzyman wrote:

    Additionally, extending IronPython from C# is orders of magnitude
    easier than extending CPython from C.
    Given the existence of Pyrex, that statement is pretty difficult to
    substantiate.

    -Mike
  • Fuzzyman at Apr 24, 2007 at 2:59 pm ⇧

    On Apr 23, 9:52 pm, Klaas wrote:
    On Apr 21, 5:14 pm, Fuzzyman wrote:

    Additionally, extending IronPython from C# is orders of magnitude
    easier than extending CPython from C.
    Given the existence of Pyrex, that statement is pretty difficult to
    substantiate.
    With Pyrex you still need to do memory management for non-Python
    types. Additionally compiling C# from within IronPython is *very*easy.

    C# is a much easier language to use than C - even with the help of
    Pyrex. You use types defined in C# *natively* within IronPython.

    All the best,

    Fuzzyman
    http://www.voidspace.org.uk/python/articles.shtml
    -Mike
  • Jorgen Grahn at Apr 24, 2007 at 3:39 pm ⇧

    On 23 Apr 2007 13:52:52 -0700, Klaas wrote:
    On Apr 21, 5:14 pm, Fuzzyman wrote:

    Additionally, extending IronPython from C# is orders of magnitude
    easier than extending CPython from C.
    Given the existence of Pyrex, that statement is pretty difficult to
    substantiate.
    IMHO, you don't even need Pyrex. When you've done it manually once,
    you have the blueprint for your other extension modules. Most of the
    work should be normal "fake OOP in C" programming.

    Doing it for the first time is another matter, however.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
    \X/ snipabacken.dyndns.org> R'lyeh wgah'nagl fhtagn!
  • Fuzzyman at Apr 22, 2007 at 2:36 pm ⇧

    On Apr 22, 1:03 am, Neil Hodgson wrote:
    Fuzzyman:
    IronPython is *definitely* not restricted by the GIL.
    IronPython is currently mostly slower than CPython although the
    particular problem should be tested to see if IronPython helps.

    Some recent benchmarks between IronPython and CPython:http://sparcs.kaist.ac.kr/~tinuviel/pybench/
    Not when running on the .NET platform, there is not a lot of
    difference overall:

    http://www.voidspace.org.uk/python/weblog/arch_d7_2007_04_21.shtml#e688

    Fuzzyman
    http://www.voidspace.org.uk/ironpython/index.shtml
    Neil
  • Caleb Hattingh at Apr 21, 2007 at 11:24 pm ⇧

    On Apr 21, 11:02 pm, danfan1... at yahoo.com wrote:
    Hi,
    I am using Python Thread library for my parallel processing course
    project. I am doing matrix convolution on a multi-processor machine
    running Solaris. I just found out that no speed-up is obtained with
    threading. It is probably because of something called GIL in Python.
    How can I get around
    that GIL and get speed-up?
    Thanks in advance.
    Daniel
    Perhaps try

    http://www.parallelpython.com/

    or

    http://www.its.caltech.edu/~astraw/seppo.html

    Caleb
  • John Nagle at Apr 21, 2007 at 11:36 pm ⇧

    Caleb Hattingh wrote:
    On Apr 21, 11:02 pm, danfan1... at yahoo.com wrote:

    Hi,
    I am using Python Thread library for my parallel processing course
    project. I am doing matrix convolution on a multi-processor machine
    running Solaris. I just found out that no speed-up is obtained with
    threading. It is probably because of something called GIL in Python.
    How can I get around
    that GIL and get speed-up?
    Thanks in advance.
    Daniel
    If you're actually doing the convolution in Python, you need
    optimization before you need more CPUs. There's a numerics library
    for Python called NumPy, but it doesn't have a convolution function,
    although it has an FFT, which may be useful.

    But this is just homework. Do something reasonable and turn it
    in. A high performance solution to this problem is probably more
    work than it's worth.

    John Nagle
  • Robert Kern at Apr 22, 2007 at 3:09 am ⇧

    John Nagle wrote:
    There's a numerics library
    for Python called NumPy, but it doesn't have a convolution function,
    although it has an FFT, which may be useful.
    In [1]: from numpy import *

    In [2]: convolve?
    Type: function
    Base Class: <type 'function'>
    Namespace: Interactive
    File:
    /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy-1.0.3.dev3714-py2.5-macosx-10.3-fat.egg/numpy/core/numeric.py
    Definition: convolve(a, v, mode='full')
    Docstring:
    Returns the discrete, linear convolution of 1-D sequences a and v; mode
    can be 'valid', 'same', or 'full' to specify size of the resulting sequence.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
  • Cameron Laird at Apr 22, 2007 at 3:08 pm ⇧
    In article <EdxWh.5224$2v1.4151 at newssvr14.news.prodigy.net>,
    John Nagle wrote:
    Caleb Hattingh wrote:
    On Apr 21, 11:02 pm, danfan1... at yahoo.com wrote:

    Hi,
    I am using Python Thread library for my parallel processing course
    project. I am doing matrix convolution on a multi-processor machine
    running Solaris. I just found out that no speed-up is obtained with
    threading. It is probably because of something called GIL in Python.
    How can I get around
    that GIL and get speed-up?
    Thanks in advance.
    Daniel
    If you're actually doing the convolution in Python, you need
    optimization before you need more CPUs. There's a numerics library
    for Python called NumPy, but it doesn't have a convolution function,
    although it has an FFT, which may be useful.

    But this is just homework. Do something reasonable and turn it
    in. A high performance solution to this problem is probably more
    work than it's worth.

    John Nagle
    Along with the excellent advice given by Dennis, John, and the
    rest, please be aware that *process*-level parallelization of
    a problem sometimes is a benefit. As already recommended, <URL:
    http://wiki.python.org/moin/ParallelProcessing > touches on most
    of the pertinent concepts.
  • Grant Edwards at Apr 22, 2007 at 2:44 am ⇧

    On 2007-04-21, danfan1981 at yahoo.com wrote:

    I am using Python Thread library for my parallel processing
    course project. I am doing matrix convolution on a
    multi-processor machine running Solaris. I just found out that
    no speed-up is obtained with threading. It is probably because
    of something called GIL in Python. How can I get around that
    GIL and get speed-up?
    Not much of a parallel processing course. It appears they
    haven't taught you anything about parallel processing.

    http://wiki.python.org/moin/ParallelProcessing


    --
    Grant Edwards grante Yow! There's a SALE on
    at STRETCH SOCKS down at the
    visi.com "7-11"!!

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedApr 21, '07 at 9:02p
activeApr 24, '07 at 3:39p
posts15
users11
websitepython.org

11 users in discussion

Fuzzyman: 4 posts Danfan1981: 2 posts Caleb Hattingh: 1 post Neil Hodgson: 1 post Grant Edwards: 1 post John Nagle: 1 post Klaas: 1 post Stefan Behnel: 1 post Robert Kern: 1 post Cameron Laird: 1 post Jorgen Grahn: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase