FAQ
The yield statement looks to be a big step toward some kind of
lightweight concurrency-oriented programming in python. Is there
any similarly nice way to do the opposite of yield, just sit around
(perhaps in the middle of a loop) and wait until some other routine
(unknown to the waiting module) hurls a value in?

If this is not implemented now, is there any chance that it's on the
list of things to come?


Al

Search Discussions

  • Bob Gailer at Sep 10, 2003 at 7:56 pm
    At 01:40 PM 9/10/2003, achrist at easystreet.com wrote:

    The yield statement looks to be a big step toward some kind of
    lightweight concurrency-oriented programming in python. Is there
    any similarly nice way to do the opposite of yield, just sit around
    (perhaps in the middle of a loop) and wait until some other routine
    (unknown to the waiting module) hurls a value in?
    Sounds like a task for a thread that sleeps a while then checks for the data.

    Also reminiscent (a long time ago) someone proposed a COME FROM statement
    for FORTRAN to be the inverse of GO TO.

    Perhaps a new Python statement "suck"?

    Bob Gailer
    bgailer at alum.rpi.edu
    303 442 2625
    -------------- next part --------------

    ---
    Outgoing mail is certified Virus Free.
    Checked by AVG anti-virus system (http://www.grisoft.com).
    Version: 6.0.506 / Virus Database: 303 - Release Date: 8/1/2003
  • Erik Max Francis at Sep 10, 2003 at 7:59 pm

    achrist at easystreet.com wrote:

    The yield statement looks to be a big step toward some kind of
    lightweight concurrency-oriented programming in python. Is there
    any similarly nice way to do the opposite of yield, just sit around
    (perhaps in the middle of a loop) and wait until some other routine
    (unknown to the waiting module) hurls a value in?
    Yep:

    generator.next()

    --
    Erik Max Francis && max at alcyone.com && http://www.alcyone.com/max/
    __ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
    / \ Dead men have no victory.
    \__/ Euripides
  • Achrist at Sep 10, 2003 at 9:39 pm

    Erik Max Francis wrote:

    Yep:

    generator.next()
    This style supports only output-driven processes. The generator,
    being on the input side, is always the server, and the output
    side is always the client. It doesn't fit the situation where the
    input is heterogeneous and input has to get spewed in several
    directions, depending on its content.


    Al
  • Bengt Richter at Sep 10, 2003 at 10:07 pm

    On Wed, 10 Sep 2003 14:39:14 -0700, achrist at easystreet.com wrote:
    Erik Max Francis wrote:

    Yep:

    generator.next()
    This style supports only output-driven processes. The generator,
    being on the input side, is always the server, and the output
    side is always the client. It doesn't fit the situation where the
    input is heterogeneous and input has to get spewed in several
    directions, depending on its content.
    Could you make some fantasy-code to illustrate what you mean?

    Regards,
    Bengt Richter
  • Stuart D. Gathman at Sep 10, 2003 at 10:15 pm

    On Wed, 10 Sep 2003 17:39:14 -0400, achrist wrote:

    Erik Max Francis wrote:
    Yep:

    generator.next()
    This style supports only output-driven processes. The generator, being
    on the input side, is always the server, and the output side is always
    the client. It doesn't fit the situation where the input is
    heterogeneous and input has to get spewed in several directions,
    depending on its content.
    
    It does if you add a dispatcher framework. There is a python project for
    just such a framework, but I can't remember the name. The idea is dirt
    simple, but the implementation gets knarly with exceptions and such.

    Found it - Charming Python 
    http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html
  • Erik Max Francis at Sep 10, 2003 at 10:17 pm

    achrist at easystreet.com wrote:

    This style supports only output-driven processes. The generator,
    being on the input side, is always the server, and the output
    side is always the client. It doesn't fit the situation where the
    input is heterogeneous and input has to get spewed in several
    directions, depending on its content.
    The distinction you're making is push vs. pull processing. To get what
    you want, you should use some sort of asynchronous mechanism with
    synchronization when data is transferred. Threading with the use of the
    Queue module comes to mind immediately. Other forms of asynchronous
    processing are certainly possible, as well.

    That being said, calling the next method of a generator _is_ the obvious
    inverse process of a generator yielding a value.

    --
    Erik Max Francis && max at alcyone.com && http://www.alcyone.com/max/
    __ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
    / \ Forever we / Infinitely
    \__/ Sandra St. Victor
  • Peter Hansen at Sep 10, 2003 at 8:19 pm

    achrist at easystreet.com wrote:
    The yield statement looks to be a big step toward some kind of
    lightweight concurrency-oriented programming in python. Is there
    any similarly nice way to do the opposite of yield, just sit around
    (perhaps in the middle of a loop) and wait until some other routine
    (unknown to the waiting module) hurls a value in?

    If this is not implemented now, is there any chance that it's on the
    list of things to come?
    Queue.Queue.get() ...

    -Peter
  • Achrist at Sep 10, 2003 at 9:20 pm

    Peter Hansen wrote:

    Queue.Queue.get() ...
    That looks good. But that needs a thread to block, right?
    A generator manages to run without being a thread (at least
    overtly). If we have the suck() statement, we also need
    something that's the opposite of a generator (a Sucker?)
    and something that's the opposite of an iterator (a Suckee?).
    I'm starting to get an idea why this is completely not all
    there.

    The main question this raises is "How lightweight are threads?"
    Can I program with dozens or hundreds of python threads in a
    program (for example under Windows) and not notice that this is
    an inefficient or inept coding style?


    Al
  • Peter Hansen at Sep 10, 2003 at 9:41 pm

    achrist at easystreet.com wrote:
    Peter Hansen wrote:

    Queue.Queue.get() ...
    That looks good. But that needs a thread to block, right? Yep.
    A generator manages to run without being a thread (at least
    overtly). If we have the suck() statement, we also need
    something that's the opposite of a generator (a Sucker?)
    and something that's the opposite of an iterator (a Suckee?).
    I'm starting to get an idea why this is completely not all
    there.

    The main question this raises is "How lightweight are threads?"
    Can I program with dozens or hundreds of python threads in a
    program (for example under Windows) and not notice that this is
    an inefficient or inept coding style?
    The real question might be why would you want to?

    If you don't want a thread, but you want something which takes
    input from elsewhere and does some processing, then returns
    control to some other place until more data is available (which
    is what one might assume if threads aren't good for you), there
    is already a convenient solution: the subroutine. ;-)

    Seriously, what's the requirement driving the need for this?
    (I suspect Erik Max's answer is still what you need, and you
    misunderstand the nature of what he was suggesting, but if you'll
    explain the specific case you have in mind we'll know for sure.)

    -Peter
  • Achrist at Sep 10, 2003 at 11:29 pm

    Peter Hansen wrote:
    The real question might be why would you want to?

    If you don't want a thread, but you want something which takes
    input from elsewhere and does some processing, then returns
    control to some other place until more data is available (which
    is what one might assume if threads aren't good for you), there
    is already a convenient solution: the subroutine. ;-)

    Seriously, what's the requirement driving the need for this?
    (I suspect Erik Max's answer is still what you need, and you
    misunderstand the nature of what he was suggesting, but if you'll
    explain the specific case you have in mind we'll know for sure.)
    No, I didn't say that I didn't want a thread. If there are no
    serious drawbacks to relying heavily on threads, I'd be glad to
    use them. This would be deluxe.

    There really isn't any need for a generator, but it is a very
    nice way to construct the input side of a program (the caller
    doesn't have to worry about expicitly calling initialization and
    finalization). Some kind of through-the-looking-glass version
    of a generator would be a very nice way to construct the output
    side.

    I decided to test threads to see what happens:

    -------------Warning, Code Follows---------------------

    import Queue
    import thread
    import time

    queues = [Queue.Queue()]
    threads = []

    appendLock = thread.allocate_lock()

    def TransferData(queues, atPos):
    while len(queues) < atPos + 1:
    pass
    inputQueue = queues[atPos]
    outputQueue = Queue.Queue()
    appendLock.acquire()
    queues.append(outputQueue)
    appendLock.release()
    while 1:
    data = inputQueue.get(True)
    outputQueue.put(data, True)

    startTime = time.time()
    for i in range(1000):
    print i,
    threads.append(thread.start_new_thread(TransferData, (queues, i)))
    queues[0].put(i, True)
    appendLock.acquire()
    print queues[-1].get(True)
    appendLock.release()

    print time.time() - startTime


    --------------Warning Discussion Follows -------------------------

    Running this on my Windows NT 200 NHz machine gave decent results.
    With 1000 threads and 1000 Queues, memory used was 24MB. Python
    takes almost 10 MB to start, so the overhead per thread + Queue
    is only a little over 10 kb. Passing all the messages through
    the chain took 2.5 minutes, so we are down around a quarter millisec
    for each put or get.

    That should be pretty good for many uses.


    Al
  • Mike C. Fletcher at Sep 10, 2003 at 9:42 pm

    achrist at easystreet.com wrote:
    Peter Hansen wrote:

    Queue.Queue.get() ...

    That looks good. But that needs a thread to block, right?
    A generator manages to run without being a thread (at least
    overtly). If we have the suck() statement, we also need
    something that's the opposite of a generator (a Sucker?)
    and something that's the opposite of an iterator (a Suckee?).
    I'm starting to get an idea why this is completely not all
    there.
    You're probably looking for "watcher" or "dispatcher" patterns (using
    "pattern" loosely), i.e. something which is registered to respond to a
    particular type of "event" and process the results only when the event
    occurs. See Patrick 's pydispatcher project for a fairly good framework
    on which to build such things.

    http://pydispatcher.sourceforge.net/

    In a more general sense, this is asynchronous programming, so at the
    high-end you could check out Twisted Matrix, which is a network-oriented
    asynchronous event system (pydispatcher is primarily an in-process system).
    The main question this raises is "How lightweight are threads?"
    Can I program with dozens or hundreds of python threads in a
    program (for example under Windows) and not notice that this is
    an inefficient or inept coding style?
    Large numbers of threads tend to cause problems eventually unless you're
    using micro-threads under Stackless Python. Since AFAIK, micro-threads
    haven't been re-implemented in Stackless 3.0, massively-threading
    approaches aren't likely to be a great approach just yet. As an aside,
    micro-threading, particularly when combined with continuations, allows
    for programming some of the most beautiful ways to deal with
    asynchronous events I've ever seen.

    That said, a few dozen, or even hundred threads may run quite nicely
    under Windows. Tim could likely give a better estimate, but I seem to
    recall him saying (way back when Stackless was 1.0) that Win95 could
    handle a far higher thread-load than Linux (possibly a few thousand
    threads?) I really wouldn't try running a few thousand threads on most
    boxes, however, the OSes just aren't designed to handle it well.

    HTH,
    Mike

    _______________________________________
    Mike C. Fletcher
    Designer, VR Plumber, Coder
    http://members.rogers.com/mcfletch/
  • Chris Liechti at Sep 10, 2003 at 9:53 pm

    achrist at easystreet.com wrote in news:3F5F95B9.A8CB5786 at easystreet.com:

    Peter Hansen wrote:

    Queue.Queue.get() ...
    That looks good. But that needs a thread to block, right? yes
    A generator manages to run without being a thread (at least
    overtly). If we have the suck() statement, we also need
    something that's the opposite of a generator (a Sucker?)
    and something that's the opposite of an iterator (a Suckee?).
    I'm starting to get an idea why this is completely not all
    there.
    basicaly you poll every generator (or more precicly "iterable" as that is
    what a generator returns), one after the other... enjoy the link below.
    The main question this raises is "How lightweight are threads?"
    Can I program with dozens or hundreds of python threads in a
    program (for example under Windows) and not notice that this is
    an inefficient or inept coding style?
    this one is for you :-)
    http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html

    you wont get happy with hundrets of native threads, but with the light ones
    from above you can have thousands...

    generaly you will find interesting stuff here (one line):
    http://www-
    106.ibm.com/developerworks/views/linux/articles.jsp?sort_order=desc&expand=
    &sort_by=Date&show_abstract=true&view_by=Search&search_by=charming+python%3
    A

    chris
    --
    Chris <cliechti at gmx.net>
  • Jan Decaluwe at Sep 11, 2003 at 7:43 am

    achrist at easystreet.com wrote:
    Peter Hansen wrote:

    Queue.Queue.get() ...
    That looks good. But that needs a thread to block, right?
    A generator manages to run without being a thread (at least
    overtly). If we have the suck() statement, we also need
    something that's the opposite of a generator (a Sucker?)
    and something that's the opposite of an iterator (a Suckee?).
    I'm starting to get an idea why this is completely not all
    there.

    The main question this raises is "How lightweight are threads?"
    Can I program with dozens or hundreds of python threads in a
    program (for example under Windows) and not notice that this is
    an inefficient or inept coding style?

    Al
    You may want to check out how such things are done with my MyHDL
    package. The target application is hardware design, but the idea
    is more probably general.

    http://jandecaluwe.com/Tools/MyHDL/manual/model-obj.html

    Jan

    --
    Jan Decaluwe - Resources bvba - http://jandecaluwe.com
    Losbergenlaan 16, B-3010 Leuven, Belgium
    Bored with EDA the way it is? Check this:
    http://jandecaluwe.com/Tools/MyHDL/Overview.html
  • Dave Benjamin at Sep 10, 2003 at 8:19 pm
    "Bob Gailer" <bgailer at alum.rpi.edu> wrote in message
    news:mailman.1063223955.7295.python-list at python.org...
    At 01:40 PM 9/10/2003, achrist at easystreet.com wrote:
    The yield statement looks to be a big step toward some kind of
    lightweight concurrency-oriented programming in python. Is there
    any similarly nice way to do the opposite of yield, just sit around
    (perhaps in the middle of a loop) and wait until some other routine
    (unknown to the waiting module) hurls a value in?
    ...

    Perhaps a new Python statement "suck"?
    Oh come on, now, just because Java does it...
  • Christos TZOTZIOY Georgiou at Sep 10, 2003 at 9:03 pm
    On Wed, 10 Sep 2003 20:19:35 GMT, rumours say that "Dave Benjamin"
    <dave at 3dex.com> might have written:
    Perhaps a new Python statement "suck"?
    Oh come on, now, just because Java does it...
    QOTW +1 :)
    --
    TZOTZIOY, I speak England very best,
    Microsoft Security Alert: the Matrix began as open source.
  • Samuel Bronson at Sep 10, 2003 at 11:16 pm

    On Wed, 10 Sep 2003 12:40:37 -0700, achris wrote:
    The yield statement looks to be a big step toward some kind of lightweight
    concurrency-oriented programming in python. Is there any similarly nice
    way to do the opposite of yield, just sit around (perhaps in the middle of
    a loop) and wait until some other routine (unknown to the waiting module)
    hurls a value in?

    If this is not implemented now, is there any chance that it's on the list
    of things to come?


    Al
    Well, aside from Stackless's microthreading, you might want to check out
    Twisted -- it isn't as magical as yield, but it gets the job done pretty
    well.
  • Oren Tirosh at Sep 11, 2003 at 7:17 am

    On Wed, Sep 10, 2003 at 12:40:37PM -0700, achrist at easystreet.com wrote:
    The yield statement looks to be a big step toward some kind of
    lightweight concurrency-oriented programming in python. Is there
    any similarly nice way to do the opposite of yield, just sit around
    (perhaps in the middle of a loop) and wait until some other routine
    (unknown to the waiting module) hurls a value in?
    Here is a way of converting a generator function into a consumer using
    a single item list as a mailbox:

    from __future__ import generators

    def consumer():
    print "This code runs before any data is fed to the consumer"
    mailbox = []; yield mailbox
    while mailbox:
    print "Here's what I got:", mailbox.pop()
    yield mailbox
    print "This code runs after end of data is signalled"

    # How to feed a consumer:

    i = 0
    for mailbox in consumer():
    if i < 5:
    mailbox.append(i) # feed next item
    i += 1


    Output:
    This code runs before any data is fed to the consumer
    Here's what I got: 0
    Here's what I got: 1
    Here's what I got: 2
    Here's what I got: 3
    Here's what I got: 4
    This code runs after end of data is signalled


    For a more comprehensive solution try Christian Tismer's Stackless Python.

    Oren
  • Greg Chapman at Sep 12, 2003 at 3:00 pm

    On Wed, 10 Sep 2003 12:40:37 -0700, achrist at easystreet.com wrote:
    The yield statement looks to be a big step toward some kind of
    lightweight concurrency-oriented programming in python. Is there
    any similarly nice way to do the opposite of yield, just sit around
    (perhaps in the middle of a loop) and wait until some other routine
    (unknown to the waiting module) hurls a value in?

    If this is not implemented now, is there any chance that it's on the
    list of things to come?
    You can use a generator as the body of a function. That is, the generator is
    only ever accessed through the function, and the function and the generator have
    some agreed on way to pass parameters. I.e.:

    def getline(output):
    '''getline accumulates text until it has a complete line (signaled
    by either a newline (\\n) or a null (\\0)). It then passes the
    line (without trailing \n) to output'''

    genargs = [None]
    def callgen(text):
    genargs[0] = text
    return gennext()
    def gen():
    line = ''
    while True:
    text, = genargs
    if text:
    lines = text.splitlines(True)
    if line:
    lines[0] = line+lines[0]
    for line in lines[:-1]:
    output(line[:-1])
    line = lines[-1]
    if line[-1] in '\n\0':
    if line[-1] == '\n':
    line = line[:-1]
    output(line)
    line = ''
    yield None
    gennext = gen().next
    return callgen

    def printer(text):
    print text

    lineprinter = getline(printer)

    lineprinter('hello') #nothing output
    lineprinter(' world\n') #prints 'hello world'


    ---
    Greg Chapman
  • Jacek Generowicz at Sep 17, 2003 at 11:09 am

    Bob Gailer <bgailer at alum.rpi.edu> writes:

    Also reminiscent (a long time ago) someone proposed a COME FROM
    statement for FORTRAN to be the inverse of GO TO.
    http://c2.com/cgi/wiki?ComeFrom

Related Discussions

People

Translate

site design / logo © 2022 Grokbase