FAQ
Hi folks,

I've seen the following issue come up in multiple posts to this
mailing list:

I have a python program that spawns a child process with popen or
popen2 or popen3 or popen2.popen2 etc.
the child process is interactive: it asks for input then spits out
some output, asks for more input then spits out some output. for
example, consider the trivial child program:

print "welcome"
print ">",
s=raw_input()
while s!='exit':
print "you entered:",s
print ">",
s=raw_input()

Now I may be completely wrong about this (I did play with popen for a
very long time before writing this message), but it appears that none
of the popen variants allow for a sequence of reads and writes to/from
this child. that is, if I read from the open pipe's output I will
never be able to write the input for the child because the parent
program will block on read until eof (I will have similar blocking
problems if I start with write - using readline does not seem to
help).

the standard proposed remedy I have seen on this list is to use Unix-
only select/fctl, or otherwise dig into the bowls of the win32 api, or
download some half-complete sourceforge process control project. All
well and good, but unsatisfying for writing platform independent code.

it turns out that there is at least one open source multi-platform
(read: win32/linux) api that does handle synchronous I/O with the
child: wxWidgets and wxPython using the class wxProcess. Now the
wxWidgets implementation is far from perfect, but it at least allows a
program to test for new input on the child's stdout and read stdout/
write stdout in a non-blocking way. However, I find it frustrating
that I have to import wx just to have useable interactive pipes in my
python scripts when I would expect this to be part of the native
python implementation.

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

Search Discussions

  • Nick Craig-Wood at Jun 7, 2007 at 5:30 pm

    dmoore wrote:
    I've seen the following issue come up in multiple posts to this
    mailing list:

    I have a python program that spawns a child process with popen or
    popen2 or popen3 or popen2.popen2 etc.
    the child process is interactive: it asks for input then spits out
    some output, asks for more input then spits out some output. for
    example, consider the trivial child program:

    print "welcome"
    print ">",
    s=raw_input()
    while s!='exit':
    print "you entered:",s
    print ">",
    s=raw_input()

    Now I may be completely wrong about this (I did play with popen for a
    very long time before writing this message), but it appears that none
    of the popen variants allow for a sequence of reads and writes to/from
    this child. that is, if I read from the open pipe's output I will
    never be able to write the input for the child because the parent
    program will block on read until eof (I will have similar blocking
    problems if I start with write - using readline does not seem to
    help).
    You are correct.
    the standard proposed remedy I have seen on this list is to use Unix-
    only select/fctl, or otherwise dig into the bowls of the win32 api, or
    download some half-complete sourceforge process control project.
    If you are referring to pexpect I've found it works well - unix only
    though. I've not noticed it being half complete.

    There is also a recipe for a non-blocking subprocess - see

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
    All well and good, but unsatisfying for writing platform independent code.

    it turns out that there is at least one open source multi-platform
    (read: win32/linux) api that does handle synchronous I/O with the
    child: wxWidgets and wxPython using the class wxProcess. Now the
    wxWidgets implementation is far from perfect, but it at least allows a
    program to test for new input on the child's stdout and read stdout/
    write stdout in a non-blocking way.
    Interesting I didn't know about that - I shall try it!
    However, I find it frustrating
    that I have to import wx just to have useable interactive pipes in my
    python scripts when I would expect this to be part of the native
    python implementation.

    Anybody have any thoughts on this? Do I have my story straight? (the
    popen variants can't handle this case and there are no other
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
    The non-blocking subprocess would make a good start for a stdlib
    submission.

    It should really optionally use ptys under unix too otherwise you'll
    never be able to script passwd etc. An interface a bit like pexpect
    wpuld be useful too (ie scan for these regexps or timeout and return a
    match object).

    --
    Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick
  • Terry Reedy at Jun 7, 2007 at 6:29 pm
    "dmoore" <damienlmoore at gmail.com> wrote in message
    news:1181232071.928835.301710 at w5g2000hsg.googlegroups.com...
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
    http://sourceforge.net/tracker/?group_idT70

    But don't hold your breath for a volunteer to respond. Discussion here is
    good until you have a concrete request and support/rationale.
  • Noah at Jun 8, 2007 at 12:59 am
    On Jun 7, 9:01 am, dmoore wrote:

    popen and friends will never do what you want it to do. Down that path
    lies bitter disappointment.
    You need pseduo-ttys and non-blocking IO. I don't know how to do this
    on Windows, but I know it's possible.
    Cygwin does it.
    Anybody have any thoughts on this? Do I have my story straight? (the
    popen variants can't handle this case and there are no other
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
    Try Pexpect http://pexpect.sourceforge.net/
    It's been around for a long time and is quite complete and stable.

    The catch is that it's UNIX-only. If you want to tease out the magic
    code from wxProcess that
    does non-blocking reads on win32 then I'd be happy to integrate that
    into the current development branch
    of Pexpect. If someone can provide a pure Python drop-in replacement
    for the read_nonblocking() function
    I use for UNIX systems then it would be easy. I have a whole test
    framework that I can run it through to
    see if it performs the same as the UNIX flavor.

    I'm sure this is feasible without any C extensions -- it might require
    some ctypes hacking.
    I know Windows has pretty decent async IO, but I don't know what they
    have as an equivalent for a pty.
    Maybe it isn't necessary. A pty is only necessary on UNIX because the
    standard c library, stdio, behaves
    differently when it's talking to a plain pipe versus a terminal -- it
    switches buffering
    between block and line oriented buffer. You don't want block buffering
    on interactive applications.
    This is why popen eventually breaks down. No, there is no way to
    select this behavior from
    the calling side... unless you can trick it into dynamically linking
    to your specially hacked libc.
    But that's getting ahead because that's what happens on UNIX -- it
    might be a non-issue on Windows.

    The read_nonblocking() function I use has an interface like this:
    <pre>
    def read_nonblocking (self, size = 1, timeout = -1):

    """This reads at most size characters from the child
    application. It
    includes a timeout. If the read does not complete within the
    timeout
    period then a TIMEOUT exception is raised. If the end of file
    is read
    then an EOF exception will be raised. If a log file was set
    using
    setlog() then all data will also be written to the log file.

    If timeout is None then the read may block indefinitely. If
    timeout is -1
    then the self.timeout value is used. If timeout is 0 then the
    child is
    polled and if there was no data immediately ready then this
    will raise
    a TIMEOUT exception.

    The timeout refers only to the amount of time to read at least
    one
    character. This is not effected by the 'size' parameter, so if
    you call
    read_nonblocking(size0, timeout0) and only one character
    is
    available right away then one character will be returned
    immediately.
    It will not wait for 30 seconds for another 99 characters to
    come in.

    This is a wrapper around os.read(). It uses select.select() to
    implement the timeout. """
    </pre>

    Yours,
    Noah
  • Dmoore at Jun 8, 2007 at 1:50 am
    thanks for all of your responses. i'll look more closely at pexpect
    (The version I originally saw was much much older). I do want a cross-
    platform solution. I'm pretty busy so getting the wxProcess magic into
    useful shape will take me some time (at least on the order of weeks)
    so other interest parties feel free to step up.

    cheers
    Damien
    On Jun 7, 8:59 pm, Noah wrote:
    On Jun 7, 9:01 am, dmoore wrote:

    popen and friends will never do what you want it to do. Down that path
    lies bitter disappointment.
    You need pseduo-ttys and non-blocking IO. I don't know how to do this
    on Windows, but I know it's possible.
    Cygwin does it.
    Anybody have any thoughts on this? Do I have my story straight? (the
    popen variants can't handle this case and there are no other
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
    Try Pexpecthttp://pexpect.sourceforge.net/
    It's been around for a long time and is quite complete and stable.

    The catch is that it's UNIX-only. If you want to tease out the magic
    code from wxProcess that
    does non-blocking reads on win32 then I'd be happy to integrate that
    into the current development branch
    of Pexpect. If someone can provide a pure Python drop-in replacement
    for the read_nonblocking() function
    I use for UNIX systems then it would be easy. I have a whole test
    framework that I can run it through to
    see if it performs the same as the UNIX flavor.

    I'm sure this is feasible without any C extensions -- it might require
    some ctypes hacking.
    I know Windows has pretty decent async IO, but I don't know what they
    have as an equivalent for a pty.
    Maybe it isn't necessary. A pty is only necessary on UNIX because the
    standard c library, stdio, behaves
    differently when it's talking to a plain pipe versus a terminal -- it
    switches buffering
    between block and line oriented buffer. You don't want block buffering
    on interactive applications.
    This is why popen eventually breaks down. No, there is no way to
    select this behavior from
    the calling side... unless you can trick it into dynamically linking
    to your specially hacked libc.
    But that's getting ahead because that's what happens on UNIX -- it
    might be a non-issue on Windows.

    The read_nonblocking() function I use has an interface like this:
    <pre>
    def read_nonblocking (self, size = 1, timeout = -1):

    """This reads at most size characters from the child
    application. It
    includes a timeout. If the read does not complete within the
    timeout
    period then a TIMEOUT exception is raised. If the end of file
    is read
    then an EOF exception will be raised. If a log file was set
    using
    setlog() then all data will also be written to the log file.

    If timeout is None then the read may block indefinitely. If
    timeout is -1
    then the self.timeout value is used. If timeout is 0 then the
    child is
    polled and if there was no data immediately ready then this
    will raise
    a TIMEOUT exception.

    The timeout refers only to the amount of time to read at least
    one
    character. This is not effected by the 'size' parameter, so if
    you call
    read_nonblocking(size0, timeout0) and only one character
    is
    available right away then one character will be returned
    immediately.
    It will not wait for 30 seconds for another 99 characters to
    come in.

    This is a wrapper around os.read(). It uses select.select() to
    implement the timeout. """
    </pre>

    Yours,
    Noah
  • Nick Craig-Wood at Jun 8, 2007 at 4:30 pm

    Noah wrote:
    On Jun 7, 9:01 am, dmoore wrote:

    popen and friends will never do what you want it to do. Down that path
    lies bitter disappointment.
    You need pseduo-ttys and non-blocking IO. I don't know how to do this
    on Windows, but I know it's possible.
    Cygwin does it.
    Anybody have any thoughts on this? Do I have my story straight? (the
    popen variants can't handle this case and there are no other
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
    Try Pexpect http://pexpect.sourceforge.net/
    It's been around for a long time and is quite complete and stable.

    The catch is that it's UNIX-only. If you want to tease out the magic
    code from wxProcess that
    does non-blocking reads on win32 then I'd be happy to integrate that
    into the current development branch
    of Pexpect.
    Windows has a really strange idea of non-blocking IO - it uses
    something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
    flag when you create the file/pipe. You then pass in overlap buffers
    for reading writing.

    I implemented this for serial ports in C a while ago. If you look at
    the code in pyserial (in serialwin32.py) you'll see a very similar
    implementation of non blocking IO. I assume that the same things will
    work for pipes, but I've only direct experience with serial ports!
    I'm sure this is feasible without any C extensions -- it might require
    some ctypes hacking.
    pyserial uses win32file and win32event
    I know Windows has pretty decent async IO, but I don't know what they
    have as an equivalent for a pty.
    Maybe it isn't necessary. A pty is only necessary on UNIX because the
    standard c library, stdio, behaves
    differently when it's talking to a plain pipe versus a terminal -- it
    switches buffering
    between block and line oriented buffer. You don't want block buffering
    on interactive applications.
    Pty's probably aren't needed on Windows.

    BTW I'd love to see pexpect working on windows and also in the
    standard library. It is the proper answer to controlling other
    interactive processes IMHO.

    --
    Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick
  • Dmoore at Jun 8, 2007 at 7:14 pm

    On Jun 8, 12:30 pm, Nick Craig-Wood wrote:
    Windows has a really strange idea of non-blocking IO - it uses
    something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
    flag when you create the file/pipe. You then pass in overlap buffers
    for reading writing.
    the wx guys appear to do it differently (unless FILE_FLAG_OVERLAPPED
    is implicit in the calls they make)

    take a look at:
    http://cvs.wxwidgets.org/viewcvs.cgi/wxWidgets/src/msw/utilsexc.cpp?rev=1.88&content-type=text/vnd.viewcvs-markup

    a reasonably well-documented wxExecute function handles all the messy
    win32 api calls and wraps the process in a wxProcess object and the
    streams in wxInputStream / wxOutputStream (also see the wxExecuteDDE
    function)
  • Nick Craig-Wood at Jun 10, 2007 at 5:36 am

    dmoore wrote:
    On Jun 8, 12:30 pm, Nick Craig-Wood wrote:
    Windows has a really strange idea of non-blocking IO - it uses
    something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
    flag when you create the file/pipe. You then pass in overlap buffers
    for reading writing.
    the wx guys appear to do it differently (unless FILE_FLAG_OVERLAPPED
    is implicit in the calls they make)

    take a look at:
    http://cvs.wxwidgets.org/viewcvs.cgi/wxWidgets/src/msw/utilsexc.cpp?rev=1.88&content-type=text/vnd.viewcvs-markup

    a reasonably well-documented wxExecute function handles all the messy
    win32 api calls and wraps the process in a wxProcess object and the
    streams in wxInputStream / wxOutputStream (also see the wxExecuteDDE
    function)
    You are right, named pipes seem to have a non blocking mode.

    Here is the lowdown from
    http://msdn2.microsoft.com/en-US/library/aa365605.aspx

    Both pipe clients and pipe servers can change a pipe handle's wait
    mode by specifying either PIPE_WAIT or PIPE_NOWAIT in a call to the
    SetNamedPipeHandleState function.

    Note The nonblocking-wait mode is supported for compatibility with
    Microsoft? LAN Manager version 2.0. This mode should not be used to
    achieve overlapped input and output (I/O) with named
    pipes. Overlapped I/O should be used instead, because it enables
    time-consuming operations to run in the background after the
    function returns. For more information about overlapped I/O, see
    Synchronous and Overlapped Input and Output.

    So it has a nonblocking mode but you shouldn't use it!

    --
    Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick
  • Hendrik van Rooyen at Jun 8, 2007 at 6:59 am
    "dmoore" wrote:

    8< --------------- <description of full duplex problem> ------------
    Anybody have any thoughts on this? Do I have my story straight? (the
    popen variants can't handle this case and there are no other
    alternatives in the standard python distro) Is there some place I can
    submit this as a feature request? (Python dev?)
    I think this is a hassle in the file implementation - I have had it also on
    serial ports, where it acts as if the file "driver" is inherently half duplex.

    It manifests as: "My characters don't come out"

    I don't know if it is python or the underlying stuff.

    I use fcntl to unblock and generally mess around to solve it, but
    I don't think my solution is portable.

    It is a real issue though - you are not imagining the dragon, it is
    right at the door...

    - Hendrik

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 7, '07 at 4:01p
activeJun 10, '07 at 5:36a
posts9
users5
websitepython.org

People

Translate

site design / logo © 2022 Grokbase