FAQ
Hi all,
Consider this scenario, where in I need to use subprocess to execute a
command like 'ping 127.0.0.1' which will have a continuous non-
terminating output in Linux.

# code
import subprocess
process = subprocess.Popen('ping 127.0.0.1', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print process.pid # returns pid
print process.poll() # returns None!!! this is strange.
print process.stdout.read()
# This hangs at this point.
How should I handle these kind of commands (ping 127.0.0.1) with
subprocess module. I am using subprocess, instead of os.system because
at anypoint in time, I need access to stdout and stderr of execution.

Thanks,
Senthil

Search Discussions

  • Zacherates at Jul 4, 2007 at 12:09 pm

    How should I handle these kind of commands (ping 127.0.0.1) with
    subprocess module. I am using subprocess, instead of os.system because
    at anypoint in time, I need access to stdout and stderr of execution.
    Ping, for one, allows you to set an upper bound on how long it runs
    (the -c option). This is probably the cleanest approach if it's
    available.

    You can also send the subprocess signals if you need it to exit
    (although, this is a unix thing so I'm not sure how portable it is).
    You could emulate having a timeout on child.stdout.read by registering
    a callback with Timer to kill the child.

    Cheers,
    Aaron
  • O.R.Senthil Kumaran at Jul 4, 2007 at 2:36 pm

    * zacherates [2007-07-04 12:09:03]:

    How should I handle these kind of commands (ping 127.0.0.1) with
    subprocess module. I am using subprocess, instead of os.system because
    at anypoint in time, I need access to stdout and stderr of execution.
    Ping, for one, allows you to set an upper bound on how long it runs
    (the -c option). This is probably the cleanest approach if it's
    available.
    Yes, I am aware of the ping -c option. But again even that does not help.
    try
    process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE,
    shell=True)
    process.stdout.read() # This will hang again.

    I am not sure, why subprocess is behaving so.
    You can also send the subprocess signals if you need it to exit
    (although, this is a unix thing so I'm not sure how portable it is).
    Yes, I have tried to kill and then get the standard output result.
    But the result has been the same. I could not read the Popen returned file
    object.
    You could emulate having a timeout on child.stdout.read by registering
    a callback with Timer to kill the child.
    I dont know how to do this. I shall give it a try ( by looking around ) and
    trying.

    --
    O.R.Senthil Kumaran
    http://uthcode.sarovar.org
  • Jerry Hill at Jul 4, 2007 at 3:23 pm

    On 7/4/07, O.R.Senthil Kumaran wrote:
    Yes, I am aware of the ping -c option. But again even that does not help.
    try
    process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE,
    shell=True)
    process.stdout.read() # This will hang again.
    When I try that, it doesn't hang. Instead, I get the output of the
    ping command pruinted to the screen, then the following exception:

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    AttributeError: 'NoneType' object has no attribute 'read'

    That's because you tied stdin to a pipe in your Popen call, but then
    tried to read from stdout. Try this instead:
    process = subprocess.Popen("ping -c 10 127.0.0.1",
    stdout=subprocess.PIPE, shell=True)
    process.stdout.readlines()
    ['PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.\n', '64 bytes from
    127.0.0.1: icmp_seq=1 ttld time=0.049 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=2 ttld time=0.033 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=3 ttld time=0.040 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=4 ttld time=0.033 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=5 ttld time=0.033 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=6 ttld time=0.030 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=7 ttld time=0.032 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=8 ttld time=0.028 ms\n', '64 bytes from
    127.0.0.1: icmp_seq=9 ttld time=0.030 ms\n', '64 bytes from
    127.0.0.1: icmp_seq ttld time=0.039 ms\n', '\n', '--- 127.0.0.1
    ping statistics ---\n', '10 packets transmitted, 10 received, 0%
    packet loss, time 8991ms\n', 'rtt min/avg/max/mdev 0.028/0.034/0.049/0.009 ms\n']

    --
    Jerry
  • O.R.Senthil Kumaran at Jul 4, 2007 at 7:29 pm
    * Jerry Hill [2007-07-04 11:23:33]:
    That's because you tied stdin to a pipe in your Popen call, but then
    tried to read from stdout. Try this instead:
    My mistake. I had just 'typed' the command in the mail itself and forgot to
    include the stdin, stdout, and stderr and mentioned it as hung based on some
    recollection.
    process = subprocess.Popen("ping -c 10 127.0.0.1",
    stdout=subprocess.PIPE, shell=True)
    process.stdout.readlines()
    I tried it again and found that giving the -c 10 returns a well defined
    output.
    Only when the program has executed and the output available, subprocess can
    read through PIPE's stdout it seems ( not at any other time).
    With killing, I loose the output.
    process = subprocess.Popen('ping 10 127.0.0.1', stdin=subprocess.PIPE,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,shell=True)
    process.pid
    3475
    import os
    import signal
    os.kill(process.pid,signal.SIGINT)
    process.stdout.read()
    ''
    # required output is lost!

    --
    O.R.Senthil Kumaran
    http://uthcode.sarovar.org
  • Jerry Hill at Jul 5, 2007 at 3:30 am

    On 7/4/07, O.R.Senthil Kumaran wrote:
    Only when the program has executed and the output available, subprocess can
    read through PIPE's stdout it seems ( not at any other time).
    With killing, I loose the output.
    I think you have to read the data from the process's stdout before you
    kill it. If you want to process the output of the program before it's
    done, do something like this:

    import subprocess
    process = subprocess.Popen("ping 127.0.0.1", stdout=subprocess.PIPE, shell=True)
    for i in xrange(10):
    line = process.stdout.readline()
    print "line:", repr(line)

    Then you can decide to kill the subprocess when you have enough data,
    or whatever.

    Also, even though the output from ping appears to be pretty easy to
    read, there can be problems with buffering when you connect a pipe to
    the stdout of some programs. The pexpect faq talks a little bit about
    that - http://pexpect.sourceforge.net/#faq . If you're working under
    a unix system, pexpect is really useful for automating interactive
    programs that are otherwise difficult to work with.

    --
    Jerry
  • Zacherates at Jul 4, 2007 at 8:19 pm

    Only when the program has executed and the output available, subprocess can
    read through PIPE's stdout it seems ( not at any other time).
    With killing, I loose the output.
    This is untrue.
    process.stdout.read() # Blocks until end of stream.
    process.stdout.read(1) # Reads one character, only blocks if that character is unavailable.
    As such you can read the needed chars from the child's STDOUT one at a
    time. For example:

    import os
    import signal
    import subprocess
    import threading
    import sys

    stop = False
    ping = subprocess.Popen('ping 127.0.0.1', shell = True, stdout =
    subprocess.PIPE)

    def kill():
    global stop
    stop = True
    os.kill(ping.pid, signal.SIGTERM)

    threading.Timer(5, kill).start()

    while not stop:
    sys.stdout.write(ping.stdout.read(1))

    This solution let's you read from the stdout of a program that may
    never terminate and time out after a certain amount of time but it's
    not pretty. It's unix specific and introduces threads into a program
    that doesn't need them. I'd go with trying to limit the time the
    child runs through command line options if at all possible.

    Cheers,
    Aaron



    From http Wed Jul 4 22:22:12 2007
    From: http (Paul Rubin)
    Date: 04 Jul 2007 13:22:12 -0700
    Subject: PEP 3107 and stronger typing (note: probably a newbie question)
    References: <5dveb4F360un0U1@mid.individual.net>
    <mailman.9346.1182435283.32031.python-list@python.org>
    <1182858785.886469.93060@n60g2000hse.googlegroups.com>
    <46824d8d$0$2324$426a74cc@news.free.fr>
    <h1wgi.9029$c06.5169@newssvr22.news.prodigy.net>
    <4682eddc$0$23164$426a74cc@news.free.fr>
    <5ehokpF38ov3jU2@mid.individual.net>
    <4683d5cd$0$8993$426a74cc@news.free.fr>
    <7xps3e3g9x.fsf@ruckus.brouhaha.com>
    <468565a0$0$3686$426a74cc@news.free.fr>
    <7xhcopj10g.fsf@ruckus.brouhaha.com>
    <46863ea7$0$970$426a34cc@news.free.fr>
    <1i0ikru.1uccqtm1efnzn3N%aleax@mac.com>
    <donn-B9CA13.13464802072007@gnus01.u.washington.edu>
    <1i0n0tr.ve0o791347ugyN%aleax@mac.com>
    <7xd4za1086.fsf@ruckus.brouhaha.com>
    <5f0mhuF3b0stbU1@mid.individual.net>
    <7xd4z837e2.fsf@ruckus.brouhaha.com>
    <468b7426$0$9063$426a34cc@news.free.fr>
    <7x4pkkun56.fsf@ruckus.brouhaha.com>
    <1183575597.272150.152200@w5g2000hsg.googlegroups.com>
    <foSii.3330$rL1.2022@newssvr19.news.prodigy.net>
    Message-ID: <7xzm2budfv.fsf@ruckus.brouhaha.com>

    John Nagle <nagle at animats.com> writes:
    This has been tried. Original K&R C had non-enforced static typing.
    All "struct" pointers were equivalent. It wasn't pretty.

    It takes strict programmer discipline to make non-enforced static
    typing work. I've seen it work in an aerospace company, but the Python
    crowd probably doesn't want that level of engineering discipline.
    I think even enforced static types wouldn't cure what I see as the
    looseness in Python. There is not enough composability of small
    snippets of code. For example, the "for" statement clobbers its index
    variable and then leaks it to the outside of the loop. That may be
    more of a culprit than dynamic types. Perl and C++ both fix this with
    syntax like

    for (my $i in ...) ... (perl) or
    for (int i = 0; i < n; i++) ... (C++, Java)

    making a temporary scope for the index variable. Python even leaks
    the index variable of list comprehensions (I've mostly stopped using
    them because of this), though that's a recognized wart and is due to
    be fixed.

    Python would be helped a lot by some way to introduce temporary
    scopes. We had some discussion of this recently, concluding that
    there should be a compiler warning message if a variable in a
    temporary scope shadows one from a surrounding scope.
  • Prikar20 at Jul 4, 2007 at 8:55 pm

    On Jul 4, 12:29 pm, "O.R.Senthil Kumaran" wrote:
    * Jerry Hill [2007-07-04 11:23:33]:


    That's because you tied stdin to a pipe in your Popen call, but then
    tried to read from stdout. Try this instead:
    My mistake. I had just 'typed' the command in the mail itself and forgot to
    include the stdin, stdout, and stderr and mentioned it as hung based on some
    recollection.


    process = subprocess.Popen("ping -c 10 127.0.0.1",
    stdout=subprocess.PIPE, shell=True)
    process.stdout.readlines()
    I tried it again and found that giving the -c 10 returns a well defined
    output.
    Only when the program has executed and the output available, subprocess can
    read through PIPE's stdout it seems ( not at any other time).
    With killing, I loose the output.
    process = subprocess.Popen('ping 10 127.0.0.1', stdin=subprocess.PIPE,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,shell=True)
    I think you meant ping -c 10 (and not ping 10). If you pass first arg
    as string, are you sure you didn't get an exception? I get one if I
    use 'ping -c 10 <ip-addr>' -- because looks like subprocess is
    searching for an executable named as "ping -c 10 ..." (not just
    'ping').
    So I sent in a sequence and it all worked as expected (I didn't have
    shell=True, but it shouldn't matter in getting the required output).

    Try the sequence as first arg.
    process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/auto/xxxkarthikxxx/python251/lib/python2.5/subprocess.py",
    line 593, in __init__
    errread, errwrite)
    File "/auto/xxxkarthikxxx/python251/lib/python2.5/subprocess.py",
    line 1079, in _execute_child
    raise child_exception
    OSError: [Errno 2] No such file or directory
    process = subprocess.Popen('ping -c 10 127.0.0.1'.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    print process
    <subprocess.Popen object at 0xb7580aac>
    print process.pid
    13435
    print process.stdout.read()
    PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
    64 bytes from 127.0.0.1: icmp_seq=0 ttld time=0.025 ms
    64 bytes from 127.0.0.1: icmp_seq=1 ttld time=0.006 ms
    64 bytes from 127.0.0.1: icmp_seq=2 ttld time=0.005 ms
    64 bytes from 127.0.0.1: icmp_seq=3 ttld time=0.004 ms
    64 bytes from 127.0.0.1: icmp_seq=4 ttld time=0.011 ms
    64 bytes from 127.0.0.1: icmp_seq=5 ttld time=0.007 ms
    64 bytes from 127.0.0.1: icmp_seq=6 ttld time=0.020 ms
    64 bytes from 127.0.0.1: icmp_seq=7 ttld time=0.006 ms
    64 bytes from 127.0.0.1: icmp_seq=8 ttld time=0.006 ms
    64 bytes from 127.0.0.1: icmp_seq=9 ttld time=0.006 ms

    --- 127.0.0.1 ping statistics ---
    10 packets transmitted, 10 received, 0% packet loss, time 9013ms
    rtt min/avg/max/mdev = 0.004/0.009/0.025/0.007 ms, pipe 2

    >>>

    -- Karthik
    process.pid
    3475
    import os
    import signal
    os.kill(process.pid,signal.SIGINT)
    process.stdout.read()
    ''
    # required output is lost!
    --
    O.R.Senthil Kumaranhttp://uthcode.sarovar.org
  • Steve Holden at Jul 4, 2007 at 3:59 pm

    O.R.Senthil Kumaran wrote:
    * zacherates [2007-07-04 12:09:03]:
    How should I handle these kind of commands (ping 127.0.0.1) with
    subprocess module. I am using subprocess, instead of os.system because
    at anypoint in time, I need access to stdout and stderr of execution.
    Ping, for one, allows you to set an upper bound on how long it runs
    (the -c option). This is probably the cleanest approach if it's
    available.
    Yes, I am aware of the ping -c option. But again even that does not help.
    try
    process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE,
    shell=True)
    process.stdout.read() # This will hang again.

    I am not sure, why subprocess is behaving so.
    You can also send the subprocess signals if you need it to exit
    (although, this is a unix thing so I'm not sure how portable it is).
    Yes, I have tried to kill and then get the standard output result.
    But the result has been the same. I could not read the Popen returned file
    object.
    You could emulate having a timeout on child.stdout.read by registering
    a callback with Timer to kill the child.
    I dont know how to do this. I shall give it a try ( by looking around ) and
    trying.
    Is it possible your ping implementation outputs to stderr?

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    --------------- Asciimercial ------------------
    Get on the web: Blog, lens and tag the Internet
    Many services currently offer free registration
    ----------- Thank You for Reading -------------
  • Karthik Gurusamy at Jul 4, 2007 at 9:52 pm

    On Jul 4, 4:38 am, Phoe6 wrote:
    Hi all,
    Consider this scenario, where in I need to use subprocess to execute a
    command like 'ping 127.0.0.1' which will have a continuous non-
    terminating output in Linux.

    # code
    import subprocess
    process = subprocess.Popen('ping 127.0.0.1', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    print process.pid # returns pid
    print process.poll() # returns None!!! this is strange.
    It's expected behavior. It means the child process is still running.
    print process.stdout.read()
    # This hangs at this point.
    This too is expected behavior. 'ping <ip-addr>' runs forever
    generating continuous output. It doesn't stop by itself.

    read() reads until there is data available in the file object. Thus it
    doesn't ever finish since the child never stops generating data.

    If you do read(n), then it reads n bytes and returns.

    How should I handle these kind of commands (ping 127.0.0.1) with
    subprocess module. I am using subprocess, instead of os.system because
    at anypoint in time, I need access to stdout and stderr of execution.
    Using subprocess is good. Just ensure your child stops data generation
    at some point. For ping, you can use '-c <num>' or some other
    application, you can try closing it's stdin (e.g. cat, bc, gdb)

    Thanks,
    Karthik
    Thanks,
    Senthil

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJul 4, '07 at 11:38a
activeJul 5, '07 at 3:30a
posts10
users7
websitepython.org

People

Translate

site design / logo © 2022 Grokbase