FAQ

pdbdir = os.popen('ls /home/data/pdbs') # if this directory
contains
10000 files, the code does not work.

files=pdbdir.readlines()
pdbdir.close()
(o,i) = popen2.popen2("/home/programs/profit ")

i.write("reference /usr/local/3D_Dock/progs/Complex_1g.pdb ")
# This the
reference molecule.

for PDB in files:
i.write(" mobile /home/data/pdbs/" + PDB + "\n") # A new
molecule.
i.write("fit\n") # compares reference and new molecules.

i.close()
result= o.read()
o.close()
You read from the input pipe only upon writing all your data,
but maybe the pipe is full?

How about changing the loop with this:


log = StringIO.StringIO()

for PDB in files:
i.write("mobile /home/data/pdbs/%s\n" % PDB)
i.write("fit\n")
log.write(o.read())

i.close()
o.close()


So now your results are in the log object.

-gus

Search Discussions

  • Gustavo Cordova at Feb 22, 2002 at 10:45 pm
    Yeah, Jeff Shannon wrote something about
    pipes hanging in a .read() operation. :-/

    So... any other clues yet?

    -gus
    -----Original Message-----
    From: husam [mailto:h.jehadalwan at student.kun.nl]
    Sent: Viernes, 22 de Febrero de 2002 04:48 p.m.
    To: python-list at python.org
    Subject: Re: is their any limit on the size of the data passed to a
    progra m via popen2?


    Gustavo Cordova wrote:
    pdbdir = os.popen('ls /home/data/pdbs') # if this directory
    contains
    10000 files, the code does not work.

    files=pdbdir.readlines()
    pdbdir.close()
    (o,i) = popen2.popen2("/home/programs/profit ")

    i.write("reference /usr/local/3D_Dock/progs/Complex_1g.pdb ")
    # This the
    reference molecule.

    for PDB in files:
    i.write(" mobile /home/data/pdbs/" + PDB + "\n") # A new
    molecule.
    i.write("fit\n") # compares reference
    and new molecules.
    i.close()
    result= o.read()
    o.close()
    You read from the input pipe only upon writing all your data,
    but maybe the pipe is full?

    How about changing the loop with this:


    log = StringIO.StringIO()

    for PDB in files:
    i.write("mobile /home/data/pdbs/%s\n" % PDB)
    i.write("fit\n")
    log.write(o.read())

    i.close()
    o.close()


    So now your results are in the log object.

    -gus
    hi Gustaveo,
    You'r solution did not work either. The code hangs at the first loop,
    specifically at "log.write(o.read())".

    --
    http://mail.python.org/mailman/listinfo/python-list
  • Husam at Feb 22, 2002 at 10:48 pm

    Gustavo Cordova wrote:

    pdbdir = os.popen('ls /home/data/pdbs') # if this directory
    contains
    10000 files, the code does not work.

    files=pdbdir.readlines()
    pdbdir.close()
    (o,i) = popen2.popen2("/home/programs/profit ")

    i.write("reference /usr/local/3D_Dock/progs/Complex_1g.pdb ")
    # This the
    reference molecule.

    for PDB in files:
    i.write(" mobile /home/data/pdbs/" + PDB + "\n") # A new
    molecule.
    i.write("fit\n") # compares reference and new molecules.

    i.close()
    result= o.read()
    o.close()
    You read from the input pipe only upon writing all your data,
    but maybe the pipe is full?

    How about changing the loop with this:


    log = StringIO.StringIO()

    for PDB in files:
    i.write("mobile /home/data/pdbs/%s\n" % PDB)
    i.write("fit\n")
    log.write(o.read())

    i.close()
    o.close()


    So now your results are in the log object.

    -gus
    hi Gustaveo,
    You'r solution did not work either. The code hangs at the first loop,
    specifically at "log.write(o.read())".
  • Carel Fellinger at Feb 23, 2002 at 12:34 am

    husam wrote:

    hi Gustaveo,
    You'r solution did not work either. The code hangs at the first loop,
    specifically at "log.write(o.read())".
    You don't seem to be having a Python problem, so what about trying to
    get some output from the program? Instead of reading the log into a
    var, what about trying it with `os.popen("/home/programs/profit", "w)'
    and have the output directed at the terminal for direct inspection.
    Maybe you're just missing error messages:)

    An other problem that might buck you (but I don't see how this would
    be the case as your program seems to work for a few files) is that
    pipes are buffered. So uses `i.flush()' to be sure that what you
    wrote really is transmitted to the other end of the pipe.

    And as a last resort you could try reading the `ls' output a line at a
    time.

    i = os.popen("/home/programs/profit ", "w")

    Should this ---------------------------^ space be there?

    i.write("reference /usr/local/3D_Dock/progs/Complex_1g.pdb ")
    for PDB in os.popen('ls /home/data/pdbs'):
    i.write(" mobile /home/data/pdbs/" + PDB + "\n")

    Should this -----^ space be there?

    i.write("fit\n")
    i.flush()
    i.close()
    --
    groetjes, carel
  • Husam at Feb 23, 2002 at 9:33 am

    Carel Fellinger wrote:

    husam wrote:

    hi Gustaveo,
    You'r solution did not work either. The code hangs at the first loop,
    specifically at "log.write(o.read())".
    You don't seem to be having a Python problem, so what about trying to
    get some output from the program? Instead of reading the log into a
    var, what about trying it with `os.popen("/home/programs/profit", "w)'
    Off course, it works this way. But, then I have to execute another
    script to process the output. Therefor I chosen for "popen2" so that I
    can process the output in the same script.


    and have the output directed at the terminal for direct inspection.
    Maybe you're just missing error messages:)

    An other problem that might buck you (but I don't see how this would
    be the case as your program seems to work for a few files) is that
    pipes are buffered. So uses `i.flush()' to be sure that what you

    I'v tried "flush()" already, no go.

    wrote really is transmitted to the other end of the pipe.

    And as a last resort you could try reading the `ls' output a line at a
    time.

    i = os.popen("/home/programs/profit ", "w")

    Should this ---------------------------^ space be there?

    The space is needed here, otherwise the command and it's arguments
    passed to the progam, get passed like this: "CommandArguments" instead
    of " Command Arguments"

    i.write("reference /usr/local/3D_Dock/progs/Complex_1g.pdb ")
    for PDB in os.popen('ls /home/data/pdbs'):
    i.write(" mobile /home/data/pdbs/" + PDB + "\n")

    Should this -----^ space be there?

    The same explanation as above!
    i.write("fit\n")
    i.flush()
    i.close()
  • Carel Fellinger at Feb 23, 2002 at 5:37 pm

    husam wrote:
    Carel Fellinger wrote:
    husam wrote:
    ...
    var, what about trying it with `os.popen("/home/programs/profit", "w)'
    Off course, it works this way. But, then I have to execute another
    script to process the output. Therefor I chosen for "popen2" so that I
    can process the output in the same script.
    Okee, so it works with lots of files and popen and fails for popen2.

    A way to circumvent the problem then might be to use popen and redirect
    the output to a temp file like:

    os.popen("/home/programs/profit >/tmp/tempfile", "w")

    and read this file in the same srcipt when `profit' is done.

    But this would deprive us from understanding what went wrong:)

    i = os.popen("/home/programs/profit ", "w")

    Should this ---------------------------^ space be there?
    The space is needed here, otherwise the command and it's arguments
    passed to the progam, get passed like this: "CommandArguments" instead
    of " Command Arguments"
    I don't get this. All arguments to `/home/programs/profit' have to be
    given in the popen* call. I don't see arguments in that call so I
    don't see need of the space here. Or are you using a different script
    than what you showed us here? Even slight differences might be
    relevant, so if you're running different sripts how about showing
    them?
    i.write(" mobile /home/data/pdbs/" + PDB + "\n")

    Should this -----^ space be there?
    The same explanation as above!
    When you write to `i' you don't add arguments to the commandline used
    to start `profit', you're writing to the inputstream of `pdbs', quit a
    different thing. So I still don't see the need for this space.



    --
    groetjes, carel
  • Carel Fellinger at Feb 23, 2002 at 5:47 pm

    Carel Fellinger wrote:

    When you write to `i' you don't add arguments to the commandline used
    to start `profit', you're writing to the inputstream of `pdbs', quit a
    oeps, `pdbs' should read `profit'

    --
    groetjes, carel
  • Husam at Feb 24, 2002 at 9:28 am

    Carel Fellinger wrote:

    Carel Fellinger wrote:

    When you write to `i' you don't add arguments to the commandline used
    to start `profit', you're writing to the inputstream of `pdbs', quit a
    oeps, `pdbs' should read `profit'
    hee, NEDERLANDER, leuk je te ontmoeten hier. Jammer dat er geen
    Nederlandse newsgroup is voor python.


    groeten.
  • Carel Fellinger at Feb 24, 2002 at 5:00 pm
    husam wrote:
    ...
    hee, NEDERLANDER, leuk je te ontmoeten hier. Jammer dat er geen
    Nederlandse newsgroup is voor python.
    Ah, maar dit *is* de Nederlandse newsgroep voor Python.
    Zo ongeveer iedereen hier beheerst Nederlands alsof het zijn
    moers taal is, per slot komt Python ook uit Nederland:)


    --
    groetjes, carel
  • Husam at Feb 22, 2002 at 11:30 pm

    Gustavo Cordova wrote:

    Yeah, Jeff Shannon wrote something about
    pipes hanging in a .read() operation. :-/

    So... any other clues yet?

    -gus
    sorry, I don't get it! What did Jeff wrote? and where?
  • Gustavo Cordova at Feb 22, 2002 at 11:33 pm
    That doing a read() on pipes hangs, so you have to
    define a timeout or something, so that you can take
    control back.

    But I dinnae kno how ta do such a thang.

    :-/

    -gus

    pd: I *think* it was jeff.
  • Husam at Feb 23, 2002 at 2:58 pm

    Gustavo Cordova wrote:

    That doing a read() on pipes hangs, so you have to
    define a timeout or something, so that you can take
    control back.

    But I dinnae kno how ta do such a thang.

    :-/

    -gus

    pd: I *think* it was jeff.
    Ya, I red that one. But, now the problem is that running the code
    succefully really depends on the number of the files passed to the
    application in one "popen2" session. I found that the number of the
    files must not exceed 250. I worked around this by splitting the 10000
    files into 50 lists each of them contains 200 files and for each list
    opening a "popen2" session and close it at the end in a "for" loop. Now
    the code works fine. I have searched the internet for informations
    related to the size of the pipe and how to modify it, but I found
    nothing except "select.select()". At the moment I really don't have time
    to implement this.

    any way, thanks for the contribution.
  • Steve Holden at Mar 4, 2002 at 3:39 pm
    "husam" <h.jehadalwan at student.kun.nl> wrote in message
    news:3C77AE2E.3020301 at student.kun.nl...
    Gustavo Cordova wrote:
    That doing a read() on pipes hangs, so you have to
    define a timeout or something, so that you can take
    control back.

    But I dinnae kno how ta do such a thang.

    :-/

    -gus

    pd: I *think* it was jeff.
    Ya, I red that one. But, now the problem is that running the code
    succefully really depends on the number of the files passed to the
    application in one "popen2" session. I found that the number of the
    files must not exceed 250. I worked around this by splitting the 10000
    files into 50 lists each of them contains 200 files and for each list
    opening a "popen2" session and close it at the end in a "for" loop. Now
    the code works fine. I have searched the internet for informations
    related to the size of the pipe and how to modify it, but I found
    nothing except "select.select()". At the moment I really don't have time
    to implement this.

    any way, thanks for the contribution.
    I'm coming in to this conversation a little late, but is there a possibilty
    that you could actually transmit the contents of multiple files via a single
    stream, allowing you to feed files into the popen()'d subprocesses as each
    consumes input? This might save a lot a process- and file-creation overhead
    if all you really need is the computaional output. You might use a netstring
    representation, for example.

    Even if all the above is true, don't be surprised to hit hard limits on
    other annoying resources like threads-per-process or processes per subgroup.

    That "250", by they way? Might it be 254 or 255, really? Or possibly
    (255-number of other command line arguments)? This would be a typical result
    of some element of the pipeline limiting the length of the argument vector
    so a byte could be used to select each element. So, you are probably the
    victim of thoughtless design, or of some historical features of the code you
    use.

    Even so there are always limitations, whether it's long arithmetic slowdown
    or inappropriate representation, and TheOnlyWayForward is to
    MercilesslyRefactor. Every environment has limits, and sometimes it's a
    question of learning what limits you can live with. Onward and downward, as
    the Pascal standardization committee might have declared.

    wondering-what-happened-to-tony-addyman-ly y'rs - steve
    --
    Consulting, training, speaking: http://www.holdenweb.com/
    Author, Python Web Programming: http://pydish.holdenweb.com/pwp/

    "This is Python. We don't care much about theory, except where it
    intersects with useful practice." Aahz Maruch on c.l.py

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedFeb 22, '02 at 5:59p
activeMar 4, '02 at 3:39p
posts13
users4
websitepython.org

People

Translate

site design / logo © 2021 Grokbase