FAQ
It goes like this:
Im converting a perl script that executes a command on each result of
a directory listing. In perl, all is well. In Python, using
approximately
the same logic creates a seemingly random problem I can't pin down.
I hesitated to ask this question since it's so Slackware
Linux-centric,
and im new to Python. It's hard to ask in a generic way, but im out of
options...

The command is Slackware's package creation utility, makepkg. It uses
the
current working directory as package source and takes a filename as an
argument to output the newly created package to. Makepkg is
essentially
a tarball creator, wrapped in a nice command. Simple stuff. Here is my
code:

I have a for loop that reads a file list, like so:
pkg_srcdir = '/packages_source'
pkg_dir = '/packages_tgzs'
ls = os.listdir(pkg_srcdir)

for name in ls:
build_src = pkg_srcdir + '/' + name
new_filename = pkg_dir + '/' + name + '.tgz'

# Valid directory?
if not os.path.isdir(build_src):
sys.exit("\nInvalid directory \'" +
pkg_srcdir + "\'\n")

# Go into the new directory
try: os.chdir(build_src)
except os.error, value:
sys.exit("\nInvalid directory: \'" +
build_src + "\': " + value[1])

# I then execute the command on each result,
like so:
# new_filename is correctly assembled, which i
can verify
# from printing it out on each loop iteration
os.system('/sbin/makepkg --chown n --linkadd y
' + new_filename)

When all is done running, which take a good 7 minutes, I get varied
results.
It looks like its working... pauses, has lots of disk activity on the
drive
(verified with ps and vmstat), but in the end only a couple of
packages are
created. I can actually watch the output of makepkg and see it
creating the
tarballs. Here is the truly whacky part: Changing pkg_dir to anything
else,
say /tmp/beavis, will create the packages correctly for a while, then
mysteriously stop with no warning or change to the script - just run
it a
few times and it stops working.

Im running as root, so its not a permission problem - its far to
erratic
anyway.

I dont think this problem has anything to do with os.system, per say,
since it also happens with os.popen.

I thought it might be a buffering problem, so i tried using Python's
-u
switch and flushing stdout with sys.stdout.flush() - no go.

I tried sleeping for long periods of time in each loop iteration - no
go.

I tried writing each individual makepkg command out to a file and
running it with perl, doing all the proper filehandling. I also
tried bash for kicks - no go

In the end, I put an ampersand at the end of the makepkg command, like
so:
os.system('/sbin/makepkg --chown n --linkadd y ' +
new_filename + ' &')

This sort of works: it creates 95% of the packages most of the time -
results varies. It always seems to leave out the same packages.
The packages it misses are in the middle of the list, so its not
just skipping the first or the last few.

Im baffled.

Any ideas?

Search Discussions

  • Donn Cave at Jan 15, 2004 at 9:09 pm
    In article <604fc0ca.0401141750.4dc57782 at posting.google.com>,
    even at pondie.com (Ben Floyd) wrote:
    ...
    The command is Slackware's package creation utility, makepkg. It uses
    the
    current working directory as package source and takes a filename as an
    argument to output the newly created package to. Makepkg is
    essentially
    a tarball creator, wrapped in a nice command. Simple stuff. Here is my
    code:

    I have a for loop that reads a file list, like so:
    pkg_srcdir = '/packages_source'
    pkg_dir = '/packages_tgzs'
    ls = os.listdir(pkg_srcdir)

    for name in ls:
    build_src = pkg_srcdir + '/' + name
    new_filename = pkg_dir + '/' + name + '.tgz'

    # Valid directory?
    if not os.path.isdir(build_src):
    sys.exit("\nInvalid directory \'" + pkg_srcdir + "\'\n")

    # Go into the new directory
    try: os.chdir(build_src)
    except os.error, value:
    sys.exit("\nInvalid directory: \'" + build_src + "\': " + value[1])

    # I then execute the command on each result, like so:
    # new_filename is correctly assembled, which i can verify
    # from printing it out on each loop iteration
    os.system('/sbin/makepkg --chown n --linkadd y ' + new_filename)
    Well, I don't know. In casual inspection I don't notice
    anything obviously wrong, but then for all I know this
    isn't the exact code that has the problem anyway. What
    I could suggest is substitute your own shell script for
    makepkg, something that prints out its arguments, current
    working directory and whatever you think might be interesting.
    The object is to find out exactly what you're asking makepkg
    to do, since that is evidently a problem.

    Actually a Python program might be a little better, because
    "print sys.argv" gives you less ambiguous information than
    "echo $*", more like "for arg do echo $arg; done" but on
    one line.

    The two usage points that occur to me are
    1. chdir() affects Python module path resolution. If you
    chdir to a directory with an "os.py", that would be an
    obvious if unlikely problem, but also some installations
    rely on Python's library finding heuristics in a way
    that is kind of fragile. I may be wrong about that,
    I don't even want to think about wasting time on this
    misguided feature.

    2. system() can be replaced with spawnv() for an increase
    in command parameter list reliability. What if some
    file has spaces? In spawnv(), you don't care, but in
    system() it's your job to quote it. This is also a
    security related issue.

    Donn Cave, donn at u.washington.edu

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJan 15, '04 at 1:50a
activeJan 15, '04 at 9:09p
posts2
users2
websitepython.org

2 users in discussion

Ben Floyd: 1 post Donn Cave: 1 post

People

Translate

site design / logo © 2022 Grokbase