FAQ
Hello


I've a list of tasks to perform. Each of them is a threading.Thread.
Basically I have :

while task_list :
task = task_list[0]
task.run()
task_list.remove(task)


Now I want, in some circumstance to raise errors that make the loop stop.

In order IOError to make stop the loop, I tried this :

no_error = True
while task_list and no_error:
task = task_list[0]
try :
task.run()
except IOError :
no_error = False
task_list.remove(task)

But it does not work (I think the exception is raised too late).

I prefer that the taske are not aware to be included in a loop, but I
can make them raise personnal exceptions.
How can I do ?


Thanks
Laurent

Search Discussions

  • Chris Angelico at Jun 19, 2011 at 3:19 pm

    On Mon, Jun 20, 2011 at 12:42 AM, Laurent Claessens wrote:
    ? Hello


    I've a list of tasks to perform. Each of them is a threading.Thread.
    Basically I have :

    while task_list :
    ? task = task_list[0]
    ? task.run()
    ? task_list.remove(task)
    I'm not understanding what you're doing with threads here. Are you
    using threading.Thread but then calling its run() method
    synchronously?

    Normally threads are used for asynchronous operations. You would then
    use the start() method to spin the thread off; it will return almost
    immediately, and the thread will run to completion in parallel with
    you. But then you can't halt the main loop, because it will have
    already finished by the time you detect the IOError (starting a bunch
    of threads is pretty quick). On the other hand, the code you're
    showing seems to simply call each thread's run() method one by one,
    which should propagate any exceptions in the same way that function
    calls usually do.

    Can you share the code for one of the tasks, and show what happens
    when it raises an exception?

    Chris Angelico
  • Laurent Claessens at Jun 19, 2011 at 3:39 pm

    Le 19/06/2011 17:19, Chris Angelico a ?crit :
    On Mon, Jun 20, 2011 at 12:42 AM, Laurent Claessenswrote:
    Hello


    I've a list of tasks to perform. Each of them is a threading.Thread.
    Basically I have :

    while task_list :
    task = task_list[0]
    task.run()
    task_list.remove(task)
    I'm not understanding what you're doing with threads here. Are you
    using threading.Thread but then calling its run() method
    synchronously?
    Woops yes. I missprinted my example. I was using task.start() of course.

    The aim is to copy the content of a repertory (with some conditions on
    each file, so I cannot use shutils or something).

    I've one thread that runs over the repertory and fill the list
    'task_list' with taskes from the following class :


    class FileCopyTask(threading.Thread):
    def __init__(self,source,destination,old_version):
    threading.Thread.__init__(self)
    self.source = source
    self.destination = destination
    def run(self):
    try :
    shutil.copy(self.source,self.destination)
    except (IOError,OSError),data :
    <WHAT TO PUT HERE ??>
    else :
    print "file copied"


    In the same time I've a thread that read the list and perform the
    operations:

    def run():
    while task_list :
    task = task_list[0]
    task_list.remove(task)
    task.start()


    My problem is that when FileToCopyTask raises an error, the program does
    not stop.
    In fact when the error is Disk Full, I want to stop the whole program
    because I know that the next task will fail too.

    thanks for any help
    Laurent
  • Laurent Claessens at Jun 19, 2011 at 3:54 pm
    I read the library documentation. I think that if I get a trick to kill
    a thread, then I'm done.

    Is there a way ?

    Laurent

    Le 19/06/2011 17:39, Laurent Claessens a ?crit :
    Le 19/06/2011 17:19, Chris Angelico a ?crit :
    On Mon, Jun 20, 2011 at 12:42 AM, Laurent Claessenswrote:
    Hello


    I've a list of tasks to perform. Each of them is a threading.Thread.
    Basically I have :
    >
  • Chris Angelico at Jun 19, 2011 at 4:03 pm

    On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens wrote:
    My problem is that when FileToCopyTask raises an error, the program does not
    stop.
    In fact when the error is Disk Full, I want to stop the whole program
    because I know that the next task will fail too.
    If you're starting a thread for every file you're copying, you're
    starting a huge number of threads that probably will just end up
    fighting over the disk. To get a reasonably efficient early-abort, I'd
    recommend having a fixed pool of worker threads (say, ten of them),
    and have each thread (a) check if the early-abort flag is set, and
    then (b) start copying the next file in queue. Once the queue's empty
    or the early-abort flag is set, all ten threads will terminate when
    they finish their current transfers.

    (The ten threads figure is arbitrary. Optimum value for performance
    will come by adjusting this.)

    ChrisA
  • Terry Reedy at Jun 19, 2011 at 4:58 pm

    On 6/19/2011 12:03 PM, Chris Angelico wrote:
    On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessenswrote:
    My problem is that when FileToCopyTask raises an error, the program does not
    stop.
    In fact when the error is Disk Full, I want to stop the whole program
    because I know that the next task will fail too.
    If you're starting a thread for every file you're copying, you're
    starting a huge number of threads that probably will just end up
    fighting over the disk. To get a reasonably efficient early-abort, I'd
    recommend having a fixed pool of worker threads (say, ten of them),
    and have each thread (a) check if the early-abort flag is set, and
    then (b) start copying the next file in queue. Once the queue's empty
    or the early-abort flag is set, all ten threads will terminate when
    they finish their current transfers.

    (The ten threads figure is arbitrary. Optimum value for performance
    will come by adjusting this.)
    I an not convinced that much of anything is gained by having multiple
    copying threads. It certainly adds complication -- hence this thread. I
    would just use shutil to copy files or directories of files.

    Note that copytree has an option to pass a filter function:
    "If ignore is given, it must be a callable that will receive as its
    arguments the directory being visited by copytree(), and a list of its
    contents, as returned by os.listdir(). Since copytree() is called
    recursively, the ignore callable will be called once for each directory
    that is copied. The callable must return a sequence of directory and
    file names relative to the current directory (i.e. a subset of the items
    in its second argument); these names will then be ignored in the copy
    process. ignore_patterns() can be used to create such a callable that
    ignores names based on glob-style patterns."

    --
    Terry Jan Reedy
  • Laurent Claessens at Jun 19, 2011 at 4:08 pm

    Le 19/06/2011 18:03, Chris Angelico a ?crit :
    On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessenswrote:
    My problem is that when FileToCopyTask raises an error, the program does not
    stop.
    In fact when the error is Disk Full, I want to stop the whole program
    because I know that the next task will fail too.
    If you're starting a thread for every file you're copying, you're
    starting a huge number of threads that probably will just end up
    fighting over the disk. To get a reasonably efficient early-abort, I'd
    recommend having a fixed pool of worker threads (say, ten of them),
    and have each thread (a) check if the early-abort flag is set, and
    then (b) start copying the next file in queue. Once the queue's empty
    or the early-abort flag is set, all ten threads will terminate when
    they finish their current transfers.
    Yes, my example was simplified, but I do that :)

    Laurent
  • Terry Reedy at Jun 19, 2011 at 4:38 pm

    On 6/19/2011 11:39 AM, Laurent Claessens wrote:

    In the same time I've a thread that read the list and perform the
    operations:

    def run():
    while task_list :
    task = task_list[0]
    task_list.remove(task)
    task.start()
    Popping task off the end of the list is more efficient:

    while task_list:
    task_list.pop().start()

    or if the list is static

    for task in task_list:
    task.start()


    --
    Terry Jan Reedy
  • Laurent Claessens at Jun 19, 2011 at 4:52 pm

    Popping task off the end of the list is more efficient:
    while task_list:
    task_list.pop().start()
    That's cool. In my case it's better to do
    task_list.pop(0).start

    in order to pop the first element.
    or if the list is static
    No, my list is dynamic and is feeded by an other thread (which also has
    to be closed).

    Finally, I solved the problem by using a global flag looked at each
    iteration of the loop. A break is done if the global flag reports an error.

    That was suggested by Alain Ketterlin from the French speaking python's
    usenet and Chris Angelico here.

    Thanks all
    Laurent
  • Lie Ryan at Jun 19, 2011 at 5:04 pm

    On 06/20/11 02:52, Laurent Claessens wrote:
    Popping task off the end of the list is more efficient:
    while task_list:
    task_list.pop().start()
    That's cool. In my case it's better to do
    task_list.pop(0).start

    in order to pop the first element.
    then you really wanted a queue instead of a list.

    There is a thread-safe `queue` module in the standard library.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 19, '11 at 2:42p
activeJun 19, '11 at 5:04p
posts10
users4
websitepython.org

People

Translate

site design / logo © 2022 Grokbase