FAQ
Hi,

the Python threading module does not seem to provide a means to cancel
a running thread. There are many discussions on the web dealing with
this issue and many solutions are offered, but none of them seems to
be applicable to my situation, which is as follows:

I have a C library which does some very computationally intensive
stuff. Some functions in this library might run for a long time, and
the code in the library is optimized for speed. The library is used
in C programs and also from a PyGTK GUI program. In the GUI, you can
choose the computation parameters and than start the computation. It
is run in a separate thread, which calls a function in the C library
via ctypes.

Now it should be possible to cancel the computation from the GUI. In
the C programs, I can just kill or cancel the thread (there is no
cleanup to be done -- the thread does not use any system resources
apart from CPU time). In Python, this is not possible. The solutions
for this problem I found on the web usually recommend to have the
thread regularly check some variable and exit if this variable
indicates to do so. This would have to be included in the C library
and would include quite a bit of code refactoring. The check cannot
be included in the inner loop because it would lead to significant
performance loss. Furthermore, I just do not want to mess up an
elegant and efficient library design which works perfect when used
from C just to make up for shortcomings in Python.

So do I really have to refactor my C library just because Python
Thread objects lack a cancel method? Is there really no other way?
And why on earth doesn't that cancel method exist? There *are* good
reasons to cancel a thread, just google for "terminate a Python
thread" for tons of examples.

I would be grateful for any suggestions.

Greetings from Germany and have a nice day,
Sven

Search Discussions

  • Carl Banks at Nov 8, 2009 at 3:27 am

    On Nov 7, 6:04?pm, Sven Marnach wrote:
    So do I really have to refactor my C library just because Python
    Thread objects lack a cancel method? ?Is there really no other way?
    It doesn't sound like the thread is communicating with the process
    much. Therefore:

    1. Run the C code in a separate process, or
    2. Create the thread from a C extension, maybe even straight from
    ctypes, and kill it from C or ctypes.

    And why on earth doesn't that cancel method exist? There *are* good
    reasons to cancel a thread, just google for "terminate a Python
    thread" for tons of examples.
    Arguing that there are good reasons to allow killing threads isn't
    going to get you very far. The language developers already know
    killing a thread is useful, yet the disallowed it anyway. The
    drawbacks were judged too severe (it makes enforcing invariants pretty
    much impossible).


    Carl Banks
  • John Nagle at Nov 8, 2009 at 5:14 am

    Carl Banks wrote:
    Arguing that there are good reasons to allow killing threads isn't
    going to get you very far. The language developers already know
    killing a thread is useful, yet the disallowed it anyway. The
    drawbacks were judged too severe (it makes enforcing invariants pretty
    much impossible).
    While outright thread cancellation is generally unsafe, it would
    be useful if there was a way to force another thread to unblock from
    a wait condition, like a blocking read, with an exception. This is,
    among other things, why control-C won't terminate some threaded programs.

    Python 2.6 and 3 have some steps in this direction. There's
    "signal.set_wakeup_fd(fd)", and "siginterrupt()". But the real
    problem, that signals are associated only with the first thread, hasn't
    been addressed.

    Question: if "signal.set_wakeup_fd(fd)" is used, and the thread
    waiting on "fd" is not the main thread, will a signal cause the
    waiting thread to get a read completion? Or is this another "first
    thread only" thing"?

    John Nagle
  • Sven at Nov 8, 2009 at 12:40 pm

    On Nov 8, 4:27?am, Carl Banks wrote:
    It doesn't sound like the thread is communicating with the process
    much. ?Therefore:
    There is quite a bit of communication -- the computation results are
    visulized while they are generated.
    1. Run the C code in a separate process, or
    2. Create the thread from a C extension, maybe even straight from
    ctypes, and kill it from C or ctypes.
    The second option is a good idea. Thanks a lot, Carl!
    And why on earth doesn't that cancel method exist? There *are* good
    reasons to cancel a thread, just google for "terminate a Python
    thread" for tons of examples.
    Arguing that there are good reasons to allow killing threads isn't
    going to get you very far. ?The language developers already know
    killing a thread is useful, yet the disallowed it anyway. ?The
    drawbacks were judged too severe (it makes enforcing invariants pretty
    much impossible).
    I really don't get that. If the reason would be that it is too much
    work to
    implement, then I could accept it. But saying: We know it is useful,
    but we
    won't allow to do it, just does not seem reasonable. Thread
    cancellation
    might be generally unsafe, but there are cases when it is safe. It
    should be
    up to the user to decide it. There are many things that do harm if
    you don't
    use them correctly, and of course it would be a bad idea to remove all
    of
    them from Python.

    Well, I won't complain too much. At least some volunteers created
    that
    great language and gave it away for free :)

    Thanks again for your suggestion.

    Cheers,
    Sven
  • Exarkun at Nov 8, 2009 at 1:50 pm

    On 12:40 pm, sven at uni-hd.de wrote:
    On Nov 8, 4:27?am, Carl Banks wrote:
    It doesn't sound like the thread is communicating with the process
    much. ?Therefore:
    There is quite a bit of communication -- the computation results are
    visulized while they are generated.
    I'm curious how this visualization works, since earlier you said
    something to the affect that there were no shared resources. If you
    kill a thread and it had opened a window and was drawing on it, with
    most toolkits, you'll end up with a window stuck in your screen, won't
    you?
    [snip]

    I really don't get that. If the reason would be that it is too much
    work to
    implement, then I could accept it. But saying: We know it is useful,
    but we
    won't allow to do it, just does not seem reasonable. Thread
    cancellation
    might be generally unsafe, but there are cases when it is safe. It
    should be
    up to the user to decide it. There are many things that do harm if
    you don't
    use them correctly, and of course it would be a bad idea to remove all
    of
    them from Python.
    The CPython philosophy sort of follows the guideline that you should be
    allowed to do bad stuff if you want, except when that bad stuff would
    crash the interpreter (clearly ctypes is an exception to this rule of
    thumb). I think this is the argument that has been applied in
    opposition to adding thread termination in the past, though I don't
    remember for sure.

    Jean-Paul
  • Antoine Pitrou at Nov 8, 2009 at 1:52 pm

    Le Sun, 08 Nov 2009 04:40:26 -0800, sven a ?crit?:
    I really don't get that. If the reason would be that it is too much
    work to
    implement, then I could accept it.
    It would probably be a lot of work and even then it would still be unsafe.

    Read for example:
    http://msdn.microsoft.com/en-us/library/ms686717%28VS.85%29.aspx

    ? TerminateThread is a dangerous function that should only be used in the
    most extreme cases. You should call TerminateThread only if you know
    exactly what the target thread is doing, and you control all of the code
    that the target thread could possibly be running at the time of the
    termination. For example, TerminateThread can result in the following
    problems:

    * If the target thread owns a critical section, the critical section
    will not be released.
    * If the target thread is allocating memory from the heap, the heap
    lock will not be released.
    * If the target thread is executing certain kernel32 calls when it is
    terminated, the kernel32 state for the thread's process could be
    inconsistent.
    * If the target thread is manipulating the global state of a shared
    DLL, the state of the DLL could be destroyed, affecting other users of
    the DLL. ?
  • John Nagle at Nov 8, 2009 at 6:39 pm

    Antoine Pitrou wrote:
    Le Sun, 08 Nov 2009 04:40:26 -0800, sven a ?crit :
    I really don't get that. If the reason would be that it is too much
    work to
    implement, then I could accept it.
    It would probably be a lot of work and even then it would still be unsafe.

    Read for example:
    http://msdn.microsoft.com/en-us/library/ms686717%28VS.85%29.aspx
    I'd argue against general thread cancellation. Inter-thread
    signals, though, have safety problems no worse than the first-thread
    only signals we have now. You're allowed to raise an exception
    in a signal handler, which is effectively thread cancellation.

    So right now, you can kill the first thread from another thread.

    John Nagle
  • Antoine Pitrou at Nov 8, 2009 at 7:34 pm

    John Nagle <nagle <at> animats.com> writes:

    I'd argue against general thread cancellation. Inter-thread
    signals, though, have safety problems no worse than the first-thread
    only signals we have now. You're allowed to raise an exception
    in a signal handler, which is effectively thread cancellation.
    Can you give an example of such "cancellation"?
    In any case, this would be a side-effect of the current implementation, not
    officially supported behaviour.

    Regards

    Antoine.
  • John Nagle at Nov 9, 2009 at 5:04 am

    Antoine Pitrou wrote:
    John Nagle <nagle <at> animats.com> writes:
    I'd argue against general thread cancellation. Inter-thread
    signals, though, have safety problems no worse than the first-thread
    only signals we have now. You're allowed to raise an exception
    in a signal handler, which is effectively thread cancellation.
    Can you give an example of such "cancellation"?
    In any case, this would be a side-effect of the current implementation, not
    officially supported behaviour.
    It's not only documented behavior, it's an example in the official
    documentation. See

    http://docs.python.org/library/signal.html#example

    where an exception is raised in a signal handler.

    John Nagle
  • Antoine Pitrou at Nov 9, 2009 at 12:20 pm

    Le Sun, 08 Nov 2009 21:04:06 -0800, John Nagle a ?crit?:
    Antoine Pitrou wrote:
    John Nagle <nagle <at> animats.com> writes:
    I'd argue against general thread cancellation. Inter-thread
    signals, though, have safety problems no worse than the first-thread
    only signals we have now. You're allowed to raise an exception in a
    signal handler, which is effectively thread cancellation.
    Can you give an example of such "cancellation"? In any case, this would
    be a side-effect of the current implementation, not officially
    supported behaviour.
    It's not only documented behavior, it's an example in the official
    documentation. See

    http://docs.python.org/library/signal.html#example
    Well, the only supported behaviour is to send signals to the main thread.
    Besides, it doesn't "cancel" the thread, it just raises an exception in
    it, which can be caught and silenced. Just try the following:


    import signal, time

    def handler(signum, frame):
    print 'Signal handler called with signal', signum
    raise IOError

    # Set the signal handler and a 5-second alarm
    signal.signal(signal.SIGALRM, handler)
    signal.alarm(2)

    try:
    time.sleep(10)
    except IOError:
    print "got IOError!"
  • Sven at Nov 8, 2009 at 11:59 pm

    On Nov 8, 2:50?pm, exar... at twistedmatrix.com wrote:
    I'm curious how this visualization works, since earlier you said
    something to the affect that there were no shared resources. ?If you
    kill a thread and it had opened a window and was drawing on it, with
    most toolkits, you'll end up with a window stuck in your screen, won't
    you?
    The Python code passes an array to the C library which in the end
    contains the computation results. This array only contains some kind
    of counts which are incremented during the computation. The PyGTK GUI
    simply visualizes the current state of this array in regular
    intervals. The C library does not do any GUI stuff. Cancelling the
    thread really would not do any harm -- the thread only allocates
    memory on the stack, and the stack will vanish with the thread.
    The CPython philosophy sort of follows the guideline that you should be
    allowed to do bad stuff if you want, except when that bad stuff would
    crash the interpreter (clearly ctypes is an exception to this rule of
    thumb).
    Well, I am really glad about that exception. And following the hint
    of Carl, I will use it to call pthread_cancel or pthread_kill directly
    from the library. (Most certainly I also have to use ctypes to create
    my threads to be able to do this.)

    Cheers,
    Sven

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedNov 8, '09 at 2:04a
activeNov 9, '09 at 12:20p
posts11
users6
websitepython.org

People

Translate

site design / logo © 2022 Grokbase