FAQ
How would you approach the following?
In a multithreaded realtime data acquisition system (all python v2.4),
after hours of running without a snag, without warning python hangs at
once without leaving any error message or error traceback at all. After
the incident, the OS itself (linux) appears to remain fully functional.
Thank you!
malv

Search Discussions

  • Peter Hansen at Dec 29, 2005 at 1:39 pm

    malv wrote:
    How would you approach the following?
    In a multithreaded realtime data acquisition system (all python v2.4),
    after hours of running without a snag, without warning python hangs at
    once without leaving any error message or error traceback at all. After
    the incident, the OS itself (linux) appears to remain fully functional.
    1. Define "hangs" more precisely. It's rare that an app can hang
    without any behaviour that can tell you something more about it. For
    example, can you Ctrl-C out of it? Does it have sockets open still? Is
    it using any CPU time? etc... While you're at it, define "realtime"
    too, since many people use it incorrectly and it might mean something
    different to you than it does to everyone else reading it...

    2. Use logging. "import logging" and proceed from there... without
    logging, you have no idea where the problem occurred, and are going to
    spend lots of time guessing where the cause is.

    3. Make sure you are using Queues exclusively for inter-thread
    communication.

    4. Consider whether your external source of data (assuming there is one)
    is the cause. How are you interfacing to it? Serial port? ctypes?
    Something SWIGged for Python? A true "hang" is much more likely in an
    external package than in Python code.

    5. Tell us more about the platform. Which Linux (including version),
    does it have a GUI, what's the basic architecture of the app as it
    relates to the threading and data acquisition stuff, etc. (Of course,
    the problem might not come from threads or data acquisition, but until
    we learn more who could say?)

    -Peter
  • Malv at Dec 29, 2005 at 5:47 pm
    Hi Peter,
    Thank you for your extensive reply.
    In the meantime, I solved the problem. I had started out to trace the
    path of the randomly arriving external data through the different
    threaded processing stages. After one of those 'hangs', with some luck,
    I was able to spot the cause of the trouble: a piece of old code that
    wasn't supposed to be out there! Really dumb.
    FYI, to my surprise, Python really quit. No more CPU time, no message,
    nothing. I can't recall having seen this before. You are quite right
    about queues. I use these extensively and carefully. This kind of
    enabled me to trace the path of the externally triggered events without
    too much of a hassle.
    Thanks again,
    malv
  • Peter Hansen at Dec 29, 2005 at 6:25 pm

    malv wrote:
    After one of those 'hangs', with some luck,
    I was able to spot the cause of the trouble: a piece of old code that
    wasn't supposed to be out there! Really dumb.
    FYI, to my surprise, Python really quit. No more CPU time, no message,
    nothing. I can't recall having seen this before.
    If you post the relevant "old code" it's very likely someone could point
    out exactly why it failed, and how it wasn't *Python* that "quit", but
    merely something in your code that was blocking. (In other words,
    Python probably did exactly what you told it to do.)

    In general, Python does not "quit". It's very robust. When you get a
    failure like that, always suspect your own code structure.

    -Peter
  • Malv at Dec 29, 2005 at 8:45 pm
    What happened in fact was that a graphics plotting function for
    monitoring got called twice frorm a different thread, due to an 'old'
    instruction inadvertently left. Due to the quasi random nature of the
    incoming data streams, this happened rather very sparingly. As the
    plotting itself involves quite a bit of code, there are probably
    several if not many reasons why problems could arise. It was never my
    intention to reenter this function and make it thread safe. Problem
    solved.

    I was indeed a surprise for me to see Python quit this way.
    Thanks again for your commentary.
    malv

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedDec 29, '05 at 8:08a
activeDec 29, '05 at 8:45p
posts5
users2
websitepython.org

2 users in discussion

Malv: 3 posts Peter Hansen: 2 posts

People

Translate

site design / logo © 2022 Grokbase