FAQ
I am seeing very perplexing segfaults and standard allocation exceptions in my native code (.so files passed to the distributed cace) which is called via JNI from the map task. This code runs perfectly fine (on the same data) outside Hadoop. Even when run in a Hadoop standalone mode (no cluster), it still segfaults. The memory footprint is quite small and inspection at run time reveals there is plenty of memory left, yet I get segfaults and exceptions.

I'm starting to wonder if this is a thread issue.

The native code is not *specifically* thread safe (not compiled with pthreads or anything like that).

However, it is also not run in any concurrent fashion except w.r.t. to the JVM itself. For example, my map task doesn't make parallel calls through JNI to the native code on concurrent threads at the Java level, nor does the native code itself spawn any threads (like I said, it isn't even compiled with pthreads).

However, there are clearly other "threads" of execution. For example, the JVM itself is running, including whatever supplemental threads the JVM involves (the garbage collector?). In addition, my Java mapper is running two Java threads at the time of the native code. One calls the native code and effectively blocks until the native code returns through JNI. The other just spins and sends reports and statuses to the job tracker at regular intervals to prevent the task from being killed, but it doesn't do anything else particularly memory-related, certainly no JNI/native calls, it's very basic, just sleep 'n report, sleep 'n report.

So, the question is, in the scenario I have described, is there any reason to suspect that the cause of my problems is some sort of thread trampling between the native code and something else in the surrounding environment (the JVM or something like that), especially in the context of the surrounding Hadoop infrastructure? It doesn't really make any sense to me, but I'm running out of ideas.

I've experimented with "mapred.child.java.opts" and "mapred.child.ulimit" but nothing really seems to have any effect on the frequency of these errors.

I'm quite out of ideas. These segfaults and standard allocation exceptions (in the face of plenty of free memory) have basically brought my work to a halt and I just don't know what to do anymore.

Thanks.

________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com

"And what if we picked the wrong religion? Every week, we're just making God
madder and madder!"
-- Homer Simpson
________________________________________________________________________________

Search Discussions

  • Greg Roelofs at Jan 28, 2011 at 9:38 pm

    Keith Wiley wrote:

    However, it is also not run in any concurrent fashion except
    w.r.t. to the JVM itself. For example, my map task doesn't
    make parallel calls through JNI to the native code on concurrent
    threads at the Java level, nor does the native code itself spawn
    any threads (like I said, it isn't even compiled with pthreads).
    Is speculative execution enabled?
    So, the question is, in the scenario I have described, is there
    any reason to suspect that the cause of my problems is some
    sort of thread trampling between the native code and something
    else in the surrounding environment (the JVM or something like
    that), especially in the context of the surrounding Hadoop
    infrastructure? It doesn't really make any sense to me, but
    I'm running out of ideas.
    I don't see any obvious possibilities except speculative execution, and
    even that would depend on how the shared library was written. Does it
    contain any global or static variables? If so, it's almost certainly
    not thread-safe (unless, say, a global variable were basically write-
    only and advisory only, e.g., used only in an error message or a summary
    message at the end).

    No other ideas.

    Greg
  • Keith Wiley at Jan 28, 2011 at 10:52 pm

    On Jan 28, 2011, at 13:37 , Greg Roelofs wrote:

    Keith Wiley wrote:
    However, it is also not run in any concurrent fashion except
    w.r.t. to the JVM itself. For example, my map task doesn't
    make parallel calls through JNI to the native code on concurrent
    threads at the Java level, nor does the native code itself spawn
    any threads (like I said, it isn't even compiled with pthreads).
    Is speculative execution enabled?
    I suppose I hadn't specifically disabled that, but two points:

    (1) Speculative execution would occur on a completely different node, so there definitely isn't any thread cross-talk (in memory). So long as they don't rely on reading/writing temp files from HDFS I don't see how they could have any effect on one another.

    (2) I am also getting seg faults when I run in noncluster standalone mode, which is a single nonspeculated thread......I presume.

    Can you explain your thoughts on speculative execution w.r.t. the problems I'm having?
    So, the question is, in the scenario I have described, is there
    any reason to suspect that the cause of my problems is some
    sort of thread trampling between the native code and something
    else in the surrounding environment (the JVM or something like
    that), especially in the context of the surrounding Hadoop
    infrastructure? It doesn't really make any sense to me, but
    I'm running out of ideas.
    I don't see any obvious possibilities except speculative execution, and
    even that would depend on how the shared library was written. Does it
    contain any global or static variables? If so, it's almost certainly
    not thread-safe (unless, say, a global variable were basically write-
    only and advisory only, e.g., used only in an error message or a summary
    message at the end).

    Yes, not thread safe, but what difference could that make if I don't use the library in a multi-threaded fashion. One map task, one node, one Java thread calling JNI and using the native code? How do thread safety issues factor into this? I admit, it's my theory that threads might be involved somehow, but I don't understand how, I'm just shooting in the dark since I can't solve this problem any other way yet.

    Thanks for the input. Can you tell me what you're thinking w.r.t. speculative execution?

    I'll try it without, but I don't see how it could alter the standalone behavior.

    Cheers!

    ________________________________________________________________________________
    Keith Wiley kwiley@keithwiley.com www.keithwiley.com

    "Luminous beings are we, not this crude matter."
    -- Yoda
    ________________________________________________________________________________
  • Greg Roelofs at Jan 28, 2011 at 11:51 pm

    Keith Wiley wrote:

    (1) Speculative execution would occur on a completely different
    node, so there definitely isn't any thread cross-talk (in memory).
    So long as they don't rely on reading/writing temp files from
    HDFS I don't see how they could have any effect on one another.
    Good point.
    (2) I am also getting seg faults when I run in noncluster
    standalone mode, which is a single nonspeculated thread......I
    presume.
    That's the same as "pseudo-distributed mode"?
    Can you explain your thoughts on speculative execution w.r.t. the
    problems I'm having?
    Thoughts? You expect me to have thoughts, too??

    :-)

    I had not fully thought through the spec ex idea; it was the only thing
    I could think of that might put two (otherwise independent) JNI-using tasks
    onto the same node. But as you point out above, it wouldn't...

    Does your .so depend on any other potentially thread-unsafe .so that other
    (non-Hadoop) processes might be using? System libraries like zlib are safe
    (else they wouldn't make very good system libraries), but maybe some other
    research library or something? (That's a long shot, but I'm pretty much
    grasping at straws here.)
    Yes, not thread safe, but what difference could that make if I
    don't use the library in a multi-threaded fashion. One map task,
    one node, one Java thread calling JNI and using the native code?
    How do thread safety issues factor into this? I admit, it's
    my theory that threads might be involved somehow, but I don't
    understand how, I'm just shooting in the dark since I can't
    solve this problem any other way yet.
    Since you can reproduce it in standalone mode, can you enable core dumps
    so you can see the backtrace of the code that segfaults? Knowing what
    specifically broke and how it got there is always a big help.

    Btw, keep in mind that there are memory-related bugs that don't show up
    until there's something big in memory that pushes the code in question
    up into a region with different data patterns in it (most frequently zero
    vs. non-zero, but others are possible). IOW, maybe the code is dependent
    on uninitialized memory, but you were getting lucky when you ran it outside
    of Hadoop. Have you run it through valgrind or Purify or similar?

    Greg
  • Todd Lipcon at Jan 29, 2011 at 12:25 am
    JNI may also work fine with no GC running, but then work badly when GC kicks
    in at a bad time. For example, if you grab a pointer to a String or array,
    you need to essentially lock them so the GC doesn't relocate the objects
    underneath you. For example, maybe you're releasing one of these references
    and then continuing to use it?

    -Todd
    On Fri, Jan 28, 2011 at 3:50 PM, Greg Roelofs wrote:

    Keith Wiley wrote:
    (1) Speculative execution would occur on a completely different
    node, so there definitely isn't any thread cross-talk (in memory).
    So long as they don't rely on reading/writing temp files from
    HDFS I don't see how they could have any effect on one another.
    Good point.
    (2) I am also getting seg faults when I run in noncluster
    standalone mode, which is a single nonspeculated thread......I
    presume.
    That's the same as "pseudo-distributed mode"?
    Can you explain your thoughts on speculative execution w.r.t. the
    problems I'm having?
    Thoughts? You expect me to have thoughts, too??

    :-)

    I had not fully thought through the spec ex idea; it was the only thing
    I could think of that might put two (otherwise independent) JNI-using tasks
    onto the same node. But as you point out above, it wouldn't...

    Does your .so depend on any other potentially thread-unsafe .so that other
    (non-Hadoop) processes might be using? System libraries like zlib are safe
    (else they wouldn't make very good system libraries), but maybe some other
    research library or something? (That's a long shot, but I'm pretty much
    grasping at straws here.)
    Yes, not thread safe, but what difference could that make if I
    don't use the library in a multi-threaded fashion. One map task,
    one node, one Java thread calling JNI and using the native code?
    How do thread safety issues factor into this? I admit, it's
    my theory that threads might be involved somehow, but I don't
    understand how, I'm just shooting in the dark since I can't
    solve this problem any other way yet.
    Since you can reproduce it in standalone mode, can you enable core dumps
    so you can see the backtrace of the code that segfaults? Knowing what
    specifically broke and how it got there is always a big help.

    Btw, keep in mind that there are memory-related bugs that don't show up
    until there's something big in memory that pushes the code in question
    up into a region with different data patterns in it (most frequently zero
    vs. non-zero, but others are possible). IOW, maybe the code is dependent
    on uninitialized memory, but you were getting lucky when you ran it outside
    of Hadoop. Have you run it through valgrind or Purify or similar?

    Greg


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Keith Wiley at Jan 29, 2011 at 1:50 am
    Hmmm, none of my native code is using JNI objects, so the memory in question should have no relationship to the Java heap or any other aspect of the Java world, but I admit, I'm unclear on who Java, JNI, and native libraries divide memory between one another or how they trade responsibility for allocating and deallocating memory.

    I'll consider it, but since I'm not talking about JNI objects, I don't that can be it. Do you think I'm misunderstanding something?
    On Jan 28, 2011, at 16:24 , Todd Lipcon wrote:

    JNI may also work fine with no GC running, but then work badly when GC kicks
    in at a bad time. For example, if you grab a pointer to a String or array,
    you need to essentially lock them so the GC doesn't relocate the objects
    underneath you. For example, maybe you're releasing one of these references
    and then continuing to use it?

    -Todd

    ________________________________________________________________________________
    Keith Wiley kwiley@keithwiley.com www.keithwiley.com

    "Luminous beings are we, not this crude matter."
    -- Yoda
    ________________________________________________________________________________
  • Greg Roelofs at Jan 29, 2011 at 2:00 am

    Todd Lipcon wrote:

    JNI may also work fine with no GC running, but then work badly when GC kicks
    in at a bad time. For example, if you grab a pointer to a String or array,
    you need to essentially lock them so the GC doesn't relocate the objects
    underneath you. For example, maybe you're releasing one of these references
    and then continuing to use it?
    Excellent point! And one I should have remembered, too.

    Keith, take a look at the native ZlibCompressor interface to see one way of
    handling this. (It pins a number of buffers in memory and puts them into a
    pool, if I remember Chris Douglas's explanation correctly. I didn't need to
    dive into that level of detail myself for what I was working on, so I never
    touched the buffer code and might not be remembering it entirely accurately,
    but that's the gist, anyway.)

    On 0.20.x or 0.22/trunk:
    hadoop-common/src/native/src/org/apache/hadoop/io/compress
    hadoop-common/src/{core or java}/org/apache/hadoop/io/compress

    Greg
  • Keith Wiley at Jan 29, 2011 at 1:48 am

    On Jan 28, 2011, at 15:50 , Greg Roelofs wrote:

    Does your .so depend on any other potentially thread-unsafe .so that other
    (non-Hadoop) processes might be using? System libraries like zlib are safe
    (else they wouldn't make very good system libraries), but maybe some other
    research library or something? (That's a long shot, but I'm pretty much
    grasping at straws here.)
    Yeah, I dunno. It's a very complicated system that hits all kinds of popular conventional libraries: boost, eigen, countless other things. I doubt of it is being access however. This is a dedicated cluster so if my task is the only one running, then it's only concurrent with the OS itself (and the JVM and Hadoop).
    Yes, not thread safe, but what difference could that make if I
    don't use the library in a multi-threaded fashion. One map task,
    one node, one Java thread calling JNI and using the native code?
    How do thread safety issues factor into this? I admit, it's
    my theory that threads might be involved somehow, but I don't
    understand how, I'm just shooting in the dark since I can't
    solve this problem any other way yet.
    Since you can reproduce it in standalone mode, can you enable core dumps
    so you can see the backtrace of the code that segfaults? Knowing what
    specifically broke and how it got there is always a big help.
    Yep, I've got core dumps and I've run them through gdb. I know that the code often dies very deep inside ostensibly standard libraries, like eigen for example...which leads me to believe the memory corruption happened long before the code reached that point.
    Btw, keep in mind that there are memory-related bugs that don't show up
    until there's something big in memory that pushes the code in question
    up into a region with different data patterns in it (most frequently zero
    vs. non-zero, but others are possible). IOW, maybe the code is dependent
    on uninitialized memory, but you were getting lucky when you ran it outside
    of Hadoop. Have you run it through valgrind or Purify or similar?

    Valgrind has turned out to be almost useless. It can't "reach" through the JVM through JNI to the .so code. If I don't tell valgrind to following children, it obviously produces no relevant output, but if I do tell it to follow children, it can't successfully launch a VM to run Java in:

    Error occurred during initialization of VM
    Unknown x64 processor: SSE2 not supported

    Sigh...any thoughts on running Valgrind on Hadoop->JVM->JNI->native code?

    Thanks.

    ________________________________________________________________________________
    Keith Wiley kwiley@keithwiley.com www.keithwiley.com

    "Yet mark his perfect self-contentment, and hence learn his lesson, that to be
    self-contented is to be vile and ignorant, and that to aspire is better than to
    be blindly and impotently happy."
    -- Edwin A. Abbott, Flatland
    ________________________________________________________________________________
  • Greg Roelofs at Jan 29, 2011 at 2:44 am

    I wrote:

    Btw, keep in mind that there are memory-related bugs that don't show up
    until there's something big in memory that pushes the code in question
    up into a region with different data patterns in it (most frequently zero
    vs. non-zero, but others are possible). IOW, maybe the code is dependent
    on uninitialized memory, but you were getting lucky when you ran it outside
    of Hadoop. Have you run it through valgrind or Purify or similar?
    Keith Wiley wrote:
    Valgrind has turned out to be almost useless. It can't "reach"
    through the JVM through JNI to the .so code. If I don't
    tell valgrind to following children, it obviously produces
    no relevant output, but if I do tell it to follow children,
    it can't successfully launch a VM to run Java in:
    Error occurred during initialization of VM
    Unknown x64 processor: SSE2 not supported
    Sigh...any thoughts on running Valgrind on Hadoop->JVM->JNI->native code?
    I actually meant something simpler: if we posit that the bug is actually
    in the library code but isn't always triggering a segfault due to random
    memory conditions (i.e., "getting lucky"), then running valgrind on it in
    a non-Java context (i.e., what you said "runs perfectly fine outside Hadoop")
    should detect such bug(s).

    If that shows nothing, and you're not passing buffers across the JNI boundary
    (=> possible GC issues, perhaps subtle ones?), then I'm out of ideas. Again.
    Sorry. :-/

    Greg
  • Allen Wittenauer at Jan 29, 2011 at 6:16 am

    On Jan 28, 2011, at 5:47 PM, Keith Wiley wrote:
    On Jan 28, 2011, at 15:50 , Greg Roelofs wrote:

    Does your .so depend on any other potentially thread-unsafe .so that other
    (non-Hadoop) processes might be using? System libraries like zlib are safe
    (else they wouldn't make very good system libraries), but maybe some other
    research library or something? (That's a long shot, but I'm pretty much
    grasping at straws here.)
    Yeah, I dunno. It's a very complicated system that hits all kinds of popular conventional libraries: boost, eigen, countless other things. I doubt of it is being access however. This is a dedicated cluster so if my task is the only one running, then it's only concurrent with the OS itself (and the JVM and Hadoop).
    By chance, do you have jvm reuse turned on?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 28, '11 at 1:46a
activeJan 29, '11 at 6:16a
posts10
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase