FAQ
We are having difficulties running a Hadoop program making calls to external libraries - but this occurs only when we run the program on our cluster and not from within Eclipse where we are apparently running in Hadoop's standalone mode. This program invokes the Open Computer Vision libraries (OpenCV and JavaCV). (I don't think there is a problem with our cluster - we've run many Hadoop jobs on it without difficulty.)

1. I normally use Eclipse to create jar files for our Hadoop programs but I inadvertently hit the "run as Java application" button and the program ran fine, reading the input file from the eclipse workspace rather than HDFS and writing the output file to the same place. Hadoop's output appears below. (This occurred on the master Hadoop server.)

2. I then "exported" from Eclipse a "runnable jar" which "extracted required libraries" into the generated jar - presumably producing a jar file that incorporated all the required library functions. (The plain jar file for this program is 17 kB while the runnable jar is 30MB.) When I try to run this on my Hadoop cluster (including my master and slave servers) the program reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory". Now, in addition to this library being incorporated inside the runnable jar file it is also present on each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same libraries (to give Hadoop 2 shots at finding them). These include:
...
libopencv_highgui_pch_dephelp.a
libopencv_highgui.so
libopencv_highgui.so.2.2
libopencv_highgui.so.2.2.0
...

When I poke around inside the runnable jar I find javacv_linux-x86_64.jar which contains:
com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

3. I then tried adding the following to mapred-site.xml as suggested in Patch 2838 that's supposed to be included in hadoop 0.21 https://issues.apache.org/jira/browse/HADOOP-2838
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
</property>
The log is included at the bottom of this email with Hadoop now complaining about a different missing library with an out-of-memory error.

Does anyone have any ideas as to what is going wrong here? Any help would be appreciated. Thanks.

Alan


BTW: Each of our servers has 4 hard drives and many of the errors below refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and thus perhaps not a good place for Hadoop to be looking for a library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being read is 400 KB - so I hope I am not running out of memory.


1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
Running Face Program
11/03/04 12:44:10 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : 1
11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token storage with # keys=0
11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0
11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 10485760
11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
11/03/04 12:44:16 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = 104857600
11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214324(104857296); length = 73/6553600
11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > sort
11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0
11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1481 bytes
11/03/04 12:44:18 INFO mapred.LocalJobRunner:
11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
11/03/04 12:44:18 INFO mapred.LocalJobRunner:
11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
FileInputFormatCounters
BYTES_READ=411439
FileSystemCounters
FILE_BYTES_READ=825005
FILE_BYTES_WRITTEN=127557
Map-Reduce Framework
Combine input records=0
Combine output records=0
Failed Shuffles=0
GC time elapsed (ms)=88
Map input records=20
Map output bytes=1454
Map output records=19
Merged Map outputs=0
Reduce input groups=19
Reduce input records=19
Reduce output records=19
Reduce shuffle bytes=0
Shuffled Maps =0
Spilled Records=38
SPLIT_RAW_BYTES=127
0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 ],
...
0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ 210, 119, 228 ],
Job Finished in 8.679 seconds
2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

ngc@hadoop1:~$ cd hadoop-0.21.0/
ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
11/03/04 12:58:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Found 1 items
-rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 /user/ngc/Images2/JPGSequenceFile.001
ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
Running Face Program
11/03/04 12:59:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : 1
11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
11/03/04 13:00:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_0, Status : FAILED
Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:08 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_0, Status : FAILED
Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_0, Status : FAILED
Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: failed to map segment from shared object: Cannot allocate memory
11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_0, Status : FAILED
Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_0, Status : FAILED
Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: failed to map segment from shared object: Cannot allocate memory
attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
11/03/04 13:00:11 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_0, Status : FAILED
11/03/04 13:00:12 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_0, Status : FAILED
11/03/04 13:00:13 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_0, Status : FAILED
11/03/04 13:00:14 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_0, Status : FAILED
11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
11/03/04 13:00:18 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_0, Status : FAILED
Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
11/03/04 13:00:19 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_0, Status : FAILED
11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
11/03/04 13:00:20 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_0, Status : FAILED
11/03/04 13:00:21 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_0, Status : FAILED
Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:22 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_0, Status : FAILED
11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
11/03/04 13:00:23 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_0, Status : FAILED
11/03/04 13:00:24 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_0, Status : FAILED
11/03/04 13:00:25 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_0, Status : FAILED
11/03/04 13:00:36 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_1, Status : FAILED
Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:39 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_1, Status : FAILED
Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
11/03/04 13:00:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_1, Status : FAILED
11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
11/03/04 13:00:43 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_1, Status : FAILED
Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
11/03/04 13:00:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_1, Status : FAILED
11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
11/03/04 13:00:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_1, Status : FAILED
11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_1, Status : FAILED
11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000087_0, Status : FAILED
11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_2, Status : FAILED
Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
11/03/04 13:00:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_1, Status : FAILED
Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:00:52 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_1, Status : FAILED
11/03/04 13:00:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_1, Status : FAILED
11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
11/03/04 13:00:55 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_1, Status : FAILED
11/03/04 13:00:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000091_0, Status : FAILED
11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000096_0, Status : FAILED
11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_1, Status : FAILED
Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:01:01 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_1, Status : FAILED
11/03/04 13:01:03 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_1, Status : FAILED
11/03/04 13:01:04 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_1, Status : FAILED
11/03/04 13:01:06 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_1, Status : FAILED
11/03/04 13:01:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_1, Status : FAILED
11/03/04 13:01:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_2, Status : FAILED
Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
11/03/04 13:01:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_2, Status : FAILED
11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
FileInputFormatCounters
BYTES_READ=0
FileSystemCounters
FILE_BYTES_WRITTEN=3040
HDFS_BYTES_READ=1048281
Job Counters
Data-local map tasks=51
Total time spent by all maps waiting after reserving slots (ms)=0
Total time spent by all reduces waiting after reserving slots (ms)=0
Failed map tasks=1
Rack-local map tasks=86
SLOTS_MILLIS_MAPS=1091359
SLOTS_MILLIS_REDUCES=0
Launched map tasks=137
Launched reduce tasks=2
Map-Reduce Framework
Combine input records=0
Failed Shuffles=0
GC time elapsed (ms)=0
Map input records=0
Map output bytes=0
Map output records=0
Merged Map outputs=0
Spilled Records=0
SPLIT_RAW_BYTES=8960
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
at progs.HannahFace.run(HannahFace.java:137)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at progs.HannahFace.main(HannahFace.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
Running Face Program
11/03/04 15:07:11 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : 1
11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
11/03/04 15:07:38 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000016_0, Status : FAILED
Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: libxcb.so.1: failed to map segment from shared object: Cannot allocate memory
11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
11/03/04 15:07:40 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000001_0, Status : FAILED
Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: libXau.so.6: failed to map segment from shared object: Cannot allocate memory
11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
11/03/04 15:07:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000022_0, Status : FAILED
11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
11/03/04 15:07:44 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000026_0, Status : FAILED
11/03/04 15:07:45 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000011_0, Status : FAILED
11/03/04 15:07:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000034_0, Status : FAILED
11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
11/03/04 15:07:47 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000039_0, Status : FAILED
11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
11/03/04 15:07:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000082_0, Status : FAILED
Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: libxcb-render.so.0: failed to map segment from shared object: Cannot allocate memory
11/03/04 15:07:50 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000006_0, Status : FAILED
11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000044_0, Status : FAILED
11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000048_0, Status : FAILED
11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000052_0, Status : FAILED
11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000076_0, Status : FAILED
Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: libfontconfig.so.1: failed to map segment from shared object: Cannot allocate memory
11/03/04 15:07:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000057_0, Status : FAILED
11/03/04 15:07:56 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000061_0, Status : FAILED
11/03/04 15:07:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000065_0, Status : FAILED
11/03/04 15:07:59 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000069_0, Status : FAILED
attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated

Search Discussions

  • Harsh J at Mar 4, 2011 at 9:26 pm
    I'm only guessing here and might be grossly wrong about my hunch.

    Are you reusing your JVMs across tasks? Could you see if this goes
    away without reuse?

    Would be good if you can monitor your launched Tasks
    (JConsole/VisualVM/etc.) to affirm that there's either a code-based
    memory leak or some other funny issue relating to loading native
    libraries into the JVM.
    On Sat, Mar 5, 2011 at 2:23 AM, Ratner, Alan S (IS) wrote:
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
    Also, do all map tasks fail? Or do some succeed (perhaps the first wave)?

    --
    Harsh J
    www.harshj.com
  • Aaron Kimball at Mar 4, 2011 at 9:27 pm
    I don't know if putting native-code .so files inside a jar works. A
    native-code .so is not "classloaded" in the same way .class files are.

    So the correct .so files probably need to exist in some physical directory
    on the worker machines. You may want to doublecheck that the correct
    directory on the worker machines is identified in the JVM property
    'java.library.path' (instead of / in addition to $LD_LIBRARY_PATH). This can
    be manipulated in the Hadoop configuration setting mapred.child.java.opts
    (include '-Djava.library.path=/path/to/native/libs' in the string there.)

    Also, if you added your .so files to a directory that is already used by the
    tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may need to
    restart the tasktracker instance for it to take effect. (This is true of
    .jar files in the $HADOOP_HOME/lib directory; I don't know if it is true for
    native libs as well.)

    - Aaron

    On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) wrote:

    We are having difficulties running a Hadoop program making calls to
    external libraries - but this occurs only when we run the program on our
    cluster and not from within Eclipse where we are apparently running in
    Hadoop's standalone mode. This program invokes the Open Computer Vision
    libraries (OpenCV and JavaCV). (I don't think there is a problem with our
    cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs
    but I inadvertently hit the "run as Java application" button and the program
    ran fine, reading the input file from the eclipse workspace rather than HDFS
    and writing the output file to the same place. Hadoop's output appears
    below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted
    required libraries" into the generated jar - presumably producing a jar file
    that incorporated all the required library functions. (The plain jar file
    for this program is 17 kB while the runnable jar is 30MB.) When I try to
    run this on my Hadoop cluster (including my master and slave servers) the
    program reports that it is unable to locate "libopencv_highgui.so.2.2:
    cannot open shared object file: No such file or directory". Now, in
    addition to this library being incorporated inside the runnable jar file it
    is also present on each of my servers at
    hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same
    libraries (to give Hadoop 2 shots at finding them). These include:
    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find
    javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested
    in Patch 2838 that's supposed to be included in hadoop 0.21
    https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>

    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now
    complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would
    be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below
    refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for
    HDFS and thus perhaps not a good place for Hadoop to be looking for a
    library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the
    sequence file being read is 400 KB - so I hope I am not running out of
    memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User
    classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process
    : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated.
    Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token
    storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task:
    attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
    library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid
    = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend
    = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task:
    attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to
    file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41,
    68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [
    210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id is deprecated.
    Instead, use mapreduce.task.attempt.id
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21
    /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar
    progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id is deprecated.
    Instead, use mapreduce.task.attempt.id
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process
    : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated.
    Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so:
    failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so:
    failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM
    warning: Exception java.lang.OutOfMemoryError occurred dispatching signal
    SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots
    (ms)=0
    Total time spent by all reduces waiting after reserving
    slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar
    progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id is deprecated.
    Instead, use mapreduce.task.attempt.id
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process
    : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated.
    Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so:
    libxcb.so.1: failed to map segment from shared object: Cannot allocate
    memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so:
    libXau.so.6: failed to map segment from shared object: Cannot allocate
    memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so:
    libxcb-render.so.0: failed to map segment from shared object: Cannot
    allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so:
    libfontconfig.so.1: failed to map segment from shared object: Cannot
    allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM
    warning: Exception java.lang.OutOfMemoryError occurred dispatching signal
    SIGTERM to handler- the VM may need to be forcibly terminated


  • Aaron Kimball at Mar 4, 2011 at 9:30 pm
    Actually, I just misread your email and missed the difference between your
    2nd and 3rd attempts.

    Are you enforcing min/max JVM heap sizes on your tasks? Are you enforcing a
    ulimit (either through your shell configuration, or through Hadoop itself)?
    I don't know where these "cannot allocate memory" errors are coming from. If
    they're from the OS, could it be because it needs to fork() and momentarily
    exceed the ulimit before loading the native libs?

    - Aaron
    On Fri, Mar 4, 2011 at 1:26 PM, Aaron Kimball wrote:

    I don't know if putting native-code .so files inside a jar works. A
    native-code .so is not "classloaded" in the same way .class files are.

    So the correct .so files probably need to exist in some physical directory
    on the worker machines. You may want to doublecheck that the correct
    directory on the worker machines is identified in the JVM property
    'java.library.path' (instead of / in addition to $LD_LIBRARY_PATH). This can
    be manipulated in the Hadoop configuration setting mapred.child.java.opts
    (include '-Djava.library.path=/path/to/native/libs' in the string there.)

    Also, if you added your .so files to a directory that is already used by
    the tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may
    need to restart the tasktracker instance for it to take effect. (This is
    true of .jar files in the $HADOOP_HOME/lib directory; I don't know if it is
    true for native libs as well.)

    - Aaron

    On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) wrote:

    We are having difficulties running a Hadoop program making calls to
    external libraries - but this occurs only when we run the program on our
    cluster and not from within Eclipse where we are apparently running in
    Hadoop's standalone mode. This program invokes the Open Computer Vision
    libraries (OpenCV and JavaCV). (I don't think there is a problem with our
    cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs
    but I inadvertently hit the "run as Java application" button and the program
    ran fine, reading the input file from the eclipse workspace rather than HDFS
    and writing the output file to the same place. Hadoop's output appears
    below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted
    required libraries" into the generated jar - presumably producing a jar file
    that incorporated all the required library functions. (The plain jar file
    for this program is 17 kB while the runnable jar is 30MB.) When I try to
    run this on my Hadoop cluster (including my master and slave servers) the
    program reports that it is unable to locate "libopencv_highgui.so.2.2:
    cannot open shared object file: No such file or directory". Now, in
    addition to this library being incorporated inside the runnable jar file it
    is also present on each of my servers at
    hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same
    libraries (to give Hadoop 2 shots at finding them). These include:
    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find
    javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested
    in Patch 2838 that's supposed to be included in hadoop 0.21
    https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>

    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now
    complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would
    be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below
    refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for
    HDFS and thus perhaps not a good place for Hadoop to be looking for a
    library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the
    sequence file being read is 400 KB - so I hope I am not running out of
    memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE -
    SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
    for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User
    classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated.
    Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token
    storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task:
    attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
    library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length =
    6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454;
    bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584);
    kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task:
    attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to
    file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41,
    68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ],
    [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id is deprecated.
    Instead, use mapreduce.task.attempt.id
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21
    /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar
    progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id is deprecated.
    Instead, use mapreduce.task.attempt.id
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
    for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated.
    Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so:
    failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so:
    failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM
    warning: Exception java.lang.OutOfMemoryError occurred dispatching signal
    SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots
    (ms)=0
    Total time spent by all reduces waiting after reserving
    slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar
    progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id is deprecated.
    Instead, use mapreduce.task.attempt.id
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
    for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated.
    Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so:
    libxcb.so.1: failed to map segment from shared object: Cannot allocate
    memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so:
    libXau.so.6: failed to map segment from shared object: Cannot allocate
    memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so:
    libxcb-render.so.0: failed to map segment from shared object: Cannot
    allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so:
    libfontconfig.so.1: failed to map segment from shared object: Cannot
    allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM
    warning: Exception java.lang.OutOfMemoryError occurred dispatching signal
    SIGTERM to handler- the VM may need to be forcibly terminated


  • Ratner, Alan S (IS) at Mar 4, 2011 at 9:56 pm
    Aaron,

    Thanks for the rapid responses.


    * "ulimit -u unlimited" is in .bashrc.


    * HADOOP_HEAPSIZE is set to 4000 MB in hadoop-env.sh


    * Mapred.child.ulimit is set to 2048000 in mapred-site.xml


    * Mapred.child.java.opts is set to -Xmx1536m in mapred-site.xml

    I take it you are suggesting that I change the java.opts command to:

    Mapred.child.java.opts is <value> -Xmx1536m -Djava.library.path=/path/to/native/libs </value>


    Alan Ratner
    Northrop Grumman Information Systems
    Manager of Large-Scale Computing
    9020 Junction Drive
    Annapolis Junction, MD 20701
    (410) 707-8605 (cell)

    From: Aaron Kimball
    Sent: Friday, March 04, 2011 4:30 PM
    To: common-user@hadoop.apache.org
    Cc: Ratner, Alan S (IS)
    Subject: EXT :Re: Problem running a Hadoop program with external libraries

    Actually, I just misread your email and missed the difference between your 2nd and 3rd attempts.

    Are you enforcing min/max JVM heap sizes on your tasks? Are you enforcing a ulimit (either through your shell configuration, or through Hadoop itself)? I don't know where these "cannot allocate memory" errors are coming from. If they're from the OS, could it be because it needs to fork() and momentarily exceed the ulimit before loading the native libs?

    - Aaron

    On Fri, Mar 4, 2011 at 1:26 PM, Aaron Kimball wrote:
    I don't know if putting native-code .so files inside a jar works. A native-code .so is not "classloaded" in the same way .class files are.

    So the correct .so files probably need to exist in some physical directory on the worker machines. You may want to doublecheck that the correct directory on the worker machines is identified in the JVM property 'java.library.path' (instead of / in addition to $LD_LIBRARY_PATH). This can be manipulated in the Hadoop configuration setting mapred.child.java.opts (include '-Djava.library.path=/path/to/native/libs' in the string there.)

    Also, if you added your .so files to a directory that is already used by the tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may need to restart the tasktracker instance for it to take effect. (This is true of .jar files in the $HADOOP_HOME/lib directory; I don't know if it is true for native libs as well.)

    - Aaron

    On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) wrote:
    We are having difficulties running a Hadoop program making calls to external libraries - but this occurs only when we run the program on our cluster and not from within Eclipse where we are apparently running in Hadoop's standalone mode. This program invokes the Open Computer Vision libraries (OpenCV and JavaCV). (I don't think there is a problem with our cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs but I inadvertently hit the "run as Java application" button and the program ran fine, reading the input file from the eclipse workspace rather than HDFS and writing the output file to the same place. Hadoop's output appears below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted required libraries" into the generated jar - presumably producing a jar file that incorporated all the required library functions. (The plain jar file for this program is 17 kB while the runnable jar is 30MB.) When I try to run this on my Hadoop cluster (including my master and slave servers) the program reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory". Now, in addition to this library being incorporated inside the runnable jar file it is also present on each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same libraries (to give Hadoop 2 shots at finding them). These include:
    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested in Patch 2838 that's supposed to be included in hadoop 0.21 https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and thus perhaps not a good place for Hadoop to be looking for a library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being read is 400 KB - so I hope I am not running out of memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 10485760
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots (ms)=0
    Total time spent by all reduces waiting after reserving slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: libxcb.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: libXau.so.6: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: libxcb-render.so.0: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: libfontconfig.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
  • Brian Bockelman at Mar 4, 2011 at 10:13 pm
    Hi,

    Check your kernel's overcommit settings. This will prevent the JVM from allocating memory even when there's free RAM.

    Brian
    On Mar 4, 2011, at 3:55 PM, Ratner, Alan S (IS) wrote:

    Aaron,

    Thanks for the rapid responses.


    * "ulimit -u unlimited" is in .bashrc.


    * HADOOP_HEAPSIZE is set to 4000 MB in hadoop-env.sh


    * Mapred.child.ulimit is set to 2048000 in mapred-site.xml


    * Mapred.child.java.opts is set to -Xmx1536m in mapred-site.xml

    I take it you are suggesting that I change the java.opts command to:

    Mapred.child.java.opts is <value> -Xmx1536m -Djava.library.path=/path/to/native/libs </value>


    Alan Ratner
    Northrop Grumman Information Systems
    Manager of Large-Scale Computing
    9020 Junction Drive
    Annapolis Junction, MD 20701
    (410) 707-8605 (cell)

    From: Aaron Kimball
    Sent: Friday, March 04, 2011 4:30 PM
    To: common-user@hadoop.apache.org
    Cc: Ratner, Alan S (IS)
    Subject: EXT :Re: Problem running a Hadoop program with external libraries

    Actually, I just misread your email and missed the difference between your 2nd and 3rd attempts.

    Are you enforcing min/max JVM heap sizes on your tasks? Are you enforcing a ulimit (either through your shell configuration, or through Hadoop itself)? I don't know where these "cannot allocate memory" errors are coming from. If they're from the OS, could it be because it needs to fork() and momentarily exceed the ulimit before loading the native libs?

    - Aaron

    On Fri, Mar 4, 2011 at 1:26 PM, Aaron Kimball wrote:
    I don't know if putting native-code .so files inside a jar works. A native-code .so is not "classloaded" in the same way .class files are.

    So the correct .so files probably need to exist in some physical directory on the worker machines. You may want to doublecheck that the correct directory on the worker machines is identified in the JVM property 'java.library.path' (instead of / in addition to $LD_LIBRARY_PATH). This can be manipulated in the Hadoop configuration setting mapred.child.java.opts (include '-Djava.library.path=/path/to/native/libs' in the string there.)

    Also, if you added your .so files to a directory that is already used by the tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may need to restart the tasktracker instance for it to take effect. (This is true of .jar files in the $HADOOP_HOME/lib directory; I don't know if it is true for native libs as well.)

    - Aaron

    On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) wrote:
    We are having difficulties running a Hadoop program making calls to external libraries - but this occurs only when we run the program on our cluster and not from within Eclipse where we are apparently running in Hadoop's standalone mode. This program invokes the Open Computer Vision libraries (OpenCV and JavaCV). (I don't think there is a problem with our cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs but I inadvertently hit the "run as Java application" button and the program ran fine, reading the input file from the eclipse workspace rather than HDFS and writing the output file to the same place. Hadoop's output appears below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted required libraries" into the generated jar - presumably producing a jar file that incorporated all the required library functions. (The plain jar file for this program is 17 kB while the runnable jar is 30MB.) When I try to run this on my Hadoop cluster (including my master and slave servers) the program reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory". Now, in addition to this library being incorporated inside the runnable jar file it is also present on each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same libraries (to give Hadoop 2 shots at finding them). These include:
    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested in Patch 2838 that's supposed to be included in hadoop 0.21 https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and thus perhaps not a good place for Hadoop to be looking for a library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being read is 400 KB - so I hope I am not running out of memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots (ms)=0
    Total time spent by all reduces waiting after reserving slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: libxcb.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: libXau.so.6: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: libxcb-render.so.0: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: libfontconfig.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated


  • Lance Norskog at Mar 4, 2011 at 11:20 pm
    I have never heard of putting a native code shared library in a Java jar. I doubt that it works. But it's a cool idea!

    A Unix binary program loads shared libraries from the paths given in the environment variable LD_LIBRARY_PATH. This has to be set to the directory with the OpenCV .so file when you start Java.

    Lance
    On Mar 4, 2011, at 2:13 PM, Brian Bockelman wrote:

    Hi,

    Check your kernel's overcommit settings. This will prevent the JVM from allocating memory even when there's free RAM.

    Brian
    On Mar 4, 2011, at 3:55 PM, Ratner, Alan S (IS) wrote:

    Aaron,

    Thanks for the rapid responses.


    * "ulimit -u unlimited" is in .bashrc.


    * HADOOP_HEAPSIZE is set to 4000 MB in hadoop-env.sh


    * Mapred.child.ulimit is set to 2048000 in mapred-site.xml


    * Mapred.child.java.opts is set to -Xmx1536m in mapred-site.xml

    I take it you are suggesting that I change the java.opts command to:

    Mapred.child.java.opts is <value> -Xmx1536m -Djava.library.path=/path/to/native/libs </value>


    Alan Ratner
    Northrop Grumman Information Systems
    Manager of Large-Scale Computing
    9020 Junction Drive
    Annapolis Junction, MD 20701
    (410) 707-8605 (cell)

    From: Aaron Kimball
    Sent: Friday, March 04, 2011 4:30 PM
    To: common-user@hadoop.apache.org
    Cc: Ratner, Alan S (IS)
    Subject: EXT :Re: Problem running a Hadoop program with external libraries

    Actually, I just misread your email and missed the difference between your 2nd and 3rd attempts.

    Are you enforcing min/max JVM heap sizes on your tasks? Are you enforcing a ulimit (either through your shell configuration, or through Hadoop itself)? I don't know where these "cannot allocate memory" errors are coming from. If they're from the OS, could it be because it needs to fork() and momentarily exceed the ulimit before loading the native libs?

    - Aaron

    On Fri, Mar 4, 2011 at 1:26 PM, Aaron Kimball wrote:
    I don't know if putting native-code .so files inside a jar works. A native-code .so is not "classloaded" in the same way .class files are.

    So the correct .so files probably need to exist in some physical directory on the worker machines. You may want to doublecheck that the correct directory on the worker machines is identified in the JVM property 'java.library.path' (instead of / in addition to $LD_LIBRARY_PATH). This can be manipulated in the Hadoop configuration setting mapred.child.java.opts (include '-Djava.library.path=/path/to/native/libs' in the string there.)

    Also, if you added your .so files to a directory that is already used by the tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may need to restart the tasktracker instance for it to take effect. (This is true of .jar files in the $HADOOP_HOME/lib directory; I don't know if it is true for native libs as well.)

    - Aaron

    On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) wrote:
    We are having difficulties running a Hadoop program making calls to external libraries - but this occurs only when we run the program on our cluster and not from within Eclipse where we are apparently running in Hadoop's standalone mode. This program invokes the Open Computer Vision libraries (OpenCV and JavaCV). (I don't think there is a problem with our cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs but I inadvertently hit the "run as Java application" button and the program ran fine, reading the input file from the eclipse workspace rather than HDFS and writing the output file to the same place. Hadoop's output appears below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted required libraries" into the generated jar - presumably producing a jar file that incorporated all the required library functions. (The plain jar file for this program is 17 kB while the runnable jar is 30MB.) When I try to run this on my Hadoop cluster (including my master and slave servers) the program reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory". Now, in addition to this library being incorporated inside the runnable jar file it is also present on each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same libraries (to give Hadoop 2 shots at finding them). These include:
    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested in Patch 2838 that's supposed to be included in hadoop 0.21 https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and thus perhaps not a good place for Hadoop to be looking for a library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being read is 400 KB - so I hope I am not running out of memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots (ms)=0
    Total time spent by all reduces waiting after reserving slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id<http://mapred.task.id> is deprecated. Instead, use mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: libxcb.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: libXau.so.6: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: libxcb-render.so.0: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: libfontconfig.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated


  • Alejandro Abdelnur at Mar 5, 2011 at 11:48 am
    Why don't you put your native library in HDFS and use the DistributedCache
    to make them avail to the tasks. For example:

    Copy 'foo.so' to 'hdfs://localhost:8020/tmp/foo.so', then added to the job
    distributed cache:

    DistributedCache.addCacheFile("hdfs://localhost:8020/tmp/foo.so#foo.so",
    jobConf);
    DistributedCache.createSymlink(conf);

    Note that the #foo.so will create a soflink in the task running dir. And the
    task running dir is in LD_PATH of your task.

    Alejandro
    On Sat, Mar 5, 2011 at 7:19 AM, Lance Norskog wrote:

    I have never heard of putting a native code shared library in a Java jar. I
    doubt that it works. But it's a cool idea!

    A Unix binary program loads shared libraries from the paths given in the
    environment variable LD_LIBRARY_PATH. This has to be set to the directory
    with the OpenCV .so file when you start Java.

    Lance
    On Mar 4, 2011, at 2:13 PM, Brian Bockelman wrote:

    Hi,

    Check your kernel's overcommit settings. This will prevent the JVM from
    allocating memory even when there's free RAM.
    Brian
    On Mar 4, 2011, at 3:55 PM, Ratner, Alan S (IS) wrote:

    Aaron,

    Thanks for the rapid responses.


    * "ulimit -u unlimited" is in .bashrc.


    * HADOOP_HEAPSIZE is set to 4000 MB in hadoop-env.sh


    * Mapred.child.ulimit is set to 2048000 in mapred-site.xml


    * Mapred.child.java.opts is set to -Xmx1536m in mapred-site.xml

    I take it you are suggesting that I change the java.opts command to:

    Mapred.child.java.opts is <value> -Xmx1536m
    -Djava.library.path=/path/to/native/libs </value>

    Alan Ratner
    Northrop Grumman Information Systems
    Manager of Large-Scale Computing
    9020 Junction Drive
    Annapolis Junction, MD 20701
    (410) 707-8605 (cell)

    From: Aaron Kimball
    Sent: Friday, March 04, 2011 4:30 PM
    To: common-user@hadoop.apache.org
    Cc: Ratner, Alan S (IS)
    Subject: EXT :Re: Problem running a Hadoop program with external
    libraries
    Actually, I just misread your email and missed the difference between
    your 2nd and 3rd attempts.
    Are you enforcing min/max JVM heap sizes on your tasks? Are you
    enforcing a ulimit (either through your shell configuration, or through
    Hadoop itself)? I don't know where these "cannot allocate memory" errors are
    coming from. If they're from the OS, could it be because it needs to fork()
    and momentarily exceed the ulimit before loading the native libs?
    - Aaron

    On Fri, Mar 4, 2011 at 1:26 PM, Aaron Kimball wrote:
    I don't know if putting native-code .so files inside a jar works. A
    native-code .so is not "classloaded" in the same way .class files are.
    So the correct .so files probably need to exist in some physical
    directory on the worker machines. You may want to doublecheck that the
    correct directory on the worker machines is identified in the JVM property
    'java.library.path' (instead of / in addition to $LD_LIBRARY_PATH). This can
    be manipulated in the Hadoop configuration setting mapred.child.java.opts
    (include '-Djava.library.path=/path/to/native/libs' in the string there.)
    Also, if you added your .so files to a directory that is already used by
    the tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may
    need to restart the tasktracker instance for it to take effect. (This is
    true of .jar files in the $HADOOP_HOME/lib directory; I don't know if it is
    true for native libs as well.)
    - Aaron

    On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) <
    Alan.Ratner@ngc.com wrote:
    We are having difficulties running a Hadoop program making calls to
    external libraries - but this occurs only when we run the program on our
    cluster and not from within Eclipse where we are apparently running in
    Hadoop's standalone mode. This program invokes the Open Computer Vision
    libraries (OpenCV and JavaCV). (I don't think there is a problem with our
    cluster - we've run many Hadoop jobs on it without difficulty.)
    1. I normally use Eclipse to create jar files for our Hadoop
    programs but I inadvertently hit the "run as Java application" button and
    the program ran fine, reading the input file from the eclipse workspace
    rather than HDFS and writing the output file to the same place. Hadoop's
    output appears below. (This occurred on the master Hadoop server.)
    2. I then "exported" from Eclipse a "runnable jar" which "extracted
    required libraries" into the generated jar - presumably producing a jar file
    that incorporated all the required library functions. (The plain jar file
    for this program is 17 kB while the runnable jar is 30MB.) When I try to
    run this on my Hadoop cluster (including my master and slave servers) the
    program reports that it is unable to locate "libopencv_highgui.so.2.2:
    cannot open shared object file: No such file or directory". Now, in
    addition to this library being incorporated inside the runnable jar file it
    is also present on each of my servers at
    hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same
    libraries (to give Hadoop 2 shots at finding them). These include:
    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find
    javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as
    suggested in Patch 2838 that's supposed to be included in hadoop 0.21
    https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now
    complaining about a different missing library with an out-of-memory error.
    Does anyone have any ideas as to what is going wrong here? Any help
    would be appreciated. Thanks.
    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below
    refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for
    HDFS and thus perhaps not a good place for Hadoop to be looking for a
    library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the
    sequence file being read is 400 KB - so I hope I am not running out of
    memory.

    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE -
    SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
    for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set.
    User classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is
    deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token
    storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task:
    attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load
    native-hadoop library for your platform... using builtin-java classes where
    applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi
    26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length =
    6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454;
    bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584);
    kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
    is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569
    sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task:
    attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor
    complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with
    1 segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0
    is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0
    is allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to
    file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84,
    41, 68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ],
    [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id<
    http://mapred.task.id> is deprecated. Instead, use
    mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21
    /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar
    progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id<
    http://mapred.task.id> is deprecated. Instead, use
    mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
    for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is
    deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so:
    failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so:
    failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM
    warning: Exception java.lang.OutOfMemoryError occurred dispatching signal
    SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error:
    /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so:
    libopencv_highgui.so.2.2: cannot open shared object file: No such file or
    directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete:
    job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots
    (ms)=0
    Total time spent by all reduces waiting after reserving
    slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar
    progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping
    impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
    cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id<
    http://mapred.task.id> is deprecated. Instead, use
    mapreduce.task.attempt.id<http://mapreduce.task.attempt.id>
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
    for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is
    deprecated. Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following
    namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error:
    /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so:
    libxcb.so.1: failed to map segment from shared object: Cannot allocate
    memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so:
    libXau.so.6: failed to map segment from shared object: Cannot allocate
    memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error:
    /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so:
    libxcb-render.so.0: failed to map segment from shared object: Cannot
    allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error:
    /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so:
    libfontconfig.so.1: failed to map segment from shared object: Cannot
    allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id :
    attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM
    warning: Exception java.lang.OutOfMemoryError occurred dispatching signal
    SIGTERM to handler- the VM may need to be forcibly terminated

  • Ratner, Alan S (IS) at Mar 8, 2011 at 9:22 pm
    Thanks to all who suggested solutions to our problem of running a Java MR job using both external Java and C++ libraries.

    We got it to work by moving all our .so files into an archive (http://hadoop.apache.org/mapreduce/docs/r0.21.0/hadoop_archives.html)
    and publishing it to our MR app with a single statement:
    DistributedCache.createSymlink(conf).

    We found that we had to use Eclipse to generate a "runnable" jar file in "extract" mode; running an ordinary jar did not work. (We tried putting our external jars in the archive file but a plain jar still did not work - perhaps I haven't assembled the complete set of jars into the archive.)

    We had tried putting all the libraries directly in HDFS with a pointer in mapred-site.xml:
    <property><name>mapred.child.env</name><value>LD_LIBRARY_PATH=/user/ngc/lib</value></property>
    as described in https://issues.apache.org/jira/browse/HADOOP-2838 but this did not work for us.

    The bottom line of all this is that we managed to write a Hadoop job in Java that invokes the OpenCV (Open Computer Vision) C++ libraries (http://opencv.willowgarage.com/wiki/) using the JavaCV Java wrapper (http://code.google.com/p/javacv/). OpenCV includes over 500 image processing algorithms.




    -----Original Message-----
    From: Ratner, Alan S (IS)
    Sent: Friday, March 04, 2011 3:53 PM
    To: common-user@hadoop.apache.org
    Subject: EXT :Problem running a Hadoop program with external libraries

    We are having difficulties running a Hadoop program making calls to external libraries - but this occurs only when we run the program on our cluster and not from within Eclipse where we are apparently running in Hadoop's standalone mode. This program invokes the Open Computer Vision libraries (OpenCV and JavaCV). (I don't think there is a problem with our cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs but I inadvertently hit the "run as Java application" button and the program ran fine, reading the input file from the eclipse workspace rather than HDFS and writing the output file to the same place. Hadoop's output appears below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted required libraries" into the generated jar - presumably producing a jar file that incorporated all the required library functions. (The plain jar file for this program is 17 kB while the runnable jar is 30MB.) When I try to run this on my Hadoop cluster (including my master and slave servers) the program reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory". Now, in addition to this library being incorporated inside the runnable jar file it is also present on each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same libraries (to give Hadoop 2 shots at finding them). These include:

    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested in Patch 2838 that's supposed to be included in hadoop 0.21 https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and thus perhaps not a good place for Hadoop to be looking for a library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being read is 400 KB - so I hope I am not running out of memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 10485760
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots (ms)=0
    Total time spent by all reduces waiting after reserving slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: libxcb.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: libXau.so.6: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: libxcb-render.so.0: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: libfontconfig.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
  • Ratner, Alan S (IS) at Mar 8, 2011 at 9:32 pm
    One other thing: We were getting out-of-memory errors with these external libraries and we had to reduce the value of <name>mapred.child.java.opts</name> found in mapred-site.xml. We had originally been using 2 GB (our servers have 24-48 GB RAM) and eliminated the out-of-memory errors by reducing this value to 1.28 GB.

    Alan


    -----Original Message-----
    From: Ratner, Alan S (IS)
    Sent: Tuesday, March 08, 2011 4:22 PM
    To: common-user@hadoop.apache.org
    Cc: Gerlach, Hannah L (IS); Andrew Levine
    Subject: EXT :RE: Problem running a Hadoop program with external libraries

    Thanks to all who suggested solutions to our problem of running a Java MR job using both external Java and C++ libraries.

    We got it to work by moving all our .so files into an archive (http://hadoop.apache.org/mapreduce/docs/r0.21.0/hadoop_archives.html)
    and publishing it to our MR app with a single statement:
    DistributedCache.createSymlink(conf).

    We found that we had to use Eclipse to generate a "runnable" jar file in "extract" mode; running an ordinary jar did not work. (We tried putting our external jars in the archive file but a plain jar still did not work - perhaps I haven't assembled the complete set of jars into the archive.)

    We had tried putting all the libraries directly in HDFS with a pointer in mapred-site.xml:
    <property><name>mapred.child.env</name><value>LD_LIBRARY_PATH=/user/ngc/lib</value></property>
    as described in https://issues.apache.org/jira/browse/HADOOP-2838 but this did not work for us.

    The bottom line of all this is that we managed to write a Hadoop job in Java that invokes the OpenCV (Open Computer Vision) C++ libraries (http://opencv.willowgarage.com/wiki/) using the JavaCV Java wrapper (http://code.google.com/p/javacv/). OpenCV includes over 500 image processing algorithms.




    -----Original Message-----
    From: Ratner, Alan S (IS)
    Sent: Friday, March 04, 2011 3:53 PM
    To: common-user@hadoop.apache.org
    Subject: EXT :Problem running a Hadoop program with external libraries

    We are having difficulties running a Hadoop program making calls to external libraries - but this occurs only when we run the program on our cluster and not from within Eclipse where we are apparently running in Hadoop's standalone mode. This program invokes the Open Computer Vision libraries (OpenCV and JavaCV). (I don't think there is a problem with our cluster - we've run many Hadoop jobs on it without difficulty.)

    1. I normally use Eclipse to create jar files for our Hadoop programs but I inadvertently hit the "run as Java application" button and the program ran fine, reading the input file from the eclipse workspace rather than HDFS and writing the output file to the same place. Hadoop's output appears below. (This occurred on the master Hadoop server.)

    2. I then "exported" from Eclipse a "runnable jar" which "extracted required libraries" into the generated jar - presumably producing a jar file that incorporated all the required library functions. (The plain jar file for this program is 17 kB while the runnable jar is 30MB.) When I try to run this on my Hadoop cluster (including my master and slave servers) the program reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory". Now, in addition to this library being incorporated inside the runnable jar file it is also present on each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have loaded the same libraries (to give Hadoop 2 shots at finding them). These include:


    ...
    libopencv_highgui_pch_dephelp.a
    libopencv_highgui.so
    libopencv_highgui.so.2.2
    libopencv_highgui.so.2.2.0
    ...

    When I poke around inside the runnable jar I find javacv_linux-x86_64.jar which contains:
    com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so

    3. I then tried adding the following to mapred-site.xml as suggested in Patch 2838 that's supposed to be included in hadoop 0.21 https://issues.apache.org/jira/browse/HADOOP-2838
    <property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value>
    </property>
    The log is included at the bottom of this email with Hadoop now complaining about a different missing library with an out-of-memory error.

    Does anyone have any ideas as to what is going wrong here? Any help would be appreciated. Thanks.

    Alan


    BTW: Each of our servers has 4 hard drives and many of the errors below refer to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and thus perhaps not a good place for Hadoop to be looking for a library file. My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being read is 400 KB - so I hope I am not running out of memory.


    1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS
    Running Face Program
    11/03/04 12:44:10 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
    11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1
    11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token storage with # keys=0
    11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks
    11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: attempt_local_0001_m_000000_0
    11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor
    11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1
    11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
    11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
    11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080
    11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 10485760
    11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
    11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:44:16 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > map
    11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output
    11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output
    11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = 104857600
    11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214324(104857296); length = 73/6553600
    11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local_0001_m_000000_0
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete.
    11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments
    11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1481 bytes
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
    11/03/04 12:44:18 INFO mapred.LocalJobRunner:
    11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
    11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput
    11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort
    11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
    11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100%
    11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001
    11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18
    FileInputFormatCounters
    BYTES_READ=411439
    FileSystemCounters
    FILE_BYTES_READ=825005
    FILE_BYTES_WRITTEN=127557
    Map-Reduce Framework
    Combine input records=0
    Combine output records=0
    Failed Shuffles=0
    GC time elapsed (ms)=88
    Map input records=20
    Map output bytes=1454
    Map output records=19
    Merged Map outputs=0
    Reduce input groups=19
    Reduce input records=19
    Reduce output records=19
    Reduce shuffle bytes=0
    Shuffled Maps =0
    Spilled Records=38
    SPLIT_RAW_BYTES=127
    0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 ],
    ...
    0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ 210, 119, 228 ],
    Job Finished in 8.679 seconds
    2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE

    ngc@hadoop1:~$ cd hadoop-0.21.0/
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag*
    11/03/04 12:58:18 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:58:18 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    Found 1 items
    -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 /user/ngc/Images2/JPGSequenceFile.001
    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 12:59:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 12:59:40 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051
    11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0%
    11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0%
    11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0%
    11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0%
    11/03/04 13:00:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:08 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0%
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: failed to map segment from shared object: Cannot allocate memory
    11/03/04 13:00:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_0, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: failed to map segment from shared object: Cannot allocate memory
    attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
    11/03/04 13:00:11 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_0, Status : FAILED
    11/03/04 13:00:12 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_0, Status : FAILED
    11/03/04 13:00:13 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_0, Status : FAILED
    11/03/04 13:00:14 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_0, Status : FAILED
    11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0%
    11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0%
    11/03/04 13:00:18 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0%
    11/03/04 13:00:19 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_0, Status : FAILED
    11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15%
    11/03/04 13:00:20 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_0, Status : FAILED
    11/03/04 13:00:21 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:22 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_0, Status : FAILED
    11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23%
    11/03/04 13:00:23 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_0, Status : FAILED
    11/03/04 13:00:24 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_0, Status : FAILED
    11/03/04 13:00:25 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_0, Status : FAILED
    11/03/04 13:00:36 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:39 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_1, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23%
    11/03/04 13:00:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_1, Status : FAILED
    11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24%
    11/03/04 13:00:43 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000048_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24%
    11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25%
    11/03/04 13:00:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000001_1, Status : FAILED
    11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25%
    11/03/04 13:00:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000052_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000044_1, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000087_0, Status : FAILED
    11/03/04 13:00:49 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000011_2, Status : FAILED
    Error: /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25%
    11/03/04 13:00:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000006_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:00:52 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000026_1, Status : FAILED
    11/03/04 13:00:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000022_1, Status : FAILED
    11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26%
    11/03/04 13:00:55 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000016_1, Status : FAILED
    11/03/04 13:00:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000091_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000096_0, Status : FAILED
    11/03/04 13:00:58 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000034_1, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:01 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000065_1, Status : FAILED
    11/03/04 13:01:03 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000057_1, Status : FAILED
    11/03/04 13:01:04 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000076_1, Status : FAILED
    11/03/04 13:01:06 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000039_1, Status : FAILED
    11/03/04 13:01:07 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000061_1, Status : FAILED
    11/03/04 13:01:09 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000069_2, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: libopencv_highgui.so.2.2: cannot open shared object file: No such file or directory
    11/03/04 13:01:10 INFO mapreduce.Job: Task Id : attempt_201103021428_0051_m_000082_2, Status : FAILED
    11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051
    11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21
    FileInputFormatCounters
    BYTES_READ=0
    FileSystemCounters
    FILE_BYTES_WRITTEN=3040
    HDFS_BYTES_READ=1048281
    Job Counters
    Data-local map tasks=51
    Total time spent by all maps waiting after reserving slots (ms)=0
    Total time spent by all reduces waiting after reserving slots (ms)=0
    Failed map tasks=1
    Rack-local map tasks=86
    SLOTS_MILLIS_MAPS=1091359
    SLOTS_MILLIS_REDUCES=0
    Launched map tasks=137
    Launched reduce tasks=2
    Map-Reduce Framework
    Combine input records=0
    Failed Shuffles=0
    GC time elapsed (ms)=0
    Map input records=0
    Map output bytes=0
    Map output records=0
    Merged Map outputs=0
    Spilled Records=0
    SPLIT_RAW_BYTES=8960
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
    at progs.HannahFace.run(HannahFace.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at progs.HannahFace.main(HannahFace.java:162)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

    3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE

    ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar progs/HannahFace
    Running Face Program
    11/03/04 15:07:11 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
    11/03/04 15:07:11 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : 1
    11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100
    11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
    11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069
    11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0%
    11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0%
    11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0%
    11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0%
    11/03/04 15:07:38 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000016_0, Status : FAILED
    Error: /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: libxcb.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0%
    11/03/04 15:07:40 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000001_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: libXau.so.6: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0%
    11/03/04 15:07:42 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000022_0, Status : FAILED
    11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0%
    11/03/04 15:07:44 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000026_0, Status : FAILED
    11/03/04 15:07:45 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000011_0, Status : FAILED
    11/03/04 15:07:46 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000034_0, Status : FAILED
    11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13%
    11/03/04 15:07:47 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000039_0, Status : FAILED
    11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13%
    11/03/04 15:07:48 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000082_0, Status : FAILED
    Error: /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: libxcb-render.so.0: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:50 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000006_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13%
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000044_0, Status : FAILED
    11/03/04 15:07:51 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000048_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000052_0, Status : FAILED
    11/03/04 15:07:53 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000076_0, Status : FAILED
    Error: /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: libfontconfig.so.1: failed to map segment from shared object: Cannot allocate memory
    11/03/04 15:07:54 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000057_0, Status : FAILED
    11/03/04 15:07:56 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000061_0, Status : FAILED
    11/03/04 15:07:57 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000065_0, Status : FAILED
    11/03/04 15:07:59 INFO mapreduce.Job: Task Id : attempt_201103021428_0069_m_000069_0, Status : FAILED
    attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
  • Allen Wittenauer at Mar 12, 2011 at 1:03 am

    On Mar 8, 2011, at 1:21 PM, Ratner, Alan S (IS) wrote:
    We had tried putting all the libraries directly in HDFS with a pointer in mapred-site.xml:
    <property><name>mapred.child.env</name><value>LD_LIBRARY_PATH=/user/ngc/lib</value></property>
    as described in https://issues.apache.org/jira/browse/HADOOP-2838 but this did not work for us.
    Correct. This isn't expected to work.

    HDFS files are not directly accessible from the shell without some sort of action having taken place. In order for the above to work, anything reading the LD_LIBRARY_PATH environment variable would have to know that '/user/...' is a) inside HDFS and b) know how to access it. The reason why the distributed cache method works is because it pulls files from HDFS and places them in the local UNIX file system. From there, UNIX processes can now access them.

    HADOOP-2838 is really about providing a way for applications to get to libraries that are already installed at the UNIX level. (Although, in reality, it would likely be better if applications were linked with a better value provided for the runtime library search path -R/-rpath/ld.so.conf/crle/etc rather than using LD_LIBRARY_PATH.)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 4, '11 at 8:53p
activeMar 12, '11 at 1:03a
posts11
users7
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase