[Hadoop-common-user] Hadoop C++ Task Fails during runtime

-----Original Message-----
From: Kumar, Amit H.
Sent: Wednesday, March 02, 2011 10:51 AM
To: [email protected]
Subject: Hadoop C++ Task Fails during runtime

Hi All,

I am trying to follow first steps on getting a simple C++ program to
work using Hadoop Pipes. And I get the following error while running
it. Can anybody help me understand what could I be doing wrong?
Used the following code.
http://wiki.apache.org/hadoop/C%2B%2BWordCount

# hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=true -input cppinput -output coutput -
program bin/wordc
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.

11/03/02 10:47:23 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/03/02 10:47:23 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id
11/03/02 10:47:23 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the
same.
11/03/02 10:47:23 WARN mapreduce.JobSubmitter: No job jar file set.
User classes may not be found. See Job or Job#setJar(String).
11/03/02 10:47:23 INFO mapred.FileInputFormat: Total input paths to
process : 1
11/03/02 10:47:24 WARN conf.Configuration: mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
11/03/02 10:47:24 INFO mapreduce.JobSubmitter: number of splits:2
11/03/02 10:47:24 INFO mapreduce.JobSubmitter: adding the following
namenodes' delegation tokens:null
11/03/02 10:47:24 INFO mapreduce.Job: Running job:
job_201102161124_0012
11/03/02 10:47:25 INFO mapreduce.Job: map 0% reduce 0%
11/03/02 10:47:40 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000000_0: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:47:40 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000001_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000001_0: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:47:52 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000000_1, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000000_1: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:47:52 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000001_1, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000001_1: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:48:04 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000000_2, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000000_2: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:48:04 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000001_2, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000001_2: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:48:20 INFO mapreduce.Job: Job complete:
job_201102161124_0012
11/03/02 10:48:20 INFO mapreduce.Job: Counters: 8
Job Counters
Data-local map tasks=1
Total time spent by all maps waiting after reserving
slots (ms)=0
Total time spent by all reduces waiting after reserving
slots (ms)=0
Failed map tasks=1
Rack-local map tasks=7
SLOTS_MILLIS_MAPS=89957
SLOTS_MILLIS_REDUCES=0
Launched map tasks=8
Exception in thread "main" java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
at
org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264)
at
org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:504)
at
org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:519)

Any help greatly appreciated,
Thanks,
Amit



--
BEGIN-ANTISPAM-VOTING-LINKS
------------------------------------------------------

Teach CanIt if this mail (ID 451862814) is spam:
Spam:
https://www.spamtrap.odu.edu/b.php?i=451862814&m=b4ce7ffcbfc1&t=2011030
2&c=s
Not spam:
https://www.spamtrap.odu.edu/b.php?i=451862814&m=b4ce7ffcbfc1&t=2011030
2&c=n
Forget vote:
https://www.spamtrap.odu.edu/b.php?i=451862814&m=b4ce7ffcbfc1&t=2011030
2&c=f
------------------------------------------------------
END-ANTISPAM-VOTING-LINKS

-----Original Message-----
From: Kumar, Amit H.
Sent: Wednesday, March 02, 2011 10:51 AM
To: [email protected]
Subject: Hadoop C++ Task Fails during runtime

Hi All,

I am trying to follow first steps on getting a simple C++ program to
work using Hadoop Pipes. And I get the following error while running
it. Can anybody help me understand what could I be doing wrong?
Used the following code.
http://wiki.apache.org/hadoop/C%2B%2BWordCount

# hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=true -input cppinput -output coutput -
program bin/wordc
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.

11/03/02 10:47:23 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/03/02 10:47:23 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id
11/03/02 10:47:23 WARN mapreduce.JobSubmitter: Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the
same.
11/03/02 10:47:23 WARN mapreduce.JobSubmitter: No job jar file set.
User classes may not be found. See Job or Job#setJar(String).
11/03/02 10:47:23 INFO mapred.FileInputFormat: Total input paths to
process : 1
11/03/02 10:47:24 WARN conf.Configuration: mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
11/03/02 10:47:24 INFO mapreduce.JobSubmitter: number of splits:2
11/03/02 10:47:24 INFO mapreduce.JobSubmitter: adding the following
namenodes' delegation tokens:null
11/03/02 10:47:24 INFO mapreduce.Job: Running job:
job_201102161124_0012
11/03/02 10:47:25 INFO mapreduce.Job: map 0% reduce 0%
11/03/02 10:47:40 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000000_0: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:47:40 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000001_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000001_0: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:47:52 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000000_1, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000000_1: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:47:52 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000001_1, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000001_1: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:48:04 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000000_2, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000000_2: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:48:04 INFO mapreduce.Job: Task Id :
attempt_201102161124_0012_m_000001_2, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:160)
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:1
02)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
on.java:742)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:8
6)
... 7 more

attempt_201102161124_0012_m_000001_2: Hadoop Pipes Exception: Aborted
by driver
11/03/02 10:48:20 INFO mapreduce.Job: Job complete:
job_201102161124_0012
11/03/02 10:48:20 INFO mapreduce.Job: Counters: 8
Job Counters
Data-local map tasks=1
Total time spent by all maps waiting after reserving
slots (ms)=0
Total time spent by all reduces waiting after reserving
slots (ms)=0
Failed map tasks=1
Rack-local map tasks=7
SLOTS_MILLIS_MAPS=89957
SLOTS_MILLIS_REDUCES=0
Launched map tasks=8
Exception in thread "main" java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782)
at
org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264)
at
org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:504)
at
org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:519)

Any help greatly appreciated,
Thanks,
Amit



--
BEGIN-ANTISPAM-VOTING-LINKS
------------------------------------------------------

Teach CanIt if this mail (ID 451862814) is spam:
Spam:
https://www.spamtrap.odu.edu/b.php?i=451862814&m=b4ce7ffcbfc1&t=2011030
2&c=s
Not spam:
https://www.spamtrap.odu.edu/b.php?i=451862814&m=b4ce7ffcbfc1&t=2011030
2&c=n
Forget vote:
https://www.spamtrap.odu.edu/b.php?i=451862814&m=b4ce7ffcbfc1&t=2011030
2&c=f
------------------------------------------------------
END-ANTISPAM-VOTING-LINKS