FAQ
2 nodes: "ibmT43", "gentoo1".

ibmT43 = NameNode + JobTracker + TaskTracker + DataNode
gentoo1 = TaskTracker + DataNode

===conf/hadoop-site.xml=== - identical on 2 hosts:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://ibmT43:9000/</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>ibmT43:9001</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>mapred.task.timeout</name>
<value>20000</value>
</property>

<property>
<name>hadoop.pipes.executable</name>
<value>/bin/logparser</value>
</property>

<property>
<name>mapred.map.tasks</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
</property>
</configuration>

---------------------------

/etc/hosts on all two hosts also identical:

192.168.2.1 ibmT43
192.168.2.22 gentoo1
# 127.0.0.1 localhost
# it will have no effect if i will uncomment "localhost"
--------------------------

binary hdfs://bin/logparser is correct - it have been working in past.

--------------------------

bin/hadoop pipes -input /input -output /output1 -conf 123test.xml

====123test.xml===:
<?xml version="1.0"?>
<configuration>

<!--
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
</property>
-->

<property>
<name>hadoop.pipes.java.recordreader</name>
<value>true</value>
</property>

<property>
<name>hadoop.pipes.java.recordwriter</name>
<value>true</value>
</property>

</configuration>

-------------------------------------

Running job:
hadoop@ibmT43 ~/hadoop-0.18.3 $ bin/hadoop pipes -input /input -output
/output1 -conf 123test.xml
09/06/18 22:38:50 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
09/06/18 22:38:50 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
09/06/18 22:38:50 INFO mapred.FileInputFormat: Total input paths to
process : 4
09/06/18 22:38:50 INFO mapred.FileInputFormat: Total input paths to
process : 4
09/06/18 22:38:51 INFO mapred.JobClient: Running job: job_200906182236_0001
09/06/18 22:38:52 INFO mapred.JobClient: map 0% reduce 0%
09/06/18 22:39:03 INFO mapred.JobClient: map 5% reduce 0%
09/06/18 22:39:12 INFO mapred.JobClient: map 11% reduce 0%
09/06/18 22:39:23 INFO mapred.JobClient: map 5% reduce 0%
09/06/18 22:39:23 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000002_0, Status : FAILED
Task attempt_200906182236_0001_m_000002_0 failed to report status for 22
seconds. Killing!
09/06/18 22:39:24 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000003_0, Status : FAILED
Task attempt_200906182236_0001_m_000003_0 failed to report status for 22
seconds. Killing!
09/06/18 22:39:33 INFO mapred.JobClient: map 0% reduce 0%
09/06/18 22:39:33 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000000_0, Status : FAILED
Task attempt_200906182236_0001_m_000000_0 failed to report status for 23
seconds. Killing!
attempt_200906182236_0001_m_000000_0: Hadoop Pipes Exception: write error
to file: Connection reset by peer at SerialUtils.cc:129 in virtual void
Hadoo
pUtils::FileOutStream::write(const void*, size_t)
09/06/18 22:39:33 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000001_0, Status : FAILED
Task attempt_200906182236_0001_m_000001_0 failed to report status for 23
seconds. Killing!
09/06/18 22:39:34 INFO mapred.JobClient: map 2% reduce 0%
09/06/18 22:39:38 INFO mapred.JobClient: map 5% reduce 0%
09/06/18 22:39:43 INFO mapred.JobClient: map 8% reduce 0%
09/06/18 22:39:48 INFO mapred.JobClient: map 12% reduce 0%
09/06/18 22:39:54 INFO mapred.JobClient: map 9% reduce 0%
09/06/18 22:39:54 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000004_0, Status : FAILED
Task attempt_200906182236_0001_m_000004_0 failed to report status for 22
seconds. Killing!
attempt_200906182236_0001_m_000004_0: Hadoop Pipes Exception: write error
to file: Connection reset by peer at SerialUtils.cc:129 in virtual void
Hadoo
pUtils::FileOutStream::write(const void*, size_t)
09/06/18 22:39:59 INFO mapred.JobClient: map 6% reduce 0%
09/06/18 22:39:59 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000005_0, Status : FAILED
Task attempt_200906182236_0001_m_000005_0 failed to report status for 22
seconds. Killing!
attempt_200906182236_0001_m_000005_0: Hadoop Pipes Exception: write error
to file: Connection reset by peer at SerialUtils.cc:129 in virtual void
Hadoo
pUtils::FileOutStream::write(const void*, size_t)
09/06/18 22:40:03 INFO mapred.JobClient: Task Id :
attempt_200906182236_0001_m_000003_1, Status : FAILED
Task attempt_200906182236_0001_m_000003_1 failed to report status for 20
seconds. Killing!

-------------------------
20 sec, 50 sec or 999999 sec - not important.
----------------------------

thank you

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 18, '09 at 7:02p
activeJun 18, '09 at 7:02p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Pavel kolodin: 1 post

People

Translate

site design / logo © 2022 Grokbase