FAQ
I search on the internet and most of threads pointing that the problem
might come from the configuration of /etc/hosts, /etc/hostname file.

I check my /etc/hosts which contains all machines, including master
and slaves, and /etc/hostname has correct name setup. Also, firewall
opens for ports that are required for hadoop system, such as 9000,
9001, and so on.

In addition to these setting, any other place I can check for
investigating such problem?

I appreciate any suggestion.

My testing environment setting is as below:

Machines used: 1 master, 3 slaves
master: debian
slaves: cloud1, cloud2, cloud3

And all machines share the same user name - cloud

OS debian testing, java 1.6.0_23, hadoop 0.20.2.

/etc/hosts on all machines
127.0.0.1 localhost
###.###.###.### debian.domain.com debian
###.###.###.### cloud1.domain.com cloud1
###.###.###.### cloud2.domain.com cloud2
###.###.###.### cloud3.domain.com cloud3

hadoop conf in debian (master)

core-site.xml

<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>

hdfs-site.xml

<property>
<name>dfs.name.dir</name>
<value>/home/cloud/dfs/name</value>
</property>

mapred-site.xml

<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>

hadoop conf in cloud1 (slave):

core-site.xml:

<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>

hdfs-site.xml:

<property>
<name>dfs.data.dir</name>
<value>/home/cloud/dfs/data</value>
</property>

mapred-site.xml

<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>

hadoop conf in cloud2 (slave):

core-site.xml:

<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>

hdfs-site.xml:

<property>
<name>dfs.data.dir</name>
<value>/home/cloud/dfs/data</value>
</property>

mapred-site.xml:

<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>

hadoop conf in cloud3 (slave):

core-site.xml:

<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>

hdfs-site.xml:

<property>
<name>dfs.data.dir</name>
<value>/home/cloud/dfs/data</value>
</property>

mapred-site:xml

<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 19, '11 at 8:38a
activeFeb 19, '11 at 8:38a
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Thomas Anderson: 1 post

People

Translate

site design / logo © 2022 Grokbase