[Hadoop-common-user] Two questions.

I don't think the first two options can work, even you stop the tasktracker these to-be-retired nodes are still connected to the namenode.
Option 3 can work. You only need to add this exclude file on the namenode, and it is an regular file. Add a key named dfs.hosts.exclude to your conf/hadoop-site.xml file,The value associated with this key provides the full path to a file on the NameNode's local file system which contains a list of machines which are not permitted to connect to HDFS.

Then you can run the command bin/hadoop dfsadmin -refreshNodes, then the cluster will decommission the nodes in the exclude file.This might take a period of time as the cluster need to move data from those retired nodes to left nodes.

After this you can use these retired nodes as a new cluster.But remember to remove those nodes from the slave nodes file and you can delete the exclude file afterward.


2010-11-04



shangan




发件人： Raj V
发送时间： 2010-11-04 10:05:44
收件人： common-user
抄送：
主题： Two questions.

1. I have a 512 node cluster. I need to have 32 nodes do something else. They
can be datanodes but I cannot run any map or reduce jobs on them. So I see three
options.
1. Stop the tasktracker on those nodes. leave the datanode running.
2. Set mapred.tasktracker.reduce.tasks.maximum and
mapred.tasktracker.map.tasks.maximum to 0 on these nodes and make these final.
3. Use the parameter mapred.hosts.exclude.
I am assuming that any of the three methods would work. To start with, I went
with option 3. I used a local file /home/hadoop/myjob.exclude and the file
myjob.exclude had the hostname of one host per line ( hadoop-480 .. hadoop-511.
But I see both map and reduce jobs being scheduled to all the 511 nodes.
I understand there is an inherent inefficieny by running only the data node on
these 32 nodess.
Here are my questions.
1. Will all three methods work?
2. If I choose method 3, does this file exist as a dfs file or a regular file.
If regular file , does it need to exist on all the nodes or only the node where
teh job is submitted?
Many thanks in advance/
Raj
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5574 (20101029) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com

Option 1 = good

Sent from my mobile. Please excuse the typos.

Both option 1 and 3 will work.

The parameter mapred.hosts.exclude has existed in documentation for many
versions, but I do not believe it has ever been implemented in the
actual code.

If you do not want to run tasks, I'd recommend not running the
tasktracker. It will take up some resources, even if it does not run
any tasks.

David

mradmin -refreshNodes was added in 0.21. So code definitely exists for it.

That said, if you aren't using it now, probably better to just kill the tasktracker.