I don't think the first two options can work, even you stop the tasktracker these to-be-retired nodes are still connected to the namenode.
Option 3 can work. You only need to add this exclude file on the namenode, and it is an regular file. Add a key named dfs.hosts.exclude to your conf/hadoop-site.xml file,The value associated with this key provides the full path to a file on the NameNode's local file system which contains a list of machines which are not permitted to connect to HDFS.
Then you can run the command bin/hadoop dfsadmin -refreshNodes, then the cluster will decommission the nodes in the exclude file.This might take a period of time as the cluster need to move data from those retired nodes to left nodes.
After this you can use these retired nodes as a new cluster.But remember to remove those nodes from the slave nodes file and you can delete the exclude file afterward.
发件人： Raj V
发送时间： 2010-11-04 10:05:44
主题： Two questions.
1. I have a 512 node cluster. I need to have 32 nodes do something else. They
can be datanodes but I cannot run any map or reduce jobs on them. So I see three
1. Stop the tasktracker on those nodes. leave the datanode running.
2. Set mapred.tasktracker.reduce.tasks.maximum and
mapred.tasktracker.map.tasks.maximum to 0 on these nodes and make these final.
3. Use the parameter mapred.hosts.exclude.
I am assuming that any of the three methods would work. To start with, I went
with option 3. I used a local file /home/hadoop/myjob.exclude and the file
myjob.exclude had the hostname of one host per line ( hadoop-480 .. hadoop-511.
But I see both map and reduce jobs being scheduled to all the 511 nodes.
I understand there is an inherent inefficieny by running only the data node on
these 32 nodess.
Here are my questions.
1. Will all three methods work?
2. If I choose method 3, does this file exist as a dfs file or a regular file.
If regular file , does it need to exist on all the nodes or only the node where
teh job is submitted?
Many thanks in advance/
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5574 (20101029) __________
The message was checked by ESET NOD32 Antivirus.http://www.eset.com