FAQ
1. I have a 512 node cluster. I need to have 32 nodes do something else. They
can be datanodes but I cannot run any map or reduce jobs on them. So I see three
options.

1. Stop the tasktracker on those nodes. leave the datanode running.
2. Set mapred.tasktracker.reduce.tasks.maximum and
mapred.tasktracker.map.tasks.maximum to 0 on these nodes and make these final.
3. Use the parameter mapred.hosts.exclude.

I am assuming that any of the three methods would work. To start with, I went
with option 3. I used a local file /home/hadoop/myjob.exclude and the file
myjob.exclude had the hostname of one host per line ( hadoop-480 .. hadoop-511.
But I see both map and reduce jobs being scheduled to all the 511 nodes.


I understand there is an inherent inefficieny by running only the data node on
these 32 nodess.

Here are my questions.

1. Will all three methods work?
2. If I choose method 3, does this file exist as a dfs file or a regular file.
If regular file , does it need to exist on all the nodes or only the node where
teh job is submitted?

Many thanks in advance/

Raj

Search Discussions

  • Shangan at Nov 4, 2010 at 2:27 am
    I don't think the first two options can work, even you stop the tasktracker these to-be-retired nodes are still connected to the namenode.
    Option 3 can work. You only need to add this exclude file on the namenode, and it is an regular file. Add a key named dfs.hosts.exclude to your conf/hadoop-site.xml file,The value associated with this key provides the full path to a file on the NameNode's local file system which contains a list of machines which are not permitted to connect to HDFS.

    Then you can run the command bin/hadoop dfsadmin -refreshNodes, then the cluster will decommission the nodes in the exclude file.This might take a period of time as the cluster need to move data from those retired nodes to left nodes.

    After this you can use these retired nodes as a new cluster.But remember to remove those nodes from the slave nodes file and you can delete the exclude file afterward.


    2010-11-04



    shangan




    发件人: Raj V
    发送时间: 2010-11-04 10:05:44
    收件人: common-user
    抄送:
    主题: Two questions.

    1. I have a 512 node cluster. I need to have 32 nodes do something else. They
    can be datanodes but I cannot run any map or reduce jobs on them. So I see three
    options.
    1. Stop the tasktracker on those nodes. leave the datanode running.
    2. Set mapred.tasktracker.reduce.tasks.maximum and
    mapred.tasktracker.map.tasks.maximum to 0 on these nodes and make these final.
    3. Use the parameter mapred.hosts.exclude.
    I am assuming that any of the three methods would work. To start with, I went
    with option 3. I used a local file /home/hadoop/myjob.exclude and the file
    myjob.exclude had the hostname of one host per line ( hadoop-480 .. hadoop-511.
    But I see both map and reduce jobs being scheduled to all the 511 nodes.
    I understand there is an inherent inefficieny by running only the data node on
    these 32 nodess.
    Here are my questions.
    1. Will all three methods work?
    2. If I choose method 3, does this file exist as a dfs file or a regular file.
    If regular file , does it need to exist on all the nodes or only the node where
    teh job is submitted?
    Many thanks in advance/
    Raj
    __________ Information from ESET NOD32 Antivirus, version of virus signature database 5574 (20101029) __________
    The message was checked by ESET NOD32 Antivirus.
    http://www.eset.com
  • James Seigel at Nov 4, 2010 at 2:29 am
    Option 1 = good

    Sent from my mobile. Please excuse the typos.
    On 2010-11-03, at 8:27 PM, "shangan" wrote:

    I don't think the first two options can work, even you stop the tasktracker these to-be-retired nodes are still connected to the namenode.
    Option 3 can work. You only need to add this exclude file on the namenode, and it is an regular file. Add a key named dfs.hosts.exclude to your conf/hadoop-site.xml file,The value associated with this key provides the full path to a file on the NameNode's local file system which contains a list of machines which are not permitted to connect to HDFS.

    Then you can run the command bin/hadoop dfsadmin -refreshNodes, then the cluster will decommission the nodes in the exclude file.This might take a period of time as the cluster need to move data from those retired nodes to left nodes.

    After this you can use these retired nodes as a new cluster.But remember to remove those nodes from the slave nodes file and you can delete the exclude file afterward.


    2010-11-04



    shangan




    发件人: Raj V
    发送时间: 2010-11-04 10:05:44
    收件人: common-user
    抄送:
    主题: Two questions.

    1. I have a 512 node cluster. I need to have 32 nodes do something else. They
    can be datanodes but I cannot run any map or reduce jobs on them. So I see three
    options.
    1. Stop the tasktracker on those nodes. leave the datanode running.
    2. Set mapred.tasktracker.reduce.tasks.maximum and
    mapred.tasktracker.map.tasks.maximum to 0 on these nodes and make these final.
    3. Use the parameter mapred.hosts.exclude.
    I am assuming that any of the three methods would work. To start with, I went
    with option 3. I used a local file /home/hadoop/myjob.exclude and the file
    myjob.exclude had the hostname of one host per line ( hadoop-480 .. hadoop-511.
    But I see both map and reduce jobs being scheduled to all the 511 nodes.
    I understand there is an inherent inefficieny by running only the data node on
    these 32 nodess.
    Here are my questions.
    1. Will all three methods work?
    2. If I choose method 3, does this file exist as a dfs file or a regular file.
    If regular file , does it need to exist on all the nodes or only the node where
    teh job is submitted?
    Many thanks in advance/
    Raj
    __________ Information from ESET NOD32 Antivirus, version of virus signature database 5574 (20101029) __________
    The message was checked by ESET NOD32 Antivirus.
    http://www.eset.com
  • He Chen at Nov 4, 2010 at 2:39 am
    Both option 1 and 3 will work.
    On Wed, Nov 3, 2010 at 9:28 PM, James Seigel wrote:

    Option 1 = good

    Sent from my mobile. Please excuse the typos.
    On 2010-11-03, at 8:27 PM, "shangan" wrote:

    I don't think the first two options can work, even you stop the
    tasktracker these to-be-retired nodes are still connected to the namenode.
    Option 3 can work. You only need to add this exclude file on the
    namenode, and it is an regular file. Add a key named dfs.hosts.exclude to
    your conf/hadoop-site.xml file,The value associated with this key provides
    the full path to a file on the NameNode's local file system which contains a
    list of machines which are not permitted to connect to HDFS.
    Then you can run the command bin/hadoop dfsadmin -refreshNodes, then the
    cluster will decommission the nodes in the exclude file.This might take a
    period of time as the cluster need to move data from those retired nodes to
    left nodes.
    After this you can use these retired nodes as a new cluster.But remember
    to remove those nodes from the slave nodes file and you can delete the
    exclude file afterward.

    2010-11-04



    shangan




    发件人: Raj V
    发送时间: 2010-11-04 10:05:44
    收件人: common-user
    抄送:
    主题: Two questions.

    1. I have a 512 node cluster. I need to have 32 nodes do something else. They
    can be datanodes but I cannot run any map or reduce jobs on them. So I see three
    options.
    1. Stop the tasktracker on those nodes. leave the datanode running.
    2. Set mapred.tasktracker.reduce.tasks.maximum and
    mapred.tasktracker.map.tasks.maximum to 0 on these nodes and make these final.
    3. Use the parameter mapred.hosts.exclude.
    I am assuming that any of the three methods would work. To start with, I went
    with option 3. I used a local file /home/hadoop/myjob.exclude and the file
    myjob.exclude had the hostname of one host per line ( hadoop-480 ..
    hadoop-511.
    But I see both map and reduce jobs being scheduled to all the 511 nodes.
    I understand there is an inherent inefficieny by running only the data node on
    these 32 nodess.
    Here are my questions.
    1. Will all three methods work?
    2. If I choose method 3, does this file exist as a dfs file or a regular file.
    If regular file , does it need to exist on all the nodes or only the node where
    teh job is submitted?
    Many thanks in advance/
    Raj
    __________ Information from ESET NOD32 Antivirus, version of virus
    signature database 5574 (20101029) __________
    The message was checked by ESET NOD32 Antivirus.
    http://www.eset.com
  • David B. Ritch at Nov 4, 2010 at 2:27 am
    The parameter mapred.hosts.exclude has existed in documentation for many
    versions, but I do not believe it has ever been implemented in the
    actual code.

    If you do not want to run tasks, I'd recommend not running the
    tasktracker. It will take up some resources, even if it does not run
    any tasks.

    David
  • Allen Wittenauer at Nov 4, 2010 at 9:27 pm

    On Nov 3, 2010, at 7:27 PM, David B. Ritch wrote:

    The parameter mapred.hosts.exclude has existed in documentation for many
    versions, but I do not believe it has ever been implemented in the
    actual code.
    mradmin -refreshNodes was added in 0.21. So code definitely exists for it.

    That said, if you aren't using it now, probably better to just kill the tasktracker.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 4, '10 at 2:05a
activeNov 4, '10 at 9:27p
posts6
users6
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase