FAQ
I have set the MapReduce Max Java Heap Size to be 2GB and I/O Sort Memory
Buffer to 512MB under the Gateway (Base) configurations. I left the
settings under the TaskTracker (Base) configuration as default. After
restarting the mapreduce service in CM4.5 and deploying client
configurations, I see that across the cluster the settings are 1GB and
256MB respectively in /etc/hadoop/conf/mapred-site.xml. I guess this is to
be expected. But, when I check the gateway node, I expected the settings to
be 2GB and 512MB respectively, but they are not. They are 1GB and 256MB
respectively. Can someone tell me if there is something wrong here?

Thanks,
Ben

Search Discussions

  • Darren Lo at Mar 15, 2013 at 6:09 pm
    Hi Ben,

    /etc/hadoop/conf should reflect your gateway configs once you deploy client
    configs. All roles act as gateways, so this should be the on your TT node
    as well as your node with just a gateway. TT doesn't use the config in this
    directory.

    I'm assuming you meant you changed the gateway config "MapReduce Child Java
    Maximum Heap Size" to 2GB.

    I tried this in CM4.5, and it correctly gave me the following in
    /etc/hadoop/conf/mapred-site.xml after deploying client configs:
    <property>
    <name>mapred.child.java.opts</name>
    <value> -Xmx2147483648</value>
    </property>
    ...
    <property>
    <name>io.sort.mb</name>
    <value>512</value>
    </property>

    It sounds like you did everything correctly, but I'm not able to reproduce
    your issue. You may want to double-check your alternatives priority for
    gateways for hdfs, mapreduce, and yarn. Normally mapreduce wins with 92
    priority. You can also run "alternatives --display hadoop-conf" to make
    sure it's picking up the right one.

    Can you try deploying client configs and trying again? You can also try
    downloading client configs and making sure it looks correct (also tested
    this worked for me).

    Thanks,
    Darren

    On Thu, Mar 14, 2013 at 7:36 PM, Benjamin Kim wrote:

    I have set the MapReduce Max Java Heap Size to be 2GB and I/O Sort Memory
    Buffer to 512MB under the Gateway (Base) configurations. I left the
    settings under the TaskTracker (Base) configuration as default. After
    restarting the mapreduce service in CM4.5 and deploying client
    configurations, I see that across the cluster the settings are 1GB and
    256MB respectively in /etc/hadoop/conf/mapred-site.xml. I guess this is to
    be expected. But, when I check the gateway node, I expected the settings to
    be 2GB and 512MB respectively, but they are not. They are 1GB and 256MB
    respectively. Can someone tell me if there is something wrong here?

    Thanks,
    Ben


    --
    Thanks,
    Darren
  • Benjamin Kim at Mar 15, 2013 at 6:39 pm
    Hi Darren,

    I was able to fix this. You are correct in saying that it should also be on
    the TT nodes. I fixed this by entering the values directly into the
    configuration text boxes and saving them. Before, I was using the
    overrides, and this didn't work.

    Now, I am facing another issue. I am trying to increase the java heap size
    for datanodes under hdfs > datanodes > resource management by directly
    entering the 2GB value. I save the configuration changes and restart the
    hdfs service. I ssh into any datanode and see that the hadoop-env.sh didn't
    change. I do a "ps -ef | grep java" and see that parameter for -Xmx is
    still 1000m. Am I missing something here?

    Thanks,
    Ben
    On Friday, March 15, 2013 11:09:38 AM UTC-7, Darren Lo wrote:

    Hi Ben,

    /etc/hadoop/conf should reflect your gateway configs once you deploy
    client configs. All roles act as gateways, so this should be the on your TT
    node as well as your node with just a gateway. TT doesn't use the config in
    this directory.

    I'm assuming you meant you changed the gateway config "MapReduce Child
    Java Maximum Heap Size" to 2GB.

    I tried this in CM4.5, and it correctly gave me the following in
    /etc/hadoop/conf/mapred-site.xml after deploying client configs:
    <property>
    <name>mapred.child.java.opts</name>
    <value> -Xmx2147483648</value>
    </property>
    ...
    <property>
    <name>io.sort.mb</name>
    <value>512</value>
    </property>

    It sounds like you did everything correctly, but I'm not able to reproduce
    your issue. You may want to double-check your alternatives priority for
    gateways for hdfs, mapreduce, and yarn. Normally mapreduce wins with 92
    priority. You can also run "alternatives --display hadoop-conf" to make
    sure it's picking up the right one.

    Can you try deploying client configs and trying again? You can also try
    downloading client configs and making sure it looks correct (also tested
    this worked for me).

    Thanks,
    Darren


    On Thu, Mar 14, 2013 at 7:36 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    I have set the MapReduce Max Java Heap Size to be 2GB and I/O Sort Memory
    Buffer to 512MB under the Gateway (Base) configurations. I left the
    settings under the TaskTracker (Base) configuration as default. After
    restarting the mapreduce service in CM4.5 and deploying client
    configurations, I see that across the cluster the settings are 1GB and
    256MB respectively in /etc/hadoop/conf/mapred-site.xml. I guess this is to
    be expected. But, when I check the gateway node, I expected the settings to
    be 2GB and 512MB respectively, but they are not. They are 1GB and 256MB
    respectively. Can someone tell me if there is something wrong here?

    Thanks,
    Ben


    --
    Thanks,
    Darren
  • Benjamin Kim at Mar 15, 2013 at 6:49 pm
    Darren,

    Please disregard my comments about the datanode configuration settings. I
    was looking in the wrong place. I see that change in the process now. I had
    to read further down the process string.

    -Xmx2147483648

    Thanks,
    Ben
    On Friday, March 15, 2013 11:09:38 AM UTC-7, Darren Lo wrote:

    Hi Ben,

    /etc/hadoop/conf should reflect your gateway configs once you deploy
    client configs. All roles act as gateways, so this should be the on your TT
    node as well as your node with just a gateway. TT doesn't use the config in
    this directory.

    I'm assuming you meant you changed the gateway config "MapReduce Child
    Java Maximum Heap Size" to 2GB.

    I tried this in CM4.5, and it correctly gave me the following in
    /etc/hadoop/conf/mapred-site.xml after deploying client configs:
    <property>
    <name>mapred.child.java.opts</name>
    <value> -Xmx2147483648</value>
    </property>
    ...
    <property>
    <name>io.sort.mb</name>
    <value>512</value>
    </property>

    It sounds like you did everything correctly, but I'm not able to reproduce
    your issue. You may want to double-check your alternatives priority for
    gateways for hdfs, mapreduce, and yarn. Normally mapreduce wins with 92
    priority. You can also run "alternatives --display hadoop-conf" to make
    sure it's picking up the right one.

    Can you try deploying client configs and trying again? You can also try
    downloading client configs and making sure it looks correct (also tested
    this worked for me).

    Thanks,
    Darren


    On Thu, Mar 14, 2013 at 7:36 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    I have set the MapReduce Max Java Heap Size to be 2GB and I/O Sort Memory
    Buffer to 512MB under the Gateway (Base) configurations. I left the
    settings under the TaskTracker (Base) configuration as default. After
    restarting the mapreduce service in CM4.5 and deploying client
    configurations, I see that across the cluster the settings are 1GB and
    256MB respectively in /etc/hadoop/conf/mapred-site.xml. I guess this is to
    be expected. But, when I check the gateway node, I expected the settings to
    be 2GB and 512MB respectively, but they are not. They are 1GB and 256MB
    respectively. Can someone tell me if there is something wrong here?

    Thanks,
    Ben


    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 15, '13 at 2:37a
activeMar 15, '13 at 6:49p
posts4
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Benjamin Kim: 3 posts Darren Lo: 1 post

People

Translate

site design / logo © 2022 Grokbase