FAQ
Hi all,

I have CDH3 11 node cluster setup by Cloudera Manager.
In order to tune my cluster in my way, I changed the *
mapred.child.java.opts*, *mapred.map.child.java.opts* and *
mapred.reduce.child.java.opts* properties values to -Xmx2048m, -Xmx1024m
and -Xmx1024m respectively from CM MapReduce configuration page under the
default as well as client settings section.
Then I deployed client configuration to the cluster and restarted my CDH3
cluster.

Now, when I am running any job and from JobTracker Web UI I go to the Map
Task Attempts, I can see the following warning messages:
**
*2012-09-06 11:03:25,004 WARN org.apache.hadoop.conf.Configuration:
/mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
attempt to override final parameter: mapred.child.java.opts; Ignoring.
2012-09-06 11:03:25,006 WARN org.apache.hadoop.conf.Configuration:
/mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
attempt to override final parameter: mapred.map.child.java.opts; Ignoring.
2012-09-06 11:03:25,007 WARN org.apache.hadoop.conf.Configuration:
/mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
attempt to override final parameter: mapred.reduce.child.java.opts;
Ignoring.*
**
Although, in the Job.xml, I can see the values reflecting to what I set.
Is it not accepting the values I have specified? In that case, how and
where can I set these values inside CM so that it reflects to my job?

Thanks,
Gaurav Dasgupta

Search Discussions

  • Harsh J at Sep 8, 2012 at 1:11 pm
    There is a feature in CM that lets admins hard-clamp some client-end
    properties such that they are never overridable. It seems like you
    have activated these. These options are marked in their names with
    "(Client Override)" and are under the service settings rather than the
    client ones. Please see such fields' descriptions to get a more
    clearer idea for what they are for.

    I'd first make sure that none of these "(Client Override)" settings
    pertaining to any "Java Opts" configurations are set. After removing
    all of those and setting only the configs for java opts under the
    mapreduce1 -> Configuration -> Client (left sidebar classifying
    option) -> Performance config set, you should not see this issue of
    overriding being prevented anymore.

    (You'll need to restart MR after reverting these changes)
    On Thu, Sep 6, 2012 at 10:07 PM, Gaurav Dasgupta wrote:
    Hi all,

    I have CDH3 11 node cluster setup by Cloudera Manager.
    In order to tune my cluster in my way, I changed the
    mapred.child.java.opts, mapred.map.child.java.opts and
    mapred.reduce.child.java.opts properties values to -Xmx2048m, -Xmx1024m and
    -Xmx1024m respectively from CM MapReduce configuration page under the
    default as well as client settings section.
    Then I deployed client configuration to the cluster and restarted my CDH3
    cluster.

    Now, when I am running any job and from JobTracker Web UI I go to the Map
    Task Attempts, I can see the following warning messages:

    2012-09-06 11:03:25,004 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.child.java.opts; Ignoring.
    2012-09-06 11:03:25,006 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.map.child.java.opts; Ignoring.
    2012-09-06 11:03:25,007 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.reduce.child.java.opts;
    Ignoring.

    Although, in the Job.xml, I can see the values reflecting to what I set.
    Is it not accepting the values I have specified? In that case, how and where
    can I set these values inside CM so that it reflects to my job?

    Thanks,
    Gaurav Dasgupta


    --
    Harsh J
  • Ben Nantaco at May 7, 2014 at 8:49 am
    My apologies for replying to the 2-year-old post.
    I'm using CM 4.6, and have similar issue with Gaurav.

    Harsh's hint about "Client Override" does help me understand what's it all
    about.
    Now I want to set the default heap size of tasks to 1GB, and let clients
    override it.
    but Im not able to find Configuration -> Client on CM4.6 to set the default
    heap size.

    Instead, I found Configuration -> Gateway having "Java Opts" related
    configurations.
    Changing these values changes what's in the mapred-site.xml on each server,
    but the jobs run from Cloudera Hue wouldn't take these settings.

    So, what would be the best way to set the default heap size of tasks to
    1GB, and let both HUE and CLI clients override it?

    Thank you for all of your supports.

    Best,
    Ben
    On Saturday, September 8, 2012 10:04:43 PM UTC+9, Harsh J wrote:

    There is a feature in CM that lets admins hard-clamp some client-end
    properties such that they are never overridable. It seems like you
    have activated these. These options are marked in their names with
    "(Client Override)" and are under the service settings rather than the
    client ones. Please see such fields' descriptions to get a more
    clearer idea for what they are for.

    I'd first make sure that none of these "(Client Override)" settings
    pertaining to any "Java Opts" configurations are set. After removing
    all of those and setting only the configs for java opts under the
    mapreduce1 -> Configuration -> Client (left sidebar classifying
    option) -> Performance config set, you should not see this issue of
    overriding being prevented anymore.

    (You'll need to restart MR after reverting these changes)
    On Thu, Sep 6, 2012 at 10:07 PM, Gaurav Dasgupta wrote:
    Hi all,

    I have CDH3 11 node cluster setup by Cloudera Manager.
    In order to tune my cluster in my way, I changed the
    mapred.child.java.opts, mapred.map.child.java.opts and
    mapred.reduce.child.java.opts properties values to -Xmx2048m, -Xmx1024m and
    -Xmx1024m respectively from CM MapReduce configuration page under the
    default as well as client settings section.
    Then I deployed client configuration to the cluster and restarted my CDH3
    cluster.

    Now, when I am running any job and from JobTracker Web UI I go to the Map
    Task Attempts, I can see the following warning messages:

    2012-09-06 11:03:25,004 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.child.java.opts; Ignoring.
    2012-09-06 11:03:25,006 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.map.child.java.opts; Ignoring.
    2012-09-06 11:03:25,007 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.reduce.child.java.opts;
    Ignoring.

    Although, in the Job.xml, I can see the values reflecting to what I set.
    Is it not accepting the values I have specified? In that case, how and where
    can I set these values inside CM so that it reflects to my job?

    Thanks,
    Gaurav Dasgupta


    --
    Harsh J
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Darren Lo at May 7, 2014 at 4:43 pm
    Hi Ben,

    It sounds like you want to set configuration in the Gateway (NOT client
    override) so that MR clients use 1 GB by default, but can change that value
    as they see fit. The relevant property (assuming you want the same for map
    and reduce tasks) is "MapReduce Child Java Maximum Heap Size", under
    Gateway -> Resource Management. You can also use the search bar in the
    upper left of the config page to find it. You should ensure that you don't
    have any of the relevant Client Override parameters set, and that you don't
    have map or reduce memory / java options set, since those will take
    precedence.

    Remember to re-deploy client configuration (from the menu next to your
    cluster name on the home page) and restart hue and Hive to make sure these
    changes take effect.

    You should find that your MapReduce client configuration (Gateway
    configuration) is used by Hue. I'm not quite sure how to get Hue to
    override these settings. Hue experts may be able to chime in.

    Thanks,
    Darren

    On Wed, May 7, 2014 at 1:49 AM, Ben Nantaco wrote:

    My apologies for replying to the 2-year-old post.
    I'm using CM 4.6, and have similar issue with Gaurav.

    Harsh's hint about "Client Override" does help me understand what's it all
    about.
    Now I want to set the default heap size of tasks to 1GB, and let clients
    override it.
    but Im not able to find Configuration -> Client on CM4.6 to set the
    default heap size.

    Instead, I found Configuration -> Gateway having "Java Opts" related
    configurations.
    Changing these values changes what's in the mapred-site.xml on each
    server,
    but the jobs run from Cloudera Hue wouldn't take these settings.

    So, what would be the best way to set the default heap size of tasks to
    1GB, and let both HUE and CLI clients override it?

    Thank you for all of your supports.

    Best,
    Ben
    On Saturday, September 8, 2012 10:04:43 PM UTC+9, Harsh J wrote:

    There is a feature in CM that lets admins hard-clamp some client-end
    properties such that they are never overridable. It seems like you
    have activated these. These options are marked in their names with
    "(Client Override)" and are under the service settings rather than the
    client ones. Please see such fields' descriptions to get a more
    clearer idea for what they are for.

    I'd first make sure that none of these "(Client Override)" settings
    pertaining to any "Java Opts" configurations are set. After removing
    all of those and setting only the configs for java opts under the
    mapreduce1 -> Configuration -> Client (left sidebar classifying
    option) -> Performance config set, you should not see this issue of
    overriding being prevented anymore.

    (You'll need to restart MR after reverting these changes)

    On Thu, Sep 6, 2012 at 10:07 PM, Gaurav Dasgupta <gdsa...@gmail.com>
    wrote:
    Hi all,

    I have CDH3 11 node cluster setup by Cloudera Manager.
    In order to tune my cluster in my way, I changed the
    mapred.child.java.opts, mapred.map.child.java.opts and
    mapred.reduce.child.java.opts properties values to -Xmx2048m, -Xmx1024m and
    -Xmx1024m respectively from CM MapReduce configuration page under the
    default as well as client settings section.
    Then I deployed client configuration to the cluster and restarted my CDH3
    cluster.

    Now, when I am running any job and from JobTracker Web UI I go to the Map
    Task Attempts, I can see the following warning messages:

    2012-09-06 11:03:25,004 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.child.java.opts; Ignoring.
    2012-09-06 11:03:25,006 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.map.child.java.opts; Ignoring.
    2012-09-06 11:03:25,007 WARN org.apache.hadoop.conf.Configuration:
    /mnt/data/mapred/local/taskTracker/hdfs/jobcache/job_201209061049_0003/job.xml:a
    attempt to override final parameter: mapred.reduce.child.java.opts;
    Ignoring.

    Although, in the Job.xml, I can see the values reflecting to what I set.
    Is it not accepting the values I have specified? In that case, how and where
    can I set these values inside CM so that it reflects to my job?

    Thanks,
    Gaurav Dasgupta


    --
    Harsh J
    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedSep 6, '12 at 4:37p
activeMay 7, '14 at 4:43p
posts4
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase