Grokbase › Groups › Hadoop › scm-users › November 2012
FAQ
Hey Mark,

Moving this to [email protected] as this is CM specific.

These configs are service-side configs, used only by the TaskTracker.
They are irrelevant to a client and hence will not appear in the
generated client configurations.

However, if you wish to see service configurations in any case, visit
in the UI something along these paths: Home -> mapreduce1 -> Any
tasktracker -> Process tab -> Expand the "Show" under Configuration
Files -> Click on mapred-site.xml.
On Mon, Nov 19, 2012 at 8:50 PM, Mark wrote:
Trying to find these two settings as Cloudera Manager tells me they are set
to a non-default value (8,24). But, not finding them in the normal config
locations (/etc/hadoop/conf/*-site.xml). Looked on the client, DNs, JT, NN
etc, but no sign of these settings. Just trying to understand where these
configs could possibly live.

Thanks

--



--
Harsh J

Search Discussions

  • Ian Wrigley at Nov 20, 2012 at 12:51 am ⇧
    Hi Mark

    You'll find the settings in /var/run/cloudera-scm-agent/process. There's a subdirectory per role under that directory. So on my cluster, for example, on a slave node running the TaskTracker daemon I see
    /var/run/cloudera-scm-agent/process/126-mapreduce-TASKTRACKER
    and under that directory the *-site.xml files.

    Cloudera Manager pushes the configs out to the nodes in directories under /var/run/cloudera-scm-agent, and the agent process on the nodes starts the daemon up using the relevant subdirectory as its config directory.

    (Note: don't edit those files directly on the nodes...)

    Ian.
    On Nov 19, 2012, at 4:43 PM, Mark wrote:

    Thanks so much for the reply Harsh. That location in CM is very helpful for seeing configuration settings. Curious as well about where this setting is on the TT itself (where does CM query from). I'm not seeing in the mapred-site.xml files I am finding on the TT. Performing a thorough search now on the TT - grep -r -H "mapred.tasktracker.map.tasks.maximum" /etc/.


    On Monday, November 19, 2012 5:46:52 PM UTC-5, Harsh J wrote:
    Hey Mark,

    Moving this to [email protected] as this is CM specific.

    These configs are service-side configs, used only by the TaskTracker.
    They are irrelevant to a client and hence will not appear in the
    generated client configurations.

    However, if you wish to see service configurations in any case, visit
    in the UI something along these paths: Home -> mapreduce1 -> Any
    tasktracker -> Process tab -> Expand the "Show" under Configuration
    Files -> Click on mapred-site.xml.
    On Mon, Nov 19, 2012 at 8:50 PM, Mark wrote:
    Trying to find these two settings as Cloudera Manager tells me they are set
    to a non-default value (8,24). But, not finding them in the normal config
    locations (/etc/hadoop/conf/*-site.xml). Looked on the client, DNs, JT, NN
    etc, but no sign of these settings. Just trying to understand where these
    configs could possibly live.

    Thanks

    --



    --
    Harsh J

    --


    ---
    Ian Wrigley
    Curriculum Manager
    Cloudera, Inc
    Cell: (323) 819 4075
  • Ian Wrigley at Nov 20, 2012 at 1:18 am ⇧
    When a setting is changed in CM, new config values are sent to the client when the daemon in question is restarted. (You'll see, for example, the MapReduce service reporting that it is 'running with outdated configuration' after you've made a config change until you restart the relevant daemons.)
    On Nov 19, 2012, at 5:12 PM, Mark wrote:

    Thanks Ian. Found it and that makes sense. Are the multiple dameon directories (##-hdfs-DATANODE, ##-mapreduce-TASKTRACKER) under /var/run/cloudera-scm-agent/process due to a continuous config refresh by CM or something?

    Thanks again.

    On Monday, November 19, 2012 7:51:03 PM UTC-5, Ian Wrigley wrote:
    Hi Mark

    You'll find the settings in /var/run/cloudera-scm-agent/process. There's a subdirectory per role under that directory. So on my cluster, for example, on a slave node running the TaskTracker daemon I see
    /var/run/cloudera-scm-agent/process/126-mapreduce-TASKTRACKER
    and under that directory the *-site.xml files.

    Cloudera Manager pushes the configs out to the nodes in directories under /var/run/cloudera-scm-agent, and the agent process on the nodes starts the daemon up using the relevant subdirectory as its config directory.

    (Note: don't edit those files directly on the nodes...)

    Ian.
    On Nov 19, 2012, at 4:43 PM, Mark wrote:

    Thanks so much for the reply Harsh. That location in CM is very helpful for seeing configuration settings. Curious as well about where this setting is on the TT itself (where does CM query from). I'm not seeing in the mapred-site.xml files I am finding on the TT. Performing a thorough search now on the TT - grep -r -H "mapred.tasktracker.map.tasks.maximum" /etc/.


    On Monday, November 19, 2012 5:46:52 PM UTC-5, Harsh J wrote:
    Hey Mark,

    Moving this to [email protected] as this is CM specific.

    These configs are service-side configs, used only by the TaskTracker.
    They are irrelevant to a client and hence will not appear in the
    generated client configurations.

    However, if you wish to see service configurations in any case, visit
    in the UI something along these paths: Home -> mapreduce1 -> Any
    tasktracker -> Process tab -> Expand the "Show" under Configuration
    Files -> Click on mapred-site.xml.
    On Mon, Nov 19, 2012 at 8:50 PM, Mark wrote:
    Trying to find these two settings as Cloudera Manager tells me they are set
    to a non-default value (8,24). But, not finding them in the normal config
    locations (/etc/hadoop/conf/*-site.xml). Looked on the client, DNs, JT, NN
    etc, but no sign of these settings. Just trying to understand where these
    configs could possibly live.

    Thanks

    --



    --
    Harsh J

    --


    ---
    Ian Wrigley
    Curriculum Manager
    Cloudera, Inc
    Cell: (323) 819 4075


    --


    ---
    Ian Wrigley
    Curriculum Manager
    Cloudera, Inc
    Cell: (323) 819 4075

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedNov 19, '12 at 10:46p
activeNov 20, '12 at 1:18a
posts3
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Ian Wrigley: 2 posts Harsh J: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase