Grokbase › Groups › Hadoop › scm-users › December 2013
FAQ
I have successfully install CDH4 using cloudera manager. Nodes are working
fine, also I can upload file to HDFS and browse that file.

But now I want to run my map reduce program, which need a custom xml and
properties files. Also I want to change the default hdfs and mapred port
from 8020 and 8021 to something else. Therefore, to make this properties
file and configuration available I have done below steps:

I have create a directory say clientconf in /etc/hadoop directory
Run below command:
*alternatives --set hadoop-conf /etc/hadoop/clientconf*

And put the updated core-site.xml and mapred-site.xml in
/etc/hadoop/clientconf directory. Also put my project dependent xml and
properties files in /etc/hadoop/clientconf

Then from cloudera manager UI, I click on the "deploy client
configuration". But it has no effect on the actual installation and also my
M/R job can't found the custom xml or properties files in
/var/run/cloudera-scm-agent/process/<processid>-mapreduce-TASKTRACKER
folder.

Do I miss something?

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Prodiptag at Dec 6, 2013 at 6:29 am ⇧
    my custom mapreduce job can be run by putting custom xml and properties
    files
    in /var/run/cloudera-scm-agent/process/<processid>-mapreduce-TASKTRACKER
    folder in each node.

    But is it the correct way of deploying my project specific custom xml and
    properties files to each node?
    On Friday, December 6, 2013 9:49:58 AM UTC+5:30, prod...@2pirad.com wrote:

    I have successfully install CDH4 using cloudera manager. Nodes are working
    fine, also I can upload file to HDFS and browse that file.

    But now I want to run my map reduce program, which need a custom xml and
    properties files. Also I want to change the default hdfs and mapred port
    from 8020 and 8021 to something else. Therefore, to make this properties
    file and configuration available I have done below steps:

    I have create a directory say clientconf in /etc/hadoop directory
    Run below command:
    *alternatives --set hadoop-conf /etc/hadoop/clientconf*

    And put the updated core-site.xml and mapred-site.xml in
    /etc/hadoop/clientconf directory. Also put my project dependent xml and
    properties files in /etc/hadoop/clientconf

    Then from cloudera manager UI, I click on the "deploy client
    configuration". But it has no effect on the actual installation and also my
    M/R job can't found the custom xml or properties files in
    /var/run/cloudera-scm-agent/process/<processid>-mapreduce-TASKTRACKER
    folder.

    Do I miss something?
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Philip Zeyliger at Dec 6, 2013 at 5:31 pm ⇧
    Typically, users either include their XML and properties files as resources
    of the jar that contains their MR job, or they use the Distributed Cache (
    https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#DistributedCache)
    which is a fancy name for shipping some additional files (whether they be
    jars or other files) along with your MR job. Centrally placing these
    things on individual nodes doesn't scale with multiple users and causes
    upgrade headaches.

    The default ports should be configured in CM, and your MR job should be
    able to get at them via new Configuration().get("...").

    -- Philip

    On Thu, Dec 5, 2013 at 8:19 PM, wrote:

    I have successfully install CDH4 using cloudera manager. Nodes are working
    fine, also I can upload file to HDFS and browse that file.

    But now I want to run my map reduce program, which need a custom xml and
    properties files. Also I want to change the default hdfs and mapred port
    from 8020 and 8021 to something else. Therefore, to make this properties
    file and configuration available I have done below steps:

    I have create a directory say clientconf in /etc/hadoop directory
    Run below command:
    *alternatives --set hadoop-conf /etc/hadoop/clientconf*

    And put the updated core-site.xml and mapred-site.xml in
    /etc/hadoop/clientconf directory. Also put my project dependent xml and
    properties files in /etc/hadoop/clientconf

    Then from cloudera manager UI, I click on the "deploy client
    configuration". But it has no effect on the actual installation and also my
    M/R job can't found the custom xml or properties files in
    /var/run/cloudera-scm-agent/process/<processid>-mapreduce-TASKTRACKER
    folder.

    Do I miss something?

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Prodiptag at Dec 10, 2013 at 8:56 am ⇧
    Thanks. Its working now using the DistributedCache.
    On Friday, December 6, 2013 11:01:03 PM UTC+5:30, Philip Zeyliger wrote:

    Typically, users either include their XML and properties files as
    resources of the jar that contains their MR job, or they use the
    Distributed Cache (
    https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#DistributedCache)
    which is a fancy name for shipping some additional files (whether they be
    jars or other files) along with your MR job. Centrally placing these
    things on individual nodes doesn't scale with multiple users and causes
    upgrade headaches.

    The default ports should be configured in CM, and your MR job should be
    able to get at them via new Configuration().get("...").

    -- Philip

    On Thu, Dec 5, 2013 at 8:19 PM, <prod...@2pirad.com <javascript:>> wrote:

    I have successfully install CDH4 using cloudera manager. Nodes are
    working fine, also I can upload file to HDFS and browse that file.

    But now I want to run my map reduce program, which need a custom xml and
    properties files. Also I want to change the default hdfs and mapred port
    from 8020 and 8021 to something else. Therefore, to make this properties
    file and configuration available I have done below steps:

    I have create a directory say clientconf in /etc/hadoop directory
    Run below command:
    *alternatives --set hadoop-conf /etc/hadoop/clientconf*

    And put the updated core-site.xml and mapred-site.xml in
    /etc/hadoop/clientconf directory. Also put my project dependent xml and
    properties files in /etc/hadoop/clientconf

    Then from cloudera manager UI, I click on the "deploy client
    configuration". But it has no effect on the actual installation and also my
    M/R job can't found the custom xml or properties files in
    /var/run/cloudera-scm-agent/process/<processid>-mapreduce-TASKTRACKER
    folder.

    Do I miss something?

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedDec 6, '13 at 4:20a
activeDec 10, '13 at 8:56a
posts4
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Prodiptag: 3 posts Philip Zeyliger: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase