FAQ
Hello,

I've installed a cluster via cloudera manager, and now I need to launch the
cluster manually.

I've been using the following command:

$ sudo -u hdfs hadoop namenode / datanode / jobtracker

But then the dfs.name.dir is set up /tmp. I can't seem to find where
cloudera manager has the HDFS config files. The ones in
/usr/lib/hadoop-02*/conf seem to be minimal. They're missing the* **
dfs.name.dir* which is what I'm looking for particularly. I'm on an RHLE 6
system, by the way. Being lazy, I though I could just copy over cloudera
manager's HDFS config files, so I don't have to manually create them, the
copy them over to 6 nodes :)

Thanks,

Sam

Search Discussions

  • Sam at Oct 30, 2012 at 4:59 pm
    Joey,

    Let me be even more clear. My instances are on EC2. When I stop the
    instance, I get a new local and public DNS's, and therefore, cloudera
    manager is unable to locate the cluster. Is there a way around this as
    well?

    Thanks,

    Sam
    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the* **
    dfs.name.dir* which is what I'm looking for particularly. I'm on an RHLE
    6 system, by the way. Being lazy, I though I could just copy over cloudera
    manager's HDFS config files, so I don't have to manually create them, the
    copy them over to 6 nodes :)

    Thanks,

    Sam

  • Joey Echeverria at Oct 30, 2012 at 4:55 pm
    The only solution that I'm aware of here is to use an elastic ip on
    the instances to prevent the re-ip.

    -Joey
    On Tue, Oct 30, 2012 at 12:51 PM, Sam wrote:
    Joey,

    Let me be even more clear. My instances are on EC2. When I stop the
    instance, I get a new local and public DNS's, and therefore, cloudera
    manager is unable to locate the cluster. Is there a way around this as
    well?

    Thanks,

    Sam
    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the
    dfs.name.dir which is what I'm looking for particularly. I'm on an RHLE 6
    system, by the way. Being lazy, I though I could just copy over cloudera
    manager's HDFS config files, so I don't have to manually create them, the
    copy them over to 6 nodes :)

    Thanks,

    Sam


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Mark Schnegelberger at Oct 30, 2012 at 6:42 pm
    Set up static Elastic IP's for the nodes to avoid getting new IP's after
    restarts. That will help you avoid that headache and ensure that Cloudera
    Manager can still resolve the hosts as-added initially.

    Regards,
    --
    On Tue, Oct 30, 2012 at 12:51 PM, Sam wrote:

    Joey,

    Let me be even more clear. My instances are on EC2. When I stop the
    instance, I get a new local and public DNS's, and therefore, cloudera
    manager is unable to locate the cluster. Is there a way around this as
    well?

    Thanks,

    Sam
    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the* **
    dfs.name.dir* which is what I'm looking for particularly. I'm on an
    RHLE 6 system, by the way. Being lazy, I though I could just copy over
    cloudera manager's HDFS config files, so I don't have to manually create
    them, the copy them over to 6 nodes :)

    Thanks,

    Sam


    --
    *Mark Schnegelberger*
    Customer Operations Engineer

    *Phone*: 919.436.3814
    *Email*: smark@cloudera.com
    *Twitter*: @markschneg
    *Data*: http://www.cloudera.com
  • Sam Mohamed at Oct 30, 2012 at 5:43 pm
    Mark,

    Joey,

    Many thanks for the prompt answers. Will look into elastic IP today.

    Joey,

    Can you elaborate a little more on the safety valve feature? Possibly a good link as a starting point?

    Thanks,

    Sam

    On Oct 30, 2012, at 9:55 AM, Mark Schnegelberger wrote:

    Set up static Elastic IP's for the nodes to avoid getting new IP's after restarts. That will help you avoid that headache and ensure that Cloudera Manager can still resolve the hosts as-added initially.

    Regards,
    --

    On Tue, Oct 30, 2012 at 12:51 PM, Sam wrote:
    Joey,

    Let me be even more clear. My instances are on EC2. When I stop the instance, I get a new local and public DNS's, and therefore, cloudera manager is unable to locate the cluster. Is there a way around this as well?

    Thanks,

    Sam

    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:
    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where cloudera manager has the HDFS config files. The ones in /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the dfs.name.dir which is what I'm looking for particularly. I'm on an RHLE 6 system, by the way. Being lazy, I though I could just copy over cloudera manager's HDFS config files, so I don't have to manually create them, the copy them over to 6 nodes :)

    Thanks,

    Sam





    --
    Mark Schnegelberger
    Customer Operations Engineer

    Phone: 919.436.3814
    Email: smark@cloudera.com
    Twitter: @markschneg
    Data: http://www.cloudera.com

    <cloudera_AskBiggerQuestions.png>
  • Mark Schnegelberger at Oct 30, 2012 at 5:43 pm
    Hi Sam,

    Here's a good starting link on configuration management and safety valves
    in Cloudera Manager Free edition -
    https://ccp.cloudera.com/display/FREE41DOC/Changing+Service+Configurations

    As an abstract, a "safety valve" is a place to be able to set
    configurations for roles/services/etc that are not necessarily explicitly
    exposed in the Cloudera Manager interface, and they'll be written into the
    configuration location specified per-safety valve.

    Regards,
    --

    On Tue, Oct 30, 2012 at 1:13 PM, Sam Mohamed wrote:

    Mark,

    Joey,

    Many thanks for the prompt answers. Will look into elastic IP today.

    Joey,

    Can you elaborate a little more on the safety valve feature? Possibly a
    good link as a starting point?

    Thanks,

    Sam


    On Oct 30, 2012, at 9:55 AM, Mark Schnegelberger wrote:

    Set up static Elastic IP's for the nodes to avoid getting new IP's after
    restarts. That will help you avoid that headache and ensure that Cloudera
    Manager can still resolve the hosts as-added initially.

    Regards,
    --
    On Tue, Oct 30, 2012 at 12:51 PM, Sam wrote:

    Joey,

    Let me be even more clear. My instances are on EC2. When I stop the
    instance, I get a new local and public DNS's, and therefore, cloudera
    manager is unable to locate the cluster. Is there a way around this as
    well?

    Thanks,

    Sam
    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the* **
    dfs.name.dir* which is what I'm looking for particularly. I'm on an
    RHLE 6 system, by the way. Being lazy, I though I could just copy over
    cloudera manager's HDFS config files, so I don't have to manually create
    them, the copy them over to 6 nodes :)

    Thanks,

    Sam


    --
    *Mark Schnegelberger*
    Customer Operations Engineer

    *Phone*: 919.436.3814
    *Email*: smark@cloudera.com
    *Twitter*: @markschneg
    *Data*: http://www.cloudera.com

    <cloudera_AskBiggerQuestions.png>


    --
    *Mark Schnegelberger*
    Customer Operations Engineer

    *Phone*: 919.436.3814
    *Email*: smark@cloudera.com
    *Twitter*: @markschneg
    *Data*: http://www.cloudera.com
  • Vinithra Varadharajan at Oct 30, 2012 at 6:23 pm
    Sam,

    In case you aren't already referring to it, the security guide is a useful
    resource to setup a secure Hadoop cluster:
    https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide.
    On Tue, Oct 30, 2012 at 10:42 AM, Mark Schnegelberger wrote:

    Hi Sam,

    Here's a good starting link on configuration management and safety valves
    in Cloudera Manager Free edition -
    https://ccp.cloudera.com/display/FREE41DOC/Changing+Service+Configurations

    As an abstract, a "safety valve" is a place to be able to set
    configurations for roles/services/etc that are not necessarily explicitly
    exposed in the Cloudera Manager interface, and they'll be written into the
    configuration location specified per-safety valve.

    Regards,
    --

    On Tue, Oct 30, 2012 at 1:13 PM, Sam Mohamed wrote:

    Mark,

    Joey,

    Many thanks for the prompt answers. Will look into elastic IP today.

    Joey,

    Can you elaborate a little more on the safety valve feature? Possibly a
    good link as a starting point?

    Thanks,

    Sam


    On Oct 30, 2012, at 9:55 AM, Mark Schnegelberger <smark@cloudera.com>
    wrote:

    Set up static Elastic IP's for the nodes to avoid getting new IP's after
    restarts. That will help you avoid that headache and ensure that Cloudera
    Manager can still resolve the hosts as-added initially.

    Regards,
    --
    On Tue, Oct 30, 2012 at 12:51 PM, Sam wrote:

    Joey,

    Let me be even more clear. My instances are on EC2. When I stop the
    instance, I get a new local and public DNS's, and therefore, cloudera
    manager is unable to locate the cluster. Is there a way around this as
    well?

    Thanks,

    Sam
    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the* **
    dfs.name.dir* which is what I'm looking for particularly. I'm on an
    RHLE 6 system, by the way. Being lazy, I though I could just copy over
    cloudera manager's HDFS config files, so I don't have to manually create
    them, the copy them over to 6 nodes :)

    Thanks,

    Sam


    --
    *Mark Schnegelberger*
    Customer Operations Engineer

    *Phone*: 919.436.3814
    *Email*: smark@cloudera.com
    *Twitter*: @markschneg
    *Data*: http://www.cloudera.com

    <cloudera_AskBiggerQuestions.png>


    --
    *Mark Schnegelberger*
    Customer Operations Engineer

    *Phone*: 919.436.3814
    *Email*: smark@cloudera.com
    *Twitter*: @markschneg
    *Data*: http://www.cloudera.com


  • Joey Echeverria at Oct 30, 2012 at 5:46 pm
    HI Sam,

    Cloudera Manager pushes out fresh config files on every service
    restart to a temporary directory under
    /var/run/cloudera-scm-agent/process. If you want to start/stop
    services from the commandline, I'd suggest you take a look at the CM
    python shell that we've made available on our github site:

    https://github.com/cloudera/cm_api/blob/master/SHELL_README.md

    -Joey
    On Tue, Oct 30, 2012 at 12:40 PM, Sam wrote:
    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch the
    cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the
    dfs.name.dir which is what I'm looking for particularly. I'm on an RHLE 6
    system, by the way. Being lazy, I though I could just copy over cloudera
    manager's HDFS config files, so I don't have to manually create them, the
    copy them over to 6 nodes :)

    Thanks,

    Sam


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Sam at Oct 30, 2012 at 6:38 pm
    Hi Joey,

    Thanks for the prompt reply.

    The reason I can't use the clouder manager free edition anymore is because
    I need to add Kerberos setup. If I use the command line client, will I be
    able to add Kerberso configs?

    Thanks,

    Sam
    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the* **
    dfs.name.dir* which is what I'm looking for particularly. I'm on an RHLE
    6 system, by the way. Being lazy, I though I could just copy over cloudera
    manager's HDFS config files, so I don't have to manually create them, the
    copy them over to 6 nodes :)

    Thanks,

    Sam

  • Joey Echeverria at Oct 30, 2012 at 4:57 pm
    Ah. You should be abel to use the safety valve to add kerberos
    configuration with the free edition.

    -Joey
    On Tue, Oct 30, 2012 at 12:47 PM, Sam wrote:
    Hi Joey,

    Thanks for the prompt reply.

    The reason I can't use the clouder manager free edition anymore is because I
    need to add Kerberos setup. If I use the command line client, will I be
    able to add Kerberso configs?

    Thanks,

    Sam

    On Tuesday, October 30, 2012 9:40:41 AM UTC-7, Sam wrote:

    Hello,

    I've installed a cluster via cloudera manager, and now I need to launch
    the cluster manually.

    I've been using the following command:

    $ sudo -u hdfs hadoop namenode / datanode / jobtracker

    But then the dfs.name.dir is set up /tmp. I can't seem to find where
    cloudera manager has the HDFS config files. The ones in
    /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the
    dfs.name.dir which is what I'm looking for particularly. I'm on an RHLE 6
    system, by the way. Being lazy, I though I could just copy over cloudera
    manager's HDFS config files, so I don't have to manually create them, the
    copy them over to 6 nodes :)

    Thanks,

    Sam


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedOct 30, '12 at 4:55p
activeOct 30, '12 at 6:42p
posts10
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase