FAQ
I've joined this list some weeks ago to learn about Cloudera Manager,
but I don't want to be doing everything through a GUI.
Ideally I want my Hadoop cluster set up through Puppet. I see that
Cloudera have some puppet configs called puppet-scm.
Is this the right place to discuss them? I've not heard of ANYONE
using them yet...

CHeers

Alex

Search Discussions

  • Philip Langdale at Jan 10, 2013 at 10:31 pm
    Hi Alex,

    There's a couple of separate issues bundled up in your question, which I'll
    try and separate out to answer.

    1) Can I set up a Hadoop cluster using just puppet?

    Of course you can, but this isn't the forum for that. This is for
    discussion of Cloudera Manager and how you
    might use it to achieve various things. If you really want to install CDH
    without using Cloudera Manager, the
    cdh-user mailing list would be a good resource.

    2) What's this puppet-scm thing?

    puppet-scm is/was a set of puppet recipes for installing Cloudera Manager
    itself. Once that's done, you would
    use Cloudera Manager to complete the set up of the Hadoop cluster. However,
    puppet-scm is stale and will
    not correctly set up the latest versions of CM, so it's not a turn-key
    solution, unfortunately.

    3) Can I use Cloudera Manager to set up a cluster without using the UI for
    everything?

    The short answer is yes, but you have to get CM installed first. The
    scm-installer.bin program can be
    automated with a few command line arguments to get the server installed,
    but installing the CM agent
    on your cluster nodes has to be done with the UI or with an external tool
    like puppet, which brings us back
    to 2).

    Once you've addressed that, one way or another, you can use the CM API
    http://cloudera.github.com/cm_api/
    to automate setting up of services, roles, configs, etc.

    3b) Note that in CM releases prior to 4.5, you are also responsible for
    installing CDH packages if you cannot
    or won't use the wizard UI - there is no API automation for this. In 4.5,
    it becomes possible to do through the
    parcel mechanism which is exposed through the API. This is all part of the
    v3 API which we may not have
    properly published the python bindings for yet.

    Hope that helps,

    --phil

    --phil

    On 10 January 2013 08:23, Alex McLintock wrote:

    I've joined this list some weeks ago to learn about Cloudera Manager,
    but I don't want to be doing everything through a GUI.
    Ideally I want my Hadoop cluster set up through Puppet. I see that
    Cloudera have some puppet configs called puppet-scm.
    Is this the right place to discuss them? I've not heard of ANYONE
    using them yet...

    CHeers

    Alex
  • Bala Venkatrao at Jan 10, 2013 at 10:49 pm
    Alex,

    Phil has summarized this well.

    We are working on a blog post to better illustrate how to you can use
    Puppet along with CM API to setup your cluster. We already have several
    customers doing this successfully. So stay tuned!

    Thanks,
    Bala
    On Thu, Jan 10, 2013 at 2:25 PM, Philip Langdale wrote:

    Hi Alex,

    There's a couple of separate issues bundled up in your question, which
    I'll try and separate out to answer.

    1) Can I set up a Hadoop cluster using just puppet?

    Of course you can, but this isn't the forum for that. This is for
    discussion of Cloudera Manager and how you
    might use it to achieve various things. If you really want to install CDH
    without using Cloudera Manager, the
    cdh-user mailing list would be a good resource.

    2) What's this puppet-scm thing?

    puppet-scm is/was a set of puppet recipes for installing Cloudera Manager
    itself. Once that's done, you would
    use Cloudera Manager to complete the set up of the Hadoop cluster.
    However, puppet-scm is stale and will
    not correctly set up the latest versions of CM, so it's not a turn-key
    solution, unfortunately.

    3) Can I use Cloudera Manager to set up a cluster without using the UI for
    everything?

    The short answer is yes, but you have to get CM installed first. The
    scm-installer.bin program can be
    automated with a few command line arguments to get the server installed,
    but installing the CM agent
    on your cluster nodes has to be done with the UI or with an external tool
    like puppet, which brings us back
    to 2).

    Once you've addressed that, one way or another, you can use the CM API
    http://cloudera.github.com/cm_api/
    to automate setting up of services, roles, configs, etc.

    3b) Note that in CM releases prior to 4.5, you are also responsible for
    installing CDH packages if you cannot
    or won't use the wizard UI - there is no API automation for this. In 4.5,
    it becomes possible to do through the
    parcel mechanism which is exposed through the API. This is all part of the
    v3 API which we may not have
    properly published the python bindings for yet.

    Hope that helps,

    --phil

    --phil

    On 10 January 2013 08:23, Alex McLintock wrote:

    I've joined this list some weeks ago to learn about Cloudera Manager,
    but I don't want to be doing everything through a GUI.
    Ideally I want my Hadoop cluster set up through Puppet. I see that
    Cloudera have some puppet configs called puppet-scm.
    Is this the right place to discuss them? I've not heard of ANYONE
    using them yet...

    CHeers

    Alex
  • Asher Devuyst at Feb 9, 2013 at 2:18 am
    Hi Bala,

    Do you have a link to the blog post?

    Thanks,

    Asher
    On Thursday, January 10, 2013 5:48:57 PM UTC-5, Bala Venkatrao wrote:

    Alex,

    Phil has summarized this well.

    We are working on a blog post to better illustrate how to you can use
    Puppet along with CM API to setup your cluster. We already have several
    customers doing this successfully. So stay tuned!

    Thanks,
    Bala

    On Thu, Jan 10, 2013 at 2:25 PM, Philip Langdale <phi...@cloudera.com<javascript:>
    wrote:
    Hi Alex,

    There's a couple of separate issues bundled up in your question, which
    I'll try and separate out to answer.

    1) Can I set up a Hadoop cluster using just puppet?

    Of course you can, but this isn't the forum for that. This is for
    discussion of Cloudera Manager and how you
    might use it to achieve various things. If you really want to install CDH
    without using Cloudera Manager, the
    cdh-user mailing list would be a good resource.

    2) What's this puppet-scm thing?

    puppet-scm is/was a set of puppet recipes for installing Cloudera Manager
    itself. Once that's done, you would
    use Cloudera Manager to complete the set up of the Hadoop cluster.
    However, puppet-scm is stale and will
    not correctly set up the latest versions of CM, so it's not a turn-key
    solution, unfortunately.

    3) Can I use Cloudera Manager to set up a cluster without using the UI
    for everything?

    The short answer is yes, but you have to get CM installed first. The
    scm-installer.bin program can be
    automated with a few command line arguments to get the server installed,
    but installing the CM agent
    on your cluster nodes has to be done with the UI or with an external tool
    like puppet, which brings us back
    to 2).

    Once you've addressed that, one way or another, you can use the CM API
    http://cloudera.github.com/cm_api/
    to automate setting up of services, roles, configs, etc.

    3b) Note that in CM releases prior to 4.5, you are also responsible for
    installing CDH packages if you cannot
    or won't use the wizard UI - there is no API automation for this. In 4.5,
    it becomes possible to do through the
    parcel mechanism which is exposed through the API. This is all part of
    the v3 API which we may not have
    properly published the python bindings for yet.

    Hope that helps,

    --phil

    --phil


    On 10 January 2013 08:23, Alex McLintock <al...@owal.co.uk <javascript:>>wrote:
    I've joined this list some weeks ago to learn about Cloudera Manager,
    but I don't want to be doing everything through a GUI.
    Ideally I want my Hadoop cluster set up through Puppet. I see that
    Cloudera have some puppet configs called puppet-scm.
    Is this the right place to discuss them? I've not heard of ANYONE
    using them yet...

    CHeers

    Alex
  • 王伟 at Apr 9, 2013 at 9:56 am
    Hi Phil,
    "Installing the CM agent on your cluster nodes has to be done with the UI"
    Is there any way to install CM agent on new cluster node with API
    now(without web UI)?
    Thank you very much.

    在 2013年1月11日星期五UTC+8上午6时25分21秒,Philip Langdale写道:
    Hi Alex,

    There's a couple of separate issues bundled up in your question, which
    I'll try and separate out to answer.

    1) Can I set up a Hadoop cluster using just puppet?

    Of course you can, but this isn't the forum for that. This is for
    discussion of Cloudera Manager and how you
    might use it to achieve various things. If you really want to install CDH
    without using Cloudera Manager, the
    cdh-user mailing list would be a good resource.

    2) What's this puppet-scm thing?

    puppet-scm is/was a set of puppet recipes for installing Cloudera Manager
    itself. Once that's done, you would
    use Cloudera Manager to complete the set up of the Hadoop cluster.
    However, puppet-scm is stale and will
    not correctly set up the latest versions of CM, so it's not a turn-key
    solution, unfortunately.

    3) Can I use Cloudera Manager to set up a cluster without using the UI for
    everything?

    The short answer is yes, but you have to get CM installed first. The
    scm-installer.bin program can be
    automated with a few command line arguments to get the server installed,
    but installing the CM agent
    on your cluster nodes has to be done with the UI or with an external tool
    like puppet, which brings us back
    to 2).

    Once you've addressed that, one way or another, you can use the CM API
    http://cloudera.github.com/cm_api/
    to automate setting up of services, roles, configs, etc.

    3b) Note that in CM releases prior to 4.5, you are also responsible for
    installing CDH packages if you cannot
    or won't use the wizard UI - there is no API automation for this. In 4.5,
    it becomes possible to do through the
    parcel mechanism which is exposed through the API. This is all part of the
    v3 API which we may not have
    properly published the python bindings for yet.

    Hope that helps,

    --phil

    --phil


    On 10 January 2013 08:23, Alex McLintock <al...@owal.co.uk <javascript:>>wrote:
    I've joined this list some weeks ago to learn about Cloudera Manager,
    but I don't want to be doing everything through a GUI.
    Ideally I want my Hadoop cluster set up through Puppet. I see that
    Cloudera have some puppet configs called puppet-scm.
    Is this the right place to discuss them? I've not heard of ANYONE
    using them yet...

    CHeers

    Alex
  • Philip Zeyliger at Apr 9, 2013 at 5:08 pm
    Hi Alex,

    Check out the instructions here:
    http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.1/Cloudera-Manager-Enterprise-Edition-Installation-Guide/cmeeig_topic_6_6.html?scroll=cmeeig_topic_6_6_6_unique_1

    The short answer is "yum install cloudera-manager-agent" and edit
    /etc/cloudera-scm-agent/config.ini.

    Cheers,

    -- Philip

    On Tue, Apr 9, 2013 at 2:56 AM, 王伟 wrote:

    Hi Phil,
    "Installing the CM agent on your cluster nodes has to be done with the UI"
    Is there any way to install CM agent on new cluster node with API
    now(without web UI)?
    Thank you very much.

    在 2013年1月11日星期五UTC+8上午6时25分21秒,Philip Langdale写道:
    Hi Alex,

    There's a couple of separate issues bundled up in your question, which
    I'll try and separate out to answer.

    1) Can I set up a Hadoop cluster using just puppet?

    Of course you can, but this isn't the forum for that. This is for
    discussion of Cloudera Manager and how you
    might use it to achieve various things. If you really want to install CDH
    without using Cloudera Manager, the
    cdh-user mailing list would be a good resource.

    2) What's this puppet-scm thing?

    puppet-scm is/was a set of puppet recipes for installing Cloudera Manager
    itself. Once that's done, you would
    use Cloudera Manager to complete the set up of the Hadoop cluster.
    However, puppet-scm is stale and will
    not correctly set up the latest versions of CM, so it's not a turn-key
    solution, unfortunately.

    3) Can I use Cloudera Manager to set up a cluster without using the UI
    for everything?

    The short answer is yes, but you have to get CM installed first. The
    scm-installer.bin program can be
    automated with a few command line arguments to get the server installed,
    but installing the CM agent
    on your cluster nodes has to be done with the UI or with an external tool
    like puppet, which brings us back
    to 2).

    Once you've addressed that, one way or another, you can use the CM API
    http://cloudera.github.**com/cm_api/ <http://cloudera.github.com/cm_api/>
    to automate setting up of services, roles, configs, etc.

    3b) Note that in CM releases prior to 4.5, you are also responsible for
    installing CDH packages if you cannot
    or won't use the wizard UI - there is no API automation for this. In 4.5,
    it becomes possible to do through the
    parcel mechanism which is exposed through the API. This is all part of
    the v3 API which we may not have
    properly published the python bindings for yet.

    Hope that helps,

    --phil

    --phil

    On 10 January 2013 08:23, Alex McLintock wrote:

    I've joined this list some weeks ago to learn about Cloudera Manager,
    but I don't want to be doing everything through a GUI.
    Ideally I want my Hadoop cluster set up through Puppet. I see that
    Cloudera have some puppet configs called puppet-scm.
    Is this the right place to discuss them? I've not heard of ANYONE
    using them yet...

    CHeers

    Alex

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJan 10, '13 at 4:23p
activeApr 9, '13 at 5:08p
posts6
users6
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase