Grokbase › Groups › Hadoop › scm-users › May 2013
FAQ
Hi all,

I am using the free version of Cloudera Manager 4.5 to install all the
services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
While on another cluster I had no major troubles and all was nicely
configured and working out of the box, on the current cluster (exactly the
same as the first one) the system is behaving differently.
In the oozie-site.xml the jobtracker whitelist port is set to 8032 while in
the mapreduce settings the jobtracker port is actually 8021. Having the
free version of cloudera manager, I cannot find how can I edit
oozie-site.xml to allow any jobtracker in the whitelist.
The jobtracker is on the namenode and I can see that both 8021 and 8032 are
being used and I just cannot explain that behaviour. The cloudera manager
was installed on a clean CentOS and the roles are exactly the same as my
other cluster where things work fine.
I really hope someone can help me as this is really frustrating. I even
cleaned-up all the machines and reinstalled Cloudera manager but still
ended up with the same wrong configuration.
Kind regards,

Robert

Search Discussions

  • Robert at May 8, 2013 at 4:29 pm ⇧
    The OS is actually "Red Hat Enterprise Linux Server release 6.3 (Santiago)"
    and not CentOS
    On Wednesday, 8 May 2013 18:23:36 UTC+2, Robert wrote:

    Hi all,

    I am using the free version of Cloudera Manager 4.5 to install all the
    services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
    While on another cluster I had no major troubles and all was nicely
    configured and working out of the box, on the current cluster (exactly the
    same as the first one) the system is behaving differently.
    In the oozie-site.xml the jobtracker whitelist port is set to 8032 while
    in the mapreduce settings the jobtracker port is actually 8021. Having the
    free version of cloudera manager, I cannot find how can I edit
    oozie-site.xml to allow any jobtracker in the whitelist.
    The jobtracker is on the namenode and I can see that both 8021 and 8032
    are being used and I just cannot explain that behaviour. The cloudera
    manager was installed on a clean CentOS and the roles are exactly the same
    as my other cluster where things work fine.
    I really hope someone can help me as this is really frustrating. I even
    cleaned-up all the machines and reinstalled Cloudera manager but still
    ended up with the same wrong configuration.
    Kind regards,

    Robert
  • Vikram Srivastava at May 8, 2013 at 5:47 pm ⇧
    To set the JobTracker port number, you need to goto your MapReduce service
    in CM, and then "Configuration". On the left side you'll see configuration
    sections for "JobTracker (Base)" > "Ports and Addresses". You can set the
    port numbers there. Remember to restart MR and Oozie after changing any
    ports.

    Regarding oozie - Can you tell me the location of your oozie-site.xml where
    you see 8032 port for JT?
    On Wed, May 8, 2013 at 9:29 AM, Robert wrote:

    The OS is actually "Red Hat Enterprise Linux Server release 6.3
    (Santiago)" and not CentOS

    On Wednesday, 8 May 2013 18:23:36 UTC+2, Robert wrote:

    Hi all,

    I am using the free version of Cloudera Manager 4.5 to install all the
    services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
    While on another cluster I had no major troubles and all was nicely
    configured and working out of the box, on the current cluster (exactly the
    same as the first one) the system is behaving differently.
    In the oozie-site.xml the jobtracker whitelist port is set to 8032 while
    in the mapreduce settings the jobtracker port is actually 8021. Having the
    free version of cloudera manager, I cannot find how can I edit
    oozie-site.xml to allow any jobtracker in the whitelist.
    The jobtracker is on the namenode and I can see that both 8021 and 8032
    are being used and I just cannot explain that behaviour. The cloudera
    manager was installed on a clean CentOS and the roles are exactly the same
    as my other cluster where things work fine.
    I really hope someone can help me as this is really frustrating. I even
    cleaned-up all the machines and reinstalled Cloudera manager but still
    ended up with the same wrong configuration.
    Kind regards,

    Robert
  • Robert at May 8, 2013 at 6:20 pm ⇧
    Hi and thanks for the quick reply :-)
    The JobTracker port number is set to 8021 (I think that is the default
    value) and it is not my intention to change that.
    Regarding oozie, the oozie-site.xml is located at
    "myServerFE:7180/cmf/process/153/config?filename=oozie-site.xml" and for
    some reason in the whitelist it has the following:

       <property>
         <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
         <value>myServerNameNode:8032</value>
       </property>


    I never changed the job tracker port during the installation or after.
    The only strange thing during the installation was that for some of the
    hosts there was a different Role group automatically assigned, like
    DataNode(1) instead of DataNode(Base). Cloudera Manager was claiming that
    this was due to a different hardware configuration while my hosts are all
    exactly the same virtual machines.
    It would be good if you could explain me what could've possibly happened
    and if there is a way to fix the configuration without another re-install
    of the system.
    Thanks and kind regards,

    Robert
    On Wednesday, 8 May 2013 19:47:21 UTC+2, Vikram Srivastava wrote:

    To set the JobTracker port number, you need to goto your MapReduce service
    in CM, and then "Configuration". On the left side you'll see configuration
    sections for "JobTracker (Base)" > "Ports and Addresses". You can set the
    port numbers there. Remember to restart MR and Oozie after changing any
    ports.

    Regarding oozie - Can you tell me the location of your oozie-site.xml
    where you see 8032 port for JT?

    On Wed, May 8, 2013 at 9:29 AM, Robert <robert...@gmail.com <javascript:>>wrote:
    The OS is actually "Red Hat Enterprise Linux Server release 6.3
    (Santiago)" and not CentOS

    On Wednesday, 8 May 2013 18:23:36 UTC+2, Robert wrote:

    Hi all,

    I am using the free version of Cloudera Manager 4.5 to install all the
    services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
    While on another cluster I had no major troubles and all was nicely
    configured and working out of the box, on the current cluster (exactly the
    same as the first one) the system is behaving differently.
    In the oozie-site.xml the jobtracker whitelist port is set to 8032 while
    in the mapreduce settings the jobtracker port is actually 8021. Having the
    free version of cloudera manager, I cannot find how can I edit
    oozie-site.xml to allow any jobtracker in the whitelist.
    The jobtracker is on the namenode and I can see that both 8021 and 8032
    are being used and I just cannot explain that behaviour. The cloudera
    manager was installed on a clean CentOS and the roles are exactly the same
    as my other cluster where things work fine.
    I really hope someone can help me as this is really frustrating. I even
    cleaned-up all the machines and reinstalled Cloudera manager but still
    ended up with the same wrong configuration.
    Kind regards,

    Robert
  • Vikram Srivastava at May 8, 2013 at 6:25 pm ⇧
    I think I know what is happening. You've set your oozie to use Yarn instead
    of MR1 and the default port for Yarn resource manager is 8032. You can
    select MR1 for Oozie by going to its Configuration page in CM.

    About the different role config groups: even though all your hardware is
    identical, on your first host, you have extra roles like NN and JT. So CM
    automatically assigns lower heap size for DN on that host compared to other
    hosts.
    On Wed, May 8, 2013 at 11:20 AM, Robert wrote:

    Hi and thanks for the quick reply :-)
    The JobTracker port number is set to 8021 (I think that is the default
    value) and it is not my intention to change that.
    Regarding oozie, the oozie-site.xml is located at
    "myServerFE:7180/cmf/process/153/config?filename=oozie-site.xml" and for
    some reason in the whitelist it has the following:

    <property>
    <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
    <value>myServerNameNode:8032</value>
    </property>


    I never changed the job tracker port during the installation or after.
    The only strange thing during the installation was that for some of the
    hosts there was a different Role group automatically assigned, like
    DataNode(1) instead of DataNode(Base). Cloudera Manager was claiming that
    this was due to a different hardware configuration while my hosts are all
    exactly the same virtual machines.
    It would be good if you could explain me what could've possibly happened
    and if there is a way to fix the configuration without another re-install
    of the system.
    Thanks and kind regards,

    Robert
    On Wednesday, 8 May 2013 19:47:21 UTC+2, Vikram Srivastava wrote:

    To set the JobTracker port number, you need to goto your MapReduce
    service in CM, and then "Configuration". On the left side you'll see
    configuration sections for "JobTracker (Base)" > "Ports and Addresses". You
    can set the port numbers there. Remember to restart MR and Oozie after
    changing any ports.

    Regarding oozie - Can you tell me the location of your oozie-site.xml
    where you see 8032 port for JT?
    On Wed, May 8, 2013 at 9:29 AM, Robert wrote:

    The OS is actually "Red Hat Enterprise Linux Server release 6.3
    (Santiago)" and not CentOS

    On Wednesday, 8 May 2013 18:23:36 UTC+2, Robert wrote:

    Hi all,

    I am using the free version of Cloudera Manager 4.5 to install all the
    services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
    While on another cluster I had no major troubles and all was nicely
    configured and working out of the box, on the current cluster (exactly the
    same as the first one) the system is behaving differently.
    In the oozie-site.xml the jobtracker whitelist port is set to 8032
    while in the mapreduce settings the jobtracker port is actually 8021.
    Having the free version of cloudera manager, I cannot find how can I edit
    oozie-site.xml to allow any jobtracker in the whitelist.
    The jobtracker is on the namenode and I can see that both 8021 and 8032
    are being used and I just cannot explain that behaviour. The cloudera
    manager was installed on a clean CentOS and the roles are exactly the same
    as my other cluster where things work fine.
    I really hope someone can help me as this is really frustrating. I even
    cleaned-up all the machines and reinstalled Cloudera manager but still
    ended up with the same wrong configuration.
    Kind regards,

    Robert
  • Robert at May 8, 2013 at 6:32 pm ⇧
    Correct, I've set it up to use YARN as actually we wanted to use that to
    run our workflow. So I suppose if I set oozie to use MRv1, the whitelist
    would revert back to the port 8021.
    I think the "port number mistery" is now explained, thanks for that!!!

    After getting my colleague to change her workflow to use port 8032 for the
    jobtracker, she run the workflow but it is failing giving the following
    message:

    Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:397)

    Is this something missing in the yarn configuration? CM is telling me Yarn
    is all healthy and running.
    Thanks again,

    Robert

    On Wednesday, 8 May 2013 20:25:39 UTC+2, Vikram Srivastava wrote:

    I think I know what is happening. You've set your oozie to use Yarn
    instead of MR1 and the default port for Yarn resource manager is 8032. You
    can select MR1 for Oozie by going to its Configuration page in CM.

    About the different role config groups: even though all your hardware is
    identical, on your first host, you have extra roles like NN and JT. So CM
    automatically assigns lower heap size for DN on that host compared to other
    hosts.

    On Wed, May 8, 2013 at 11:20 AM, Robert <robert...@gmail.com <javascript:>
    wrote:
    Hi and thanks for the quick reply :-)
    The JobTracker port number is set to 8021 (I think that is the default
    value) and it is not my intention to change that.
    Regarding oozie, the oozie-site.xml is located at
    "myServerFE:7180/cmf/process/153/config?filename=oozie-site.xml" and for
    some reason in the whitelist it has the following:

    <property>
    <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
    <value>myServerNameNode:8032</value>
    </property>


    I never changed the job tracker port during the installation or after.
    The only strange thing during the installation was that for some of the
    hosts there was a different Role group automatically assigned, like
    DataNode(1) instead of DataNode(Base). Cloudera Manager was claiming that
    this was due to a different hardware configuration while my hosts are all
    exactly the same virtual machines.
    It would be good if you could explain me what could've possibly happened
    and if there is a way to fix the configuration without another re-install
    of the system.
    Thanks and kind regards,

    Robert
    On Wednesday, 8 May 2013 19:47:21 UTC+2, Vikram Srivastava wrote:

    To set the JobTracker port number, you need to goto your MapReduce
    service in CM, and then "Configuration". On the left side you'll see
    configuration sections for "JobTracker (Base)" > "Ports and Addresses". You
    can set the port numbers there. Remember to restart MR and Oozie after
    changing any ports.

    Regarding oozie - Can you tell me the location of your oozie-site.xml
    where you see 8032 port for JT?
    On Wed, May 8, 2013 at 9:29 AM, Robert wrote:

    The OS is actually "Red Hat Enterprise Linux Server release 6.3
    (Santiago)" and not CentOS

    On Wednesday, 8 May 2013 18:23:36 UTC+2, Robert wrote:

    Hi all,

    I am using the free version of Cloudera Manager 4.5 to install all the
    services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
    While on another cluster I had no major troubles and all was nicely
    configured and working out of the box, on the current cluster (exactly the
    same as the first one) the system is behaving differently.
    In the oozie-site.xml the jobtracker whitelist port is set to 8032
    while in the mapreduce settings the jobtracker port is actually 8021.
    Having the free version of cloudera manager, I cannot find how can I edit
    oozie-site.xml to allow any jobtracker in the whitelist.
    The jobtracker is on the namenode and I can see that both 8021 and
    8032 are being used and I just cannot explain that behaviour. The cloudera
    manager was installed on a clean CentOS and the roles are exactly the same
    as my other cluster where things work fine.
    I really hope someone can help me as this is really frustrating. I
    even cleaned-up all the machines and reinstalled Cloudera manager but still
    ended up with the same wrong configuration.
    Kind regards,

    Robert
  • Vikram Srivastava at May 8, 2013 at 7:26 pm ⇧
    +cdh-user@cloudera.org which should be able to answer your question about
    the failure you are seeing while using Oozie with Yarn.
    On Wed, May 8, 2013 at 11:32 AM, Robert wrote:

    Correct, I've set it up to use YARN as actually we wanted to use that to
    run our workflow. So I suppose if I set oozie to use MRv1, the whitelist
    would revert back to the port 8021.
    I think the "port number mistery" is now explained, thanks for that!!!

    After getting my colleague to change her workflow to use port 8032 for the
    jobtracker, she run the workflow but it is failing giving the following
    message:

    Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:397)

    Is this something missing in the yarn configuration? CM is telling me Yarn
    is all healthy and running.
    Thanks again,

    Robert

    On Wednesday, 8 May 2013 20:25:39 UTC+2, Vikram Srivastava wrote:

    I think I know what is happening. You've set your oozie to use Yarn
    instead of MR1 and the default port for Yarn resource manager is 8032. You
    can select MR1 for Oozie by going to its Configuration page in CM.

    About the different role config groups: even though all your hardware is
    identical, on your first host, you have extra roles like NN and JT. So CM
    automatically assigns lower heap size for DN on that host compared to other
    hosts.
    On Wed, May 8, 2013 at 11:20 AM, Robert wrote:

    Hi and thanks for the quick reply :-)
    The JobTracker port number is set to 8021 (I think that is the default
    value) and it is not my intention to change that.
    Regarding oozie, the oozie-site.xml is located at
    "myServerFE:7180/cmf/process/**153/config?filename=oozie-**site.xml"
    and for some reason in the whitelist it has the following:

    <property>
    <name>oozie.service.**HadoopAccessorService.**jobTracker.whitelist</name>
    <value>myServerNameNode:8032</**value>
    </property>


    I never changed the job tracker port during the installation or after.
    The only strange thing during the installation was that for some of the
    hosts there was a different Role group automatically assigned, like
    DataNode(1) instead of DataNode(Base). Cloudera Manager was claiming that
    this was due to a different hardware configuration while my hosts are all
    exactly the same virtual machines.
    It would be good if you could explain me what could've possibly happened
    and if there is a way to fix the configuration without another re-install
    of the system.
    Thanks and kind regards,

    Robert
    On Wednesday, 8 May 2013 19:47:21 UTC+2, Vikram Srivastava wrote:

    To set the JobTracker port number, you need to goto your MapReduce
    service in CM, and then "Configuration". On the left side you'll see
    configuration sections for "JobTracker (Base)" > "Ports and Addresses". You
    can set the port numbers there. Remember to restart MR and Oozie after
    changing any ports.

    Regarding oozie - Can you tell me the location of your oozie-site.xml
    where you see 8032 port for JT?
    On Wed, May 8, 2013 at 9:29 AM, Robert wrote:

    The OS is actually "Red Hat Enterprise Linux Server release 6.3
    (Santiago)" and not CentOS

    On Wednesday, 8 May 2013 18:23:36 UTC+2, Robert wrote:

    Hi all,

    I am using the free version of Cloudera Manager 4.5 to install all
    the services on my cluster composed by 8 machines (1 FE, 6 DN, 1 NN).
    While on another cluster I had no major troubles and all was nicely
    configured and working out of the box, on the current cluster (exactly the
    same as the first one) the system is behaving differently.
    In the oozie-site.xml the jobtracker whitelist port is set to 8032
    while in the mapreduce settings the jobtracker port is actually 8021.
    Having the free version of cloudera manager, I cannot find how can I edit
    oozie-site.xml to allow any jobtracker in the whitelist.
    The jobtracker is on the namenode and I can see that both 8021 and
    8032 are being used and I just cannot explain that behaviour. The cloudera
    manager was installed on a clean CentOS and the roles are exactly the same
    as my other cluster where things work fine.
    I really hope someone can help me as this is really frustrating. I
    even cleaned-up all the machines and reinstalled Cloudera manager but still
    ended up with the same wrong configuration.
    Kind regards,

    Robert

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMay 8, '13 at 4:23p
activeMay 8, '13 at 7:26p
posts7
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Robert: 4 posts Vikram Srivastava: 3 posts

Content

People

Support

Translate

site design / logo © 2022 Grokbase