FAQ
I noticed that when I want to deploy configuration changes to a Gateway box
that the mapped-site.xml does not get pushed to /etc/hadoop/conf/. Am I
missing something or is this the way it should be? I had to manually
download it from Client Configuration URLs and save it there. I noticed it
when using Hive to run a query. It kept doing a local mapreduce job. After
copying it over, it now uses the cluster.

Thanks,
Ben

Search Discussions

  • bc Wong at Nov 29, 2012 at 7:52 pm
    Can you run `update-alternatives --display hadoop-conf' and tell me what
    you see?

    Cheers,
    bc
    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a Gateway
    box that the mapped-site.xml does not get pushed to /etc/hadoop/conf/. Am I
    missing something or is this the way it should be? I had to manually
    download it from Client Configuration URLs and save it there. I noticed it
    when using Hive to run a query. It kept doing a local mapreduce job. After
    copying it over, it now uses the cluster.

    Thanks,
    Ben
  • Benjamin Kim at Nov 29, 2012 at 8:18 pm
    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where Hive
    is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me what
    you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    I noticed that when I want to deploy configuration changes to a Gateway
    box that the mapped-site.xml does not get pushed to /etc/hadoop/conf/. Am I
    missing something or is this the way it should be? I had to manually
    download it from Client Configuration URLs and save it there. I noticed it
    when using Hive to run a query. It kept doing a local mapreduce job. After
    copying it over, it now uses the cluster.

    Thanks,
    Ben
  • Brian Burton at Nov 29, 2012 at 8:22 pm
    Hi Benjamin,

    I suspect you only added an HDFS Gateway role. You'll need to add one for
    MapReduce. In fact, since MapReduce is a superset of HDFS configs, you
    don't even need an HDFS Gateway role, just a MapReduce one.

    You add it the same way you would for an HDFS Gateway role.

    Thanks,

    Brian Burton
    Customer Operations Engineer, Cloudera

    On Thu, Nov 29, 2012 at 3:10 PM, Benjamin Kim wrote:

    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where Hive
    is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me what
    you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a Gateway
    box that the mapped-site.xml does not get pushed to /etc/hadoop/conf/. Am I
    missing something or is this the way it should be? I had to manually
    download it from Client Configuration URLs and save it there. I noticed it
    when using Hive to run a query. It kept doing a local mapreduce job. After
    copying it over, it now uses the cluster.

    Thanks,
    Ben
  • Benjamin Kim at Nov 29, 2012 at 8:38 pm
    Hi Brian,

    I have the same box as both the hdfs and mapreduce gateway. Could this be
    causing a conflict? It's also the hbase gateway.

    Thanks,
    Ben
    On Thursday, November 29, 2012 12:22:26 PM UTC-8, bdb wrote:

    Hi Benjamin,

    I suspect you only added an HDFS Gateway role. You'll need to add one for
    MapReduce. In fact, since MapReduce is a superset of HDFS configs, you
    don't even need an HDFS Gateway role, just a MapReduce one.

    You add it the same way you would for an HDFS Gateway role.

    Thanks,

    Brian Burton
    Customer Operations Engineer, Cloudera


    On Thu, Nov 29, 2012 at 3:10 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where Hive
    is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me what
    you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a Gateway
    box that the mapped-site.xml does not get pushed to /etc/hadoop/conf/. Am I
    missing something or is this the way it should be? I had to manually
    download it from Client Configuration URLs and save it there. I noticed it
    when using Hive to run a query. It kept doing a local mapreduce job. After
    copying it over, it now uses the cluster.

    Thanks,
    Ben
  • bc Wong at Nov 29, 2012 at 8:44 pm

    On Thu, Nov 29, 2012 at 12:38 PM, Benjamin Kim wrote:

    Hi Brian,

    I have the same box as both the hdfs and mapreduce gateway. Could this be
    causing a conflict? It's also the hbase gateway.
    There is no conflict. As you can see, the mapreduce config has a higher
    alternatives priority than hdfs's.

    bc

    On Thursday, November 29, 2012 12:22:26 PM UTC-8, bdb wrote:

    Hi Benjamin,

    I suspect you only added an HDFS Gateway role. You'll need to add one for
    MapReduce. In fact, since MapReduce is a superset of HDFS configs, you
    don't even need an HDFS Gateway role, just a MapReduce one.

    You add it the same way you would for an HDFS Gateway role.

    Thanks,

    Brian Burton
    Customer Operations Engineer, Cloudera

    On Thu, Nov 29, 2012 at 3:10 PM, Benjamin Kim wrote:

    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.**mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.**hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.**mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.**mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where
    Hive is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me
    what you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a
    Gateway box that the mapped-site.xml does not get pushed to
    /etc/hadoop/conf/. Am I missing something or is this the way it should be?
    I had to manually download it from Client Configuration URLs and save it
    there. I noticed it when using Hive to run a query. It kept doing a local
    mapreduce job. After copying it over, it now uses the cluster.

    Thanks,
    Ben
  • Brian Burton at Nov 29, 2012 at 8:44 pm
    Benjamin,

    That shouldn't be a problem. The MapReduce config, as evidenced by the
    alternatives configuration, takes precedence as it should. Since you have
    that configured properly, I would focus on bc's recommendations. Run 'ls
    -ld /etc/hadoop/conf' and see if it's linked to anything other than
    /etc/alternatives/hadoop-conf. If so, you will need to correct that so that
    /etc/hadoop/conf is a link to /etc/alternatives/hadoop-conf.

    You can verify that mapred-site.xml is being deployed properly by looking
    in /etc/hadoop/conf.cloudera.mapreduce1.

    Thank you,

    Brian Burton
    Customer Operations Engineer, Cloudera

    On Thu, Nov 29, 2012 at 3:38 PM, Benjamin Kim wrote:

    Hi Brian,

    I have the same box as both the hdfs and mapreduce gateway. Could this be
    causing a conflict? It's also the hbase gateway.

    Thanks,
    Ben

    On Thursday, November 29, 2012 12:22:26 PM UTC-8, bdb wrote:

    Hi Benjamin,

    I suspect you only added an HDFS Gateway role. You'll need to add one for
    MapReduce. In fact, since MapReduce is a superset of HDFS configs, you
    don't even need an HDFS Gateway role, just a MapReduce one.

    You add it the same way you would for an HDFS Gateway role.

    Thanks,

    Brian Burton
    Customer Operations Engineer, Cloudera

    On Thu, Nov 29, 2012 at 3:10 PM, Benjamin Kim wrote:

    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.**mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.**hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.**mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.**mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where
    Hive is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me
    what you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a
    Gateway box that the mapped-site.xml does not get pushed to
    /etc/hadoop/conf/. Am I missing something or is this the way it should be?
    I had to manually download it from Client Configuration URLs and save it
    there. I noticed it when using Hive to run a query. It kept doing a local
    mapreduce job. After copying it over, it now uses the cluster.

    Thanks,
    Ben
  • Benjamin Kim at Nov 29, 2012 at 10:04 pm
    Hi Brian,

    This is what I get.

    /etc/hadoop/conf -> /etc/alternatives/hadoop-conf

    I think it's correct. It must be just Hive that's the problem. Doesn't
    Clouder Manager automatically configure Hive? It installs it.

    Thanks,
    Ben
    On Thursday, November 29, 2012 12:44:56 PM UTC-8, bdb wrote:

    Benjamin,

    That shouldn't be a problem. The MapReduce config, as evidenced by the
    alternatives configuration, takes precedence as it should. Since you have
    that configured properly, I would focus on bc's recommendations. Run 'ls
    -ld /etc/hadoop/conf' and see if it's linked to anything other than
    /etc/alternatives/hadoop-conf. If so, you will need to correct that so that
    /etc/hadoop/conf is a link to /etc/alternatives/hadoop-conf.

    You can verify that mapred-site.xml is being deployed properly by looking
    in /etc/hadoop/conf.cloudera.mapreduce1.

    Thank you,

    Brian Burton
    Customer Operations Engineer, Cloudera


    On Thu, Nov 29, 2012 at 3:38 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Hi Brian,

    I have the same box as both the hdfs and mapreduce gateway. Could this be
    causing a conflict? It's also the hbase gateway.

    Thanks,
    Ben

    On Thursday, November 29, 2012 12:22:26 PM UTC-8, bdb wrote:

    Hi Benjamin,

    I suspect you only added an HDFS Gateway role. You'll need to add one
    for MapReduce. In fact, since MapReduce is a superset of HDFS configs, you
    don't even need an HDFS Gateway role, just a MapReduce one.

    You add it the same way you would for an HDFS Gateway role.

    Thanks,

    Brian Burton
    Customer Operations Engineer, Cloudera

    On Thu, Nov 29, 2012 at 3:10 PM, Benjamin Kim wrote:

    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.**mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.**hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.**mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.**mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where
    Hive is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me
    what you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a
    Gateway box that the mapped-site.xml does not get pushed to
    /etc/hadoop/conf/. Am I missing something or is this the way it should be?
    I had to manually download it from Client Configuration URLs and save it
    there. I noticed it when using Hive to run a query. It kept doing a local
    mapreduce job. After copying it over, it now uses the cluster.

    Thanks,
    Ben
  • Benjamin Kim at Nov 30, 2012 at 12:49 am
    Hi Brian,

    I removed mapred-site.xml from /etc/hadoop/conf/ and did a deploy of client
    configurations only for the mapreduce service. This fixed it. It looks like
    deploying the client configurations for the hdfs service changes the
    alternatives back to its folder, which does not have the mapped-site.xml
    file.

    I guess it should be known that, when setting the gateway role, one should
    choose either hdfs or map reduce. Is this a correct assumption?

    Thanks,
    Ben
    On Thursday, November 29, 2012 12:44:56 PM UTC-8, bdb wrote:

    Benjamin,

    That shouldn't be a problem. The MapReduce config, as evidenced by the
    alternatives configuration, takes precedence as it should. Since you have
    that configured properly, I would focus on bc's recommendations. Run 'ls
    -ld /etc/hadoop/conf' and see if it's linked to anything other than
    /etc/alternatives/hadoop-conf. If so, you will need to correct that so that
    /etc/hadoop/conf is a link to /etc/alternatives/hadoop-conf.

    You can verify that mapred-site.xml is being deployed properly by looking
    in /etc/hadoop/conf.cloudera.mapreduce1.

    Thank you,

    Brian Burton
    Customer Operations Engineer, Cloudera


    On Thu, Nov 29, 2012 at 3:38 PM, Benjamin Kim <bbui...@gmail.com<javascript:>
    wrote:
    Hi Brian,

    I have the same box as both the hdfs and mapreduce gateway. Could this be
    causing a conflict? It's also the hbase gateway.

    Thanks,
    Ben

    On Thursday, November 29, 2012 12:22:26 PM UTC-8, bdb wrote:

    Hi Benjamin,

    I suspect you only added an HDFS Gateway role. You'll need to add one
    for MapReduce. In fact, since MapReduce is a superset of HDFS configs, you
    don't even need an HDFS Gateway role, just a MapReduce one.

    You add it the same way you would for an HDFS Gateway role.

    Thanks,

    Brian Burton
    Customer Operations Engineer, Cloudera

    On Thu, Nov 29, 2012 at 3:10 PM, Benjamin Kim wrote:

    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.**mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.**hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.**mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.**mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.

    How do I make Hive automatically look there? How do I find out where
    Hive is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me
    what you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a
    Gateway box that the mapped-site.xml does not get pushed to
    /etc/hadoop/conf/. Am I missing something or is this the way it should be?
    I had to manually download it from Client Configuration URLs and save it
    there. I noticed it when using Hive to run a query. It kept doing a local
    mapreduce job. After copying it over, it now uses the cluster.

    Thanks,
    Ben
  • bc Wong at Nov 29, 2012 at 8:30 pm

    On Thu, Nov 29, 2012 at 12:10 PM, Benjamin Kim wrote:

    Hi bc,

    This is what I get.

    hadoop-conf - status is auto.
    link currently points to /etc/hadoop/conf.cloudera.mapreduce1
    /etc/hadoop/conf.empty - priority 10
    /etc/hadoop/conf.cloudera.hdfs1 - priority 90
    /etc/hadoop/conf.cloudera.mapreduce1 - priority 91
    Current `best' version is /etc/hadoop/conf.cloudera.mapreduce1.

    It looks like Hive does not look in this directory. The mapped-site.xml
    plus other config files are there.
    Your `/etc/hadoop/conf' directory should be a symlink to
    `/etc/alternatives/hadoop-conf', which is a symlink to the selected
    alternative (mapreduce). That's how the alternatives system works.

    Is it possible that '/etc/hadoop/conf' points elsewhere, due to manual
    override or modification?

    Cheers,
    bc


    How do I make Hive automatically look there? How do I find out where Hive
    is looking?

    Thanks,
    Ben
    On Thursday, November 29, 2012 11:51:38 AM UTC-8, bc Wong wrote:

    Can you run `update-alternatives --display hadoop-conf' and tell me what
    you see?

    Cheers,
    bc

    On Thu, Nov 29, 2012 at 11:34 AM, Benjamin Kim wrote:

    I noticed that when I want to deploy configuration changes to a Gateway
    box that the mapped-site.xml does not get pushed to /etc/hadoop/conf/. Am I
    missing something or is this the way it should be? I had to manually
    download it from Client Configuration URLs and save it there. I noticed it
    when using Hive to run a query. It kept doing a local mapreduce job. After
    copying it over, it now uses the cluster.

    Thanks,
    Ben

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedNov 29, '12 at 7:34p
activeNov 30, '12 at 12:49a
posts10
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase