FAQ
I'm using the python library and API to configure my cluster. Anyone know
where (in role or service) I can set the fs.default.name property.

I've tried something like the following:

MAPRED_CONF = {
     'hdfs_service': HDFS_NAME,
     'io.file.buffer.size':65536,
     'fs.default.name':ZOOKEEPER_IP,
     }

mapred = apiCluster.create_service(MAPRED_NAME, MAPREDUCE_SERVICE_TYPE)
mapred.update_config(MAPRED_CONF)

but this results in a cm_api.api_client.ApiException: Unknown configuration
attribute 'fs.default.name'. (error 400)

any ideas where I can set this attribute?

Thanks in advance
Adam

Search Discussions

  • bc Wong at Aug 22, 2012 at 10:25 pm

    On Tue, Aug 21, 2012 at 1:33 PM, Scuba Hunter wrote:

    I'm using the python library and API to configure my cluster. Anyone know
    where (in role or service) I can set the fs.default.name property.

    I've tried something like the following:

    MAPRED_CONF = {
    'hdfs_service': HDFS_NAME,
    'io.file.buffer.size':65536,
    'fs.default.name':ZOOKEEPER_IP,
    }

    mapred = apiCluster.create_service(MAPRED_NAME, MAPREDUCE_SERVICE_TYPE)
    mapred.update_config(MAPRED_CONF)

    but this results in a cm_api.api_client.ApiException: Unknown
    configuration attribute 'fs.default.name'. (error 400)

    any ideas where I can set this attribute?
    Hi Adam,

    The CM service config names are not necessarily the same as the raw Hadoop
    names. Here's how you can find out what the mapping is:

    service_configs, roletype_configs = mapred.get_config(view='full')
    print "\n=========== Service Level ============="
    for c in service_configs.values():
         print "---\n%s: %s\n %s" % (c.name, c.relatedName, c.description)
    for rt, confmap in roletype_configs.items():
         print "\n===========", rt, "============="
         for c in confmap.values():
             print "---\n%s: %s\n %s" % (c.name, c.relatedName, c.description)

    To answer your questions specifically:

        - You don't need to specify fs.default.name for MR. You need to specify
        the HDFS service that MR depends on, via the service config `hdfs_service'.
        - You can do that by:
        mapred.update_config(dict(hdfs_service="my_hdfs_service_name"))

    Cheers,
    bc

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedAug 21, '12 at 8:33p
activeAug 22, '12 at 10:25p
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

bc Wong: 1 post Scuba Hunter: 1 post

People

Translate

site design / logo © 2022 Grokbase