FAQ
Hi all,

I have installed CDH3 in my cluster using Cloudera Manager and found that
the values of certain Hadoop metrics are not quivalent to the
"hdfs-default.xml" & "mapred-default.xml" (i.e., they are not equivalent to
the default values given by "hdfs-default.xaml" & "mapred-default.xml")

Following are some of the metrics along with their values (installed from
Cloudera Manager):

dfs.block.size = 128MB (default should be 64MB)
mapred.child.java.opts = 474.22MB (default should be 200MB)
io.sort.mb = 118MB (default should be 100MB)
mapred.reduce.parallel.copies = 10 (default should be 5)

Above are some of the properties which I found different that what it
should be as the default value.

My question is that how & why Cloudera Manager configures the default
cluster. Why these values are different when Hadoop is installed using
Cloudera Manager.
Will these values differ if I install CDH3 using Cloudera Manager in some
other cluster of different size and hardware configuration? (I mean dows
cloudera manager instantly decide while installing, how to configure Hadoop
depending on the cluster environment?)

Can someone please explain this?


Thanks,
Gaurav Dasgupta

Search Discussions

  • Philip Zeyliger at Sep 5, 2012 at 3:51 pm
    Hi Gaurav,

    CM's defaults have evolved according to Cloudera's best practices.
    Some of these are calculated based on how much RAM you have and how
    many machines you have (io.sort.mb, mapred.reduce.parallel.copies) and
    some are simply values we think are better defaults (dfs.block.size).
    All of the values we choose are in the configuration screens in CM,
    and you're welcome to tweak them as you see fit.

    Our configuration engine is run when you create services as part of
    the installation wizard, and yes, if you use different hardware,
    you'll get different values.

    Cheers,

    -- Philip
    On Wed, Sep 5, 2012 at 2:12 AM, Gaurav Dasgupta wrote:
    Hi all,

    I have installed CDH3 in my cluster using Cloudera Manager and found that
    the values of certain Hadoop metrics are not quivalent to the
    "hdfs-default.xml" & "mapred-default.xml" (i.e., they are not equivalent to
    the default values given by "hdfs-default.xaml" & "mapred-default.xml")

    Following are some of the metrics along with their values (installed from
    Cloudera Manager):

    dfs.block.size = 128MB (default should be 64MB)
    mapred.child.java.opts = 474.22MB (default should be 200MB)
    io.sort.mb = 118MB (default should be 100MB)
    mapred.reduce.parallel.copies = 10 (default should be 5)

    Above are some of the properties which I found different that what it should
    be as the default value.

    My question is that how & why Cloudera Manager configures the default
    cluster. Why these values are different when Hadoop is installed using
    Cloudera Manager.
    Will these values differ if I install CDH3 using Cloudera Manager in some
    other cluster of different size and hardware configuration? (I mean dows
    cloudera manager instantly decide while installing, how to configure Hadoop
    depending on the cluster environment?)

    Can someone please explain this?


    Thanks,
    Gaurav Dasgupta

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedSep 5, '12 at 10:19a
activeSep 5, '12 at 3:51p
posts2
users2
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase