FAQ
Hello,
I had previously emailed regarding heap size issue and have discovered
that the hadoop-site.xml is not loading completely, i.e
Configuration defaults = new Configuration();
JobConf jobConf = new JobConf(defaults, XYZ.class);
System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
System.out.println("2:"+jobConf.get("mapred.map.tasks"));
System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
hadoop-site.xml are very different.

Is there a way for hadoop to dump the parameters read in from
hadoop-site.xml and hadoop-default.xml?

--
Saptarshi Guha - saptarshi.guha@gmail.com

Search Discussions

  • Amareshwari Sriramadasu at Dec 30, 2008 at 3:37 am

    Saptarshi Guha wrote:
    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?
    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari
  • G00dn3ss at Dec 30, 2008 at 10:25 pm
    Hey all,

    I have a similar issue. I am specifically having problems with the config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
    to print out the ENV values and it has the correct value for HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
    the JobConf passed to the mapper and it has my specified values for
    fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!


    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu wrote:

    Saptarshi Guha wrote:
    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));

    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?

    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari
  • Saptarshi Guha at Jan 5, 2009 at 7:27 pm
    Hello,
    I have set my HADOOP_CONF_DIR to the conf folder and still not
    loading. I have to manually set the options when I create my conf.
    Have you resolved this?

    Regards
    Saptarshi
    On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss wrote:
    Hey all,

    I have a similar issue. I am specifically having problems with the config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
    to print out the ENV values and it has the correct value for HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
    the JobConf passed to the mapper and it has my specified values for
    fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!



    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
    amarsri@yahoo-inc.com> wrote:
    Saptarshi Guha wrote:
    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));

    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?

    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari


    --
    Saptarshi Guha - saptarshi.guha@gmail.com
  • Saptarshi Guha at Jan 5, 2009 at 8:51 pm
    For some strange reason, neither hadoop-default.xml nor
    hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
    folder.
    HADOOP_CONF_DIR points to thi
    In some java code, (not a job), i did
    Configuration cf = new Configuration();
    String[] xu= new
    String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
    for(String x : xu){
    LOG.info(x+": "+cf.get(x));
    }

    All the values returned were default values(as noted in
    hadoop-default.xml). Changing the value in hadoop-default.xml were not
    reflected here. Grepping the source reveal the defaults are
    hardcoded. So it seems neither is hadoop-default.xml not
    hadooop-site.xml loading.
    Also, Configuration.java mentionds that hadoop-site.xml is depecated
    (thought still loaded)

    Any suggestions?
    Regards
    Saptarshi

    On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha wrote:
    Hello,
    I have set my HADOOP_CONF_DIR to the conf folder and still not
    loading. I have to manually set the options when I create my conf.
    Have you resolved this?

    Regards
    Saptarshi
    On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss wrote:
    Hey all,

    I have a similar issue. I am specifically having problems with the config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
    to print out the ENV values and it has the correct value for HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
    the JobConf passed to the mapper and it has my specified values for
    fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!



    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
    amarsri@yahoo-inc.com> wrote:
    Saptarshi Guha wrote:
    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));

    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?

    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari


    --
    Saptarshi Guha - saptarshi.guha@gmail.com


    --
    Saptarshi Guha - saptarshi.guha@gmail.com
  • Jason Venner at Jan 5, 2009 at 10:10 pm
    somehow you have alternate versions of the file earlier in the class path.

    Perhaps someone's empty copies are bundled into one of your application
    jar files.

    Or perhaps the configurationfiles are not distributed to the datanodes
    in the expected locations.

    Saptarshi Guha wrote:
    For some strange reason, neither hadoop-default.xml nor
    hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
    folder.
    HADOOP_CONF_DIR points to thi
    In some java code, (not a job), i did
    Configuration cf = new Configuration();
    String[] xu= new
    String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
    for(String x : xu){
    LOG.info(x+": "+cf.get(x));
    }

    All the values returned were default values(as noted in
    hadoop-default.xml). Changing the value in hadoop-default.xml were not
    reflected here. Grepping the source reveal the defaults are
    hardcoded. So it seems neither is hadoop-default.xml not
    hadooop-site.xml loading.
    Also, Configuration.java mentionds that hadoop-site.xml is depecated
    (thought still loaded)

    Any suggestions?
    Regards
    Saptarshi

    On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha wrote:

    Hello,
    I have set my HADOOP_CONF_DIR to the conf folder and still not
    loading. I have to manually set the options when I create my conf.
    Have you resolved this?

    Regards
    Saptarshi
    On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss wrote:

    Hey all,

    I have a similar issue. I am specifically having problems with the config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
    to print out the ENV values and it has the correct value for HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
    the JobConf passed to the mapper and it has my specified values for
    fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!



    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
    amarsri@yahoo-inc.com> wrote:

    Saptarshi Guha wrote:

    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));

    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?


    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari
    --
    Saptarshi Guha - saptarshi.guha@gmail.com

  • Saptarshi Guha at Jan 6, 2009 at 2:12 am
    Possibly. When i force load the configuration
    Configuration cf = new Configuration();
    LOG.info("Adding new resource");
    cf.addResource("hadoop-site.xml")

    It doesn't load, though hadoop-site.xml is present in
    hadoop-0.19.0/conf, and the hadoop-0.19 folder is replicated across
    all the machines. However if load it via
    cf.addResource(new
    Path("/home/godhuli/custom/hadoop-0.19.0/conf/hadoop-site.xml"));
    It works! So clearly, there is some other hadoop-site.xml in the
    classpath, but where? Should I add
    /home/godhuli/custom/hadoop-0.19.0/conf/ to the Hadoop classpath in
    hadoop-env.sh?

    Thanks
    Saptarshi
    On Mon, Jan 5, 2009 at 5:09 PM, Jason Venner wrote:
    somehow you have alternate versions of the file earlier in the class path.

    Perhaps someone's empty copies are bundled into one of your application jar
    files.

    Or perhaps the configurationfiles are not distributed to the datanodes in
    the expected locations.

    Saptarshi Guha wrote:
    For some strange reason, neither hadoop-default.xml nor
    hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
    folder.
    HADOOP_CONF_DIR points to thi
    In some java code, (not a job), i did
    Configuration cf = new Configuration();
    String[] xu= new

    String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
    for(String x : xu){
    LOG.info(x+": "+cf.get(x));
    }

    All the values returned were default values(as noted in
    hadoop-default.xml). Changing the value in hadoop-default.xml were not
    reflected here. Grepping the source reveal the defaults are
    hardcoded. So it seems neither is hadoop-default.xml not
    hadooop-site.xml loading.
    Also, Configuration.java mentionds that hadoop-site.xml is depecated
    (thought still loaded)

    Any suggestions?
    Regards
    Saptarshi


    On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com>
    wrote:
    Hello,
    I have set my HADOOP_CONF_DIR to the conf folder and still not
    loading. I have to manually set the options when I create my conf.
    Have you resolved this?

    Regards
    Saptarshi
    On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss wrote:


    Hey all,

    I have a similar issue. I am specifically having problems with the
    config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it uses
    -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option
    was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map
    task
    to print out the ENV values and it has the correct value for
    HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values
    in
    the JobConf passed to the mapper and it has my specified values for
    fs.default.name and mapred.job.tracker. Other settings like
    dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!



    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
    amarsri@yahoo-inc.com> wrote:

    Saptarshi Guha wrote:

    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));


    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));

    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?


    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR)
    directory?

    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari
    --
    Saptarshi Guha - saptarshi.guha@gmail.com



    --
    Saptarshi Guha - saptarshi.guha@gmail.com
  • Saptarshi Guha at Jan 6, 2009 at 3:28 pm
    I have checked, the classpath and $HADOOP_CONF_DIR respectively
    contain and point to my conf folder.
    Regards
    Saptarshi
    On Jan 6, 2009, at 12:53 AM, Sharad Agarwal wrote:

    bin/hadoop script does that. It puts the $HADOOP_CONF_DIR in the
    classpath. Try to echo $HADOOP_CONF_DIR and $CLASSPATH in the bin/
    hadoop script and see whether those getting picked up correctly.

    Saptarshi Guha wrote:
    Possibly. When i force load the configuration
    Configuration cf = new Configuration();
    LOG.info("Adding new resource");
    cf.addResource("hadoop-site.xml")

    It doesn't load, though hadoop-site.xml is present in
    hadoop-0.19.0/conf, and the hadoop-0.19 folder is replicated across
    all the machines. However if load it via
    cf.addResource(new
    Path("/home/godhuli/custom/hadoop-0.19.0/conf/hadoop-site.xml"));
    It works! So clearly, there is some other hadoop-site.xml in the
    classpath, but where? Should I add
    /home/godhuli/custom/hadoop-0.19.0/conf/ to the Hadoop classpath in
    hadoop-env.sh?

    Thanks
    Saptarshi

    On Mon, Jan 5, 2009 at 5:09 PM, Jason Venner <jason@attributor.com>
    wrote:
    somehow you have alternate versions of the file earlier in the
    class path.
    Perhaps someone's empty copies are bundled into one of your
    application jar
    files.

    Or perhaps the configurationfiles are not distributed to the
    datanodes in
    the expected locations.

    Saptarshi Guha wrote:
    For some strange reason, neither hadoop-default.xml nor
    hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
    folder.
    HADOOP_CONF_DIR points to thi
    In some java code, (not a job), i did
    Configuration cf = new Configuration();
    String[] xu= new

    String[]
    {"io
    .file
    .buffer
    .size
    ","io
    .sort
    .mb
    ","io
    .sort
    .factor
    ","mapred
    .tasktracker
    .map
    .tasks
    .maximum
    ","hadoop
    .logfile
    .count
    ","hadoop
    .logfile
    .size
    ","hadoop
    .tmp
    .dir
    ","dfs
    .replication
    ","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
    for(String x : xu){
    LOG.info(x+": "+cf.get(x));
    }

    All the values returned were default values(as noted in
    hadoop-default.xml). Changing the value in hadoop-default.xml
    were not
    reflected here. Grepping the source reveal the defaults are
    hardcoded. So it seems neither is hadoop-default.xml not
    hadooop-site.xml loading.
    Also, Configuration.java mentionds that hadoop-site.xml is
    depecated
    (thought still loaded)

    Any suggestions?
    Regards
    Saptarshi


    On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com
    wrote:
    Hello,
    I have set my HADOOP_CONF_DIR to the conf folder and still not
    loading. I have to manually set the options when I create my
    conf.
    Have you resolved this?

    Regards
    Saptarshi

    On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g00dn3ss@gmail.com>
    wrote:
    Hey all,

    I have a similar issue. I am specifically having problems
    with the
    config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it
    uses
    -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure
    this option
    was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config
    is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a
    small map
    task
    to print out the ENV values and it has the correct value for
    HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the
    key/values
    in
    the JobConf passed to the mapper and it has my specified
    values for
    fs.default.name and mapred.job.tracker. Other settings like
    dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!



    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
    amarsri@yahoo-inc.com> wrote:

    Saptarshi Guha wrote:

    Hello,
    I had previously emailed regarding heap size issue and have
    discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
    System
    .out
    .println
    ("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
    returns -Xmx200m, 2,1,2 respectively, even though the
    numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?


    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR)
    directory?

    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration

    -Amareshwari
    --
    Saptarshi Guha - saptarshi.guha@gmail.com



    --
    Saptarshi Guha - saptarshi.guha@gmail.com
    Saptarshi Guha | saptarshi.guha@gmail.com | http://www.stat.purdue.edu/~sguha
    There's nothing wrong with teenagers that
    reasoning with them won't aggravate.
  • G00dn3ss at Jan 6, 2009 at 7:38 pm
    OK, I figured out my problem. As expected, this was my silly mistake. We
    have a java program that runs on a machine outside of our hadoop cluster
    that references the hadoop jar files. This machine submits jobs to our
    cluster but doesn't actually have hadoop installed. I was configuring the
    mapred.child.java.opts in the hadoop-site.xml used by the cluster. However,
    the value that the tasks actually use come from the JobConf of the submitted
    job. So I am now manually setting the options I want in our external
    program that submits the jobs. It seems to be working as expected now. I
    thought my old setup was working with the older version of hadoop, but it
    may be that our jobs just recently started hitting the 200m heap boundary.

    Not sure if this is also your problem, Saptarshi. But hopefully it will
    lead you in the right direction.

    g00dn3ss


    On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha wrote:

    Hello,
    I have set my HADOOP_CONF_DIR to the conf folder and still not
    loading. I have to manually set the options when I create my conf.
    Have you resolved this?

    Regards
    Saptarshi
    On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss wrote:
    Hey all,

    I have a similar issue. I am specifically having problems with the config
    option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
    regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
    working in the previous versions of Hadoop I was using.

    I am not explicitly setting HADOOP_CONF_DIR. My site config is in
    ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
    to print out the ENV values and it has the correct value for HADOOP_HOME,
    HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
    the JobConf passed to the mapper and it has my specified values for
    fs.default.name and mapred.job.tracker. Other settings like
    dfs.name.dir,
    dfs.data.dir, and mapred.child.java.opts do not have my values.

    Any suggestion where to look at next?

    Thanks!



    On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
    amarsri@yahoo-inc.com> wrote:
    Saptarshi Guha wrote:
    Hello,
    I had previously emailed regarding heap size issue and have discovered
    that the hadoop-site.xml is not loading completely, i.e
    Configuration defaults = new Configuration();
    JobConf jobConf = new JobConf(defaults, XYZ.class);
    System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
    System.out.println("2:"+jobConf.get("mapred.map.tasks"));
    System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
    System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
    returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
    hadoop-site.xml are very different.

    Is there a way for hadoop to dump the parameters read in from
    hadoop-site.xml and hadoop-default.xml?

    Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
    http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
    -Amareshwari


    --
    Saptarshi Guha - saptarshi.guha@gmail.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 29, '08 at 9:18p
activeJan 6, '09 at 7:38p
posts9
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase