FAQ
Hi,

I am trying to add a file to HDFS programmatically.

In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration object as follows

Configuration configuration = null;
configuration.addResource(new URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
configuration.addResource(new Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));
configuration.reloadConfiguration();
fileSystem = FileSystem.get(configuration);
System.out.println(fileSystem.getName());

This code prints file system as file:/// instead of hdfs://.

When I manually set HDFS configuration parameter like below, it works well

//configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");

How can I make my code work? Any inputs would be greatly appreciated.

Regards,
Anand.C

Search Discussions

  • Mohammad Tariq at Aug 9, 2012 at 7:59 am
    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works well



    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


  • Chandra Mohan, Ananda Vel Murugan at Aug 9, 2012 at 8:29 am
    Hi,

    I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.

    When I add

    System.out.println(configuration.get("fs.default.name"));

    I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///

    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 1:28 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works well



    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


  • Dave Beech at Aug 9, 2012 at 8:33 am
    Instead of setting the xml files programmatically, why not set the
    HADOOP_CONF_DIR env variable to '/usr/local/hadoop-1.0.2/conf/'?
    That way, you can just create a new Configuration() object and the files
    will be loaded for you without any extra work.
    On 9 August 2012 09:28, Chandra Mohan, Ananda Vel Murugan wrote:

    Hi,

    I have added other xml files too. But the issue is configuration object is
    not getting updated with the xml conents.

    When I add

    System.out.println(configuration.get("fs.default.name"));

    I am not getting my hdfs url which I have in core-site.xml. When I run my
    code in eclipse, it works. But when I make it as a runnable jar file, it
    outputs file:///

    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 1:28 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop
    Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works well


    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


  • Mohammad Tariq at Aug 9, 2012 at 8:34 am
    Try this and let me know if it works,

    Configuration conf = new Configuration();
    conf.addResource(new Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
    conf.addResource(new Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
    FileSystem fs = FileSystem.get(conf);
    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,

    I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.

    When I add

    System.out.println(configuration.get("fs.default.name"));

    I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///

    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 1:28 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works well



    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


  • Dave Beech at Aug 9, 2012 at 8:35 am
    Also - make sure the fs.default.name parameter is actually in
    core-site.xml, not hdfs-site.xml
    On 9 August 2012 09:32, Mohammad Tariq wrote:

    Try this and let me know if it works,

    Configuration conf = new Configuration();
    conf.addResource(new
    Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
    conf.addResource(new
    Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
    FileSystem fs = FileSystem.get(conf);
    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,

    I have added other xml files too. But the issue is configuration object
    is not getting updated with the xml conents.
    When I add

    System.out.println(configuration.get("fs.default.name"));

    I am not getting my hdfs url which I have in core-site.xml. When I run
    my code in eclipse, it works. But when I make it as a runnable jar file, it
    outputs file:///
    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 1:28 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop
    Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works
    well


    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


  • Mohammad Tariq at Aug 9, 2012 at 8:35 am
    Also, Hadoop jars must be in the classpath when you run the app from
    command line or as a jar.

    Regards,
    Mohammad Tariq

    On Thu, Aug 9, 2012 at 2:02 PM, Mohammad Tariq wrote:
    Try this and let me know if it works,

    Configuration conf = new Configuration();
    conf.addResource(new Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
    conf.addResource(new Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
    FileSystem fs = FileSystem.get(conf);
    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,

    I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.

    When I add

    System.out.println(configuration.get("fs.default.name"));

    I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///

    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 1:28 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works well



    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


  • Chandra Mohan, Ananda Vel Murugan at Aug 9, 2012 at 8:43 am
    Thanks all. I had a typo in core-site.xml path. It works now.

    It would be better if conf.addResource throws an exception like FileNotFoundException when it is not able to find the xml file specified.

    I like idea of setting HADOOP_CONF_DIR. I will try that too.

    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 2:04 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Also, Hadoop jars must be in the classpath when you run the app from
    command line or as a jar.

    Regards,
    Mohammad Tariq

    On Thu, Aug 9, 2012 at 2:02 PM, Mohammad Tariq wrote:
    Try this and let me know if it works,

    Configuration conf = new Configuration();
    conf.addResource(new Path("YOUR_HADOOP_HOME/conf/core-site.xml"));
    conf.addResource(new Path("YOUR_HADOOP_HOME/conf/hdfs-site.xml"));
    FileSystem fs = FileSystem.get(conf);
    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 1:58 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,

    I have added other xml files too. But the issue is configuration object is not getting updated with the xml conents.

    When I add

    System.out.println(configuration.get("fs.default.name"));

    I am not getting my hdfs url which I have in core-site.xml. When I run my code in eclipse, it works. But when I make it as a runnable jar file, it outputs file:///

    Regards,
    Anand.C

    -----Original Message-----
    From: Mohammad Tariq
    Sent: Thursday, August 09, 2012 1:28 PM
    To: user@hadoop.apache.org
    Subject: Re: Question on adding Hadoop XML to Configuration Object

    Hello there,

    Add the "conf/core-site.xml" file as well.

    Regards,
    Mohammad Tariq


    On Thu, Aug 9, 2012 at 12:37 PM, Chandra Mohan, Ananda Vel Murugan
    wrote:
    Hi,



    I am trying to add a file to HDFS programmatically.



    In my code, I am adding hdfs-site.xml and other xml to Hadoop Configuration
    object as follows



    Configuration configuration = null;

    configuration.addResource(new
    URL("file:///usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.addResource(new
    Path("usr/local/hadoop-1.0.2/conf/hdfs-site.xml"));

    configuration.reloadConfiguration();

    fileSystem = FileSystem.get(configuration);

    System.out.println(fileSystem.getName());



    This code prints file system as file:/// instead of hdfs://.



    When I manually set HDFS configuration parameter like below, it works well



    //configuration.set("fs.default.name", "hdfs://10.78.32.252:54310");



    How can I make my code work? Any inputs would be greatly appreciated.



    Regards,

    Anand.C


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshadoop
postedAug 9, '12 at 7:08a
activeAug 9, '12 at 8:43a
posts8
users3
websitehadoop.apache.org
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase