FAQ
Hey there, I need to write seqfiles to an hdfs cluster using a normal java
app but can't use "hadoop jar".
I have access to the installation of a hadoop node.
I've reached the goal launching my app with java -jar and manually changing
de configuration in the code:

Configuration hadoopConf = new Configuration();
hadoopConf.addResource(new Path("/opt/hadoop/install/conf/core-site.xml"));
hadoopConf.addResource(new Path("/opt/hadoop/install/conf/hdfs-site.xml"));
hadoopConf.reloadConfiguration();
sfw = SequenceFile.createWriter(FileSystem.get(hadoopConf), hadoopConf,
...);

** /opt/hadoop/install/conf/core-site.xml and
/opt/hadoop/install/conf/hdfs-site.xml are the conf files of the cluster.

This works but I would like to know if there's a more elegant way to do
that. Specifying hadoop env vars to the java classpath or something like
that.

Thanks in advance.

--
View this message in context: http://lucene.472066.n3.nabble.com/write-seqfiles-to-hdfs-from-a-java-app-without-hadoop-jar-command-tp2677234p2677234.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Search Discussions

  • Harsh J at Mar 15, 2011 at 3:43 am
    Adding the config path to the classpath makes Configuration load it
    automatically ('stead of you having to add resource files manually).

    java -classpath
    /opt/hadoop/install/conf:/opt/hadoop/install/hadoop-*-core.jar -jar
    yourjar.jar

    [You'll need more jars on your classpath, though - like log4j,
    jackson, etc., all found under lib/*.jar]
    On Tue, Mar 15, 2011 at 1:55 AM, Marc Sturlese wrote:
    Hey there, I need to write seqfiles to an hdfs cluster using a normal java
    app but can't use "hadoop jar".
    I have access to the installation of a hadoop node.
    I've reached the goal launching my app with java -jar and manually changing
    de configuration in the code:

    Configuration hadoopConf = new Configuration();
    hadoopConf.addResource(new Path("/opt/hadoop/install/conf/core-site.xml"));
    hadoopConf.addResource(new Path("/opt/hadoop/install/conf/hdfs-site.xml"));
    hadoopConf.reloadConfiguration();
    sfw = SequenceFile.createWriter(FileSystem.get(hadoopConf), hadoopConf,
    ...);

    ** /opt/hadoop/install/conf/core-site.xml and
    /opt/hadoop/install/conf/hdfs-site.xml are the conf files of the cluster.

    This works but I would like to know if there's a more elegant way to do
    that. Specifying hadoop env vars to the java classpath or something like
    that.

    Thanks in advance.

    --
    View this message in context: http://lucene.472066.n3.nabble.com/write-seqfiles-to-hdfs-from-a-java-app-without-hadoop-jar-command-tp2677234p2677234.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.


    --
    Harsh J
    http://harshj.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 14, '11 at 8:25p
activeMar 15, '11 at 3:43a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Harsh J: 1 post Marc Sturlese: 1 post

People

Translate

site design / logo © 2022 Grokbase