Grokbase › Groups › Hadoop › common-user › October 2009
FAQ
Hi!

I have quite odd Hadoop behavior. I wrote a client to my app that
simply is trying to talk to HDFS and do stuff. Version of Hadoop is
20.0. I still suspect CLASSPATH, but would be nice to know details.
So, here is a part of a traceback:
----------------------------------------------------
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
....
----------------------------------------------------

Now, could anybody please feel more light here what FileSystem is
trying to load in the configuration, once configuration was created
like: "... = new Configuration()"? I've looked at the source code, and
it does something this:
----------------------------------------------------
private static FileSystem createFileSystem(URI uri, Configuration conf
) throws IOException {
Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
if (clazz == null) {
throw new IOException("No FileSystem for scheme: " + uri.getScheme());
}
FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
fs.initialize(uri, conf);
return fs;
}
----------------------------------------------------

...but I think I need either long hours to hack or just ask here what
the heck is trying to load and can not. :-) It sounds to me that
CLASSPATH was screwed up, but I've put all the required libraries in
Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
However, when I am trying to write there, my thread dies with the
traceback above.

If anybody at least suggest me where to dig more, I would greatly appreciate it.

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

Search Discussions

  • Bogdan M. Maryniuk at Oct 22, 2009 at 4:29 am ⇧
    One more thing. I am creating configuration like this (if it is wrong
    — please tell me why and how is right to do it):
    -------------------------------------------
    this.conf = new Configuration();
    this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
    host, port));
    this.fs = FileSystem.get(this.conf);
    -------------------------------------------

    I don't see anything wrong with a code above, unless I just missing
    something undocumented...
    Anyone, please?

    Thanks in advance.

    --
    Bo


    On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
    wrote:
    Hi!

    I have quite odd Hadoop behavior. I wrote a client to my app that
    simply is trying to talk to HDFS and do stuff. Version of Hadoop is
    20.0. I still suspect CLASSPATH, but would be nice to know details.
    So, here is a part of a traceback:
    ----------------------------------------------------
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
    at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
    ....
    ----------------------------------------------------

    Now, could anybody please feel more light here what FileSystem is
    trying to load in the configuration, once configuration was created
    like: "... = new Configuration()"? I've looked at the source code, and
    it does something this:
    ----------------------------------------------------
    private static FileSystem createFileSystem(URI uri, Configuration conf
    ) throws IOException {
    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
    if (clazz == null) {
    throw new IOException("No FileSystem for scheme: " + uri.getScheme());
    }
    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
    fs.initialize(uri, conf);
    return fs;
    }
    ----------------------------------------------------

    ...but I think I need either long hours to hack or just ask here what
    the heck is trying to load and can not. :-) It sounds to me that
    CLASSPATH was screwed up, but I've put all the required libraries in
    Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
    However, when I am trying to write there, my thread dies with the
    traceback above.

    If anybody at least suggest me where to dig more, I would greatly appreciate it.

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.


    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.
  • Huy Phan at Oct 22, 2009 at 4:33 am ⇧
    Hi Bogdan,
    There's nothing wrong with your code, since you're talking about
    CLASSPATH, I guess you didn't execute your script the right way.
    How did you run your script anyway ? With or without hadoop command ?

    Bogdan M. Maryniuk wrote:
    One more thing. I am creating configuration like this (if it is wrong
    — please tell me why and how is right to do it):
    -------------------------------------------
    this.conf = new Configuration();
    this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
    host, port));
    this.fs = FileSystem.get(this.conf);
    -------------------------------------------

    I don't see anything wrong with a code above, unless I just missing
    something undocumented...
    Anyone, please?

    Thanks in advance.

    --
    Bo


    On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
    wrote:
    Hi!

    I have quite odd Hadoop behavior. I wrote a client to my app that
    simply is trying to talk to HDFS and do stuff. Version of Hadoop is
    20.0. I still suspect CLASSPATH, but would be nice to know details.
    So, here is a part of a traceback:
    ----------------------------------------------------
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
    at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
    ....
    ----------------------------------------------------

    Now, could anybody please feel more light here what FileSystem is
    trying to load in the configuration, once configuration was created
    like: "... = new Configuration()"? I've looked at the source code, and
    it does something this:
    ----------------------------------------------------
    private static FileSystem createFileSystem(URI uri, Configuration conf
    ) throws IOException {
    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
    if (clazz == null) {
    throw new IOException("No FileSystem for scheme: " + uri.getScheme());
    }
    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
    fs.initialize(uri, conf);
    return fs;
    }
    ----------------------------------------------------

    ...but I think I need either long hours to hack or just ask here what
    the heck is trying to load and can not. :-) It sounds to me that
    CLASSPATH was screwed up, but I've put all the required libraries in
    Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
    However, when I am trying to write there, my thread dies with the
    traceback above.

    If anybody at least suggest me where to dig more, I would greatly appreciate it.

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.

  • BM at Oct 22, 2009 at 7:59 am ⇧

    On Thu, Oct 22, 2009 at 1:33 PM, Huy Phan wrote:
    Hi Bogdan,
    There's nothing wrong with your code, since you're talking about CLASSPATH,
    I guess you didn't execute your script the right way.
    How did you run your script anyway ? With or without hadoop command ?
    Well, that's not a script really. I am trying to integrate the thing
    with the GlassFish in order apps talk to Hadoop and put/get media from
    there instead. So I wrote a client that is available over JNDI. All
    Hadoop-related libs I put into common classpath, so they *are*
    available. The only thing is probably that Catalina class loader could
    mess up things, but not likely...

    Anyway, I am looking to fix this. If somebody would fill more light
    here what this method wants? What is interesting, that a test-case
    works great, i.e. none of errors like this, file are written in the
    HDFS and read perfectly. I suspect that Hadoop does some "fun" with
    class loading, but I am not sure, since I did not invested time into
    this yet (still hope someone will tell me where could be a problem).

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.
  • #YONG YONG CHENG# at Oct 22, 2009 at 1:44 pm ⇧
    Good Day,

    Maybe you can try putting Hadoop jar files in the Extension classloader directory <domain name>/lib/ext.

    Thanks.

    ________________________________

    From: BM
    Sent: Thu 10/22/2009 3:59 PM
    To: common-user@hadoop.apache.org
    Subject: Re: Random weirdness - anyone explain, please?


    On Thu, Oct 22, 2009 at 1:33 PM, Huy Phan wrote:
    Hi Bogdan,
    There's nothing wrong with your code, since you're talking about CLASSPATH,
    I guess you didn't execute your script the right way.
    How did you run your script anyway ? With or without hadoop command ?
    Well, that's not a script really. I am trying to integrate the thing
    with the GlassFish in order apps talk to Hadoop and put/get media from
    there instead. So I wrote a client that is available over JNDI. All
    Hadoop-related libs I put into common classpath, so they *are*
    available. The only thing is probably that Catalina class loader could
    mess up things, but not likely...

    Anyway, I am looking to fix this. If somebody would fill more light
    here what this method wants? What is interesting, that a test-case
    works great, i.e. none of errors like this, file are written in the
    HDFS and read perfectly. I suspect that Hadoop does some "fun" with
    class loading, but I am not sure, since I did not invested time into
    this yet (still hope someone will tell me where could be a problem).

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.
  • BM at Oct 22, 2009 at 3:09 pm ⇧

    On Thu, Oct 22, 2009 at 10:43 PM, #YONG YONG CHENG# wrote:
    Maybe you can try putting Hadoop jar files in the Extension classloader directory <domain name>/lib/ext.
    Apparently, no. Now it works fine. Sounds like Webspace is just
    Sun-style thing: bloated, big, slow, complicated and... wrong. On a
    regular GlassFish it works fine.

    Thanks! Case closed, was not Hadoop issue. :-)

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.
  • Brien colwell at Oct 22, 2009 at 2:56 pm ⇧
    When I debug Hadoop locally in Eclipse I run jobs with the following
    configurations. I access the HDFS with paths like below . It seems to me
    this should be the same as what you have so I also suspect a classpath
    issue.

    baseConf.set("fs.default.name", "file:///");

    Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
    FileSystem fs = p.getFileSystem(baseConf) .





    Bogdan M. Maryniuk wrote:
    One more thing. I am creating configuration like this (if it is wrong
    — please tell me why and how is right to do it):
    -------------------------------------------
    this.conf = new Configuration();
    this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
    host, port));
    this.fs = FileSystem.get(this.conf);
    -------------------------------------------

    I don't see anything wrong with a code above, unless I just missing
    something undocumented...
    Anyone, please?

    Thanks in advance.

    --
    Bo


    On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
    wrote:
    Hi!

    I have quite odd Hadoop behavior. I wrote a client to my app that
    simply is trying to talk to HDFS and do stuff. Version of Hadoop is
    20.0. I still suspect CLASSPATH, but would be nice to know details.
    So, here is a part of a traceback:
    ----------------------------------------------------
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
    at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
    ....
    ----------------------------------------------------

    Now, could anybody please feel more light here what FileSystem is
    trying to load in the configuration, once configuration was created
    like: "... = new Configuration()"? I've looked at the source code, and
    it does something this:
    ----------------------------------------------------
    private static FileSystem createFileSystem(URI uri, Configuration conf
    ) throws IOException {
    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
    if (clazz == null) {
    throw new IOException("No FileSystem for scheme: " + uri.getScheme());
    }
    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
    fs.initialize(uri, conf);
    return fs;
    }
    ----------------------------------------------------

    ...but I think I need either long hours to hack or just ask here what
    the heck is trying to load and can not. :-) It sounds to me that
    CLASSPATH was screwed up, but I've put all the required libraries in
    Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
    However, when I am trying to write there, my thread dies with the
    traceback above.

    If anybody at least suggest me where to dig more, I would greatly appreciate it.

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.

  • BM at Oct 22, 2009 at 3:14 pm ⇧

    On Thu, Oct 22, 2009 at 11:56 PM, brien colwell wrote:
    When I debug Hadoop locally in Eclipse I run jobs with the following
    configurations. I access the HDFS with paths like below . It seems to me
    this should be the same as what you have so I also suspect a classpath
    issue.

    baseConf.set("fs.default.name", "file:///");

    Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
    FileSystem fs = p.getFileSystem(baseConf) .
    Well, not my case: I am accessing a real cluster. That works fine, but
    somewhat looks like Catalina class loader dislikes hadoop's forName
    tricks and in some cases (depends on a Moon phase and solar
    interference) does not works. All become OK after a regular restart.
    Which is... weird and wrong. On a regular GlassFish it works fine and
    stable, only Webspace freaks out.

    I need to dig into it anyway, but sounds like not Hadoop issue anymore.

    --
    Kind regards, BM

    Things, that are stupid at the beginning, rarely ends up wisely.
  • Philip Zeyliger at Oct 22, 2009 at 4:32 pm ⇧

    On Wed, Oct 21, 2009 at 9:23 PM, Bogdan M. Maryniuk wrote:

    Hi!

    I have quite odd Hadoop behavior. I wrote a client to my app that
    simply is trying to talk to HDFS and do stuff. Version of Hadoop is
    20.0. I still suspect CLASSPATH, but would be nice to know details.
    So, here is a part of a traceback:
    ----------------------------------------------------
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
    org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
    at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
    at
    org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
    ....
    ----------------------------------------------------

    Now, could anybody please feel more light here what FileSystem is
    trying to load in the configuration, once configuration was created
    like: "... = new Configuration()"? I've looked at the source code, and
    it does something this:
    ----------------------------------------------------
    private static FileSystem createFileSystem(URI uri, Configuration conf
    ) throws IOException {
    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
    if (clazz == null) {
    throw new IOException("No FileSystem for scheme: " + uri.getScheme());
    }
    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
    fs.initialize(uri, conf);
    return fs;
    }
    ----------------------------------------------------
    Here's what's going on here:

    The FileSystem class has a static factory which takes the "scheme" of your
    URL ("hdfs" or "s3" or "s3n" or "file"...) and maps that into an
    implementation class by looking up a configuration variable that looks like
    fs.hdfs.impl (or fs.s3.impl or fs.file.impl, ...). It then loads that
    implementation by reflection. (Configuration.getClass is just a short hand
    for read a configuration variable, and return the class that it represents.)

    One thing you might try to debug is to dump the Configuration (using
    configuration.writeXml(System.err) or similar), and making sure everything
    there looks reasonable. Typically HDFS is configured from from
    core-default.xml, like so:

    <property>
    <name>fs.hdfs.impl</name>
    <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
    <description>The FileSystem for hdfs: uris.</description>
    </property>



    -- Philip
  • Joe Hammerman at Oct 23, 2009 at 12:03 am ⇧
    Hi all,

    We are working on integrating our hadoop logs which live on the datanodes into a central syslog-ng logging server. There seems to be a dearth of information available on the internet as how to effect this; I was hoping someone here could help out, either with a link or by providing examples from their configuration.

    Thanks in advance!

    Regards,
    Joseph Hammerman
  • Brian Bockelman at Oct 23, 2009 at 12:09 am ⇧
    Hey Joe,

    We added the following to log4j.properties:

    log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
    log4j.appender.SYSLOG.facility=local0
    log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
    log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n
    log4j.appender.SYSLOG.SyslogHost=red
    log4j.appender.SYSLOG.threshold=ERROR
    log4j.appender.SYSLOG.Header=true
    log4j.appender.SYSLOG.FacilityPrinting=true

    IIRC, if you don't have have .Header=true, the generated packets are
    invalid and syslog chokes on them. Fun!

    Then, we add whatever we'd like to go to syslog; i.e.

    log4j.logger.org.apache.hadoop.hdfs=INFO,SYSLOGM

    or

    log4j.rootLogger=DEBUG,SYSLOG,console,DRFA,EventCounter

    Brian
    On Oct 22, 2009, at 7:01 PM, Joe Hammerman wrote:

    Hi all,

    We are working on integrating our hadoop logs which live on the
    datanodes into a central syslog-ng logging server. There seems to be
    a dearth of information available on the internet as how to effect
    this; I was hoping someone here could help out, either with a link
    or by providing examples from their configuration.

    Thanks in advance!

    Regards,
    Joseph Hammerman

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 22, '09 at 4:24a
activeOct 23, '09 at 12:09a
posts11
users7
websitehadoop.apache.org...
irc#hadoop

7 users in discussion

BM: 5 posts #YONG YONG CHENG#: 1 post Philip Zeyliger: 1 post Huy Phan: 1 post Brian Bockelman: 1 post Joe Hammerman: 1 post Brien colwell: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase