FAQ
Greetings,

I recently had a power failure, resulting in all my servers shutting
down. Everything appears to have recovered, but I am now unable to run
fsck:

$ hadoop fsck /
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 503 for URL: http://0.0.0.0:50070/fsck?path=%2F
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:123)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:159)

Any ideas as to what is going on? My data appears safe, and everything
else seems to be working fine.

Chris

Search Discussions

  • Christian Stucchio at Feb 20, 2011 at 4:37 pm
    Just a followup to this question.

    After further experimentation, I'm also running into errors in the
    task tracker. While running a job, I see:

    ...
    11/02/20 21:59:08 INFO mapred.JobClient: map 93% reduce 0%
    11/02/20 22:00:05 INFO mapred.JobClient: map 100% reduce 0%
    11/02/20 22:03:59 INFO mapred.JobClient: Task Id :
    attempt_201102202150_0001_m_000001_0, Status : FAILED
    Too many fetch-failures
    11/02/20 22:03:59 WARN mapred.JobClient: Error reading task
    outputServer returned HTTP response code: 503 for URL:
    http://stylewok-hadoop:50060/tasklog?plaintext=true&taskid=attempt_201102202150_0001_m_000001_0&filter=stdout
    11/02/20 22:03:59 WARN mapred.JobClient: Error reading task
    outputServer returned HTTP response code: 503 for URL:
    http://stylewok-hadoop:50060/tasklog?plaintext=true&taskid=attempt_201102202150_0001_m_000001_0&filter=stderr
    11/02/20 22:04:06 INFO mapred.JobClient: map 93% reduce 0%

    Any help with this would be greatly appreciated.


    Chris


    On Sun, Feb 20, 2011 at 9:37 AM, Christian Stucchio wrote:
    Greetings,

    I recently had a power failure, resulting in all my servers shutting
    down. Everything appears to have recovered, but I am now unable to run
    fsck:

    $ hadoop fsck /
    Exception in thread "main" java.io.IOException: Server returned HTTP
    response code: 503 for URL: http://0.0.0.0:50070/fsck?path=%2F
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:123)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:159)

    Any ideas as to what is going on? My data appears safe, and everything
    else seems to be working fine.

    Chris
  • Harsh J at Feb 20, 2011 at 4:48 pm
    Hello,

    There could be something wrong with the Jetty webserver that starts
    along with a daemon (Used by the Reducer to fetch map outputs, and
    seemingly also by fsck from NN's server). Perhaps the logs (of NN, TT)
    can tell you more on what may have happened when it was initializing
    the Jetty HTTP service. Look at the initializing part (early up in the
    logs, during start-up).

    Quite possibly, it may be down/uninitialized due to some reason. Are
    you able to view the Web UIs of each service (services that publish
    one)?

    --
    Harsh J
    www.harshj.com
  • Christian Stucchio at Feb 20, 2011 at 7:15 pm
    It's definitely uninitialized. From the namenode log:

    2011-02-20 22:38:31,789 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2011-02-20 22:38:31,835 INFO org.apache.hadoop.http.HttpServer: Port
    returned by webServer.getConnectors()[0].getLocalPort() before open()
    is -1. Opening the listener on 50070
    2011-02-20 22:38:31,836 INFO org.apache.hadoop.http.HttpServer:
    listener.getLocalPort() returned 50070
    webServer.getConnectors()[0].getLocalPort() returned 50070
    2011-02-20 22:38:31,836 INFO org.apache.hadoop.http.HttpServer: Jetty
    bound to port 50070
    2011-02-20 22:38:31,836 INFO org.mortbay.log: jetty-6.1.14
    2011-02-20 22:38:31,932 WARN org.mortbay.log: Failed startup of
    context org.mortbay.jetty.webapp.WebAppContext@7c19f9d2{/,file:/usr/local/hadoop/webapps/hdfs}
    java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.(JarFile.java:135)
    at java.util.jar.JarFile.(TagLibConfiguration.java:168)
    at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1231)
    at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
    at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:460)
    at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
    at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
    at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
    at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
    at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
    at org.mortbay.jetty.Server.doStart(Server.java:222)
    at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
    at org.apache.hadoop.http.HttpServer.start(HttpServer.java:461)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:246)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:202)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:956)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
    2011-02-20 22:38:31,939 INFO org.mortbay.log: Started
    SelectChannelConnector@0.0.0.0:50070

    Googling this error only found this page, which suggests that the
    JVM's cache of jar files might have been modified somehow. I don't
    fully understand the error.

    http://docs.codehaus.org/display/JETTY/ZIP+exception+reading+jar+or+war+files

    However, on the theory that something was wrong with a jar file
    somewhere, I reinstalled the hadoop jars on my server. This resolved
    the issue.


    Thanks for your help,



    Chris


    On Sun, Feb 20, 2011 at 11:47 AM, Harsh J wrote:
    Hello,

    There could be something wrong with the Jetty webserver that starts
    along with a daemon (Used by the Reducer to fetch map outputs, and
    seemingly also by fsck from NN's server). Perhaps the logs (of NN, TT)
    can tell you more on what may have happened when it was initializing
    the Jetty HTTP service. Look at the initializing part (early up in the
    logs, during start-up).

    Quite possibly, it may be down/uninitialized due to some reason. Are
    you able to view the Web UIs of each service (services that publish
    one)?

    --
    Harsh J
    www.harshj.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 20, '11 at 2:38p
activeFeb 20, '11 at 7:15p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Christian Stucchio: 3 posts Harsh J: 1 post

People

Translate

site design / logo © 2022 Grokbase