FAQ
Hi I'm trying to get a small cluster up with hdfs (hadoop 0.20.1)

It is working somewhat. I can create files and directories and retrieve
them. However I am seeing a random failure rate (about 25%) where I get
this error when storing a file, and the resultant file is stored as a 0
len file (as reported by hadoop fs -ls).
I set this all up with straight IP addresses because I have no access to
the DNS servers here.
However the datanodes do have hostnames that do NOT resolve in DNS.
(not sure if that matters)


Put failed: 09/11/19 15:40:09 INFO hdfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.241.4.101:50010

09/11/19 15:40:09 INFO hdfs.DFSClient: Abandoning block
blk_8325005803148307980_1082

-etc,etc

I've verified that the java process is running and listening on port
50010 (as reported by netstat -plnt) and passphraseless ssh is working fine.
(also, the namenode web page intermittently fails to connect when asked
to browse the filesystem, (works most of the time)

The logs on that datanode show a java.net.NoRouteToHostException for a
few blocks, then it seems to start receiving blocks fine.

The log ....


2009-11-19 06:12:30,149 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_6892009382281192058_1068
src: /10.241.4.101:41570 dest: /10.241.4.101:50010
2009-11-19 06:12:30,150 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_6892009382281192058_1068 rece
ived exception java.net.NoRouteToHostException: No route to host
2009-11-19 06:12:30,151 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.241.4.101:50010, st
orageID=DS-886003042-127.0.0.1-50010-1258634370537, infoPort=50075,
ipcPort=50020):DataXceiver
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Unknown Source)
2009-11-19 06:12:36,155 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-7007024210780549144_106
8 src: /10.241.4.101:41574 dest: /10.241.4.101:50010
2009-11-19 06:12:36,157 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_-7007024210780549144_1068 rec
eived exception java.net.NoRouteToHostException: No route to host
2.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Unknown Source)
2009-11-19 06:12:30,149 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_6892009382281192058_1068
src: /10.241.4.101:41570 dest: /10.241.4.101:50010
2009-11-19 06:12:30,150 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_6892009382281192058_1068 rece
ived exception java.net.NoRouteToHostException: No route to host
2009-11-19 06:12:30,151 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.241.4.101:50010,
storageID=DS-886003042-127.0.0.1-50010-1258634370537, infoPort=50075,
ipcPort=50020):DataXceiver
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Unknown Source)
2009-11-19 06:12:36,155 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-7007024210780549144_1068 src: /10.241.4.101:41574 dest:
/10.241.4.101:50010
2009-11-19 06:12:36,157 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_-7007024210780549144_1068 received exception
java.net.NoRouteToHostException: No route to host
2009-11-19 06:12:36,157 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.241.4.101:50010,
storageID=DS-886003042-127.0.0.1-50010-1258634370537, infoPort=50075,
ipcPort=50020):DataXceiver
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Unknown Source)
2009-11-19 06:15:31,776 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_6602943802956432390_1059
2009-11-19 06:37:06,201 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-3274535555692844186_1063
2009-11-19 06:45:15,117 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_753873727864345115_1067
2009-11-19 06:51:56,888 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-810392349062598446_1062
2009-11-19 07:03:17,441 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-4850126846096803878_1066
2009-11-19 07:09:26,335 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 12
blocks got processed in 3
msecs



Any ideas would be greatly appreciated.

Thanks -Bill

Search Discussions

  • Raymond Jennings III at Nov 20, 2009 at 10:09 pm
    The plugin that is included in the hadoop distribution under src/contrib/eclipse-plugin - how does that get installed as it does not appear to be in a standard plugin format. Do I have to build it first and if so can you tell me how. Thanks. Ray
  • Jason Venner at Nov 23, 2009 at 2:33 am
    At one point my admin staff added 5 machines to my cluster but accidently
    left port 50010 among onthers firewalled.
    This resulted in chaos for a while until the firewall was found.
    On Fri, Nov 20, 2009 at 1:01 PM, Bill Brune wrote:


    Hi I'm trying to get a small cluster up with hdfs (hadoop 0.20.1)

    It is working somewhat. I can create files and directories and retrieve
    them. However I am seeing a random failure rate (about 25%) where I get
    this error when storing a file, and the resultant file is stored as a 0 len
    file (as reported by hadoop fs -ls).
    I set this all up with straight IP addresses because I have no access to
    the DNS servers here. However the datanodes do have hostnames that do NOT
    resolve in DNS. (not sure if that matters)


    Put failed: 09/11/19 15:40:09 INFO hdfs.DFSClient: Exception in
    createBlockOutputStream java.io.IOException: Bad connect ack with
    firstBadLink 10.241.4.101:50010

    09/11/19 15:40:09 INFO hdfs.DFSClient: Abandoning block
    blk_8325005803148307980_1082

    -etc,etc

    I've verified that the java process is running and listening on port 50010
    (as reported by netstat -plnt) and passphraseless ssh is working fine.
    (also, the namenode web page intermittently fails to connect when asked to
    browse the filesystem, (works most of the time)

    The logs on that datanode show a java.net.NoRouteToHostException for a few
    blocks, then it seems to start receiving blocks fine.

    The log ....


    2009-11-19 06:12:30,149 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
    blk_6892009382281192058_1068
    src: /10.241.4.101:41570 dest: /10.241.4.101:50010
    2009-11-19 06:12:30,150 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
    blk_6892009382281192058_1068 rece
    ived exception java.net.NoRouteToHostException: No route to host
    2009-11-19 06:12:30,151 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    10.241.4.101:50010, st
    orageID=DS-886003042-127.0.0.1-50010-1258634370537, infoPort=50075,
    ipcPort=50020):DataXceiver
    java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
    at java.lang.Thread.run(Unknown Source)
    2009-11-19 06:12:36,155 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
    blk_-7007024210780549144_106
    8 src: /10.241.4.101:41574 dest: /10.241.4.101:50010
    2009-11-19 06:12:36,157 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
    blk_-7007024210780549144_1068 rec
    eived exception java.net.NoRouteToHostException: No route to host
    2.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
    at java.lang.Thread.run(Unknown Source)
    2009-11-19 06:12:30,149 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
    blk_6892009382281192058_1068
    src: /10.241.4.101:41570 dest: /10.241.4.101:50010
    2009-11-19 06:12:30,150 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
    blk_6892009382281192058_1068 rece
    ived exception java.net.NoRouteToHostException: No route to host
    2009-11-19 06:12:30,151 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    10.241.4.101:50010, storageID=DS-886003042-127.0.0.1-50010-1258634370537,
    infoPort=50075, ipcPort=50020):DataXceiver
    java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
    at java.lang.Thread.run(Unknown Source)
    2009-11-19 06:12:36,155 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
    blk_-7007024210780549144_1068 src: /10.241.4.101:41574 dest: /
    10.241.4.101:50010
    2009-11-19 06:12:36,157 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
    blk_-7007024210780549144_1068 received exception
    java.net.NoRouteToHostException: No route to host
    2009-11-19 06:12:36,157 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    10.241.4.101:50010, storageID=DS-886003042-127.0.0.1-50010-1258634370537,
    infoPort=50075, ipcPort=50020):DataXceiver
    java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:282)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
    at java.lang.Thread.run(Unknown Source)
    2009-11-19 06:15:31,776 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_6602943802956432390_1059
    2009-11-19 06:37:06,201 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-3274535555692844186_1063
    2009-11-19 06:45:15,117 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_753873727864345115_1067
    2009-11-19 06:51:56,888 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-810392349062598446_1062
    2009-11-19 07:03:17,441 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-4850126846096803878_1066
    2009-11-19 07:09:26,335 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 12 blocks
    got processed in 3
    msecs



    Any ideas would be greatly appreciated.

    Thanks -Bill


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 20, '09 at 9:01p
activeNov 23, '09 at 2:33a
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase