FAQ
Hi,

We are noticing the following errors in our datanode logs. We are running
hbase on top of hdfs, but are not noticing any errors in the hbase logs. So
it seems like the hdfs clients are not suffering from these errors. However,
it would be nice to understand why they appear. We have upped the number of
xcievers to 1024 and are having some throughts about there are too many
sockets and some of them are timing out because they are not being used?
Also we have set dfs.datanode.socket.write.timeout and dfs.socket.timeout to
about 10 minutes. Anyone knows why these errors appears, and perhaps how to
get rid of them?

2010-07-15 12:31:02,307 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.10.54:50010,
storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075,
ipcPort=50020):Got exception while serving blk_3305049902326993023_98358 to
/192.168.10.54:
java.net.SocketTimeoutException: 600000 millis timeout while waiting for
channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010remote=/
192.168.10.54:46276]
at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
at java.lang.Thread.run(Thread.java:619)

2010-07-15 12:31:02,308 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.10.54:50010,
storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075,
ipcPort=50020):DataXceiver
java.net.SocketTimeoutException: 600000 millis timeout while waiting for
channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010remote=/
192.168.10.54:46276]
at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
at java.lang.Thread.run(Thread.java:619)

TIA,
Peter

Search Discussions

  • Ken Goodhope at Jul 15, 2010 at 4:25 pm
    What is your file descriptor limit? If it is set to the 1024, you will want
    to up that considerably. Don't be afraid to go to 64k.
    On Thu, Jul 15, 2010 at 3:38 AM, Peter Falk wrote:

    Hi,

    We are noticing the following errors in our datanode logs. We are running
    hbase on top of hdfs, but are not noticing any errors in the hbase logs. So
    it seems like the hdfs clients are not suffering from these errors.
    However,
    it would be nice to understand why they appear. We have upped the number of
    xcievers to 1024 and are having some throughts about there are too many
    sockets and some of them are timing out because they are not being used?
    Also we have set dfs.datanode.socket.write.timeout and dfs.socket.timeout
    to
    about 10 minutes. Anyone knows why these errors appears, and perhaps how to
    get rid of them?

    2010-07-15 12:31:02,307 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.10.54:50010,
    storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075,
    ipcPort=50020):Got exception while serving blk_3305049902326993023_98358 to
    /192.168.10.54:
    java.net.SocketTimeoutException: 600000 millis timeout while waiting for
    channel to be ready for write. ch :
    java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010
    remote=/
    192.168.10.54:46276]
    at

    org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
    at

    org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
    at

    org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
    at

    org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
    at

    org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
    at

    org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
    at java.lang.Thread.run(Thread.java:619)

    2010-07-15 12:31:02,308 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.10.54:50010,
    storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075,
    ipcPort=50020):DataXceiver
    java.net.SocketTimeoutException: 600000 millis timeout while waiting for
    channel to be ready for write. ch :
    java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010
    remote=/
    192.168.10.54:46276]
    at

    org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
    at

    org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
    at

    org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
    at

    org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
    at

    org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
    at

    org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
    at java.lang.Thread.run(Thread.java:619)

    TIA,
    Peter

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 15, '10 at 10:40a
activeJul 15, '10 at 4:25p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Ken Goodhope: 1 post Peter Falk: 1 post

People

Translate

site design / logo © 2022 Grokbase