FAQ
I have setup hadoop in pseudo distributed mode with namenode, datanode,
jobtracker and tasktracker all on the same machine...
I also have a code which I use to write my data into hadoop. The code of my
mine reads data from the local disk does some preprocessing and after that
uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
FSDataOutputStreams open at one time cause I want to write data into
different files based on some logic I have.

Now the problem... While the process was writing data to hadoop I got this
error Problem renewing lease for DFSClient_1637324984. On going to name node
logs I found this
2009-02-23 10:02:57,181 FATAL
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
storage directories are inaccessible.

I have absolutely no idea as to what might have caused this. Can some one
please help.

--
Parag Dhanuka

Search Discussions

  • Mithila Nagendra at Feb 24, 2009 at 7:20 am
    Hey Parag
    Check if the namenode and the datanode are up and running. Use the 'jps'
    command to do so. If they are not running U ll have to do a stop-all and
    reformat the namenode using hadoop namenode -format (make sure u have no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log files
    for errors.

    Mithila
    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka wrote:

    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The code of my
    mine reads data from the local disk does some preprocessing and after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 7:28 am
    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I ll
    restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the 'jps'
    command to do so. If they are not running U ll have to do a stop-all and
    reformat the namenode using hadoop namenode -format (make sure u have no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <parag.dhanuka@gmail.com
    wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The code of my
    mine reads data from the local disk does some preprocessing and after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 12:00 pm
    I re ran after formatting and got this error.. Can it be because I have too
    many files opened?
    2009-02-24 02:46:19,284 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.1.201:50010, storageID=DS-1261214527-127.0
    .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
    java.io.IOException: Input/output error
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:883)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(DataXceiver.java:258)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
    at java.lang.Thread.run(Thread.java:619)
    On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka wrote:

    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I ll
    restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the 'jps'
    command to do so. If they are not running U ll have to do a stop-all and
    reformat the namenode using hadoop namenode -format (make sure u have no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <parag.dhanuka@gmail.com
    wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The code of my
    mine reads data from the local disk does some preprocessing and after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 1:32 pm
    I think I know what the problem is... It was related to maximum number of
    file descriptors I can had opened. The limit was 1024 and I was exceeding
    that limit. In order to fix it I increased the number to 65535 using help
    from here
    http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

    The process takes 2 hrs to finish but I am hoping it does finish this time
    without causing any pains.

    Parag
    On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka wrote:

    I re ran after formatting and got this error.. Can it be because I have too
    many files opened?
    2009-02-24 02:46:19,284 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.1.201:50010, storageID=DS-1261214527-127.0
    .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
    java.io.IOException: Input/output error
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:883)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
    at java.lang.Thread.run(Thread.java:619)
    On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka wrote:

    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I ll
    restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the 'jps'
    command to do so. If they are not running U ll have to do a stop-all and
    reformat the namenode using hadoop namenode -format (make sure u have no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <parag.dhanuka@gmail.com
    wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The code of my
    mine reads data from the local disk does some preprocessing and after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 2:13 pm
    It doesnot seem to be helping.... my name node got down again and on trying
    to restart it I got this error....
    2009-02-24 06:06:59,063 ERROR
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
    initialization failed.
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock
    (Input/output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
    2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 9000
    2009-02-24 06:06:59,065 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu
    /output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)



    I am lost please helpppppppppppp

    Parag
    On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka wrote:

    I think I know what the problem is... It was related to maximum number of
    file descriptors I can had opened. The limit was 1024 and I was exceeding
    that limit. In order to fix it I increased the number to 65535 using help
    from here
    http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

    The process takes 2 hrs to finish but I am hoping it does finish this time
    without causing any pains.

    Parag

    On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka wrote:

    I re ran after formatting and got this error.. Can it be because I have
    too many files opened?
    2009-02-24 02:46:19,284 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.1.201:50010, storageID=DS-1261214527-127.0
    .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
    java.io.IOException: Input/output error
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:883)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
    at java.lang.Thread.run(Thread.java:619)
    On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka wrote:

    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I ll
    restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the 'jps'
    command to do so. If they are not running U ll have to do a stop-all and
    reformat the namenode using hadoop namenode -format (make sure u have no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log
    files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <
    parag.dhanuka@gmail.com>wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The code of my
    mine reads data from the local disk does some preprocessing and after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 2:16 pm
    This is how my data log looks like..... Please help.... I am lost now :(
    2009-02-24 05:52:17,613 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
    blk_-7039044529189438628_1012 terminating
    2009-02-24 05:52:17,619 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
    192.168.1.200:55700, dest: /192.168.1.201:50010, byte
    : 22260762, op: HDFS_WRITE, cliID: DFSClient_1664136935, srvID:
    DS-1255721471-127.0.1.1-50010-1235480246191, blockid:
    blk_7602190212483678862_1017
    2009-02-24 05:52:17,619 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
    blk_7602190212483678862_1017 terminating
    2009-02-24 05:55:54,536 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-477808440580664947_1021
    2009-02-24 05:56:54,539 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_1534358347758148859_1021
    2009-02-24 06:01:38,546 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-3560215495928841638_1004
    2009-02-24 06:02:41,560 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
    to hadoop1/192.168.1.201:9000 failed on local
    exception: null
    at org.apache.hadoop.ipc.Client.call(Client.java:699)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
    at $Proxy4.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:669)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1100)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:438)

    Parag
    On Tue, Feb 24, 2009 at 7:43 PM, Parag Dhanuka wrote:

    It doesnot seem to be helping.... my name node got down again and on trying
    to restart it I got this error....
    2009-02-24 06:06:59,063 ERROR
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
    initialization failed.
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock
    (Input/output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
    2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 9000
    2009-02-24 06:06:59,065 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu
    /output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)



    I am lost please helpppppppppppp

    Parag
    On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka wrote:

    I think I know what the problem is... It was related to maximum number of
    file descriptors I can had opened. The limit was 1024 and I was exceeding
    that limit. In order to fix it I increased the number to 65535 using help
    from here
    http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

    The process takes 2 hrs to finish but I am hoping it does finish this time
    without causing any pains.

    Parag

    On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka wrote:

    I re ran after formatting and got this error.. Can it be because I have
    too many files opened?
    2009-02-24 02:46:19,284 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.1.201:50010, storageID=DS-1261214527-127.0
    .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
    java.io.IOException: Input/output error
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:883)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
    at java.lang.Thread.run(Thread.java:619)

    On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka <parag.dhanuka@gmail.com
    wrote:
    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I ll
    restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the
    'jps'
    command to do so. If they are not running U ll have to do a stop-all
    and
    reformat the namenode using hadoop namenode -format (make sure u have
    no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log
    files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <
    parag.dhanuka@gmail.com>wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The code of my
    mine reads data from the local disk does some preprocessing and after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 2:22 pm
    Sorry for spamming but I also found some interesting things in the secondary
    data node and here is the log again.
    My name node crashed at
    2009-02-24 06:02:38,771 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    whilst secondary data node had this
    2009-02-24 05:02:38,741 WARN
    org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done.
    New Image Size: 28519
    2009-02-24 06:02:38,757 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions:
    0 Total time for transactions(ms): 0 Number
    of syncs: 0 SyncTimes(ms): 0
    2009-02-24 06:02:39,089 ERROR
    org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in
    doCheckpoint:
    2009-02-24 06:02:39,089 ERROR
    org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
    java.io.IOException: Call to hadoop1/192.168.1.201:9000 failed
    :


    Can checkpointing be creating some problem..

    Parag
    On Tue, Feb 24, 2009 at 7:46 PM, Parag Dhanuka wrote:

    This is how my data log looks like..... Please help.... I am lost now :(
    2009-02-24 05:52:17,613 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
    blk_-7039044529189438628_1012 terminating
    2009-02-24 05:52:17,619 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
    192.168.1.200:55700, dest: /192.168.1.201:50010, byte
    : 22260762, op: HDFS_WRITE, cliID: DFSClient_1664136935, srvID:
    DS-1255721471-127.0.1.1-50010-1235480246191, blockid:
    blk_7602190212483678862_1017
    2009-02-24 05:52:17,619 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
    blk_7602190212483678862_1017 terminating
    2009-02-24 05:55:54,536 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-477808440580664947_1021
    2009-02-24 05:56:54,539 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_1534358347758148859_1021
    2009-02-24 06:01:38,546 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-3560215495928841638_1004
    2009-02-24 06:02:41,560 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
    to hadoop1/192.168.1.201:9000 failed on local
    exception: null
    at org.apache.hadoop.ipc.Client.call(Client.java:699)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
    at $Proxy4.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:669)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1100)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:438)

    Parag
    On Tue, Feb 24, 2009 at 7:43 PM, Parag Dhanuka wrote:

    It doesnot seem to be helping.... my name node got down again and on
    trying to restart it I got this error....
    2009-02-24 06:06:59,063 ERROR
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
    initialization failed.
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock
    (Input/output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
    2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping server
    on 9000
    2009-02-24 06:06:59,065 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu
    /output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)



    I am lost please helpppppppppppp

    Parag
    On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka wrote:

    I think I know what the problem is... It was related to maximum number of
    file descriptors I can had opened. The limit was 1024 and I was exceeding
    that limit. In order to fix it I increased the number to 65535 using help
    from here
    http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

    The process takes 2 hrs to finish but I am hoping it does finish this
    time without causing any pains.

    Parag

    On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka wrote:

    I re ran after formatting and got this error.. Can it be because I have
    too many files opened?
    2009-02-24 02:46:19,284 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.1.201:50010, storageID=DS-1261214527-127.0
    .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
    java.io.IOException: Input/output error
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:883)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
    at java.lang.Thread.run(Thread.java:619)

    On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka <
    parag.dhanuka@gmail.com> wrote:
    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I ll
    restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the
    'jps'
    command to do so. If they are not running U ll have to do a stop-all
    and
    reformat the namenode using hadoop namenode -format (make sure u have
    no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log
    files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <
    parag.dhanuka@gmail.com>wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The
    code of my
    mine reads data from the local disk does some preprocessing and
    after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka
  • Parag Dhanuka at Feb 24, 2009 at 2:36 pm
    If it helps I also have this in my namenode logs
    2009-02-24 05:31:14,161 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 8 on 9000, call getListing(/data/hadoop/mapred/system) from
    192.168.1.20
    1:40598: error: org.apache.hadoop.security.AccessControlException:
    Permission denied: user=webuser, access=READ_EXECUTE,
    inode="system":hadoop:supergr
    oup:rwx-wx-wx
    org.apache.hadoop.security.AccessControlException: Permission denied:
    user=webuser, access=READ_EXECUTE,
    inode="system":hadoop:supergroup:rwx-wx-wx
    at
    org.apache.hadoop.hdfs.server.namenode.PermissionChecker.check(PermissionChecker.java:176)
    at
    org.apache.hadoop.hdfs.server.namenode.PermissionChecker.checkPermission(PermissionChecker.java:111)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4386)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:4346)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:1969)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.getListing(NameNode.java:485)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)


    On Tue, Feb 24, 2009 at 7:51 PM, Parag Dhanuka wrote:

    Sorry for spamming but I also found some interesting things in the
    secondary data node and here is the log again.
    My name node crashed at
    2009-02-24 06:02:38,771 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    whilst secondary data node had this
    2009-02-24 05:02:38,741 WARN
    org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done.
    New Image Size: 28519
    2009-02-24 06:02:38,757 INFO
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions:
    0 Total time for transactions(ms): 0 Number
    of syncs: 0 SyncTimes(ms): 0
    2009-02-24 06:02:39,089 ERROR
    org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in
    doCheckpoint:
    2009-02-24 06:02:39,089 ERROR
    org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
    java.io.IOException: Call to hadoop1/192.168.1.201:9000 failed
    :


    Can checkpointing be creating some problem..

    Parag
    On Tue, Feb 24, 2009 at 7:46 PM, Parag Dhanuka wrote:

    This is how my data log looks like..... Please help.... I am lost now :(
    2009-02-24 05:52:17,613 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
    blk_-7039044529189438628_1012 terminating
    2009-02-24 05:52:17,619 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
    192.168.1.200:55700, dest: /192.168.1.201:50010, byte
    : 22260762, op: HDFS_WRITE, cliID: DFSClient_1664136935, srvID:
    DS-1255721471-127.0.1.1-50010-1235480246191, blockid:
    blk_7602190212483678862_1017
    2009-02-24 05:52:17,619 INFO
    org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
    blk_7602190212483678862_1017 terminating
    2009-02-24 05:55:54,536 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-477808440580664947_1021
    2009-02-24 05:56:54,539 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_1534358347758148859_1021
    2009-02-24 06:01:38,546 INFO
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
    succeeded for blk_-3560215495928841638_1004
    2009-02-24 06:02:41,560 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
    to hadoop1/192.168.1.201:9000 failed on local
    exception: null
    at org.apache.hadoop.ipc.Client.call(Client.java:699)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
    at $Proxy4.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:669)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1100)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:438)

    Parag
    On Tue, Feb 24, 2009 at 7:43 PM, Parag Dhanuka wrote:

    It doesnot seem to be helping.... my name node got down again and on
    trying to restart it I got this error....
    2009-02-24 06:06:59,063 ERROR
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
    initialization failed.
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock
    (Input/output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
    2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping
    server on 9000
    2009-02-24 06:06:59,065 ERROR
    org.apache.hadoop.hdfs.server.namenode.NameNode:
    java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu
    /output error)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
    at
    org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
    at
    org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)



    I am lost please helpppppppppppp

    Parag
    On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka wrote:

    I think I know what the problem is... It was related to maximum number
    of file descriptors I can had opened. The limit was 1024 and I was exceeding
    that limit. In order to fix it I increased the number to 65535 using help
    from here
    http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

    The process takes 2 hrs to finish but I am hoping it does finish this
    time without causing any pains.

    Parag


    On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka <parag.dhanuka@gmail.com
    wrote:
    I re ran after formatting and got this error.. Can it be because I have
    too many files opened?
    2009-02-24 02:46:19,284 ERROR
    org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
    192.168.1.201:50010, storageID=DS-1261214527-127.0
    .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
    java.io.IOException: Input/output error
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createNewFile(File.java:883)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
    at
    org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
    at java.lang.Thread.run(Thread.java:619)

    On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka <
    parag.dhanuka@gmail.com> wrote:
    Hi Mithila,
    Thanks for a quick reply...

    I did check the name node and data node logs... They have died and I
    ll restart them but I just want to know what happened before doing anything.

    On the data node the interesting error which I saw is this:


    java.io.EOFException: while trying to read 65557 bytes
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
    at
    org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
    at
    org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)

    At the same time I saw the following error on name node
    2009-02-23 10:02:57,181
    FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
    All storage directories are inaccessible.

    Can you please elaborate as to what this might be.

    Parag
    On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra wrote:

    Hey Parag
    Check if the namenode and the datanode are up and running. Use the
    'jps'
    command to do so. If they are not running U ll have to do a stop-all
    and
    reformat the namenode using hadoop namenode -format (make sure u have
    no
    data on the HDFS). Then restart hadoop using start-all.sh.

    If U have the datanode and the namenode running, then check the log
    files
    for errors.

    Mithila

    On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <
    parag.dhanuka@gmail.com>wrote:
    I have setup hadoop in pseudo distributed mode with namenode, datanode,
    jobtracker and tasktracker all on the same machine...
    I also have a code which I use to write my data into hadoop. The
    code of my
    mine reads data from the local disk does some preprocessing and
    after that
    uses (multiple) FSDataOutputStream to write data to hadoop. I have multiple
    FSDataOutputStreams open at one time cause I want to write data into
    different files based on some logic I have.

    Now the problem... While the process was writing data to hadoop I got this
    error Problem renewing lease for DFSClient_1637324984. On going to name
    node
    logs I found this
    2009-02-23 10:02:57,181 FATAL
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.

    I have absolutely no idea as to what might have caused this. Can some one
    please help.

    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka


    --
    Parag Dhanuka

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 24, '09 at 7:03a
activeFeb 24, '09 at 2:36p
posts9
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Parag Dhanuka: 8 posts Mithila Nagendra: 1 post

People

Translate

site design / logo © 2021 Grokbase