Grokbase › Groups › Hadoop › common-user › December 2009
FAQ
Hi,

Just one week since upgrading to 0.20.1, I've been hit twice by NN
crashes. The symptoms were the same. The NN log says:

2009-12-01 12:04:00,420 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
10.63.118.5
2009-12-01 12:04:00,420 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
transactions: 73801 Total time for transactions(ms): 222Number of
transactions batched in Syncs: 20359 Number of syncs: 40349
SyncTimes(ms): 12206
2009-12-01 12:04:00,421 FATAL
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
storage directories are inaccessible.
2009-12-01 12:04:00,424 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
...

It seems the NN crashed when rolling edit log from the secondary NN. On
the secondary NN side, there were a bunch of "connection rejected"
errors.

Any clue? Thanks for help.


Zhang

Search Discussions

  • Eli Collins at Dec 4, 2009 at 10:03 pm ⇧
    Hey Zhang,
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.
    Are the directories specified by dfs.namenode.[name|edits].dir
    accessible? Perhaps they're NFS mounts that are flaking out?

    Thanks,
    Eli
  • Zhang, Zhang at Dec 4, 2009 at 11:01 pm ⇧
    Eli,

    Thanks for responding to my question. I guess what you're mentioning are
    dfs.name.edits.dir and dfs.namenode.name.dir. They do not contain any
    NFS mounted directories. They were all accessible (can be seen by UNIX
    shell commands such as ls and du) when the namenode went down. The error
    message is just the symptom. It is not the cause of the failure.

    Zhang


    -----Original Message-----
    From: Eli Collins
    Sent: Friday, December 04, 2009 2:03 PM
    To: common-user@hadoop.apache.org
    Subject: Re: Namenode crashes while rolling edit log from secondary
    namenode

    Hey Zhang,
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.
    Are the directories specified by dfs.namenode.[name|edits].dir
    accessible? Perhaps they're NFS mounts that are flaking out?

    Thanks,
    Eli
  • Eli Collins at Dec 5, 2009 at 5:13 am ⇧
    Hey Zhang,

    Thanks for the info. Can you be more specific, eg by NN crash do you
    mean NPE? Relevant logs would be helpful.

    Thanks,
    Eli
    On Fri, Dec 4, 2009 at 2:59 PM, Zhang, Zhang wrote:

    Eli,

    Thanks for responding to my question. I guess what you're mentioning are
    dfs.name.edits.dir and dfs.namenode.name.dir. They do not contain any
    NFS mounted directories. They were all accessible (can be seen by UNIX
    shell commands such as ls and du) when the namenode went down. The error
    message is just the symptom. It is not the cause of the failure.

    Zhang


    -----Original Message-----
    From: Eli Collins
    Sent: Friday, December 04, 2009 2:03 PM
    To: common-user@hadoop.apache.org
    Subject: Re: Namenode crashes while rolling edit log from secondary
    namenode

    Hey Zhang,
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
    storage directories are inaccessible.
    Are the directories specified by dfs.namenode.[name|edits].dir
    accessible? Perhaps they're NFS mounts that are flaking out?

    Thanks,
    Eli

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 1, '09 at 11:53p
activeDec 5, '09 at 5:13a
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Zhang, Zhang: 2 posts Eli Collins: 2 posts

Content

People

Support

Translate

site design / logo © 2022 Grokbase