Looking at the codebase it seems to suggest that it ignores a editlog
storage directory if it encounters an error


Check lines:
Code in line 334
comment: 387 - 390
comment: 411 - 414
Comment: 433 - 436

The processIOError method is called throughout the code if it encounters an

A fatal error is only thrown if none of the storage directories is
accessible. Lines 394, 420

- Sudhir

On Aug/23/ 2:21 PM, "common-user-digest-help@hadoop.apache.org"
From: Michael Segel <michael_segel@hotmail.com>
Date: Mon, 23 Aug 2010 14:05:05 -0500
To: <common-user@hadoop.apache.org>
Subject: RE: what will happen if a backup name node folder becomes


Now you have me confused.
Everything we've seen says that writing to both a local disk and to an NFS
mounted disk would be the best way to prevent a problem.

Now you and Harsh J say that this could actually be problematic.

Which is it?
Is this now a defect that should be addressed, or should we just not use an
NFS mounted drive?



Date: Mon, 23 Aug 2010 11:42:59 -0700
From: licht_jiang@yahoo.com
Subject: Re: what will happen if a backup name node folder becomes
To: common-user@hadoop.apache.org

This makes a good argument. Actually, after seeing the previous reply, I
kindof convinced that I should go back to "sync" the meta data to a backup
location instead of using this feature, which as David mentioned, introduced
a 2nd single point of failure to hadoop, which degrades the availability of
hadoop. BTW, we are using cloudera package hadoop-0.20.2+228. Can someone
confirm whether a name node will shut down given that a backup folder listed
in "dfs.name.dir" becomes unavailable in this version?



--- On Sun, 8/22/10, David B. Ritch wrote:

From: David B. Ritch <david.ritch@gmail.com>
Subject: Re: what will happen if a backup name node folder becomes
To: common-user@hadoop.apache.org
Date: Sunday, August 22, 2010, 11:34 PM

Which version of Hadoop was this? The folks at Cloudera have assured
me that the namenode in CDH2 will continue as long as one of the
directories is still writable.

It *does* seem a bit of a waste if an availability feature - the ability
to write to multiple directories - actually reduces availability by
providing an additional single point of failure.


On 8/20/2010 5:27 PM, Harsh J wrote:
Whee, lets try it out:

Start with both paths available. ... Starts fine.
Store some files. ... Works.
rm -r the second path. ... Ouch.
Store some more files. ... Still Works. [Cuz the SNN hasn't sent us
stuff back yet]
Wait for checkpoint to hit.
And ...

2010-08-21 02:42:00,385 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
2010-08-21 02:42:00,385 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
transactions: 37 Total time for transactions(ms): 6Number of
transactions batched in Syncs: 0 Number of syncs: 26 SyncTimes(ms):
307 277
2010-08-21 02:42:00,439 FATAL
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
storage directories are inaccessible.
2010-08-21 02:42:00,440 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
SHUTDOWN_MSG: Shutting down NameNode at localhost/

So yes, as Edward says - never let this happen!
On Sat, Aug 21, 2010 at 2:26 AM, jiang licht wrote:
Using nfs folder to back up dfs meta information as follows,


where /hadoop-backup is on a backup machine and mounted on the master node.

I have a question: if somehow, the backup folder becomes unavailable, will
it freeze master node? That is, will write operation simply hang up on this
condition on the master node? Or will master node log the problem and
continues to work?



iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 2 | next ›
Discussion Overview
groupcommon-user @
postedAug 24, '10 at 5:21a
activeAug 24, '10 at 6:12a

2 users in discussion

Sudhir Vallamkondu: 1 post Harsh J: 1 post



site design / logo © 2022 Grokbase