[Hadoop-common-dev] [jira] Created: (HADOOP-3363) HDFS throws a InconsistentFSStateException when the name node starts up on a directory that isnt formatted

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595052#action_12595052 ]

Steve Loughran commented on HADOOP-3363:
----------------------------------------

stack trace. The two classes at the root are mine

Caused by: org.apache.hadoop.dfs.InconsistentFSStateException: Directory /tmp/tempdir22787 is in an inconsistent state: /tmp/tempdir22787/image does not exist.
at org.apache.hadoop.dfs.FSImage.isConversionNeeded(FSImage.java:480)
at org.apache.hadoop.dfs.Storage$StorageDirectory.analyzeStorage(Storage.java:283)
at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:148)
at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:79)
at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:254)
at org.apache.hadoop.dfs.FSNamesystem.(FSNamesystem.java:235)
at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:131)
at org.apache.hadoop.dfs.NameNode.(NameNode.java:176)
at org.apache.hadoop.dfs.NameNode.(NameNode.java:162)
at org.apache.hadoop.dfs.ExtNameNode.(ExtNameNode.java:61)
at org.apache.hadoop.dfs.ExtNameNode.createNameNode(ExtNameNode.java:44)
at org.smartfrog.services.hadoop.components.namenode.NamenodeImpl.sfStart(NamenodeImpl.java:73)

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595083#action_12595083 ]

Doug Cutting commented on HADOOP-3363:
--------------------------------------

I'm not sure what you expect here. The filesystem has not yet been formatted, right? If so, attempting to start a namenode should fail--we don't auto-format filesystems. Are you proposing that the exception should be clearer?

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595180#action_12595180 ]

Steve Loughran commented on HADOOP-3363:
----------------------------------------

I certainly expect the operation to fail, but it shouldnt be with a 'you havent upgraded' message.

Looking at the code in analyzeStorage(), there is a check for an unformatted filesystem later, which then returns a StorageState.NOT_FORMATTED response to FSImage.recoverTransitionRead(), which then chooses its actions based on filesystem state. Only it isnt getting a chance to apply its policy when not formatted, because the exception is being thrown and forcing it off on a different path.

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595329#action_12595329 ]

Doug Cutting commented on HADOOP-3363:
--------------------------------------

Okay, that makes sense. We should add a unit test that checks that when a NameNode is started in a non-existant or unformatted directory that a "not formatted" exception is thrown.

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-3363:
------------------------------------

Priority: Blocker (was: Trivial)
Fix Version/s: 0.18.0
Assignee: Konstantin Shvachko

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-3363:
----------------------------------------

Attachment: NotFormatted.patch

Yes the message is confusing. It should just say the name-node is not formatted.
The problem here is that FSImage demands the old image directory to exist even if the storage directory is not formatted.
This patch fixes the problem.
I also add verification that the message contains words "not formatted" in the exception thrown in the appropriate case.

HADOOP-2797 removed everything related to the pre-crc versions, and now we do not need to support pre-upgrade
versions and therefore don't need to check for conversions. I'll file a jira to remove this code.

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-3363:
----------------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604438#action_12604438 ]

Hadoop QA commented on HADOOP-3363:
-----------------------------------

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383886/NotFormatted.patch
against trunk revision 666620.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2647/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2647/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2647/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2647/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lohit Vijayarenu updated HADOOP-3363:
-------------------------------------

Hadoop Flags: [Reviewed]

+1 Looks good

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-3363:
----------------------------------------

Resolution: Fixed
Status: Resolved (was: Patch Available)

I just committed this.

[ https://issues.apache.org/jira/browse/HADOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605430#action_12605430 ]

Hudson commented on HADOOP-3363:
--------------------------------

Integrated in Hadoop-trunk #520 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/520/])