[Hadoop-common-dev] [jira] Created: (HADOOP-6017) NameNode and SecondaryNameNode fail to restart because of abnormal filenames.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718691#action_12718691 ]

Raghu Angadi commented on HADOOP-6017:
--------------------------------------

Stacktrace of such a failure:

{noformat}

2009-06-11 07:14:30,798 ERROR org.apache.hadoop.dfs.NameNode.Secondary:
Throwable Exception in doCheckpoint:
2009-06-11 07:14:30,798 ERROR org.apache.hadoop.dfs.NameNode.Secondary:
java.lang.IllegalArgumentException: Illegal group reference
at java.util.regex.Matcher.appendReplacement(Matcher.java:713)
at java.util.regex.Matcher.replaceFirst(Matcher.java:861)
at java.lang.String.replaceFirst(String.java:2147)
at
org.apache.hadoop.dfs.LeaseManager.changeLease(LeaseManager.java:288)
at
org.apache.hadoop.dfs.FSNamesystem.changeLease(FSNamesystem.java:4441)
at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:563)
at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
at
org.apache.hadoop.dfs.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:567)
at
org.apache.hadoop.dfs.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:464)
at
org.apache.hadoop.dfs.SecondaryNameNode.doMerge(SecondaryNameNode.java:341)
at
org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:305)
at
org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:216)
at java.lang.Thread.run(Thread.java:619)

2009-06-11 07:14:30,842 INFO org.apache.hadoop.dfs.NameNode.Secondary:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at host/ip
************************************************************/
{noformat}

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6017:
-------------------------------------------

Attachment: 6017_20090611.patch

6017_20090611.patch: we should not use String.replaceFirst(..) at all.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6017:
-------------------------------------------

Attachment: 6017_20090611b.patch

6017_20090611b.patch: changed the unit test a little bit.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raghu Angadi updated HADOOP-6017:
---------------------------------

Attachment: HADOOP-6017-branch-18.patch

+1. The patch looks good.

I am attaching the patch 0.18 branch.

There are more changes required for TestRenameWhileOpen.java since the rename test that the new test is part of was disabled in 0.18. This patch enables parts of the test.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718705#action_12718705 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-6017:
------------------------------------------------

{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718940#action_12718940 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-6017:
------------------------------------------------

I ran unit tests on trunk with 6017_20090611b.patch. The only failed test was TestReduceFetch, which seemed unrelated. Then, I ran TestReduceFetch on a clean trunk. It also failed in both my linux and windows machines.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718952#action_12718952 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-6017:
------------------------------------------------

I filed HADOOP-6029 for TestReduceFetch.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6017:
-------------------------------------------

Component/s: dfs
Description:
SecondaryNameNode (and NameNode) fail to load the edits log. I will include stack trace in next comment.

This is traced to the fact that LeaseManager uses {{String.relaceFirst()}} to replace front of a sting with another string. Unfortunately {{replaceFirst()}} uses regex, though the first argument is {{quoted}} by the code, the second argument is not. (the second arg is not really treated as regex but still gets processed for back references (as in '{{sed s/first/second/g}}')

As Nicholas suggested, it is just simpler to use {{substring()}} to replace part of the string.

was:

SecondaryNameNode (and NameNode) fail to load the edits log. I will include stack trace in next comment.

This is traced to the fact that LeaseManager uses {{String.relaceFirst()}} to replace front of a sting with another string. Unfortunately {{replaceFirst()}} uses regex, though the first argument is {{quoted}} by the code, the second argument is not. (the second arg is not really treated as regex but still gets processed for back references (as in '{{sed s/first/second/g}}')

As Nicholas suggested, it is just simpler to use {{substring()}} to replace part of the string.

Hadoop Flags: [Reviewed]

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raghu Angadi updated HADOOP-6017:
---------------------------------

Status: Patch Available (was: Open)

I am planning to commit these. Hudson is currently running Jun 11th patches.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raghu Angadi updated HADOOP-6017:
---------------------------------

Resolution: Fixed
Release Note: rename to a file name with certain characters can cause fatal exceptions in NameNode initialization and Secondary NameNode.
Status: Resolved (was: Patch Available)

I just committed this. Thanks Nicholas.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raghu Angadi updated HADOOP-6017:
---------------------------------

Attachment: HADOOP-6017-branch-20.patch

Patch for branch 0.20 is attached. The trunk patch does not apply because of path difference for hdfs tests.

This patch applies to 0.19 as well.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720277#action_12720277 ]

Allen Wittenauer commented on HADOOP-6017:
------------------------------------------

Looking at the patch, this appears to only fix input/output validation. How do we deal with our currently corrupted edits file? This patch needs to fix that as well!

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720316#action_12720316 ]

Raghu Angadi commented on HADOOP-6017:
--------------------------------------

This fixes actual bug. The edits file is not corrupted. It is just that NameNode didn't handle filenames properly for certain edit log entries. With this patch, NN and secondary NN can handle the same edit log properly.

[ https://issues.apache.org/jira/browse/HADOOP-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720844#action_12720844 ]

Hudson commented on HADOOP-6017:
--------------------------------

Integrated in Hadoop-trunk #869 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/869/])