FAQ
workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
-------------------------------------------------------------------

Key: HADOOP-4997
URL: https://issues.apache.org/jira/browse/HADOOP-4997
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.18.0
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Priority: Blocker
Fix For: 0.18.3



This is a temporary work around issues discussed in HADOOP-4663.

The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • dhruba borthakur (JIRA) at Jan 8, 2009 at 11:03 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-4997:
    -------------------------------------

    Attachment: deleteTmp-0.18.patch

    when the datanode restarts, remove all blocks from the tmp directory of the datanode. However, fsync does not throw an exception.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Jan 8, 2009 at 11:17 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662163#action_12662163 ]

    Hairong Kuang commented on HADOOP-4997:
    ---------------------------------------

    Dhruba, we should also remove the unit tests that test the tmp directory does not get removed after restarting the cluster.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Jan 8, 2009 at 11:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662177#action_12662177 ]

    Hairong Kuang commented on HADOOP-4997:
    ---------------------------------------

    The two junit tests that would fail with this patch are TestFileCreationDelete and TestRenameWhileOpen.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 9, 2009 at 12:27 am
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Raghu Angadi updated HADOOP-4997:
    ---------------------------------

    Attachment: HADOOP-4997-branch-18.patch

    Thanks Hairong.

    The updated patch disables the test Hairong mentioned with a very explicit comment that it is a temporary change and it has the original patch Dhruba attached.

    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 9, 2009 at 1:07 am
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662205#action_12662205 ]

    dhruba borthakur commented on HADOOP-4997:
    ------------------------------------------

    Hi Hairong, I saw ur comment just now about the unit tests. Thanks for fixing it.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 14, 2009 at 8:39 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Raghu Angadi reassigned HADOOP-4997:
    ------------------------------------

    Assignee: (was: Raghu Angadi)
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 16, 2009 at 7:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664642#action_12664642 ]

    Konstantin Shvachko commented on HADOOP-4997:
    ---------------------------------------------

    +1 on Raghu's patch. It is critical for 0.18.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 16, 2009 at 10:02 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Raghu Angadi reassigned HADOOP-4997:
    ------------------------------------

    Assignee: Raghu Angadi
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 16, 2009 at 10:58 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664728#action_12664728 ]

    Raghu Angadi commented on HADOOP-4997:
    --------------------------------------

    test-patch :
    ------------

    [exec] -1 overall.

    [exec] +1 @author. The patch does not contain any @author tags.

    [exec] +1 tests included. The patch appears to include 6 new or modified tests.

    [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

    ----------

    JavaDoc failures are unrelated to this patch (they are for "CyclicIterator.java").

    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 16, 2009 at 11:58 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Raghu Angadi resolved HADOOP-4997.
    ----------------------------------

    Resolution: Fixed
    Release Note: Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees.
    Hadoop Flags: [Incompatible change]

    I just committed this to 0.18 only.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 17, 2009 at 1:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664778#action_12664778 ]

    Konstantin Shvachko commented on HADOOP-4997:
    ---------------------------------------------

    Could you please file a bug against CyclicIterator.java
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 17, 2009 at 2:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664785#action_12664785 ]

    Raghu Angadi commented on HADOOP-4997:
    --------------------------------------
    Could you please file a bug against CyclicIterator.java
    done : HADOOP-5077.
    workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)
    -------------------------------------------------------------------

    Key: HADOOP-4997
    URL: https://issues.apache.org/jira/browse/HADOOP-4997
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.0
    Reporter: Raghu Angadi
    Assignee: Raghu Angadi
    Priority: Blocker
    Fix For: 0.18.3

    Attachments: deleteTmp-0.18.patch, HADOOP-4997-branch-18.patch


    This is a temporary work around issues discussed in HADOOP-4663.
    The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJan 8, '09 at 8:23p
activeJan 17, '09 at 2:06a
posts13
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Raghu Angadi (JIRA): 13 posts

People

Translate

site design / logo © 2022 Grokbase