FAQ
Tool to give the block location and to check if the block at a given datanode is corrupt or not
-----------------------------------------------------------------------------------------------

Key: HADOOP-4761
URL: https://issues.apache.org/jira/browse/HADOOP-4761
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Ramya R
Priority: Minor
Fix For: 0.20.0


It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Lohit Vijayarenu (JIRA) at Dec 3, 2008 at 5:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652867#action_12652867 ]

    Lohit Vijayarenu commented on HADOOP-4761:
    ------------------------------------------
    Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    If the approach is to read the block to check if it is corrupt, then the current behavior is like this. If a read failure is seen on the block, it would be reported to namenode as corrupt replica and added to replication queue. So, even thought the tool (assuming client end ) reports the block as corrupt, it would heal itself after discovery.
    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor
    Fix For: 0.20.0


    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Dec 3, 2008 at 5:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652868#action_12652868 ]

    Doug Cutting commented on HADOOP-4761:
    --------------------------------------

    This should be simple to add. A new command can be added to FsShell.java that calls FileSystem#getBlockLocations and prints the results.
    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor
    Fix For: 0.20.0


    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Dec 3, 2008 at 6:26 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652885#action_12652885 ]

    Konstantin Shvachko commented on HADOOP-4761:
    ---------------------------------------------

    # One can have block locations printed by calling
    {code}
    hadoop fsck /yourDir/yourFile -files -blocks -locations
    {code}
    # Data-nodes currently have a BlockScanner, which periodically on the background verifies data-node blocks and reports corrupted ones to the name-node.

    Is that what you need?
    It might be useful to add an option to fsck to do a complete file verification including its replicas on data-nodes by reading them as previously suggested.
    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor
    Fix For: 0.20.0


    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Ramya R (JIRA) at Dec 4, 2008 at 5:15 am
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653173#action_12653173 ]

    Ramya R commented on HADOOP-4761:
    ---------------------------------
    One can have block locations printed by calling fsck
    Using fsck is not that intuitive. It would be useful to have a separate tool instead. Besides, fsck does not print out the locations given a block-id. It requires a path to the filename only. Also I was just wondering if we could have the hostnames printed rather than the IP addresses. Any comments?
    Is that what you need?
    To be more precise, I was looking for a utility which would tell if a block is corrupt or not given the block-id OR given a datanode and a block-id/file, display if that replica is corrupt or not.

    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor
    Fix For: 0.20.0


    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Dec 4, 2008 at 6:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653395#action_12653395 ]

    Konstantin Shvachko commented on HADOOP-4761:
    ---------------------------------------------
    I was looking for a utility which would tell if a block is corrupt or not given the block-id OR given a datanode and a block-id/file
    Why would you need that?
    The problem here is that we do not have file system api's to access individual blocks. Which gives us (developers) flexibility to change block-ids if we need to. E.g. it was particularly changed recently when appends were introduced.
    What you propose is a backdoor to HDFS internals, which in my opinion should not be supported at all.
    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor
    Fix For: 0.20.0


    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Lei Xu (JIRA) at Dec 5, 2008 at 3:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653839#action_12653839 ]

    Lei Xu commented on HADOOP-4761:
    --------------------------------

    This issue is duplicate to HADOOP-4763 I think.
    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor
    Fix For: 0.20.0


    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Ramya R (JIRA) at Jan 23, 2009 at 5:03 am
    [ https://issues.apache.org/jira/browse/HADOOP-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Ramya R resolved HADOOP-4761.
    -----------------------------

    Resolution: Duplicate

    Marking as duplicate to HADOOP-4763.
    Tool to give the block location and to check if the block at a given datanode is corrupt or not
    -----------------------------------------------------------------------------------------------

    Key: HADOOP-4761
    URL: https://issues.apache.org/jira/browse/HADOOP-4761
    Project: Hadoop Core
    Issue Type: New Feature
    Components: dfs
    Reporter: Ramya R
    Priority: Minor

    It would be useful if we could have a command line tool which would list out the location of all the replicas of a block given a block-id/filename. Also, an utility to check if the block at a given datanode is corrupt or not would be of great help in managing the cluster.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedDec 3, '08 at 3:36p
activeJan 23, '09 at 5:03a
posts8
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Ramya R (JIRA): 8 posts

People

Translate

site design / logo © 2022 Grokbase