FAQ
'-setrep' command skips under-replicated blocks
-----------------------------------------------

Key: HADOOP-4598
URL: https://issues.apache.org/jira/browse/HADOOP-4598
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Koji Noguchi
Priority: Minor


When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling

hadoop dfs -setrep <higher_replication> <filename>

would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Hairong Kuang (JIRA) at Nov 6, 2008 at 12:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang reassigned HADOOP-4598:
    -------------------------------------

    Assignee: Hairong Kuang
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor

    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Nov 14, 2008 at 7:52 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647709#action_12647709 ]

    Hairong Kuang commented on HADOOP-4598:
    ---------------------------------------

    UnderReplicatedBlocks.update has an optimization that a under-replication block does not get added to the queue if a block was under-replicated and its previous priority is the same as the current priority. This optimization is based on the assumption that a previously under-replicated block should have already been in the queue.

    For the case that Koji filed, due to for some bugs like HADOOP-4597, under-replicated blocks did not get to place in the under-replicated block queue at NameNode startup time. Later on changing a file's replication factor to be a bigger number does not trigger the move of the file's blocks into the queue if the replication factor change does not change their priority.

    The fix for this problem is to remove the optimization.
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor

    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Nov 14, 2008 at 11:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-4598:
    ----------------------------------

    Attachment: setRep.patch

    A patch for review.
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor
    Attachments: setRep.patch


    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Nov 19, 2008 at 12:22 am
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648836#action_12648836 ]

    Konstantin Shvachko commented on HADOOP-4598:
    ---------------------------------------------

    +1 This should work.
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor
    Attachments: setRep.patch


    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Nov 19, 2008 at 1:02 am
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-4598:
    ----------------------------------

    Hadoop Flags: [Reviewed]
    Status: Patch Available (was: Open)
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor
    Attachments: setRep.patch


    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Nov 20, 2008 at 9:43 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649492#action_12649492 ]

    Hairong Kuang commented on HADOOP-4598:
    ---------------------------------------

    Ant test-core passed on my local machine.
    BUILD SUCCESSFUL
    Total time: 124 minutes 24 seconds

    Ant test-patch result:
    [exec] +1 overall.

    [exec] +1 @author. The patch does not contain any @author tags.

    [exec] +1 tests included. The patch appears to include 4 new or modified tests.

    [exec] +1 javadoc. The javadoc tool did not generate any warningmessages.

    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor
    Attachments: setRep.patch


    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Nov 20, 2008 at 10:03 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-4598:
    ----------------------------------

    Resolution: Fixed
    Fix Version/s: 0.20.0
    Status: Resolved (was: Patch Available)

    I just committed this!
    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor
    Fix For: 0.20.0

    Attachments: setRep.patch


    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Nov 21, 2008 at 4:30 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649697#action_12649697 ]

    Hudson commented on HADOOP-4598:
    --------------------------------

    Integrated in Hadoop-trunk #667 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/667/])
    . Setrep command skips under-replicated blocks. Contributed by Hairong Kuang.

    '-setrep' command skips under-replicated blocks
    -----------------------------------------------

    Key: HADOOP-4598
    URL: https://issues.apache.org/jira/browse/HADOOP-4598
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Koji Noguchi
    Assignee: Hairong Kuang
    Priority: Minor
    Fix For: 0.20.0

    Attachments: setRep.patch


    When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
    hadoop dfs -setrep <higher_replication> <filename>
    would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedNov 6, '08 at 12:17a
activeNov 21, '08 at 4:30p
posts9
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hudson (JIRA): 9 posts

People

Translate

site design / logo © 2022 Grokbase