FAQ
Blocks remain under-replicated
------------------------------

Key: HADOOP-5465
URL: https://issues.apache.org/jira/browse/HADOOP-5465
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.18.3
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Priority: Blocker
Fix For: 0.18.4


Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
2. When taking meta save in two different days, some blocks remain in under-replication queue.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Hairong Kuang (JIRA) at Mar 11, 2009 at 11:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681098#action_12681098 ]

    Hairong Kuang commented on HADOOP-5465:
    ---------------------------------------

    Thank Koji for his tireless investigation on this issue.

    When this situation occurs, the source DataNode of the block shows abnormal behavior. No blocks gets replicated from this node or no block gets removed from this node. Digging into the problem, we seet that NameNode sends the DataNode an empty replication request, i.e. a replication request with no blocks and targets as parameters, on every heartbeat reply, thus preventing sending the node any replication or deletion request. More suspiciously DataNode notifies NameNode that it has 1 replication in progress although its jstack shows that it has no replication (data transfer) thread alive.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 12:07 am
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681107#action_12681107 ]

    Hairong Kuang commented on HADOOP-5465:
    ---------------------------------------

    Two bugs in DFS contributed to the problem:
    (1). DataNode does not sync on modification to the counter "xmitsInProgress", which keeps track of the number of replication in progress. When two threads update the counter concurrently, race condition may occurs. The counter may change to be a non-zero value when no replication is going on.
    (2). Each DN is configured to have at most 2 replications in progress. When DN notifies NN that it has 1 replication in progress, NN should be able to send one block replication request to DN. But NN wrongly interprets the counter as the number of targets. When it sees that the block is scheduled to 2 targets but DN can only take 1, it sends an empty replication request to DN. As a result, blocking all replication from this DataNode. If the DataNode is the only source of an under-replicated block, the block will never gets replicated.

    Fixing either one or two could fix the problem. I think (1) is more fundamental so I will fix (1) in this jira and file a different jira to fix (2).
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 5:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Attachment: xmitsSync.patch
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4

    Attachments: xmitsSync.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Mar 12, 2009 at 6:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681425#action_12681425 ]

    Raghu Angadi commented on HADOOP-5465:
    --------------------------------------

    This implies that all the blocks that remained under replicated have only one replica and only on this specific datanode. Was that the case?
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4

    Attachments: xmitsSync.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 6:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Attachment: xmitsSync1.patch

    The previous patch synced the counter on a wrong object. This patch uses AtomicInteger to guarantee atomic modification.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 6:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Attachment: (was: xmitsSync.patch)
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 6:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681446#action_12681446 ]

    Hairong Kuang commented on HADOOP-5465:
    ---------------------------------------
    This implies that all the blocks that remained under replicated have only one replica and only on this specific datanode. Was that the case?
    Yes, most of the blocks have only one source. Those are the kind of blocks that initially triggers a DataNode into this state. But we could and our clusters do have under-replicated blocks that have two replicas and all its sources are in this state. The only exception is a block in our clusters that has two sources, one in this state but the other is replicating. This block is still under investigation.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 7:03 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Fix Version/s: 0.21.0
    0.20.0
    0.19.2
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Mar 12, 2009 at 7:57 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681463#action_12681463 ]

    Raghu Angadi commented on HADOOP-5465:
    --------------------------------------

    Thanks Hairong. Since a rare race condition is suspected, I thought there would be very few datanodes hitting such a race condition.

    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 8:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681483#action_12681483 ]

    Hairong Kuang commented on HADOOP-5465:
    ---------------------------------------
    I thought there would be very few datanodes hitting such a race condition.
    On a cluster with thousands of machines, we saw 5% of the nodes were in this state.
    The only exception is a block in our clusters that has two sources, one in this state but the other is replicating.
    It turns out that the other source that is replicating has a corrupt copy of the block.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 9:39 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681107#action_12681107 ]

    Hairong Kuang edited comment on HADOOP-5465 at 3/12/09 2:37 PM:
    ----------------------------------------------------------------

    Two bugs in DFS contributed to the problem:
    (1). DataNode does not sync on modification to the counter "xmitsInProgress", which keeps track of the number of replication in progress. When two threads update the counter concurrently, race condition may occurs. The counter may change to be a non-zero value when no replication is going on.
    (2). Each DN is configured to have at most 2 replications in progress. When DN notifies NN that it has 1 replication in progress, NN should be able to send one block replication request to DN. But NN wrongly interprets the counter as the number of targets. When it sees that the block is scheduled to 2 targets but DN can only take 1, it sends an empty replication request to DN. As a result, blocking all replications from this DataNode. If the DataNode is the only source of an under-replicated block, the block will never get replicated.

    Fixing either (1) or (2) could fix the problem. I think (1) is more fundamental so I will fix (1) in this jira and file a different jira to fix (2).

    was (Author: hairong):
    Two bugs in DFS contributed to the problem:
    (1). DataNode does not sync on modification to the counter "xmitsInProgress", which keeps track of the number of replication in progress. When two threads update the counter concurrently, race condition may occurs. The counter may change to be a non-zero value when no replication is going on.
    (2). Each DN is configured to have at most 2 replications in progress. When DN notifies NN that it has 1 replication in progress, NN should be able to send one block replication request to DN. But NN wrongly interprets the counter as the number of targets. When it sees that the block is scheduled to 2 targets but DN can only take 1, it sends an empty replication request to DN. As a result, blocking all replication from this DataNode. If the DataNode is the only source of an under-replicated block, the block will never gets replicated.

    Fixing either one or two could fix the problem. I think (1) is more fundamental so I will fix (1) in this jira and file a different jira to fix (2).
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Mar 12, 2009 at 10:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681527#action_12681527 ]

    Raghu Angadi commented on HADOOP-5465:
    --------------------------------------

    Since this is such a vital stat, may be better to decrement at the top of finally block (so that some other runtime exception does not cause this situation again).


    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 11:33 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681544#action_12681544 ]

    Hairong Kuang commented on HADOOP-5465:
    ---------------------------------------

    The patch incorporates Raghu's comment.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 11:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Status: Patch Available (was: Open)
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 12, 2009 at 11:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Attachment: xmitsSync2.patch
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Mar 12, 2009 at 11:47 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681550#action_12681550 ]

    Raghu Angadi commented on HADOOP-5465:
    --------------------------------------

    +1.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Mar 13, 2009 at 3:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681591#action_12681591 ]

    Hadoop QA commented on HADOOP-5465:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12402100/xmitsSync2.patch
    against trunk revision 753052.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    +1 contrib tests. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/82/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/82/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/82/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/82/console

    This message is automatically generated.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 13, 2009 at 8:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Resolution: Fixed
    Hadoop Flags: [Reviewed]
    Status: Resolved (was: Patch Available)

    I've just committed this.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2-br18.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 13, 2009 at 8:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hairong Kuang updated HADOOP-5465:
    ----------------------------------

    Attachment: xmitsSync2-br18.patch

    Attach a patch to 0.18.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2-br18.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Mar 14, 2009 at 3:14 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682050#action_12682050 ]

    Hudson commented on HADOOP-5465:
    --------------------------------

    Integrated in Hadoop-trunk #779 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/779/])
    . Blocks remain under-replicated. Contributed by Hairong Kuang.

    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2-br18.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hairong Kuang (JIRA) at Mar 20, 2009 at 5:55 am
    [ https://issues.apache.org/jira/browse/HADOOP-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683758#action_12683758 ]

    Hairong Kuang commented on HADOOP-5465:
    ---------------------------------------

    This jira is too trivial to add a unit test.
    Blocks remain under-replicated
    ------------------------------

    Key: HADOOP-5465
    URL: https://issues.apache.org/jira/browse/HADOOP-5465
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Affects Versions: 0.18.3
    Reporter: Hairong Kuang
    Assignee: Hairong Kuang
    Priority: Blocker
    Fix For: 0.18.4, 0.19.2, 0.20.0, 0.21.0

    Attachments: xmitsSync1.patch, xmitsSync2-br18.patch, xmitsSync2.patch


    Occasionally we see some blocks remain to be under-replicated in our production clusters. This is what we obeserved:
    1. Sometimes when increasing the replication factor of a file, some blocks belonged to this file do not get to increase to the new replication factor.
    2. When taking meta save in two different days, some blocks remain in under-replication queue.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 11, '09 at 11:05p
activeMar 20, '09 at 5:55a
posts22
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hairong Kuang (JIRA): 22 posts

People

Translate

site design / logo © 2022 Grokbase