FAQ
Block report processing should compare gneration stamp
------------------------------------------------------

Key: HADOOP-5027
URL: https://issues.apache.org/jira/browse/HADOOP-5027
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Tsz Wo (Nicholas), SZE


If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 14, 2009 at 2:37 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663605#action_12663605 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5027:
    ------------------------------------------------

    This is not a problem if both sync and append are not used.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE

    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 14, 2009 at 6:56 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-5027:
    -------------------------------------

    Fix Version/s: 0.19.1
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 14, 2009 at 7:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663862#action_12663862 ]

    dhruba borthakur commented on HADOOP-5027:
    ------------------------------------------

    A block-report processing should ignore blocks if it is the last block in an file that is under construction.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 14, 2009 at 8:15 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663875#action_12663875 ]

    Raghu Angadi commented on HADOOP-5027:
    --------------------------------------

    Doesn't NN already know what the latest genstamp should be? All the genstamp changes are made by NN.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 14, 2009 at 9:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663895#action_12663895 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5027:
    ------------------------------------------------
    Doesn't NN already know what the latest genstamp should be?
    When a block is being updated in lease recovery, some replicators might have a gs larger then the one stored in the NameNode. If the update fails, the latest gs may not be committed to the NameNode.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 15, 2009 at 12:07 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-5027:
    -------------------------------------------

    Attachment: 5027_20090114.patch

    5027_20090114.patch: check generation stamps in block report processing.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 15, 2009 at 7:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664030#action_12664030 ]

    dhruba borthakur commented on HADOOP-5027:
    ------------------------------------------

    This code looks good to me. There is already a unit test TestFleAppend2 that tests "append". Any clues on why this bug is not caught by that unit test?
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 16, 2009 at 1:29 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664351#action_12664351 ]

    Konstantin Shvachko commented on HADOOP-5027:
    ---------------------------------------------

    May be the problem is not with the block report, but rather with {{nextGenerationStamp(Block)}}, which does not update generation stamp for the block. The client should call name-node to obtain a new generation stamp any way, I believe this what Raghu meant? So why do we not changing the gen stamp of the block.
    I also would like to see a unit test catching this bug.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 16, 2009 at 1:31 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664351#action_12664351 ]

    shv edited comment on HADOOP-5027 at 1/15/09 5:29 PM:
    ----------------------------------------------------------------------

    May be the problem is not with the block report, but rather with {{nextGenerationStamp(Block)}}, which does not update generation stamp for the block. The -client- _data-node_ should call name-node to obtain a new generation stamp any way, I believe this what Raghu meant? So why do we not changing the gen stamp of the block.
    I also would like to see a unit test catching this bug.

    was (Author: shv):
    May be the problem is not with the block report, but rather with {{nextGenerationStamp(Block)}}, which does not update generation stamp for the block. The client should call name-node to obtain a new generation stamp any way, I believe this what Raghu meant? So why do we not changing the gen stamp of the block.
    I also would like to see a unit test catching this bug.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 16, 2009 at 1:35 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664356#action_12664356 ]

    Raghu Angadi commented on HADOOP-5027:
    --------------------------------------
    The client data-node should call name-node to obtain a new generation stamp any way, I believe this what Raghu meant?
    Yes. I thought all the gen-stamp changes are done by NN.

    It is a bit scary if generation can be changed by clients and datanodes. That would be very different from rest of HDFS metadata management.

    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 16, 2009 at 1:35 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664355#action_12664355 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5027:
    ------------------------------------------------
    May be the problem is not with the block report, but rather with nextGenerationStamp(Block), ...
    nextGenerationStamp(Block) should not update the block information in the NameNode since *the update may fail*. If the NameNode block gs is updated but the DataNode block gs is not and the update fails. The the block in the DataNode will be consider as invalid.
    I also would like to see a unit test catching this bug.
    Sure, working on one.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 16, 2009 at 1:35 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664355#action_12664355 ]

    szetszwo edited comment on HADOOP-5027 at 1/15/09 5:34 PM:
    -------------------------------------------------------------------------
    May be the problem is not with the block report, but rather with nextGenerationStamp(Block), ...
    nextGenerationStamp(Block) should not update the block information in the NameNode since *the update may fail*. If the NameNode block gs is updated but the DataNode block gs is not and the update fails, the the block in the DataNode will be considered as invalid.
    I also would like to see a unit test catching this bug.
    Sure, working on one.

    was (Author: szetszwo):
    May be the problem is not with the block report, but rather with nextGenerationStamp(Block), ...
    nextGenerationStamp(Block) should not update the block information in the NameNode since *the update may fail*. If the NameNode block gs is updated but the DataNode block gs is not and the update fails. The the block in the DataNode will be consider as invalid.
    I also would like to see a unit test catching this bug.
    Sure, working on one.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 16, 2009 at 1:39 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664358#action_12664358 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5027:
    ------------------------------------------------
    I thought all the gen-stamp changes are done by NN.
    What do you mean "done by NN"?
    It is a bit scary if generation can be changed by clients and datanodes...
    Clients cannot change gs. They only can request a update. Then, the selected primary datanode will request a generation stamp from the namenode.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 16, 2009 at 1:53 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664362#action_12664362 ]

    Raghu Angadi commented on HADOOP-5027:
    --------------------------------------
    What do you mean "done by NN"?
    I meant all the changes go through NN.

    Note that a patch was attached but the fix is not described anywhere in the jira. Finally I read the patch.

    Looking at the patch now, it looks like you do (correctly) consider it an error if reported gs is larger than stored gs.From the description of the jira, I thought NN might consider new reported gs as the correct one. Btw, the patch just ignores such a block with a warning..., should it also delete such a block?


    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 16, 2009 at 2:05 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664369#action_12664369 ]

    Konstantin Shvachko commented on HADOOP-5027:
    ---------------------------------------------
    nextGenerationStamp(Block) should not update the block information in the NameNode since the update may fail.
    The update has already failed, that is why you do the recovery in the first place, right? So NN already may have a wrong generation stamp. I still don't understand why nextGenerationStamp(Block) cannot change Block's gen stamp.

    Another thing if you change {{blockReport}} to understand generation stamps, then the same should be done for {{blockReceived()}}.

    I have always been under the impression that generation stamp plays role only during lease recovery, and that blockReport and blockReceived distinguish blocks by their {{id = <block#> + <genStamp>}}. I would rather keep it that simple if possible.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 16, 2009 at 2:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664372#action_12664372 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5027:
    ------------------------------------------------
    I meant all the changes go through NN.
    It is currently going through NN. BTW, you may want to take a look the javadoc in LeaseManager. It describes how the generation stamps are updated.
    Note that a patch was attached but the fix is not described anywhere in the jira. Finally I read the patch.
    The fix is described in the description and [Dhruba's comment|https://issues.apache.org/jira/browse/HADOOP-5027?focusedCommentId=12663862#action_12663862]. If you need every detail of the fix, I think reading the patch is the best way. I am glad that you have already done so.
    Looking at the patch now, it looks like you do (correctly) consider it an error if reported gs is larger than stored gs.From the description of the jira, I thought NN might consider new reported gs as the correct one.
    We are considering the case that the blocks are being written. Sorry that it is not clear.
    Btw, the patch just ignores such a block with a warning..., should it also delete such a block?
    This case should not happen. If it happens, it indicates that there are bugs somewhere. Without knowing what is going on, it is better not delete the block.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Jan 16, 2009 at 2:29 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664378#action_12664378 ]

    Raghu Angadi commented on HADOOP-5027:
    --------------------------------------
    The fix is described in the description and Dhruba's comment.
    all right.
    Btw, the patch just ignores such a block with a warning..., should it also delete such a block?
    This case should not happen. If it happens, it indicates that there are bugs somewhere. Without knowing what is going on, it is better not delete the block.
    adding something like "this is unexpected" to the message might make admins and users report it back, if they notice it.


    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 16, 2009 at 6:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664415#action_12664415 ]

    dhruba borthakur commented on HADOOP-5027:
    ------------------------------------------

    I like Raghu's comment to make the warning message something like "This is an Unexpected error, please report back to the Hadoop development community" or something like that,
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Jan 17, 2009 at 1:42 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664774#action_12664774 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-5027:
    ------------------------------------------------

    As suggested by Rob, if the reported block is ignored in NN, the data available guarantee will not be as good as before. Specifically, the existing data stored in the last block being reopened may not be replicated. I think we need to work on the design first.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 20, 2009 at 6:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665346#action_12665346 ]

    dhruba borthakur commented on HADOOP-5027:
    ------------------------------------------

    When a client is creating a block for the first time, the block is not being replicated by the NN. Only when the client is done writing to that block and the block is finalized does the NN start to replicate the block (if needed).

    The current design is to apply the above design to pre-existing block that is being opened for "append". The NN will not replicate this block until the appending-client is completely done with that block.

    I agree with Rob that the availability guarantees of a block being re-opened for write is not the same as a block that nobody is writing to. But "append" is a new feature and HADOOP-1700 states that this guarantee is acceptable. Rob's request is valid, but we can address that as an improvement over the current design and attack it in a seperate JIRA. Does that sound reasonable?

    This patch is imperative to make "appends" work as designed for 0.19.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 27, 2009 at 7:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667771#action_12667771 ]

    dhruba borthakur commented on HADOOP-5027:
    ------------------------------------------
    When I open a file for append the last block of the file is copied from the main storage to a tmp directory to be tre
    The block also gets a new generation stamp.
    And the name-node should mark it as under-replicated triggering replication process
    Namenode does not trigger re-replication for blocks that belong to files that are being written (or appended to).

    Let me try to explian this with an example. Suppose a file has one block blk_10_25. The generation stamp of this block is 25. Suppose this block is not full and is the only block in the file. Suppose a client wants to append a few bytes to this file. The client instructs the datanode to stamp this block with a generation stamp of 26. The block now becomes blk_10_26 and is moved from the real block directory to "blockBeingWritten" directory (see HADOOP-4663). The namenode metadat is also updated to record the block for this file to be blk_10_26. Block reports from this datanode will not have the block blk_10_25, but this is not a problem blk_25_10 is not a valid block anymore. These details are explained in th design doc for appends (HADOOP-1700).
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 27, 2009 at 8:05 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667782#action_12667782 ]

    Konstantin Shvachko commented on HADOOP-5027:
    ---------------------------------------------
    Block reports from this datanode will not have the block blk_10_25
    The block report from the data-node will not have neither blk_10_25 nor blk_10_26 or any block with id 10.
    But the name-node has this block in its block map. It will compare the new block report with what it thinks the data-node should have and will find out that the data-node does not have this replica anymore see reportDiff(), and will remove that replica location. Do I miss anything?
    I am looking at the code: NN calls {{removeStoredBlock()}} for this block, which removes it from the {{DatanodeDescriptor}} and calls {{updateNeededReplications()}}.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 27, 2009 at 8:15 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667731#action_12667731 ]

    Konstantin Shvachko commented on HADOOP-5027:
    ---------------------------------------------

    Since this is about block reports in the context of appends the following question should be appropriate here. May be also related to "tmp" directories discussion in HADOOP-4663.
    When I open a file for append the last block of the file is copied from the main storage to a tmp directory to be treated as if it is just being created. Then on the next blockReport since the block is not in the main storage it will not be reported to the name-node. And the name-node should mark it as under-replicated triggering replication process. Replication will not be able to proceed as all replicas on other machines are in the same state, that is in tmp directory. Is that a problem?
    The main problem here I think is that the name-node removes the location of the replica if it was not reported during a block report. If all 3 nodes report that they don't have the block, the block will become not readable, that is the name-node cannot give clients any locations of the block.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 28, 2009 at 12:49 am
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667893#action_12667893 ]

    dhruba borthakur commented on HADOOP-5027:
    ------------------------------------------

    I see your point. Thanks.

    One solution may be to make reportDiff() add a block to the "toRemove" data structure only if it belongs to a file that is not under construction. Will that work?
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 28, 2009 at 6:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668104#action_12668104 ]

    Konstantin Shvachko commented on HADOOP-5027:
    ---------------------------------------------

    May be it will work, but I would prefer to avoid complicating name-node logic if it can be handled by data-nodes.
    My [comment|http://issues.apache.org/jira/browse/HADOOP-4663#action_12663891] proposes some alternatives.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Konstantin Shvachko (JIRA) at Jan 28, 2009 at 8:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668136#action_12668136 ]

    Konstantin Shvachko commented on HADOOP-5027:
    ---------------------------------------------

    I see at least 6 currently open issues, which seem to be different, but actually being about the same problem.
    HADOOP-4663
    HADOOP-5027 (states 2 problems)
    HADOOP-5133
    HADOOP-4379
    HADOOP-4692
    The problem is *consistent handling incomplete blocks* or another way to say it *consistent handling block replication*.
    Trying to fix the issues separately is imHo a mistake.
    I am advocating to step back, get a general view, design a proposal, make sure it works and then fix issues.
    Gradually changing the code may destabilize the main hdfs functionality, which is not related to synchs or appends.
    Although I am not personally in favor of gradual changes in this particular case, you can still go this path
    e.g. in a separate branch with a potential to merging it back to trunk when it is done.
    I think people should have a choice to sacrifice certain functionality in favor of reliability if they want to.
    Gradual changes close this alternative.
    I'll link this comment from the related issues.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jan 30, 2009 at 6:27 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    dhruba borthakur updated HADOOP-5027:
    -------------------------------------

    Priority: Blocker (was: Major)

    I would like to add this to the Blocker list so that we can get a fix for this one in the next release of 0.19.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Priority: Blocker
    Fix For: 0.19.1

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Nigel Daley (JIRA) at Feb 11, 2009 at 9:01 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Nigel Daley updated HADOOP-5027:
    --------------------------------

    Fix Version/s: (was: 0.19.1)
    0.19.2

    As discussed on core-dev@ (http://www.nabble.com/Hadoop-0.19.1-td21739202.html) we will disable append in 0.19.1. Moving these append related issues to 0.19.2.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Priority: Blocker
    Fix For: 0.19.2

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Robert Chansler (JIRA) at Mar 9, 2009 at 9:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-5027:
    ------------------------------------

    Priority: Major (was: Blocker)

    Demoted: 20 shouldn't have to wait for this.
    Block report processing should compare gneration stamp
    ------------------------------------------------------

    Key: HADOOP-5027
    URL: https://issues.apache.org/jira/browse/HADOOP-5027
    Project: Hadoop Core
    Issue Type: Bug
    Components: dfs
    Reporter: Tsz Wo (Nicholas), SZE
    Fix For: 0.19.2

    Attachments: 5027_20090114.patch


    If a reported block has a different generation stamp then the one stored in the NameNode, the reported block will be considered as invalid. This is incorrect since blocks with larger generation stamp are valid.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJan 14, '09 at 2:15a
activeMar 9, '09 at 9:51p
posts30
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Robert Chansler (JIRA): 30 posts

People

Translate

site design / logo © 2022 Grokbase