FAQ
UpdateBlock fails due to unmatched file length
----------------------------------------------

Key: HDFS-1227
URL: https://issues.apache.org/jira/browse/HDFS-1227
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.20.1
Reporter: Thanh Do


- Summary: client append is not atomic, hence, it is possible that
when retrying during append, there is an exception in updateBlock
indicating unmatched file length, making append failed.

- Setup:
+ # available datanodes = 3
+ # disks / datanode = 1
+ # failures = 2
+ failure type = bad disk
+ When/where failure happens = (see below)
+ This bug is non-deterministic, to reproduce it, add a sufficient sleep before out.write() in BlockReceiver.receivePacket() in dn1 and dn2 but not dn3

- Details:
Suppose client appends 16 bytes to block X which has length 16 bytes at dn1, dn2, dn3.
Dn1 is primary. The pipeline is dn3-dn2-dn1. recoverBlock succeeds.
Client starts sending data to the dn3 - the first datanode in pipeline.
dn3 forwards the packet to downstream datanodes, and starts writing
data to its disk. Suppose there is an exception in dn3 when writing to disk.
Client gets the exception, it starts the recovery code by calling dn1.recoverBlock() again.
dn1 in turn calls dn2.getMetadataInfo() and dn1.getMetaDataInfo() to build the syncList.
Suppose at the time getMetadataInfo() is called at both datanodes (dn1 and dn2),
the previous packet (which is sent from dn3) has not come to disk yet.
Hence, the block Info given by getMetaDataInfo contains the length of 16 bytes.
But after that, the packet "comes" to disk, making the block file length now becomes 32 bytes.
Using the syncList (with contains block info with length 16 byte), dn1 calls updateBlock at
dn2 and dn1, which will failed, because the length of new block info (given by updateBlock,
which is 16 byte) does not match with its actual length on disk (which is 32 byte)

Note that this bug is non-deterministic. Its depends on the thread interleaving
at datanodes.

This bug was found by our Failure Testing Service framework:
http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-98.html
For questions, please email us: Thanh Do (thanhdo@cs.wisc.edu) and
Haryadi Gunawi (haryadi@eecs.berkeley.edu)



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Todd Lipcon (JIRA) at Jun 23, 2010 at 4:54 am
    [ https://issues.apache.org/jira/browse/HDFS-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Todd Lipcon resolved HDFS-1227.
    -------------------------------

    Resolution: Duplicate

    Going to resolve this as invalid. If you can reproduce after HDFS-1186 is committed, or provide a unit test, we can reopen.
    UpdateBlock fails due to unmatched file length
    ----------------------------------------------

    Key: HDFS-1227
    URL: https://issues.apache.org/jira/browse/HDFS-1227
    Project: Hadoop HDFS
    Issue Type: Bug
    Components: data-node
    Affects Versions: 0.20-append
    Reporter: Thanh Do

    - Summary: client append is not atomic, hence, it is possible that
    when retrying during append, there is an exception in updateBlock
    indicating unmatched file length, making append failed.

    - Setup:
    + # available datanodes = 3
    + # disks / datanode = 1
    + # failures = 2
    + failure type = bad disk
    + When/where failure happens = (see below)
    + This bug is non-deterministic, to reproduce it, add a sufficient sleep before out.write() in BlockReceiver.receivePacket() in dn1 and dn2 but not dn3

    - Details:
    Suppose client appends 16 bytes to block X which has length 16 bytes at dn1, dn2, dn3.
    Dn1 is primary. The pipeline is dn3-dn2-dn1. recoverBlock succeeds.
    Client starts sending data to the dn3 - the first datanode in pipeline.
    dn3 forwards the packet to downstream datanodes, and starts writing
    data to its disk. Suppose there is an exception in dn3 when writing to disk.
    Client gets the exception, it starts the recovery code by calling dn1.recoverBlock() again.
    dn1 in turn calls dn2.getMetadataInfo() and dn1.getMetaDataInfo() to build the syncList.
    Suppose at the time getMetadataInfo() is called at both datanodes (dn1 and dn2),
    the previous packet (which is sent from dn3) has not come to disk yet.
    Hence, the block Info given by getMetaDataInfo contains the length of 16 bytes.
    But after that, the packet "comes" to disk, making the block file length now becomes 32 bytes.
    Using the syncList (with contains block info with length 16 byte), dn1 calls updateBlock at
    dn2 and dn1, which will failed, because the length of new block info (given by updateBlock,
    which is 16 byte) does not match with its actual length on disk (which is 32 byte)

    Note that this bug is non-deterministic. Its depends on the thread interleaving
    at datanodes.
    This bug was found by our Failure Testing Service framework:
    http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-98.html
    For questions, please email us: Thanh Do (thanhdo@cs.wisc.edu) and
    Haryadi Gunawi (haryadi@eecs.berkeley.edu)
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedJun 17, '10 at 1:18p
activeJun 23, '10 at 4:54a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Todd Lipcon (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase