FAQ
Turn CRC checking off for 0 byte size and differing blocksizes
--------------------------------------------------------------

Key: HADOOP-8233
URL: https://issues.apache.org/jira/browse/HADOOP-8233
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Dave Thompson
Assignee: Dave Thompson


DistcpV2 (hadoop-tools/hadoop-distcp/..) can fail from checksum failure, sometimes when copying a 0 byte file. Root cause of this may have to do with an inconsistent nature of HDFS when creating 0 byte files, however distcp can avoid this issue by not checking CRC when size is zero.

Further, distcp fails checksum when copying from two clusters that use different blocksizes. In this case it does not make sense to check CRC, as it is a guaranteed failure.

We need to turn CRC checking off for the above two cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 30, '12 at 10:15p
activeMar 30, '12 at 10:15p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Dave Thompson (Created) (JIRA): 1 post

People

Translate

site design / logo © 2021 Grokbase