FAQ
Is it safe to use this to generate a single SequenceFile out of a set of
sequence files produced by reduce?

this seems to be the source of my damaged sequence files.

Search Discussions

  • Arun C Murthy at Dec 29, 2007 at 10:18 am

    On Fri, Dec 28, 2007 at 11:53:42AM -0800, Jason Venner wrote:
    Is it safe to use this to generate a single SequenceFile out of a set of
    sequence files produced by reduce?
    Nope.

    FileUtil.copyMerge just copies bytes of src files into one large heap of a destination file. This will break if src files are SequenceFiles since we now have multiple headers mixed with data.

    I've opened http://issues.apache.org/jira/browse/HADOOP-2501 to cover _merge_ and other useful utilities for SequenceFiles.
    this seems to be the source of my damaged sequence files.
    Arun

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 28, '07 at 7:54p
activeDec 29, '07 at 10:18a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Jason Venner: 1 post Arun C Murthy: 1 post

People

Translate

site design / logo © 2022 Grokbase