FAQ
Hello my fellow Hadoop users/developers,

I'm reading the SequenceFile source code, and there is a checkAndWriteSync()
method that writes a sync marker every so many bytes. I was wondering what's
the use of the sync marker. I know one can use it to designate the end of a
header, but it's also used in the process of writing the uncompressed data.
I wish I could have figured it out :(

Thank you so much

Search Discussions

  • Chris Douglas at Mar 21, 2011 at 8:07 pm
    It's used to align input splits of the SequenceFile. A reader can
    start at an arbitrary offset, then find the boundary of the next block
    of records by looking for the sync marker defined in the header. -C
    On Mon, Mar 21, 2011 at 7:40 AM, Weishung Chung wrote:
    Hello my fellow Hadoop users/developers,

    I'm reading the SequenceFile source code, and there is a checkAndWriteSync()
    method that writes a sync marker every so many bytes. I was wondering what's
    the use of the sync marker. I know one can use it to designate the end of a
    header, but it's also used in the process of writing the uncompressed data.
    I wish I could have figured it out :(

    Thank you so much
  • Weishung Chung at Mar 22, 2011 at 2:16 pm
    Thanks, exciting works !
    On Mon, Mar 21, 2011 at 3:07 PM, Chris Douglas wrote:

    It's used to align input splits of the SequenceFile. A reader can
    start at an arbitrary offset, then find the boundary of the next block
    of records by looking for the sync marker defined in the header. -C
    On Mon, Mar 21, 2011 at 7:40 AM, Weishung Chung wrote:
    Hello my fellow Hadoop users/developers,

    I'm reading the SequenceFile source code, and there is a
    checkAndWriteSync()
    method that writes a sync marker every so many bytes. I was wondering what's
    the use of the sync marker. I know one can use it to designate the end of a
    header, but it's also used in the process of writing the uncompressed data.
    I wish I could have figured it out :(

    Thank you so much

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 21, '11 at 2:41p
activeMar 22, '11 at 2:16p
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Weishung Chung: 2 posts Chris Douglas: 1 post

People

Translate

site design / logo © 2022 Grokbase