FAQ
Is there a reason for the key and record length included in the SequenceFile
format?

Since every Writable has to be able to read and write itself, those objects
already describe their own length (either implicitly or
explicitly). Sequence file has to resort to extra buffer copies and waste
space to get those lengths, is there any reason we cant just leave them
out?

Search Discussions

  • Doug Cutting at Jun 19, 2006 at 5:36 pm

    Paul Sutter wrote:
    Is there a reason for the key and record length included in the
    SequenceFile
    format?
    This permits code to process entries without parsing them, an important
    optimization. For example, when seeking a MapFile, keys are parsed but
    not values. When sorting, if a comparator is defined that can compare
    raw binary data, then even keys are not parsed.

    Doug
  • Paul Sutter at Jun 19, 2006 at 5:46 pm
    How can I define a comparator that can compare raw binary data? (our keys
    are raw binary data)

    On 6/19/06, Doug Cutting wrote:

    Paul Sutter wrote:
    Is there a reason for the key and record length included in the
    SequenceFile
    format?
    This permits code to process entries without parsing them, an important
    optimization. For example, when seeking a MapFile, keys are parsed but
    not values. When sorting, if a comparator is defined that can compare
    raw binary data, then even keys are not parsed.

    Doug
  • Doug Cutting at Jun 19, 2006 at 5:56 pm

    Paul Sutter wrote:
    How can I define a comparator that can compare raw binary data? (our keys
    are raw binary data)
    Implement the following method:

    http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/io/WritableComparator.html#compare(byte[],%20int,%20int,%20byte[],%20int,%20int)

    To make a comparator the default for a class, call:

    http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/io/WritableComparator.html#define(java.lang.Class,%20org.apache.hadoop.io.WritableComparator)

    This is already done for most WritableComparable implementations. The
    the bottom of, e.g., IntWritable.java for an example of the idiom.

    http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/io/IntWritable.java?view=markup

    Doug
  • Owen O'Malley at Jun 19, 2006 at 6:03 pm

    On Jun 19, 2006, at 10:46 AM, Paul Sutter wrote:

    How can I define a comparator that can compare raw binary data? (our
    keys
    are raw binary data)
    Paul,
    Look at what I did for BytesWritable (lines 147-165). You need to
    define a Comparator class and register it with WritableComparable.

    -- Owen
  • Eric Baldeschwieler at Jun 26, 2006 at 5:20 pm
    Can we turn this around and assume that writables will be given a
    stream and a length when they read? That would also let us remove
    redundant info...
    On Jun 19, 2006, at 11:02 AM, Owen O'Malley wrote:

    On Jun 19, 2006, at 10:46 AM, Paul Sutter wrote:

    How can I define a comparator that can compare raw binary data?
    (our keys
    are raw binary data)
    Paul,
    Look at what I did for BytesWritable (lines 147-165). You need
    to define a Comparator class and register it with WritableComparable.

    -- Owen
  • Doug Cutting at Jun 26, 2006 at 9:49 pm

    Eric Baldeschwieler wrote:
    Can we turn this around and assume that writables will be given a stream
    and a length when they read? That would also let us remove redundant
    info...
    Unless I misunderstand, that would make it harder to nest writables,
    since all containers would need to store the length. Currently only
    top-level containers (SequenceFile and the RPC protocol) need to write
    lengths. Even these are optional, used only to optimize things.

    Doug
  • Paul Sutter at Jun 26, 2006 at 11:35 pm
    I agree, there's no easy way around this one without separate interfaces
    (one where the caller keeps the counts, and one where the writable keeps the
    counts), and that would be silly.

    However -> It still seems to me that the key length in the sequence file is
    redundant. Since each key must write its own length, know its own length,
    or be able to figure it out - even via the high speed interface - there's no
    reason to have that key length in the file.

    Why do I care about 4 bytes per record? Because we're integrating an
    external sort, and right now it has to look at a record with two key
    lengths. And I assume that others (such as Yahoo) will want to incorporate
    an external sort. And if we're going to be reading the sequence file in
    another language, we might as well be sure about the format to use.

    Thanks!

    Paul
    On 6/26/06, Doug Cutting wrote:

    Eric Baldeschwieler wrote:
    Can we turn this around and assume that writables will be given a stream
    and a length when they read? That would also let us remove redundant
    info...
    Unless I misunderstand, that would make it harder to nest writables,
    since all containers would need to store the length. Currently only
    top-level containers (SequenceFile and the RPC protocol) need to write
    lengths. Even these are optional, used only to optimize things.

    Doug
  • Doug Cutting at Jun 27, 2006 at 1:18 am

    Paul Sutter wrote:
    However -> It still seems to me that the key length in the sequence file is
    redundant.
    What if your keys are compound, containing, say, a combination of
    floats, ints and strings? Then the key may not include a length of the
    entire key entry. So you're seeking to optimize a special (if common) case.

    Doug
  • Paul Sutter at Jun 27, 2006 at 4:07 am
    its fine either way, i just wanted to know that its deleberate. it certainly
    doesnt cost much space.

    an external sorter can work only on keytypes it understands, and our keys
    are compound and may include strings, floats, and ints.

    the length is no problem at all.
    On 6/26/06, Doug Cutting wrote:

    Paul Sutter wrote:
    However -> It still seems to me that the key length in the sequence file is
    redundant.
    What if your keys are compound, containing, say, a combination of
    floats, ints and strings? Then the key may not include a length of the
    entire key entry. So you're seeking to optimize a special (if common)
    case.

    Doug
  • Eric Baldeschwieler at Jun 27, 2006 at 3:07 am
    Yikes, but this assumes that your external code can grock whatever
    strange thing is stored in the writable. This is a non-trivial
    assumption when you go multi-lingual.

    We're going to take a crack at a block compressed format soon. This
    will vastly reduce the storage impact of this kind of issue anyway.

    On Jun 26, 2006, at 4:34 PM, Paul Sutter wrote:

    I agree, there's no easy way around this one without separate
    interfaces
    (one where the caller keeps the counts, and one where the writable
    keeps the
    counts), and that would be silly.

    However -> It still seems to me that the key length in the sequence
    file is
    redundant. Since each key must write its own length, know its own
    length,
    or be able to figure it out - even via the high speed interface -
    there's no
    reason to have that key length in the file.

    Why do I care about 4 bytes per record? Because we're integrating an
    external sort, and right now it has to look at a record with two key
    lengths. And I assume that others (such as Yahoo) will want to
    incorporate
    an external sort. And if we're going to be reading the sequence
    file in
    another language, we might as well be sure about the format to use.

    Thanks!

    Paul
    On 6/26/06, Doug Cutting wrote:

    Eric Baldeschwieler wrote:
    Can we turn this around and assume that writables will be given a stream
    and a length when they read? That would also let us remove redundant
    info...
    Unless I misunderstand, that would make it harder to nest writables,
    since all containers would need to store the length. Currently only
    top-level containers (SequenceFile and the RPC protocol) need to
    write
    lengths. Even these are optional, used only to optimize things.

    Doug

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 19, '06 at 3:43p
activeJun 27, '06 at 4:07a
posts11
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase