FAQ
Hi,
mapred.output.compress is set to true in hadoop-site.xml
My question is how can I specify different compression codecs
programmatically ?

For example, normally the output is gzip compressed. But a small portion of
output needs to be LZO compressed.

Thanks

Search Discussions

  • Alex Kozlov at Jan 18, 2010 at 9:08 pm
    You can specify the compression/codec in the file writer (is this what you
    are asking?).


    SequenceFile.createWriter(fs, conf, path, key.getClass(), value.getClass(),
    SequenceFile.CompressionType.BLOCK, codec);

    You can also create your own FileOutputFormat.
    On Mon, Jan 18, 2010 at 12:53 PM, Ted Yu wrote:

    Hi,
    mapred.output.compress is set to true in hadoop-site.xml
    My question is how can I specify different compression codecs
    programmatically ?

    For example, normally the output is gzip compressed. But a small portion of
    output needs to be LZO compressed.

    Thanks
  • Todd Lipcon at Jan 18, 2010 at 9:12 pm

    On Mon, Jan 18, 2010 at 1:08 PM, Alex Kozlov wrote:

    You can specify the compression/codec in the file writer (is this what you
    are asking?).


    SequenceFile.createWriter(fs, conf, path, key.getClass(), value.getClass(),
    SequenceFile.CompressionType.BLOCK, codec);

    You can also create your own FileOutputFormat.
    If you're using one of the built in FileOutputFormat subclasses, you can
    use FileOutputFormat.setOutputCompressorClass to set the codec. If you're
    using MultipleOutputs, you will probably have to override some functions in
    order to create different RecordWriters with different codecs, as Alex
    suggested above.

    -Todd



    On Mon, Jan 18, 2010 at 12:53 PM, Ted Yu wrote:

    Hi,
    mapred.output.compress is set to true in hadoop-site.xml
    My question is how can I specify different compression codecs
    programmatically ?

    For example, normally the output is gzip compressed. But a small portion of
    output needs to be LZO compressed.

    Thanks

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 18, '10 at 8:54p
activeJan 18, '10 at 9:12p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase