 |
Todd Lipcon |
at Jan 18, 2010 at 9:12 pm
|
⇧ | 
| |
On Mon, Jan 18, 2010 at 1:08 PM, Alex Kozlov wrote:
You can specify the compression/codec in the file writer (is this what you
are asking?).
SequenceFile.createWriter(fs, conf, path, key.getClass(), value.getClass(),
SequenceFile.CompressionType.BLOCK, codec);
You can also create your own FileOutputFormat.
If you're using one of the built in FileOutputFormat subclasses, you can
use FileOutputFormat.setOutputCompressorClass to set the codec. If you're
using MultipleOutputs, you will probably have to override some functions in
order to create different RecordWriters with different codecs, as Alex
suggested above.
-Todd
On Mon, Jan 18, 2010 at 12:53 PM, Ted Yu wrote:
Hi,
mapred.output.compress is set to true in hadoop-site.xml
My question is how can I specify different compression codecs
programmatically ?
For example, normally the output is gzip compressed. But a small portion of
output needs to be LZO compressed.
Thanks
