[Hadoop-common-user] Splittable lzo files

that's very interesting. for us poor souls using streaming, would we
be able to use it?

(right now i'm looking at a 100+ GB gzipped file ...)

Miles

2009/3/3 Johan Oskarsson <johan@oskarsson.nu>:

We use it with python (dumbo) and streaming, so it should certainly be
possible. I haven't tried it myself though, so can't give any pointers.

/Johan

Miles Osborne wrote:

Thanks for posting this Johan,

I tried unsuccessfully to handle GZip files for the reasons you state
and resorted to uncompressed. I will try the Lzo format and post the
performance difference of compressed vs uncompressed on EC2 which
seems to have very slow disk IO. We have seen really bad import
speeds (like worse than mini macs even with the largest instances) on
postgis and mysql with EC2 so I think this might be very applicable to
the EC2 users.

Cheers,

Tim