Grokbase Groups Pig user March 2013
Super excited that this is finally public. The benefits are huge, and
having an (eventually) battle tested columnar storage format developed for
a diverse set of needs will be awesome.

2013/3/12 Kevin Olson <>
Second on that. Parquet looks compelling, but I'm curious to understand why
Cloudera suddenly switched from espousing future support for Trevni to
teaming with Twitter on Parquet.

On Tue, Mar 12, 2013 at 11:01 AM, Stan Rosenberg

Please excuse my ignorance. What is/was wrong with trevni
( ?



On Tue, Mar 12, 2013 at 11:45 AM, Dmitriy Ryaboy <>
Fellow Hadoopers,

We'd like to introduce a joint project between Twitter and Cloudera
engineers -- a new columnar storage format for Hadoop called Parquet (

We created Parquet to make the advantages of compressed, efficient columnar
data representation available to any project in the Hadoop ecosystem,
regardless of the choice of data processing framework, data model, or
programming language.

Parquet is built from the ground up with complex nested data structures in
mind. We adopted the repetition/definition level approach to encoding such
data structures, as described in Google's Dremel paper; we have found this
to be a very efficient method of encoding data in non-trivial object

Parquet is built to support very efficient compression and encoding
schemes. Parquet allows compression schemes to be specified on a
level, and is future-proofed to allow adding more encodings as they are
invented and implemented. We separate the concepts of encoding and
compression, allowing parquet consumers to implement operators that
directly on encoded data without paying decompression and decoding penalty
when possible.

Parquet is built to be used by anyone. The Hadoop ecosystem is rich
data processing frameworks, and we are not interested in playing
We believe that an efficient, well-implemented columnar storage
should be useful to all frameworks without the cost of extensive and
difficult to set up dependencies.

The initial code, available at, defines the file
format, provides Java building blocks for processing columnar data, and
implements Hadoop Input/Output Formats, Pig Storers/Loaders, and an example
of a complex integration -- Input/Output formats that can convert
Parquet-stored data directly to and from Thrift objects.

A preview version of Parquet support will be available in Cloudera's Impala

Twitter is starting to convert some of its major data source to Parquet in
order to take advantage of the compression and deserialization savings.

Parquet is currently under heavy development. Parquet's near-term
* Hive SerDes (Criteo)
* Cascading Taps (Criteo)
* Support for dictionary encoding, zigzag encoding, and RLE encoding of
data (Cloudera and Twitter)
* Further improvements to Pig support (Twitter)

Company names in parenthesis indicate whose engineers signed up to do
work -- others can feel free to jump in too, of course.

We've also heard requests to provide an Avro container layer, similar
what we do with Thrift. Seeking volunteers!

We welcome all feedback, patches, and ideas; to foster community
development, we plan to contribute Parquet to the Apache Incubator when the
development is farther along.

Nong Li, Julien Le Dem, Marcel Kornacker, Todd Lipcon, Dmitriy Ryaboy,
Jonathan Coveney, and friends.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 4 of 5 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMar 12, '13 at 5:30p
activeMar 13, '13 at 7:39p



site design / logo © 2021 Grokbase