Re: Parquet row performance

One thing worth noting is that many column stores do lazy IO on
dependent columns after applying predicates on others. In this
example, it seems like your predicate on the 'a' column is restrictive
-- so it shouldn't have to do any IO on the 'b' column except where a
= 12345.

AFAIK Impala doesn't yet make this optimization in the case of
Parquet, but I imagine it's something to look forward to in future
versions.

-Todd

Is this optimization on the roadmap? Meanwhile, is there a way to mitigate
this e.g. If I change the schema so that there's more partitions, but run
the query over the same amount of data, will impala know to read less
partitions for the dependent columns?

This optimization is on the roadmap. The first step would be to simply
apply predicates before decoding the
other cols. This lets us skip all the CPU cost for the other cols. Saving
the IO costs requires a bit more
thought.

I'm not sure what you are suggesting as an attempt to mitigate this.
  Impala does partition pruning and will not
read from files that are part of filtered out partitions. You can change
the schema and this will be reflected but
changing the cols that are the partitioning cols is *not* a metadata only
change. The partitioning columns are
not in the actual data files so changing them will likely require
re-writing the data files.

I find the "-*num_threads_per_disk*" option in Impala which determines the
maximum number of the threads per disk. (The default value is 1)

I am not quite sure, but increasing this option value might help to read
the column chunks in parallel. Therefore, the query latency will be lowered.

Am I right, experts?

Thanks

This is not correct. On spinning disks, the optimal value for
num_threads_per_disk is 1, which minimizes the number of disk seeks. To
read multiple
column chunks in parallel, we'd need each column stored on a different disk
which requires some storage layer (i.e. HDFS) changes.