Grokbase Groups Hive dev June 2011
FAQ
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/935/
-----------------------------------------------------------

Review request for hive.


Summary
-------

For a table stored as RCFile, intermediate results are sometimes merged if those files are below a certain threshold. For RCFiles, we can do a block level merge that does not deserialize the blocks and is more efficient. This patch leverages the existing code used to merge for ALTER TABLE ... CONCATENATE.


This addresses bug HIVE-2035.
https://issues.apache.org/jira/browse/HIVE-2035


Diffs
-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/MergeWork.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeRecordReader.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileKeyBufferWrapper.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1134415
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1134415
trunk/ql/src/test/queries/clientpositive/rcfile_createas1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/rcfile_insert.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/rcfile_merge1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/rcfile_merge2.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/rcfile_merge3.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/rcfile_merge4.q PRE-CREATION
trunk/ql/src/test/results/clientpositive/rcfile_createas1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/rcfile_insert.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/rcfile_merge1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/rcfile_merge2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/rcfile_merge3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/rcfile_merge4.q.out PRE-CREATION

Diff: https://reviews.apache.org/r/935/diff


Testing
-------


Thanks,

Franklin

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 5 | next ›
Discussion Overview
groupdev @
categorieshive, hadoop
postedJun 17, '11 at 8:45p
activeJun 23, '11 at 6:56p
posts5
users2
websitehive.apache.org

2 users in discussion

Franklin Hu: 3 posts Siying Dong: 2 posts

People

Translate

site design / logo © 2021 Grokbase