Grokbase Groups Hive commits May 2015
FAQ
Repository: hive
Updated Branches:
   refs/heads/master 7add1a887 -> 7368cfda3


HIVE-10244: Vectorization : TPC-DS Q80 fails with java.lang.ClassCastException when hive.vectorized.execution.reduce.enabled is enabled (Matt McCline reviewed by John Pullokkaran)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/7368cfda
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/7368cfda
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/7368cfda

Branch: refs/heads/master
Commit: 7368cfda3833bd2e38d644e04bdf4be60253148a
Parents: 7add1a8
Author: Gunther Hagleitner <gunther@apache.org>
Authored: Thu May 28 15:58:34 2015 -0700
Committer: Gunther Hagleitner <gunther@apache.org>
Committed: Thu May 28 15:58:47 2015 -0700

----------------------------------------------------------------------
  data/files/store_200 | 12 +
  .../test/resources/testconfiguration.properties | 1 +
  .../hive/ql/optimizer/physical/Vectorizer.java | 4 +
  .../clientpositive/vector_grouping_sets.q | 63 +++++
  .../tez/vector_grouping_sets.q.out | 266 +++++++++++++++++++
  .../clientpositive/vector_grouping_sets.q.out | 253 ++++++++++++++++++
  6 files changed, 599 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hive/blob/7368cfda/data/files/store_200
----------------------------------------------------------------------
diff --git a/data/files/store_200 b/data/files/store_200
new file mode 100644
index 0000000..078bafd
--- /dev/null
+++ b/data/files/store_200
@@ -0,0 +1,12 @@
+1|AAAAAAAABAAAAAAA|1997-03-13||2451189|ought|245|5250760|8AM-4PM|William Ward|2|Unknown|Enough high areas stop expectations. Elaborate, local is|Charles Bartley|1|Unknown|1|Unknown|767|Spring |Wy|Suite 250|Midway|Williamson County|TN|31904|United States|-5|0.03|
+2|AAAAAAAACAAAAAAA|1997-03-13|2000-03-12||able|236|5285950|8AM-4PM|Scott Smith|8|Unknown|Parliamentary candidates wait then heavy, keen mil|David Lamontagne|1|Unknown|1|Unknown|255|Sycamore |Dr.|Suite 410|Midway|Williamson County|TN|31904|United States|-5|0.03|
+3|AAAAAAAACAAAAAAA|2000-03-13|||able|236|7557959|8AM-4PM|Scott Smith|7|Unknown|Impossible, true arms can treat constant, complete w|David Lamontagne|1|Unknown|1|Unknown|877|Park Laurel|Road|Suite T|Midway|Williamson County|TN|31904|United States|-5|0.03|
+4|AAAAAAAAEAAAAAAA|1997-03-13|1999-03-13|2451044|ese|218|9341467|8AM-4PM|Edwin Adams|4|Unknown|Events would achieve other, eastern hours. Mechanisms must not eat other, new org|Thomas Pollack|1|Unknown|1|Unknown|27|Lake |Ln|Suite 260|Midway|Williamson County|TN|31904|United States|-5|0.03|
+5|AAAAAAAAEAAAAAAA|1999-03-14|2001-03-12|2450910|anti|288|9078805|8AM-4PM|Edwin Adams|8|Unknown|Events would achieve other, eastern hours. Mechanisms must not eat other, new org|Thomas Pollack|1|Unknown|1|Unknown|27|Lee 6th|Court|Suite 80|Fairview|Williamson County|TN|35709|United States|-5|0.03|
+6|AAAAAAAAEAAAAAAA|2001-03-13|||cally|229|9026222|8AM-4PM|Edwin Adams|10|Unknown|Events would achieve other, eastern hours. Mechanisms must not eat other, new org|Thomas Pollack|1|Unknown|1|Unknown|220|6th |Lane|Suite 140|Midway|Williamson County|TN|31904|United States|-5|0.03|
+7|AAAAAAAAHAAAAAAA|1997-03-13|||ation|297|8954883|8AM-4PM|David Thomas|9|Unknown|Architects coul|Thomas Benton|1|Unknown|1|Unknown|811|Lee |Circle|Suite T|Midway|Williamson County|TN|31904|United States|-5|0.01|
+8|AAAAAAAAIAAAAAAA|1997-03-13|2000-03-12||eing|278|6995995|8AM-4PM|Brett Yates|2|Unknown|Various bars make most. Difficult levels introduce at a boots. Buildings welcome only never el|Dean Morrison|1|Unknown|1|Unknown|226|12th |Lane|Suite D|Fairview|Williamson County|TN|35709|United States|-5|0.08|
+9|AAAAAAAAIAAAAAAA|2000-03-13|||eing|271|6995995|8AM-4PM|Brett Yates|2|Unknown|Formal, psychological pounds relate reasonable, young principles. Black, |Dean Morrison|1|Unknown|1|Unknown|226|Hill |Boulevard|Suite 190|Midway|Williamson County|TN|31904|United States|-5|0.08|
+10|AAAAAAAAKAAAAAAA|1997-03-13|1999-03-13||bar|294|9294113|8AM-4PM|Raymond Jacobs|8|Unknown|Little expectations include yet forward meetings.|Michael Wilson|1|Unknown|1|Unknown|175|4th |Court|Suite C|Midway|Williamson County|TN|31904|United States|-5|0.06|
+11|AAAAAAAAKAAAAAAA|1999-03-14|2001-03-12||ought|294|9294113|8AM-4PM|Raymond Jacobs|6|Unknown|Mysterious employe|Michael Wilson|1|Unknown|1|Unknown|175|Park Green|Court|Suite 160|Midway|Williamson County|TN|31904|United States|-5|0.11|
+12|AAAAAAAAKAAAAAAA|2001-03-13|||ought|294|5219562|8AM-12AM|Robert Thompson|6|Unknown|Events develop i|Dustin Kelly|1|Unknown|1|Unknown|337|College |Boulevard|Suite 100|Fairview|Williamson County|TN|31904|United States|-5|0.01|

http://git-wip-us.apache.org/repos/asf/hive/blob/7368cfda/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
diff --git a/itests/src/test/resources/testconfiguration.properties b/itests/src/test/resources/testconfiguration.properties
index 376c7af..bdb96e8 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -220,6 +220,7 @@ minitez.query.files.shared=alter_merge_2_orc.q,\
    vector_elt.q,\
    vector_groupby_3.q,\
    vector_groupby_reduce.q,\
+ vector_grouping_sets.q,\
    vector_if_expr.q,\
    vector_inner_join.q,\
    vector_interval_1.q,\

http://git-wip-us.apache.org/repos/asf/hive/blob/7368cfda/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
index 3ed3c7e..e7b9c73 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
@@ -1118,6 +1118,10 @@ public class Vectorizer implements PhysicalPlanResolver {
        LOG.info("Grouping sets not supported in vector mode");
        return false;
      }
+ if (desc.pruneGroupingSetId()) {
+ LOG.info("Pruning grouping set id not supported in vector mode");
+ return false;
+ }
      boolean ret = validateExprNodeDesc(desc.getKeys());
      if (!ret) {
        LOG.info("Cannot vectorize groupby key expression");

http://git-wip-us.apache.org/repos/asf/hive/blob/7368cfda/ql/src/test/queries/clientpositive/vector_grouping_sets.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/vector_grouping_sets.q b/ql/src/test/queries/clientpositive/vector_grouping_sets.q
new file mode 100644
index 0000000..9449ec4
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/vector_grouping_sets.q
@@ -0,0 +1,63 @@
+SET hive.vectorized.execution.enabled=true;
+set hive.fetch.task.conversion=none;
+
+-- We are not expecting either query to vectorize due to use of pruneing grouping set id,
+-- or use of GROUPING__ID virtual column.
+create table store_txt
+(
+ s_store_sk int,
+ s_store_id string,
+ s_rec_start_date string,
+ s_rec_end_date string,
+ s_closed_date_sk int,
+ s_store_name string,
+ s_number_employees int,
+ s_floor_space int,
+ s_hours string,
+ s_manager string,
+ s_market_id int,
+ s_geography_class string,
+ s_market_desc string,
+ s_market_manager string,
+ s_division_id int,
+ s_division_name string,
+ s_company_id int,
+ s_company_name string,
+ s_street_number string,
+ s_street_name string,
+ s_street_type string,
+ s_suite_number string,
+ s_city string,
+ s_county string,
+ s_state string,
+ s_zip string,
+ s_country string,
+ s_gmt_offset decimal(5,2),
+ s_tax_precentage decimal(5,2)
+)
+row format delimited fields terminated by '|'
+stored as textfile;
+
+LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt;
+
+create table store
+stored as orc as
+select * from store_txt;
+
+explain
+select s_store_id
+ from store
+ group by s_store_id with rollup;
+
+select s_store_id
+ from store
+ group by s_store_id with rollup;
+
+explain
+select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup;
+
+select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup;
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hive/blob/7368cfda/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out b/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
new file mode 100644
index 0000000..688d2ac
--- /dev/null
+++ b/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
@@ -0,0 +1,266 @@
+PREHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
+-- or use of GROUPING__ID virtual column.
+create table store_txt
+(
+ s_store_sk int,
+ s_store_id string,
+ s_rec_start_date string,
+ s_rec_end_date string,
+ s_closed_date_sk int,
+ s_store_name string,
+ s_number_employees int,
+ s_floor_space int,
+ s_hours string,
+ s_manager string,
+ s_market_id int,
+ s_geography_class string,
+ s_market_desc string,
+ s_market_manager string,
+ s_division_id int,
+ s_division_name string,
+ s_company_id int,
+ s_company_name string,
+ s_street_number string,
+ s_street_name string,
+ s_street_type string,
+ s_suite_number string,
+ s_city string,
+ s_county string,
+ s_state string,
+ s_zip string,
+ s_country string,
+ s_gmt_offset decimal(5,2),
+ s_tax_precentage decimal(5,2)
+)
+row format delimited fields terminated by '|'
+stored as textfile
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@store_txt
+POSTHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
+-- or use of GROUPING__ID virtual column.
+create table store_txt
+(
+ s_store_sk int,
+ s_store_id string,
+ s_rec_start_date string,
+ s_rec_end_date string,
+ s_closed_date_sk int,
+ s_store_name string,
+ s_number_employees int,
+ s_floor_space int,
+ s_hours string,
+ s_manager string,
+ s_market_id int,
+ s_geography_class string,
+ s_market_desc string,
+ s_market_manager string,
+ s_division_id int,
+ s_division_name string,
+ s_company_id int,
+ s_company_name string,
+ s_street_number string,
+ s_street_name string,
+ s_street_type string,
+ s_suite_number string,
+ s_city string,
+ s_county string,
+ s_state string,
+ s_zip string,
+ s_country string,
+ s_gmt_offset decimal(5,2),
+ s_tax_precentage decimal(5,2)
+)
+row format delimited fields terminated by '|'
+stored as textfile
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@store_txt
+PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
+PREHOOK: type: LOAD
+#### A masked pattern was here ####
+PREHOOK: Output: default@store_txt
+POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
+POSTHOOK: type: LOAD
+#### A masked pattern was here ####
+POSTHOOK: Output: default@store_txt
+PREHOOK: query: create table store
+stored as orc as
+select * from store_txt
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@store_txt
+PREHOOK: Output: database:default
+PREHOOK: Output: default@store
+POSTHOOK: query: create table store
+stored as orc as
+select * from store_txt
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@store_txt
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@store
+PREHOOK: query: explain
+select s_store_id
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select s_store_id
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: store
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: s_store_id (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), '0' (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reducer 2
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ pruneGroupingSetId: true
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select s_store_id
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+PREHOOK: Input: default@store
+#### A masked pattern was here ####
+POSTHOOK: query: select s_store_id
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@store
+#### A masked pattern was here ####
+NULL
+AAAAAAAABAAAAAAA
+AAAAAAAACAAAAAAA
+AAAAAAAAEAAAAAAA
+AAAAAAAAHAAAAAAA
+AAAAAAAAIAAAAAAA
+AAAAAAAAKAAAAAAA
+PREHOOK: query: explain
+select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: store
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: s_store_id (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), '0' (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reducer 2
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+PREHOOK: Input: default@store
+#### A masked pattern was here ####
+POSTHOOK: query: select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@store
+#### A masked pattern was here ####
+NULL 0
+AAAAAAAABAAAAAAA 1
+AAAAAAAACAAAAAAA 1
+AAAAAAAAEAAAAAAA 1
+AAAAAAAAHAAAAAAA 1
+AAAAAAAAIAAAAAAA 1
+AAAAAAAAKAAAAAAA 1

http://git-wip-us.apache.org/repos/asf/hive/blob/7368cfda/ql/src/test/results/clientpositive/vector_grouping_sets.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/vector_grouping_sets.q.out b/ql/src/test/results/clientpositive/vector_grouping_sets.q.out
new file mode 100644
index 0000000..69d6187
--- /dev/null
+++ b/ql/src/test/results/clientpositive/vector_grouping_sets.q.out
@@ -0,0 +1,253 @@
+PREHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
+-- or use of GROUPING__ID virtual column.
+create table store_txt
+(
+ s_store_sk int,
+ s_store_id string,
+ s_rec_start_date string,
+ s_rec_end_date string,
+ s_closed_date_sk int,
+ s_store_name string,
+ s_number_employees int,
+ s_floor_space int,
+ s_hours string,
+ s_manager string,
+ s_market_id int,
+ s_geography_class string,
+ s_market_desc string,
+ s_market_manager string,
+ s_division_id int,
+ s_division_name string,
+ s_company_id int,
+ s_company_name string,
+ s_street_number string,
+ s_street_name string,
+ s_street_type string,
+ s_suite_number string,
+ s_city string,
+ s_county string,
+ s_state string,
+ s_zip string,
+ s_country string,
+ s_gmt_offset decimal(5,2),
+ s_tax_precentage decimal(5,2)
+)
+row format delimited fields terminated by '|'
+stored as textfile
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@store_txt
+POSTHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
+-- or use of GROUPING__ID virtual column.
+create table store_txt
+(
+ s_store_sk int,
+ s_store_id string,
+ s_rec_start_date string,
+ s_rec_end_date string,
+ s_closed_date_sk int,
+ s_store_name string,
+ s_number_employees int,
+ s_floor_space int,
+ s_hours string,
+ s_manager string,
+ s_market_id int,
+ s_geography_class string,
+ s_market_desc string,
+ s_market_manager string,
+ s_division_id int,
+ s_division_name string,
+ s_company_id int,
+ s_company_name string,
+ s_street_number string,
+ s_street_name string,
+ s_street_type string,
+ s_suite_number string,
+ s_city string,
+ s_county string,
+ s_state string,
+ s_zip string,
+ s_country string,
+ s_gmt_offset decimal(5,2),
+ s_tax_precentage decimal(5,2)
+)
+row format delimited fields terminated by '|'
+stored as textfile
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@store_txt
+PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
+PREHOOK: type: LOAD
+#### A masked pattern was here ####
+PREHOOK: Output: default@store_txt
+POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
+POSTHOOK: type: LOAD
+#### A masked pattern was here ####
+POSTHOOK: Output: default@store_txt
+PREHOOK: query: create table store
+stored as orc as
+select * from store_txt
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@store_txt
+PREHOOK: Output: database:default
+PREHOOK: Output: default@store
+POSTHOOK: query: create table store
+stored as orc as
+select * from store_txt
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@store_txt
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@store
+PREHOOK: query: explain
+select s_store_id
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select s_store_id
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: store
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: s_store_id (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), '0' (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ pruneGroupingSetId: true
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select s_store_id
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+PREHOOK: Input: default@store
+#### A masked pattern was here ####
+POSTHOOK: query: select s_store_id
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@store
+#### A masked pattern was here ####
+NULL
+AAAAAAAABAAAAAAA
+AAAAAAAACAAAAAAA
+AAAAAAAAEAAAAAAA
+AAAAAAAAHAAAAAAA
+AAAAAAAAIAAAAAAA
+AAAAAAAAKAAAAAAA
+PREHOOK: query: explain
+select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: store
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: s_store_id (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), '0' (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+PREHOOK: type: QUERY
+PREHOOK: Input: default@store
+#### A masked pattern was here ####
+POSTHOOK: query: select s_store_id, GROUPING__ID
+ from store
+ group by s_store_id with rollup
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@store
+#### A masked pattern was here ####
+NULL 0
+AAAAAAAABAAAAAAA 1
+AAAAAAAACAAAAAAA 1
+AAAAAAAAEAAAAAAA 1
+AAAAAAAAHAAAAAAA 1
+AAAAAAAAIAAAAAAA 1
+AAAAAAAAKAAAAAAA 1

Search Discussions

  • Gunther at May 28, 2015 at 11:18 pm
    Repository: hive
    Updated Branches:
       refs/heads/branch-1.2 e592e1a22 -> a47bf7369


    HIVE-10244: Vectorization : TPC-DS Q80 fails with java.lang.ClassCastException when hive.vectorized.execution.reduce.enabled is enabled (Matt McCline reviewed by John Pullokkaran)


    Project: http://git-wip-us.apache.org/repos/asf/hive/repo
    Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/a47bf736
    Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/a47bf736
    Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/a47bf736

    Branch: refs/heads/branch-1.2
    Commit: a47bf7369d7ca4e7e926c30eec8bd553664a7df4
    Parents: e592e1a
    Author: Gunther Hagleitner <gunther@apache.org>
    Authored: Thu May 28 15:58:34 2015 -0700
    Committer: Gunther Hagleitner <gunther@apache.org>
    Committed: Thu May 28 16:06:24 2015 -0700

    ----------------------------------------------------------------------
      data/files/store_200 | 12 +
      .../test/resources/testconfiguration.properties | 1 +
      .../hive/ql/optimizer/physical/Vectorizer.java | 4 +
      .../clientpositive/vector_grouping_sets.q | 63 +++++
      .../tez/vector_grouping_sets.q.out | 266 +++++++++++++++++++
      .../clientpositive/vector_grouping_sets.q.out | 253 ++++++++++++++++++
      6 files changed, 599 insertions(+)
    ----------------------------------------------------------------------


    http://git-wip-us.apache.org/repos/asf/hive/blob/a47bf736/data/files/store_200
    ----------------------------------------------------------------------
    diff --git a/data/files/store_200 b/data/files/store_200
    new file mode 100644
    index 0000000..078bafd
    --- /dev/null
    +++ b/data/files/store_200
    @@ -0,0 +1,12 @@
    +1|AAAAAAAABAAAAAAA|1997-03-13||2451189|ought|245|5250760|8AM-4PM|William Ward|2|Unknown|Enough high areas stop expectations. Elaborate, local is|Charles Bartley|1|Unknown|1|Unknown|767|Spring |Wy|Suite 250|Midway|Williamson County|TN|31904|United States|-5|0.03|
    +2|AAAAAAAACAAAAAAA|1997-03-13|2000-03-12||able|236|5285950|8AM-4PM|Scott Smith|8|Unknown|Parliamentary candidates wait then heavy, keen mil|David Lamontagne|1|Unknown|1|Unknown|255|Sycamore |Dr.|Suite 410|Midway|Williamson County|TN|31904|United States|-5|0.03|
    +3|AAAAAAAACAAAAAAA|2000-03-13|||able|236|7557959|8AM-4PM|Scott Smith|7|Unknown|Impossible, true arms can treat constant, complete w|David Lamontagne|1|Unknown|1|Unknown|877|Park Laurel|Road|Suite T|Midway|Williamson County|TN|31904|United States|-5|0.03|
    +4|AAAAAAAAEAAAAAAA|1997-03-13|1999-03-13|2451044|ese|218|9341467|8AM-4PM|Edwin Adams|4|Unknown|Events would achieve other, eastern hours. Mechanisms must not eat other, new org|Thomas Pollack|1|Unknown|1|Unknown|27|Lake |Ln|Suite 260|Midway|Williamson County|TN|31904|United States|-5|0.03|
    +5|AAAAAAAAEAAAAAAA|1999-03-14|2001-03-12|2450910|anti|288|9078805|8AM-4PM|Edwin Adams|8|Unknown|Events would achieve other, eastern hours. Mechanisms must not eat other, new org|Thomas Pollack|1|Unknown|1|Unknown|27|Lee 6th|Court|Suite 80|Fairview|Williamson County|TN|35709|United States|-5|0.03|
    +6|AAAAAAAAEAAAAAAA|2001-03-13|||cally|229|9026222|8AM-4PM|Edwin Adams|10|Unknown|Events would achieve other, eastern hours. Mechanisms must not eat other, new org|Thomas Pollack|1|Unknown|1|Unknown|220|6th |Lane|Suite 140|Midway|Williamson County|TN|31904|United States|-5|0.03|
    +7|AAAAAAAAHAAAAAAA|1997-03-13|||ation|297|8954883|8AM-4PM|David Thomas|9|Unknown|Architects coul|Thomas Benton|1|Unknown|1|Unknown|811|Lee |Circle|Suite T|Midway|Williamson County|TN|31904|United States|-5|0.01|
    +8|AAAAAAAAIAAAAAAA|1997-03-13|2000-03-12||eing|278|6995995|8AM-4PM|Brett Yates|2|Unknown|Various bars make most. Difficult levels introduce at a boots. Buildings welcome only never el|Dean Morrison|1|Unknown|1|Unknown|226|12th |Lane|Suite D|Fairview|Williamson County|TN|35709|United States|-5|0.08|
    +9|AAAAAAAAIAAAAAAA|2000-03-13|||eing|271|6995995|8AM-4PM|Brett Yates|2|Unknown|Formal, psychological pounds relate reasonable, young principles. Black, |Dean Morrison|1|Unknown|1|Unknown|226|Hill |Boulevard|Suite 190|Midway|Williamson County|TN|31904|United States|-5|0.08|
    +10|AAAAAAAAKAAAAAAA|1997-03-13|1999-03-13||bar|294|9294113|8AM-4PM|Raymond Jacobs|8|Unknown|Little expectations include yet forward meetings.|Michael Wilson|1|Unknown|1|Unknown|175|4th |Court|Suite C|Midway|Williamson County|TN|31904|United States|-5|0.06|
    +11|AAAAAAAAKAAAAAAA|1999-03-14|2001-03-12||ought|294|9294113|8AM-4PM|Raymond Jacobs|6|Unknown|Mysterious employe|Michael Wilson|1|Unknown|1|Unknown|175|Park Green|Court|Suite 160|Midway|Williamson County|TN|31904|United States|-5|0.11|
    +12|AAAAAAAAKAAAAAAA|2001-03-13|||ought|294|5219562|8AM-12AM|Robert Thompson|6|Unknown|Events develop i|Dustin Kelly|1|Unknown|1|Unknown|337|College |Boulevard|Suite 100|Fairview|Williamson County|TN|31904|United States|-5|0.01|

    http://git-wip-us.apache.org/repos/asf/hive/blob/a47bf736/itests/src/test/resources/testconfiguration.properties
    ----------------------------------------------------------------------
    diff --git a/itests/src/test/resources/testconfiguration.properties b/itests/src/test/resources/testconfiguration.properties
    index 376c7af..bdb96e8 100644
    --- a/itests/src/test/resources/testconfiguration.properties
    +++ b/itests/src/test/resources/testconfiguration.properties
    @@ -220,6 +220,7 @@ minitez.query.files.shared=alter_merge_2_orc.q,\
        vector_elt.q,\
        vector_groupby_3.q,\
        vector_groupby_reduce.q,\
    + vector_grouping_sets.q,\
        vector_if_expr.q,\
        vector_inner_join.q,\
        vector_interval_1.q,\

    http://git-wip-us.apache.org/repos/asf/hive/blob/a47bf736/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
    ----------------------------------------------------------------------
    diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
    index a36ee2c..bda2352 100644
    --- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
    +++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
    @@ -1119,6 +1119,10 @@ public class Vectorizer implements PhysicalPlanResolver {
            LOG.info("Grouping sets not supported in vector mode");
            return false;
          }
    + if (desc.pruneGroupingSetId()) {
    + LOG.info("Pruning grouping set id not supported in vector mode");
    + return false;
    + }
          boolean ret = validateExprNodeDesc(desc.getKeys());
          if (!ret) {
            LOG.info("Cannot vectorize groupby key expression");

    http://git-wip-us.apache.org/repos/asf/hive/blob/a47bf736/ql/src/test/queries/clientpositive/vector_grouping_sets.q
    ----------------------------------------------------------------------
    diff --git a/ql/src/test/queries/clientpositive/vector_grouping_sets.q b/ql/src/test/queries/clientpositive/vector_grouping_sets.q
    new file mode 100644
    index 0000000..9449ec4
    --- /dev/null
    +++ b/ql/src/test/queries/clientpositive/vector_grouping_sets.q
    @@ -0,0 +1,63 @@
    +SET hive.vectorized.execution.enabled=true;
    +set hive.fetch.task.conversion=none;
    +
    +-- We are not expecting either query to vectorize due to use of pruneing grouping set id,
    +-- or use of GROUPING__ID virtual column.
    +create table store_txt
    +(
    + s_store_sk int,
    + s_store_id string,
    + s_rec_start_date string,
    + s_rec_end_date string,
    + s_closed_date_sk int,
    + s_store_name string,
    + s_number_employees int,
    + s_floor_space int,
    + s_hours string,
    + s_manager string,
    + s_market_id int,
    + s_geography_class string,
    + s_market_desc string,
    + s_market_manager string,
    + s_division_id int,
    + s_division_name string,
    + s_company_id int,
    + s_company_name string,
    + s_street_number string,
    + s_street_name string,
    + s_street_type string,
    + s_suite_number string,
    + s_city string,
    + s_county string,
    + s_state string,
    + s_zip string,
    + s_country string,
    + s_gmt_offset decimal(5,2),
    + s_tax_precentage decimal(5,2)
    +)
    +row format delimited fields terminated by '|'
    +stored as textfile;
    +
    +LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt;
    +
    +create table store
    +stored as orc as
    +select * from store_txt;
    +
    +explain
    +select s_store_id
    + from store
    + group by s_store_id with rollup;
    +
    +select s_store_id
    + from store
    + group by s_store_id with rollup;
    +
    +explain
    +select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup;
    +
    +select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup;
    \ No newline at end of file

    http://git-wip-us.apache.org/repos/asf/hive/blob/a47bf736/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
    ----------------------------------------------------------------------
    diff --git a/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out b/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
    new file mode 100644
    index 0000000..688d2ac
    --- /dev/null
    +++ b/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
    @@ -0,0 +1,266 @@
    +PREHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
    +-- or use of GROUPING__ID virtual column.
    +create table store_txt
    +(
    + s_store_sk int,
    + s_store_id string,
    + s_rec_start_date string,
    + s_rec_end_date string,
    + s_closed_date_sk int,
    + s_store_name string,
    + s_number_employees int,
    + s_floor_space int,
    + s_hours string,
    + s_manager string,
    + s_market_id int,
    + s_geography_class string,
    + s_market_desc string,
    + s_market_manager string,
    + s_division_id int,
    + s_division_name string,
    + s_company_id int,
    + s_company_name string,
    + s_street_number string,
    + s_street_name string,
    + s_street_type string,
    + s_suite_number string,
    + s_city string,
    + s_county string,
    + s_state string,
    + s_zip string,
    + s_country string,
    + s_gmt_offset decimal(5,2),
    + s_tax_precentage decimal(5,2)
    +)
    +row format delimited fields terminated by '|'
    +stored as textfile
    +PREHOOK: type: CREATETABLE
    +PREHOOK: Output: database:default
    +PREHOOK: Output: default@store_txt
    +POSTHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
    +-- or use of GROUPING__ID virtual column.
    +create table store_txt
    +(
    + s_store_sk int,
    + s_store_id string,
    + s_rec_start_date string,
    + s_rec_end_date string,
    + s_closed_date_sk int,
    + s_store_name string,
    + s_number_employees int,
    + s_floor_space int,
    + s_hours string,
    + s_manager string,
    + s_market_id int,
    + s_geography_class string,
    + s_market_desc string,
    + s_market_manager string,
    + s_division_id int,
    + s_division_name string,
    + s_company_id int,
    + s_company_name string,
    + s_street_number string,
    + s_street_name string,
    + s_street_type string,
    + s_suite_number string,
    + s_city string,
    + s_county string,
    + s_state string,
    + s_zip string,
    + s_country string,
    + s_gmt_offset decimal(5,2),
    + s_tax_precentage decimal(5,2)
    +)
    +row format delimited fields terminated by '|'
    +stored as textfile
    +POSTHOOK: type: CREATETABLE
    +POSTHOOK: Output: database:default
    +POSTHOOK: Output: default@store_txt
    +PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
    +PREHOOK: type: LOAD
    +#### A masked pattern was here ####
    +PREHOOK: Output: default@store_txt
    +POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
    +POSTHOOK: type: LOAD
    +#### A masked pattern was here ####
    +POSTHOOK: Output: default@store_txt
    +PREHOOK: query: create table store
    +stored as orc as
    +select * from store_txt
    +PREHOOK: type: CREATETABLE_AS_SELECT
    +PREHOOK: Input: default@store_txt
    +PREHOOK: Output: database:default
    +PREHOOK: Output: default@store
    +POSTHOOK: query: create table store
    +stored as orc as
    +select * from store_txt
    +POSTHOOK: type: CREATETABLE_AS_SELECT
    +POSTHOOK: Input: default@store_txt
    +POSTHOOK: Output: database:default
    +POSTHOOK: Output: default@store
    +PREHOOK: query: explain
    +select s_store_id
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +POSTHOOK: query: explain
    +select s_store_id
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +STAGE DEPENDENCIES:
    + Stage-1 is a root stage
    + Stage-0 depends on stages: Stage-1
    +
    +STAGE PLANS:
    + Stage: Stage-1
    + Tez
    + Edges:
    + Reducer 2 <- Map 1 (SIMPLE_EDGE)
    +#### A masked pattern was here ####
    + Vertices:
    + Map 1
    + Map Operator Tree:
    + TableScan
    + alias: store
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Select Operator
    + expressions: s_store_id (type: string)
    + outputColumnNames: _col0
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Group By Operator
    + keys: _col0 (type: string), '0' (type: string)
    + mode: hash
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reduce Output Operator
    + key expressions: _col0 (type: string), _col1 (type: string)
    + sort order: ++
    + Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reducer 2
    + Reduce Operator Tree:
    + Group By Operator
    + keys: KEY._col0 (type: string), KEY._col1 (type: string)
    + mode: mergepartial
    + outputColumnNames: _col0
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + pruneGroupingSetId: true
    + File Output Operator
    + compressed: false
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + table:
    + input format: org.apache.hadoop.mapred.TextInputFormat
    + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
    + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    +
    + Stage: Stage-0
    + Fetch Operator
    + limit: -1
    + Processor Tree:
    + ListSink
    +
    +PREHOOK: query: select s_store_id
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +PREHOOK: Input: default@store
    +#### A masked pattern was here ####
    +POSTHOOK: query: select s_store_id
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +POSTHOOK: Input: default@store
    +#### A masked pattern was here ####
    +NULL
    +AAAAAAAABAAAAAAA
    +AAAAAAAACAAAAAAA
    +AAAAAAAAEAAAAAAA
    +AAAAAAAAHAAAAAAA
    +AAAAAAAAIAAAAAAA
    +AAAAAAAAKAAAAAAA
    +PREHOOK: query: explain
    +select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +POSTHOOK: query: explain
    +select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +STAGE DEPENDENCIES:
    + Stage-1 is a root stage
    + Stage-0 depends on stages: Stage-1
    +
    +STAGE PLANS:
    + Stage: Stage-1
    + Tez
    + Edges:
    + Reducer 2 <- Map 1 (SIMPLE_EDGE)
    +#### A masked pattern was here ####
    + Vertices:
    + Map 1
    + Map Operator Tree:
    + TableScan
    + alias: store
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Select Operator
    + expressions: s_store_id (type: string)
    + outputColumnNames: _col0
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Group By Operator
    + keys: _col0 (type: string), '0' (type: string)
    + mode: hash
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reduce Output Operator
    + key expressions: _col0 (type: string), _col1 (type: string)
    + sort order: ++
    + Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reducer 2
    + Reduce Operator Tree:
    + Group By Operator
    + keys: KEY._col0 (type: string), KEY._col1 (type: string)
    + mode: mergepartial
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Select Operator
    + expressions: _col0 (type: string), _col1 (type: string)
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + File Output Operator
    + compressed: false
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + table:
    + input format: org.apache.hadoop.mapred.TextInputFormat
    + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
    + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    + Execution mode: vectorized
    +
    + Stage: Stage-0
    + Fetch Operator
    + limit: -1
    + Processor Tree:
    + ListSink
    +
    +PREHOOK: query: select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +PREHOOK: Input: default@store
    +#### A masked pattern was here ####
    +POSTHOOK: query: select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +POSTHOOK: Input: default@store
    +#### A masked pattern was here ####
    +NULL 0
    +AAAAAAAABAAAAAAA 1
    +AAAAAAAACAAAAAAA 1
    +AAAAAAAAEAAAAAAA 1
    +AAAAAAAAHAAAAAAA 1
    +AAAAAAAAIAAAAAAA 1
    +AAAAAAAAKAAAAAAA 1

    http://git-wip-us.apache.org/repos/asf/hive/blob/a47bf736/ql/src/test/results/clientpositive/vector_grouping_sets.q.out
    ----------------------------------------------------------------------
    diff --git a/ql/src/test/results/clientpositive/vector_grouping_sets.q.out b/ql/src/test/results/clientpositive/vector_grouping_sets.q.out
    new file mode 100644
    index 0000000..69d6187
    --- /dev/null
    +++ b/ql/src/test/results/clientpositive/vector_grouping_sets.q.out
    @@ -0,0 +1,253 @@
    +PREHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
    +-- or use of GROUPING__ID virtual column.
    +create table store_txt
    +(
    + s_store_sk int,
    + s_store_id string,
    + s_rec_start_date string,
    + s_rec_end_date string,
    + s_closed_date_sk int,
    + s_store_name string,
    + s_number_employees int,
    + s_floor_space int,
    + s_hours string,
    + s_manager string,
    + s_market_id int,
    + s_geography_class string,
    + s_market_desc string,
    + s_market_manager string,
    + s_division_id int,
    + s_division_name string,
    + s_company_id int,
    + s_company_name string,
    + s_street_number string,
    + s_street_name string,
    + s_street_type string,
    + s_suite_number string,
    + s_city string,
    + s_county string,
    + s_state string,
    + s_zip string,
    + s_country string,
    + s_gmt_offset decimal(5,2),
    + s_tax_precentage decimal(5,2)
    +)
    +row format delimited fields terminated by '|'
    +stored as textfile
    +PREHOOK: type: CREATETABLE
    +PREHOOK: Output: database:default
    +PREHOOK: Output: default@store_txt
    +POSTHOOK: query: -- We are not expecting either query to vectorize due to use of pruneing grouping set id,
    +-- or use of GROUPING__ID virtual column.
    +create table store_txt
    +(
    + s_store_sk int,
    + s_store_id string,
    + s_rec_start_date string,
    + s_rec_end_date string,
    + s_closed_date_sk int,
    + s_store_name string,
    + s_number_employees int,
    + s_floor_space int,
    + s_hours string,
    + s_manager string,
    + s_market_id int,
    + s_geography_class string,
    + s_market_desc string,
    + s_market_manager string,
    + s_division_id int,
    + s_division_name string,
    + s_company_id int,
    + s_company_name string,
    + s_street_number string,
    + s_street_name string,
    + s_street_type string,
    + s_suite_number string,
    + s_city string,
    + s_county string,
    + s_state string,
    + s_zip string,
    + s_country string,
    + s_gmt_offset decimal(5,2),
    + s_tax_precentage decimal(5,2)
    +)
    +row format delimited fields terminated by '|'
    +stored as textfile
    +POSTHOOK: type: CREATETABLE
    +POSTHOOK: Output: database:default
    +POSTHOOK: Output: default@store_txt
    +PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
    +PREHOOK: type: LOAD
    +#### A masked pattern was here ####
    +PREHOOK: Output: default@store_txt
    +POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/store_200' OVERWRITE INTO TABLE store_txt
    +POSTHOOK: type: LOAD
    +#### A masked pattern was here ####
    +POSTHOOK: Output: default@store_txt
    +PREHOOK: query: create table store
    +stored as orc as
    +select * from store_txt
    +PREHOOK: type: CREATETABLE_AS_SELECT
    +PREHOOK: Input: default@store_txt
    +PREHOOK: Output: database:default
    +PREHOOK: Output: default@store
    +POSTHOOK: query: create table store
    +stored as orc as
    +select * from store_txt
    +POSTHOOK: type: CREATETABLE_AS_SELECT
    +POSTHOOK: Input: default@store_txt
    +POSTHOOK: Output: database:default
    +POSTHOOK: Output: default@store
    +PREHOOK: query: explain
    +select s_store_id
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +POSTHOOK: query: explain
    +select s_store_id
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +STAGE DEPENDENCIES:
    + Stage-1 is a root stage
    + Stage-0 depends on stages: Stage-1
    +
    +STAGE PLANS:
    + Stage: Stage-1
    + Map Reduce
    + Map Operator Tree:
    + TableScan
    + alias: store
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Select Operator
    + expressions: s_store_id (type: string)
    + outputColumnNames: _col0
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Group By Operator
    + keys: _col0 (type: string), '0' (type: string)
    + mode: hash
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reduce Output Operator
    + key expressions: _col0 (type: string), _col1 (type: string)
    + sort order: ++
    + Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reduce Operator Tree:
    + Group By Operator
    + keys: KEY._col0 (type: string), KEY._col1 (type: string)
    + mode: mergepartial
    + outputColumnNames: _col0
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + pruneGroupingSetId: true
    + File Output Operator
    + compressed: false
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + table:
    + input format: org.apache.hadoop.mapred.TextInputFormat
    + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
    + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    +
    + Stage: Stage-0
    + Fetch Operator
    + limit: -1
    + Processor Tree:
    + ListSink
    +
    +PREHOOK: query: select s_store_id
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +PREHOOK: Input: default@store
    +#### A masked pattern was here ####
    +POSTHOOK: query: select s_store_id
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +POSTHOOK: Input: default@store
    +#### A masked pattern was here ####
    +NULL
    +AAAAAAAABAAAAAAA
    +AAAAAAAACAAAAAAA
    +AAAAAAAAEAAAAAAA
    +AAAAAAAAHAAAAAAA
    +AAAAAAAAIAAAAAAA
    +AAAAAAAAKAAAAAAA
    +PREHOOK: query: explain
    +select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +POSTHOOK: query: explain
    +select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +STAGE DEPENDENCIES:
    + Stage-1 is a root stage
    + Stage-0 depends on stages: Stage-1
    +
    +STAGE PLANS:
    + Stage: Stage-1
    + Map Reduce
    + Map Operator Tree:
    + TableScan
    + alias: store
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Select Operator
    + expressions: s_store_id (type: string)
    + outputColumnNames: _col0
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Group By Operator
    + keys: _col0 (type: string), '0' (type: string)
    + mode: hash
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reduce Output Operator
    + key expressions: _col0 (type: string), _col1 (type: string)
    + sort order: ++
    + Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
    + Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
    + Reduce Operator Tree:
    + Group By Operator
    + keys: KEY._col0 (type: string), KEY._col1 (type: string)
    + mode: mergepartial
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + Select Operator
    + expressions: _col0 (type: string), _col1 (type: string)
    + outputColumnNames: _col0, _col1
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + File Output Operator
    + compressed: false
    + Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
    + table:
    + input format: org.apache.hadoop.mapred.TextInputFormat
    + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
    + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    +
    + Stage: Stage-0
    + Fetch Operator
    + limit: -1
    + Processor Tree:
    + ListSink
    +
    +PREHOOK: query: select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +PREHOOK: type: QUERY
    +PREHOOK: Input: default@store
    +#### A masked pattern was here ####
    +POSTHOOK: query: select s_store_id, GROUPING__ID
    + from store
    + group by s_store_id with rollup
    +POSTHOOK: type: QUERY
    +POSTHOOK: Input: default@store
    +#### A masked pattern was here ####
    +NULL 0
    +AAAAAAAABAAAAAAA 1
    +AAAAAAAACAAAAAAA 1
    +AAAAAAAAEAAAAAAA 1
    +AAAAAAAAHAAAAAAA 1
    +AAAAAAAAIAAAAAAA 1
    +AAAAAAAAKAAAAAAA 1

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommits @
categorieshive, hadoop
postedMay 28, '15 at 10:59p
activeMay 28, '15 at 11:18p
posts2
users1
websitehive.apache.org

1 user in discussion

Gunther: 2 posts

People

Translate

site design / logo © 2021 Grokbase