FAQ
Chu Tong created HIVE-7999:
------------------------------

              Summary: Hive metastore query too long when select * on table with large number of partitions
                  Key: HIVE-7999
                  URL: https://issues.apache.org/jira/browse/HIVE-7999
              Project: Hive
           Issue Type: Bug
           Components: Metastore
             Reporter: Chu Tong




When it comes to selecting * on a Hive table with large number of partitions, a query like this:

SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...

is generated and executed on metastore, however, due the query lists all the partitions in it, SQL server is unable to compile/execute such a long query and causing the whole query to fail.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Search Discussions

  • Chu Tong (JIRA) at Sep 5, 2014 at 5:19 am
    [ https://issues.apache.org/jira/browse/HIVE-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chu Tong updated HIVE-7999:
    ---------------------------
         Description:
    When it comes to selecting * on a Hive table with large number of partitions, a query like this:

    SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...

    is generated and executed on metastore, however, due the query lists all the partitions in it, SQL DB is unable to compile/execute such a long query and causing the whole query to fail.


       was:


    When it comes to selecting * on a Hive table with large number of partitions, a query like this:

    SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...

    is generated and executed on metastore, however, due the query lists all the partitions in it, SQL server is unable to compile/execute such a long query and causing the whole query to fail.


    Hive metastore query too long when select * on table with large number of partitions
    -------------------------------------------------------------------------------------

    Key: HIVE-7999
    URL: https://issues.apache.org/jira/browse/HIVE-7999
    Project: Hive
    Issue Type: Bug
    Components: Metastore
    Reporter: Chu Tong

    When it comes to selecting * on a Hive table with large number of partitions, a query like this:
    SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...
    is generated and executed on metastore, however, due the query lists all the partitions in it, SQL DB is unable to compile/execute such a long query and causing the whole query to fail.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)
  • Chu Tong (JIRA) at Sep 5, 2014 at 5:19 am
    [ https://issues.apache.org/jira/browse/HIVE-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chu Tong updated HIVE-7999:
    ---------------------------
         Description:
    When it comes to selecting * on a Hive table with large number of partitions, a query like this:

    SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...

    is generated and executed on metastore, however, due the query lists all the partitions in it, SQL server is unable to compile/execute such a long query and causing the whole query to fail.


       was:
    When it comes to selecting * on a Hive table with large number of partitions, a query like this:

    SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...

    is generated and executed on metastore, however, due the query lists all the partitions in it, SQL DB is unable to compile/execute such a long query and causing the whole query to fail.


    Hive metastore query too long when select * on table with large number of partitions
    -------------------------------------------------------------------------------------

    Key: HIVE-7999
    URL: https://issues.apache.org/jira/browse/HIVE-7999
    Project: Hive
    Issue Type: Bug
    Components: Metastore
    Reporter: Chu Tong

    When it comes to selecting * on a Hive table with large number of partitions, a query like this:
    SELECT PARTITIONS.PART_ID, SDS.SD_ID, SDS.CD_ID, SERDES.SERDE_ID, PARTITIONS.CREATE_TIME, ... SERDES.SLIB from PARTITIONS LEFT OUTER JOIN SDS ON PARTITIONS.SD_ID = SDS.SD_ID LEFT OUTER JOIN SERDES ON SDS.SERDE_ID = SERDES.SERDE_ID where PART_ID in (1,2,3,4 ... 10000000 ...
    is generated and executed on metastore, however, due the query lists all the partitions in it, SQL server is unable to compile/execute such a long query and causing the whole query to fail.


    --
    This message was sent by Atlassian JIRA
    (v6.3.4#6332)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshive, hadoop
postedSep 5, '14 at 5:18a
activeSep 5, '14 at 5:19a
posts3
users1
websitehive.apache.org

1 user in discussion

Chu Tong (JIRA): 3 posts

People

Translate

site design / logo © 2022 Grokbase