FAQ
When will Impala support non-primitive data types, especially map and array?

Thanks.

Search Discussions

  • Marcel Kornacker at Jan 30, 2013 at 12:29 am
    We can't promise specific dates yet, but we know this won't arrive in
    the GA timeframe (early April). It's high on our list, though, because
    it's been requested quite a few times, which means it'll be one of the
    first things we'll be working on after GA.

    Marcel
    On Tue, Jan 29, 2013 at 4:17 PM, Steven Wong wrote:
    When will Impala support non-primitive data types, especially map and array?

    Thanks.
  • Uri Laserson at Feb 7, 2014 at 11:38 pm
    Perhaps this is a good time to also mention that we're currently working on
    compiled Python UDFs in Impala. The idea would be that you could write a
    UDF in Python and compile it down to LLVM IR, as if it were written in C++.
      (This would NOT fire up a Python interpreter.) You would get the
    convenience of working with Python with the performance of a compiled
    language, and eventually you might get access to some of Python's
    libraries, like numpy, scikit-learn, nltk, etc.

    Some caveats: this is still highly speculative, though we see a path where
    it could work. At least to begin with, this would also only support the
    Impala primitive types.

    Uri


    On Fri, Feb 7, 2014 at 11:55 AM, Jonathan Schuff
    wrote:
    Just wanted to drop in and convey our interest in non-primitive data type
    support. Many of our use cases require the dynamic handling of arrays and
    structures, such as performing time series analyses over large datasets
    (which is coincidentally the only reason why Hive is still a primary aspect
    of our stack). If non-primitive data type support is added, I believe that
    Impala would become *much *more useful for a broad range of complex
    scientific analyses, especially in conjunction with Hive UDF support.
    Thanks for reading!

    Jon
    On Sunday, July 28, 2013 12:52:58 PM UTC-4, Marcel Kornacker wrote:

    Impala does not support non-scalar column types right now, but we are
    actively looking into adding support for that in the medium term.
    On Sat, Jul 27, 2013 at 3:25 PM, SN wrote:
    Bumping up this topic again since Its relevant in my use case. Does impala
    still not support non primitive data types...
    I have a dynamic column family in hbase that I am mapping using a
    map<string,string> in hive table

    Something of this form:
    CREATE EXTERNAL TABLE tsp_raw(row_key string,value map<string,string>) row
    format delimited fields terminated by ',' STORED BY
    'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH
    SERDEPROPERTIES
    ("hbase.columns.mapping" = ":key,cf:,") TBLPROPERTIES("hbase.table.name" =
    "tsp");

    However, when I try to read it in with the jdbc connector (via
    impalad), am
    unable to parse the value column which is a map collection...

    Any thoughts.. when I run the query directly in hive I of course do get a
    complete dump of the map. Any work arounds for the time being would also be
    very helpful.
    Thanks.

    On Tuesday, January 29, 2013 7:29:27 PM UTC-5, Marcel Kornacker wrote:

    We can't promise specific dates yet, but we know this won't arrive in
    the GA timeframe (early April). It's high on our list, though, because
    it's been requested quite a few times, which means it'll be one of the
    first things we'll be working on after GA.

    Marcel

    On Tue, Jan 29, 2013 at 4:17 PM, Steven Wong <sw...@netflix.com>
    wrote:
    When will Impala support non-primitive data types, especially map
    and
    array?

    Thanks.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to impala-user+unsubscribe@cloudera.org.


    --
    Uri Laserson, PhD
    Data Scientist, Cloudera
    Twitter/GitHub: @laserson
    +1 617 910 0447
    laserson@cloudera.com

    To unsubscribe from this group and stop receiving emails from it, send an email to impala-user+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedJan 30, '13 at 12:17a
activeFeb 7, '14 at 11:38p
posts3
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase