FAQ
Hi,

I have some transactional data, which has a few map elements, but one more
complex - array of maps. Something like this

CREATE EXTERNAL TABLE mytable
(
log_timestamp string,
price map <string,string>,
opt array <map<string, string>>
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
COLLECTION ITEMS TERMINATED BY '\004'
MAP KEYS TERMINATED BY '\002'
STORED AS TextFile;

However, the data is a bit more complex, and I also use '\003' and '\005\'
for delimiters, only I have no place to tell Hive about it.

So it seems that I can define data structures like opt array <map<string,
string>>, but I don't know how to import such data.

What would be the advice?

Thank you. Sincerely,
Mark

Search Discussions

  • Steven Wong at Oct 6, 2011 at 1:12 am
    http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201009.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B101C07A390@ExchMBX104.netflix.com%3E is a similar situation. Basically you have to use the default delimiters.


    From: Mark Kerzner
    Sent: Wednesday, October 05, 2011 3:34 PM
    To: Hive user
    Subject: Best way to import complex data into Hive?

    Hi,

    I have some transactional data, which has a few map elements, but one more complex - array of maps. Something like this

    CREATE EXTERNAL TABLE mytable
    (
    log_timestamp string,
    price map <string,string>,
    opt array <map<string, string>>
    )
    ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '\001'
    COLLECTION ITEMS TERMINATED BY '\004'
    MAP KEYS TERMINATED BY '\002'
    STORED AS TextFile;

    However, the data is a bit more complex, and I also use '\003' and '\005\' for delimiters, only I have no place to tell Hive about it.

    So it seems that I can define data structures like opt array <map<string, string>>, but I don't know how to import such data.

    What would be the advice?

    Thank you. Sincerely,
    Mark
  • Mark Kerzner at Oct 6, 2011 at 1:37 am
    Wow, Steve,

    who could have thought?! That's hidden and undocumented, but useful. I guess
    "read the code, stupid" applies to me in this case.

    Thank you,
    Mark
    On Wed, Oct 5, 2011 at 8:12 PM, Steven Wong wrote:


    http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201009.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B101C07A390@ExchMBX104.netflix.com%3Eis a similar situation. Basically you have to use the default delimiters.
    ****

    ** **

    ** **

    *From:* Mark Kerzner
    *Sent:* Wednesday, October 05, 2011 3:34 PM
    *To:* Hive user
    *Subject:* Best way to import complex data into Hive?****

    ** **

    Hi,

    I have some transactional data, which has a few map elements, but one more
    complex - array of maps. Something like this

    CREATE EXTERNAL TABLE mytable
    (
    log_timestamp string,
    price map <string,string>,
    opt array <map<string, string>>
    )
    ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '\001'
    COLLECTION ITEMS TERMINATED BY '\004'
    MAP KEYS TERMINATED BY '\002'
    STORED AS TextFile;

    However, the data is a bit more complex, and I also use '\003' and '\005\'
    for delimiters, only I have no place to tell Hive about it.

    So it seems that I can define data structures like opt array <map<string,
    string>>, but I don't know how to import such data.

    What would be the advice?

    Thank you. Sincerely,
    Mark****
  • Mark Kerzner at Oct 7, 2011 at 2:54 am
    Thank you, Steven,

    that was so good that I had to put it on my blog,
    http://shmsoft.blogspot.com/2011/10/loading-inner-maps-in-hive.html

    Mark
    On Wed, Oct 5, 2011 at 8:12 PM, Steven Wong wrote:


    http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201009.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B101C07A390@ExchMBX104.netflix.com%3Eis a similar situation. Basically you have to use the default delimiters.
    ****

    ** **

    ** **

    *From:* Mark Kerzner
    *Sent:* Wednesday, October 05, 2011 3:34 PM
    *To:* Hive user
    *Subject:* Best way to import complex data into Hive?****

    ** **

    Hi,

    I have some transactional data, which has a few map elements, but one more
    complex - array of maps. Something like this

    CREATE EXTERNAL TABLE mytable
    (
    log_timestamp string,
    price map <string,string>,
    opt array <map<string, string>>
    )
    ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '\001'
    COLLECTION ITEMS TERMINATED BY '\004'
    MAP KEYS TERMINATED BY '\002'
    STORED AS TextFile;

    However, the data is a bit more complex, and I also use '\003' and '\005\'
    for delimiters, only I have no place to tell Hive about it.

    So it seems that I can define data structures like opt array <map<string,
    string>>, but I don't know how to import such data.

    What would be the advice?

    Thank you. Sincerely,
    Mark****

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedOct 5, '11 at 10:34p
activeOct 7, '11 at 2:54a
posts4
users2
websitehive.apache.org

2 users in discussion

Mark Kerzner: 3 posts Steven Wong: 1 post

People

Translate

site design / logo © 2022 Grokbase