Grokbase Groups Hive user July 2009
FAQ
Hi all,

How do you export hive tables into oracle/mysql? through oci(Oracle Call
Interface), jdbc(oci/thin/MysqlDriver) or odbc?
Written such tools in c/c++ through HDFS native library or in java(jdbc)?
How do you translate hive tables' schema into oracle/mysql shema?

Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Search Discussions

  • Edward Capriolo at Jul 14, 2009 at 2:45 pm
    Min,

    Funny should should ask. I was talking to Aaron about sqoop, and
    un-squoop (hadoop->SQL). I am also started building a dboutput udf
    which I am about to open another thread about. It looks like we may
    open up a hadoop Jira, and possible a Hive Jiira to chat about this.
    Stay tuned!

    Edward

    On Tue, Jul 14, 2009 at 6:18 AM, Min Zhouwrote:
    Hi all,

    How do you export hive tables into oracle/mysql?  through oci(Oracle Call
    Interface), jdbc(oci/thin/MysqlDriver) or odbc?
    Written such tools in c/c++ through HDFS native library or in java(jdbc)?
    How do you translate hive tables' schema into oracle/mysql shema?

    Thanks,
    Min
    --
    My research interests are distributed systems, parallel computing and
    bytecode based virtual machine.

    My profile:
    http://www.linkedin.com/in/coderplay
    My blog:
    http://coderplay.javaeye.com
  • Peter Skomoroch at Jul 14, 2009 at 2:58 pm
    I run a cluster on EC2, at the moment I just dump tables to ctrl-A delimited
    flat files with INSERT OVERWRITE DIRECTORY and bulk load those to MySQL with
    LOAD DATA LOCAL INFILE:

    something like this:

    LOAD DATA LOCAL INFILE '/mnt/pages.txt'
    INTO TABLE new_pages
    FIELDS TERMINATED BY 0x01
    LINES TERMINATED BY '\n'
    (id, url, title, page_latest, total_pageviews, monthly_trend, daily_trend);


    There seems to be a bug where ctrl-A is the only delimiter used, regardless
    of the Hive table structure (tab delimited etc.)

    On Tue, Jul 14, 2009 at 10:45 AM, Edward Capriolo wrote:

    Min,

    Funny should should ask. I was talking to Aaron about sqoop, and
    un-squoop (hadoop->SQL). I am also started building a dboutput udf
    which I am about to open another thread about. It looks like we may
    open up a hadoop Jira, and possible a Hive Jiira to chat about this.
    Stay tuned!

    Edward

    On Tue, Jul 14, 2009 at 6:18 AM, Min Zhouwrote:
    Hi all,

    How do you export hive tables into oracle/mysql? through oci(Oracle Call
    Interface), jdbc(oci/thin/MysqlDriver) or odbc?
    Written such tools in c/c++ through HDFS native library or in java(jdbc)?
    How do you translate hive tables' schema into oracle/mysql shema?

    Thanks,
    Min
    --
    My research interests are distributed systems, parallel computing and
    bytecode based virtual machine.

    My profile:
    http://www.linkedin.com/in/coderplay
    My blog:
    http://coderplay.javaeye.com


    --
    Peter N. Skomoroch
    617.285.8348
    http://www.datawrangling.com
    http://delicious.com/pskomoroch
    http://twitter.com/peteskomoroch
  • Prasad Chakka at Jul 14, 2009 at 4:24 pm
    Peter, can you open a JIRA for the output delimiter issue?

    Also the method described by Peter is the best (you could even split the output into multiple files and use Oracle's parallel loader if you have a multi-core client and server). JDBC/ODBC/OCI will not be able to handle large data volumes.


    ________________________________
    From: Peter Skomoroch <peter.skomoroch@gmail.com>
    Reply-To: <hive-user@hadoop.apache.org>
    Date: Tue, 14 Jul 2009 07:57:32 -0700
    To: <hive-user@hadoop.apache.org>
    Subject: Re: how to export hive tables into oracle/mysql?

    I run a cluster on EC2, at the moment I just dump tables to ctrl-A delimited flat files with INSERT OVERWRITE DIRECTORY and bulk load those to MySQL with LOAD DATA LOCAL INFILE:

    something like this:

    LOAD DATA LOCAL INFILE '/mnt/pages.txt'
    INTO TABLE new_pages
    FIELDS TERMINATED BY 0x01
    LINES TERMINATED BY '\n'
    (id, url, title, page_latest, total_pageviews, monthly_trend, daily_trend);


    There seems to be a bug where ctrl-A is the only delimiter used, regardless of the Hive table structure (tab delimited etc.)


    On Tue, Jul 14, 2009 at 10:45 AM, Edward Capriolo wrote:
    Min,

    Funny should should ask. I was talking to Aaron about sqoop, and
    un-squoop (hadoop->SQL). I am also started building a dboutput udf
    which I am about to open another thread about. It looks like we may
    open up a hadoop Jira, and possible a Hive Jiira to chat about this.
    Stay tuned!

    Edward

    On Tue, Jul 14, 2009 at 6:18 AM, Min Zhouwrote:
    Hi all,

    How do you export hive tables into oracle/mysql? through oci(Oracle Call
    Interface), jdbc(oci/thin/MysqlDriver) or odbc?
    Written such tools in c/c++ through HDFS native library or in java(jdbc)?
    How do you translate hive tables' schema into oracle/mysql shema?

    Thanks,
    Min
    --
    My research interests are distributed systems, parallel computing and
    bytecode based virtual machine.

    My profile:
    http://www.linkedin.com/in/coderplay
    My blog:
    http://coderplay.javaeye.com


    --
    Peter N. Skomoroch
    617.285.8348
    http://www.datawrangling.com
    http://delicious.com/pskomoroch
    http://twitter.com/peteskomoroch
  • Peter Skomoroch at Jul 14, 2009 at 4:53 pm
    new JIRA for the output delimiter issue:

    https://issues.apache.org/jira/browse/HIVE-634
    On Tue, Jul 14, 2009 at 12:24 PM, Prasad Chakka wrote:

    Peter, can you open a JIRA for the output delimiter issue?

    Also the method described by Peter is the best (you could even split the
    output into multiple files and use Oracle’s parallel loader if you have a
    multi-core client and server). JDBC/ODBC/OCI will not be able to handle
    large data volumes.


    ------------------------------
    *From: *Peter Skomoroch <peter.skomoroch@gmail.com>
    *Reply-To: *<hive-user@hadoop.apache.org>
    *Date: *Tue, 14 Jul 2009 07:57:32 -0700
    *To: *<hive-user@hadoop.apache.org>
    *Subject: *Re: how to export hive tables into oracle/mysql?


    I run a cluster on EC2, at the moment I just dump tables to ctrl-A
    delimited flat files with INSERT OVERWRITE DIRECTORY and bulk load those to
    MySQL with LOAD DATA LOCAL INFILE:

    something like this:

    LOAD DATA LOCAL INFILE '/mnt/pages.txt'
    INTO TABLE new_pages
    FIELDS TERMINATED BY 0x01
    LINES TERMINATED BY '\n'
    (id, url, title, page_latest, total_pageviews, monthly_trend, daily_trend);


    There seems to be a bug where ctrl-A is the only delimiter used, regardless
    of the Hive table structure (tab delimited etc.)


    On Tue, Jul 14, 2009 at 10:45 AM, Edward Capriolo wrote:

    Min,

    Funny should should ask. I was talking to Aaron about sqoop, and
    un-squoop (hadoop->SQL). I am also started building a dboutput udf
    which I am about to open another thread about. It looks like we may
    open up a hadoop Jira, and possible a Hive Jiira to chat about this.
    Stay tuned!

    Edward

    On Tue, Jul 14, 2009 at 6:18 AM, Min Zhouwrote:
    Hi all,

    How do you export hive tables into oracle/mysql? through oci(Oracle Call
    Interface), jdbc(oci/thin/MysqlDriver) or odbc?
    Written such tools in c/c++ through HDFS native library or in java(jdbc)?
    How do you translate hive tables' schema into oracle/mysql shema?

    Thanks,
    Min
    --
    My research interests are distributed systems, parallel computing and
    bytecode based virtual machine.

    My profile:
    http://www.linkedin.com/in/coderplay
    My blog:
    http://coderplay.javaeye.com



    --
    Peter N. Skomoroch
    617.285.8348
    http://www.datawrangling.com
    http://delicious.com/pskomoroch
    http://twitter.com/peteskomoroch

    --
    Peter N. Skomoroch
    617.285.8348
    http://www.datawrangling.com
    http://delicious.com/pskomoroch
    http://twitter.com/peteskomoroch

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedJul 14, '09 at 10:18a
activeJul 14, '09 at 4:53p
posts5
users4
websitehive.apache.org

People

Translate

site design / logo © 2022 Grokbase