Grokbase Groups Hive user May 2011
FAQ
Hi,

I want to be able to extract data from Mysql to Hadoop without writing the
data to disk. I was thinking in the line of piping the extract and loading
it to Hadoop.

Something like this:

mysql <extract_query> | hive -e 'load data <via_pipe> into table ...

Has anyone done this before?

Search Discussions

  • Zoltan Prekopcsak at May 6, 2011 at 9:06 pm
    Hi,

    I think this is what Sqoop is for:
    http://www.cloudera.com/blog/2009/06/introducing-sqoop/
    http://www.cloudera.com/downloads/sqoop/

    Best, Zoltan


    5/6/11 7:45 PM keltezéssel, bichonfrise74 írta:
    Hi,

    I want to be able to extract data from Mysql to Hadoop without writing
    the data to disk. I was thinking in the line of piping the extract and
    loading it to Hadoop.

    Something like this:

    mysql <extract_query> | hive -e 'load data <via_pipe> into table ...

    Has anyone done this before?
  • Bichonfrise74 at May 6, 2011 at 10:57 pm
    Thanks Zoltan. I will check it out.

    2011/5/6 Zoltan Prekopcsak <preko1984@gmail.com>
    Hi,

    I think this is what Sqoop is for:
    http://www.cloudera.com/blog/2009/06/introducing-sqoop/
    http://www.cloudera.com/downloads/sqoop/

    Best, Zoltan


    5/6/11 7:45 PM keltezéssel, bichonfrise74 írta:

    Hi,

    I want to be able to extract data from Mysql to Hadoop without writing the
    data to disk. I was thinking in the line of piping the extract and loading
    it to Hadoop.

    Something like this:

    mysql <extract_query> | hive -e 'load data <via_pipe> into table ...

    Has anyone done this before?
  • Time Less at May 27, 2011 at 1:46 am
    I had trouble with Sqoop, so here's what I do (Perl):

    $cmd = qq#echo "select * from $tableName where $dateColumn >= '$dayStart
    00:00:00' and $dateColumn < '$dayEnd 00:00:00'" \\
    mysql -h $dwIP --quick -B --skip-column-names --user=$USER
    --password=$PASS $databaseName \\
    ssh hdfs\@$hadoopIP "cat | hadoop fs -put -
    /user/hive/warehouse/$tableName/$dayStart/datafile"#;
    system ($cmd);

    Then when I create my Hive table, I point each partition at
    /user/hive/warehouse/$tableName/$dayStart et voila! Everything works just
    fine and I have full control over the import.

    On Fri, May 6, 2011 at 3:57 PM, bichonfrise74 wrote:

    Thanks Zoltan. I will check it out.


    2011/5/6 Zoltan Prekopcsak <preko1984@gmail.com>
    Hi,

    I think this is what Sqoop is for:
    http://www.cloudera.com/blog/2009/06/introducing-sqoop/
    http://www.cloudera.com/downloads/sqoop/

    Best, Zoltan


    5/6/11 7:45 PM keltezéssel, bichonfrise74 írta:

    Hi,

    I want to be able to extract data from Mysql to Hadoop without writing
    the data to disk. I was thinking in the line of piping the extract and
    loading it to Hadoop.

    Something like this:

    mysql <extract_query> | hive -e 'load data <via_pipe> into table ...

    Has anyone done this before?

    --
    Tim

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedMay 6, '11 at 5:46p
activeMay 27, '11 at 1:46a
posts4
users3
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase