Grokbase › Groups › Hadoop › common-user › May 2009
FAQ
Hi,
I am want to load data in mysql using a hadoop file similar to following:
LOAD DATA INFILE 'test.txt' INTO TABLE test
FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';

But how do I load the hdfs file into the mysql comand above? Do I start the
file name with hdfd://test.txt? I am using java.
Thanks.

--
View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Search Discussions

  • Tim robertson at May 19, 2009 at 9:54 pm ⇧
    So you are using a java program to execute a "load data infile"
    command on mysql through JDBC?
    If so I *think* you would have to copy it onto the mysql machine from
    HDFS first, or the machine running the command and then try a 'load
    data local infile'.

    Or pehaps use the
    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html

    Cheers

    Tim
    On Tue, May 19, 2009 at 9:54 PM, dealmaker wrote:

    Hi,
    I am want to load data in mysql using a hadoop file similar to following:
    LOAD DATA INFILE 'test.txt' INTO TABLE test
    FIELDS TERMINATED BY ','  LINES STARTING BY 'xxx';

    But how do I load the hdfs file into the mysql comand above?  Do I start the
    file name with hdfd://test.txt?  I am using java.
    Thanks.

    --
    View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Dealmaker at May 19, 2009 at 9:58 pm ⇧
    Does DBOutputFormat have similar performance as Load Data Infile?
    Thanks.


    TimRobertson100 wrote:
    So you are using a java program to execute a "load data infile"
    command on mysql through JDBC?
    If so I *think* you would have to copy it onto the mysql machine from
    HDFS first, or the machine running the command and then try a 'load
    data local infile'.

    Or pehaps use the
    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html

    Cheers

    Tim
    On Tue, May 19, 2009 at 9:54 PM, dealmaker wrote:

    Hi,
    I am want to load data in mysql using a hadoop file similar to
    following:
    LOAD DATA INFILE 'test.txt' INTO TABLE test
    FIELDS TERMINATED BY ','  LINES STARTING BY 'xxx';

    But how do I load the hdfs file into the mysql comand above?  Do I start
    the
    file name with hdfd://test.txt?  I am using java.
    Thanks.

    --
    View this message in context:
    http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23625140.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Alex Loddengaard at May 20, 2009 at 12:20 am ⇧
    DBOutputFormat will very likely put significantly more load on your MySQL
    server vs. LOAD DATA INFILE. DBOutputFormat will trounce your MySQL server
    with at least one connection per reducer. This may be OK if you have a
    small number of reducers and a small amount of output data. LOAD DATA
    INFILE is a much safer approach, in that it won't potentially axe your MySQL
    database.

    Alex
    On Tue, May 19, 2009 at 2:58 PM, dealmaker wrote:


    Does DBOutputFormat have similar performance as Load Data Infile?
    Thanks.


    TimRobertson100 wrote:
    So you are using a java program to execute a "load data infile"
    command on mysql through JDBC?
    If so I *think* you would have to copy it onto the mysql machine from
    HDFS first, or the machine running the command and then try a 'load
    data local infile'.

    Or pehaps use the
    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html
    Cheers

    Tim
    On Tue, May 19, 2009 at 9:54 PM, dealmaker wrote:

    Hi,
    I am want to load data in mysql using a hadoop file similar to
    following:
    LOAD DATA INFILE 'test.txt' INTO TABLE test
    FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';

    But how do I load the hdfs file into the mysql comand above? Do I start
    the
    file name with hdfd://test.txt? I am using java.
    Thanks.

    --
    View this message in context:
    http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    --
    View this message in context:
    http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23625140.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
  • Marc Limotte at May 19, 2009 at 10:21 pm ⇧
    You might also try using something like Fuse-dfs (http://wiki.apache.org/hadoop/MountableHDFS) to "mount" the HDFS file system on the mysql machine. You could then use a standard unix path to specify the loadfile.

    Marc

    -----Original Message-----
    From: tim robertson
    Sent: Tuesday, May 19, 2009 1:03 PM
    To: core-user@hadoop.apache.org
    Subject: Re: Mysql Load Data Infile with Hadoop?

    So you are using a java program to execute a "load data infile"
    command on mysql through JDBC?
    If so I *think* you would have to copy it onto the mysql machine from
    HDFS first, or the machine running the command and then try a 'load
    data local infile'.

    Or pehaps use the
    http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html

    Cheers

    Tim
    On Tue, May 19, 2009 at 9:54 PM, dealmaker wrote:

    Hi,
    I am want to load data in mysql using a hadoop file similar to following:
    LOAD DATA INFILE 'test.txt' INTO TABLE test
    FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';

    But how do I load the hdfs file into the mysql comand above? Do I start the
    file name with hdfd://test.txt? I am using java.
    Thanks.

    --
    View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.
    PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
  • Sheldon Neuberger at May 19, 2009 at 9:55 pm ⇧
    You could copy the file to your local filesystem with something like

    `hadoop dfs -copyToLocal test.txt local_test.txt`
    On Tue, May 19, 2009 at 3:54 PM, dealmaker wrote:

    Hi,
    I am want to load data in mysql using a hadoop file similar to following:
    LOAD DATA INFILE 'test.txt' INTO TABLE test
    FIELDS TERMINATED BY ','  LINES STARTING BY 'xxx';

    But how do I load the hdfs file into the mysql comand above?  Do I start the
    file name with hdfd://test.txt?  I am using java.
    Thanks.

    --
    View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 19, '09 at 9:44p
activeMay 20, '09 at 12:20a
posts6
users5
websitehadoop.apache.org...
irc#hadoop

5 users in discussion

Dealmaker: 2 posts Alex Loddengaard: 1 post Marc Limotte: 1 post Sheldon Neuberger: 1 post Tim robertson: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase