FAQ
Hi,



We have MapReduce program which writes data to mysql database using
DBOutputFormat.

Our program has one reducer.

I understand that all the inserts happen during the close() operation of
the reducer.

Is it gauranteed that this operation is atomic ? i.e; what happens if
the writes fail in the middle of the operation.

Does it mean that only partial number of rows get in to mysql database?



What does it take to make the write operation atomic ?



Any suggestions around our situation( alternative solutions ) are
welcome.



Thanks,

Giridhar.

Search Discussions

  • Marcos Ortiz at May 25, 2011 at 10:51 pm

    On 05/25/2011 04:27 PM, Giridhar Addepalli wrote:
    Hi,

    We have MapReduce program which writes data to mysql database using
    DBOutputFormat.

    Our program has one reducer.

    I understand that all the inserts happen during the close() operation
    of the reducer.

    Is it gauranteed that this operation is atomic ? i.e; what happens if
    the writes fail in the middle of the operation.

    Does it mean that only partial number of rows get in to mysql database?

    What does it take to make the write operation atomic ?

    Any suggestions around our situation( alternative solutions ) are welcome.

    Thanks,

    Giridhar.
    Sqoop is done for that kind of tasks.

    Definition
    ========
    Sqoop is an open-source tool that allows users to extract data from a
    relational database into Hadoop for further processing.
    This processing can be done with MapReduce programs or other
    higher-level tools such as Hive. When the final results of an analytic
    pipeline are available, Sqoop can export
    these results back to the database for consumption by other clients.

    Source code
    ==========
    http://github.com/cloudera/sqoop

    Regards

    --
    Marcos Luis Ortiz Valmaseda
    Software Engineer (Distributed Systems)
    http://uncubanitolinuxero.blogspot.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedMay 25, '11 at 8:58p
activeMay 25, '11 at 10:51p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase