FAQ
Hello,
We had text processing libraries, which developed by C/C++, and would like
to
use Hadoop to process large data set. It looks like not much talking about
this issue
(to call C/C++ library under Hadoop Map/Reduce Java codes)
Any comment(or examples) to share how to do that? Thanks for your help.

ChaoChun
--
View this message in context: http://www.nabble.com/How-to-integrate-C-C%2B%2B-library-in-Hadoop-Map-Reduce-tf4278023.html#a12176819
Sent from the Hadoop Users mailing list archive at Nabble.com.

Search Discussions

  • Arun C Murthy at Aug 16, 2007 at 8:18 am
    Hi,

    ChaoChun Liang wrote:
    Hello,
    We had text processing libraries, which developed by C/C++, and would like
    to
    use Hadoop to process large data set. It looks like not much talking about
    this issue
    (to call C/C++ library under Hadoop Map/Reduce Java codes)
    Any comment(or examples) to share how to do that? Thanks for your help.
    I believe the hadoop *pipes* is what you need... it's coming up in the
    0.14.0 release in next few days:

    http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/package.html

    Here is some context: http://issues.apache.org/jira/browse/HADOOP-234

    Arun
    ChaoChun
  • ChaoChun Liang at Aug 27, 2007 at 3:28 am

    Arun C Murthy wrote:

    I believe the hadoop *pipes* is what you need... it's coming up in the
    0.14.0 release in next few days:

    http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/pipes/package.html
    Why not use JNI? It looks like will be easy for the implementation.

    ChaoChun
    --
    View this message in context: http://www.nabble.com/How-to-integrate-C-C%2B%2B-library-in-Hadoop-Map-Reduce-tf4278023.html#a12341115
    Sent from the Hadoop Users mailing list archive at Nabble.com.
  • Owen O'Malley at Aug 27, 2007 at 11:22 pm

    On Aug 26, 2007, at 8:28 PM, ChaoChun Liang wrote:

    Why not use JNI? It looks like will be easy for the implementation.
    One of the desires was to minimize the impact on the C++ code. In
    particular, requiring a JVM to be linked in with the application is
    pretty heavy. Additionally, the JNI interfaces are really expensive
    to move large amounts of data across.

    -- Owen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 16, '07 at 7:43a
activeAug 27, '07 at 11:22p
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase