Hi all;
I'm trying to integrate a couple of custom InputFormats into
Cascalog queries. These InputFormat definitions use the
org.apache.hadoop.mapreduce API. Here's one of the definitions:

http://j.mp/tNeiY5

While trying to create a custom Scheme in Cascading, I realized that
it uses the org.apache.hadoop.mapred API. Those two APIs appear to be
incompatible.

Is there any way to utilize these custom InputFormats with Cascalog? I
started down the path of writing an adaptor class but quickly got in
way over my head with the Hadoop API.

Does anyone have tips or tricks for this situation? Thanks much,
Brad

Search Discussions

  • Chris K Wensel at Nov 20, 2011 at 10:02 pm
    Unfortunately you cannot use the Hadoop experimental APIs and the stable APIs simultaneously in a single job. (note all deprecations have been removed from Hadoop apis in later stable maint releases).

    Cascading only uses the stable ones at this time. So you will need to port your InputFormats to use the stable packages and api calls.

    As Hadoop 0.23+ releases stabilize for general production use, we will look at migrating any required api changes then.

    chris
    On Nov 20, 2011, at 1:49 PM, Brad Chapman wrote:


    Hi all;
    I'm trying to integrate a couple of custom InputFormats into
    Cascalog queries. These InputFormat definitions use the
    org.apache.hadoop.mapreduce API. Here's one of the definitions:

    http://j.mp/tNeiY5

    While trying to create a custom Scheme in Cascading, I realized that
    it uses the org.apache.hadoop.mapred API. Those two APIs appear to be
    incompatible.

    Is there any way to utilize these custom InputFormats with Cascalog? I
    started down the path of writing an adaptor class but quickly got in
    way over my head with the Hadoop API.

    Does anyone have tips or tricks for this situation? Thanks much,
    Brad
    --
    Chris K Wensel
    chris@concurrentinc.com
    http://www.concurrentinc.com

    -- Concurrent, Inc. offers mentoring, support for Cascading

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedNov 20, '11 at 9:49p
activeNov 20, '11 at 10:02p
posts2
users2
websiteclojure.org
irc#clojure

2 users in discussion

Chris K Wensel: 1 post Brad Chapman: 1 post

People

Translate

site design / logo © 2022 Grokbase