Hi everyone,
This is my first post in this list, as I am a newb with Hadoop.
I am looking in the web for some documentation and example on how to use DI
framework with Hadoop.

Basically I want to have the mapping / reducing logic outside the mapper /
reducer classes.

I did a small prototype to show POC ofr our project.
In the prototype I had classes with static methods that did the mapping /
reducing logic.
this is not my preferable way of course.

So,
Are there any recommendations for good DI in Hadoop documentation?

Thanks,

Eyal


Eyal Golan
egolan74@gmail.com

Visit: http://jvdrums.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P Save a tree. Please don't print this e-mail unless it's really necessary

Search Discussions

  • Mahadev Konar at Dec 27, 2011 at 12:04 am
    Eyal,
    Did you check out the spring data project?

    http://www.springsource.org/spring-data/hadoop

    That might offer some help.

    thanks
    mahadev
    On Mon, Dec 26, 2011 at 2:12 AM, Eyal Golan wrote:

    Hi everyone,
    This is my first post in this list, as I am a newb with Hadoop.
    I am looking in the web for some documentation and example on how to use
    DI framework with Hadoop.

    Basically I want to have the mapping / reducing logic outside the mapper /
    reducer classes.

    I did a small prototype to show POC ofr our project.
    In the prototype I had classes with static methods that did the mapping /
    reducing logic.
    this is not my preferable way of course.

    So,
    Are there any recommendations for good DI in Hadoop documentation?

    Thanks,

    Eyal


    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary
  • Eyal Golan at Dec 28, 2011 at 12:01 am
    Thanks.
    I did see it before.
    Will check it a little bit more.


    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary


    On Tue, Dec 27, 2011 at 2:04 AM, Mahadev Konar wrote:

    Eyal,
    Did you check out the spring data project?

    http://www.springsource.org/spring-data/hadoop

    That might offer some help.

    thanks
    mahadev

    On Mon, Dec 26, 2011 at 2:12 AM, Eyal Golan wrote:

    Hi everyone,
    This is my first post in this list, as I am a newb with Hadoop.
    I am looking in the web for some documentation and example on how to use
    DI framework with Hadoop.

    Basically I want to have the mapping / reducing logic outside the mapper
    / reducer classes.

    I did a small prototype to show POC ofr our project.
    In the prototype I had classes with static methods that did the mapping /
    reducing logic.
    this is not my preferable way of course.

    So,
    Are there any recommendations for good DI in Hadoop documentation?

    Thanks,

    Eyal


    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really
    necessary
  • Owen O'Malley at Dec 30, 2011 at 5:29 pm
    Using guice in your MapReduce task is easy. Just create the injector in the
    configure/setup method and include the jars in the distributed cache.

    -- Owen

    On Dec 27, 2011, at 4:01 PM, Eyal Golan wrote:

    Thanks.
    I did see it before.
    Will check it a little bit more.


    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary


    On Tue, Dec 27, 2011 at 2:04 AM, Mahadev Konar wrote:

    Eyal,
    Did you check out the spring data project?

    http://www.springsource.org/spring-data/hadoop

    That might offer some help.

    thanks
    mahadev

    On Mon, Dec 26, 2011 at 2:12 AM, Eyal Golan wrote:

    Hi everyone,
    This is my first post in this list, as I am a newb with Hadoop.
    I am looking in the web for some documentation and example on how to use
    DI framework with Hadoop.

    Basically I want to have the mapping / reducing logic outside the mapper
    / reducer classes.

    I did a small prototype to show POC ofr our project.
    In the prototype I had classes with static methods that did the mapping /
    reducing logic.
    this is not my preferable way of course.

    So,
    Are there any recommendations for good DI in Hadoop documentation?

    Thanks,

    Eyal


    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really
    necessary
  • Costin Leau at Dec 30, 2011 at 9:53 am
    Hi,

    My name is Costin Leau and I'm the lead of Spring Hadoop (SHDP) project.

    SHDP provides DI support allowing basic POJOs to be used as
    mapper/reducers. This feature is currently developed on a dedicated
    branch [1] and we plan to merge it in master in the near future.

    In addition to the pojo/DI support, you might be interested in the SHDP
    integration with HBase/Hive/Pig - see [2]

    Everything is open-source, under ASL 2, available on github. We plan to
    release the milestone in Jan/Feb 2012.

    Feedback and ideas are more then welcome!

    [1] https://github.com/SpringSource/spring-hadoop/tree/mr-pojo
    [2] http://static.springsource.org/spring-hadoop/docs/current/reference/

    Hope this helps,
    --
    Costin
  • Eyal Golan at Dec 30, 2011 at 10:57 am
    Thank you Costin.

    I have worked with Spring before but never set up a project.
    So I guess I'll need to do a little bit more ramp-up to use Spring-Hadoop.

    Another issue that concerns me is the Hadoop cluster management.
    I don't have access for the cluster, besides adding my job and use some
    kind of CLI to schedule / trigger the jobs.

    I am wondering whether I can create a Hadoop-Spring application without
    managing the cluster.


    Thanks,

    Eyal

    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary


    On Fri, Dec 30, 2011 at 11:53 AM, Costin Leau wrote:

    Hi,

    My name is Costin Leau and I'm the lead of Spring Hadoop (SHDP) project.

    SHDP provides DI support allowing basic POJOs to be used as
    mapper/reducers. This feature is currently developed on a dedicated
    branch [1] and we plan to merge it in master in the near future.

    In addition to the pojo/DI support, you might be interested in the SHDP
    integration with HBase/Hive/Pig - see [2]

    Everything is open-source, under ASL 2, available on github. We plan to
    release the milestone in Jan/Feb 2012.

    Feedback and ideas are more then welcome!

    [1] https://github.com/SpringSource/spring-hadoop/tree/mr-pojo
    [2] http://static.springsource.org/spring-hadoop/docs/current/reference/

    Hope this helps,
    --
    Costin
  • Costin Leau at Dec 30, 2011 at 11:20 am
    Hi,

    This tends to be a problem with any library that you use inside Hadoop.

    One of the easiest way to distribute your dependencies (w/o managing) is
    to put them under lib/ folder in your job and Hadoop will take care of
    the rest - if there are any better suggestions, I'd be interested in
    hearing about them.

    Cheers,
    On 12/30/2011 12:56 PM, Eyal Golan wrote:
    Thank you Costin.

    I have worked with Spring before but never set up a project.
    So I guess I'll need to do a little bit more ramp-up to use Spring-Hadoop.

    Another issue that concerns me is the Hadoop cluster management.
    I don't have access for the cluster, besides adding my job and use some
    kind of CLI to schedule / trigger the jobs.

    I am wondering whether I can create a Hadoop-Spring application without
    managing the cluster.


    Thanks,

    Eyal

    Eyal Golan
    egolan74@gmail.com
    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary



    On Fri, Dec 30, 2011 at 11:53 AM, Costin Leau wrote:

    Hi,

    My name is Costin Leau and I'm the lead of Spring Hadoop (SHDP) project.

    SHDP provides DI support allowing basic POJOs to be used as
    mapper/reducers. This feature is currently developed on a dedicated
    branch [1] and we plan to merge it in master in the near future.

    In addition to the pojo/DI support, you might be interested in the SHDP
    integration with HBase/Hive/Pig - see [2]

    Everything is open-source, under ASL 2, available on github. We plan to
    release the milestone in Jan/Feb 2012.

    Feedback and ideas are more then welcome!

    [1] https://github.com/SpringSource/spring-hadoop/tree/mr-pojo
    [2] http://static.springsource.org/spring-hadoop/docs/current/reference/

    Hope this helps,
    --
    Costin

    --
    Costin

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 26, '11 at 10:13a
activeDec 30, '11 at 5:29p
posts7
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase