Grokbase Groups Hive user August 2009
FAQ
Can I pass parameters to custom map/reduce scripts like this:

from try_hits
select transform ip_address, aid, uid
using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid

I've tried it, and it seems to work, but I just want to be sure whether this
is guaranteed to work under all circumstances. A large part of my current
project depends on this :-)

PS: Here's what I'm trying to do -- based on a profile ID (which is passed
as a parameter), the map/reduce script is going to look up a DB and fetch a
bunch of regular expressions from there. Based on these dynamically fetched
regular expressions it'll process the input.

Saurabh.

Search Discussions

  • Prasad Chakka at Aug 4, 2009 at 4:26 pm
    Quotes should work.

    Prasad


    ________________________________
    From: Saurabh Nanda <saurabhnanda@gmail.com>
    Reply-To: <hive-user@hadoop.apache.org>
    Date: Tue, 4 Aug 2009 05:43:34 -0700
    To: <hive-user@hadoop.apache.org>
    Subject: Passing parameters to custom map/reduce scripts

    Can I pass parameters to custom map/reduce scripts like this:

    from try_hits
    select transform ip_address, aid, uid
    using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid

    I've tried it, and it seems to work, but I just want to be sure whether this is guaranteed to work under all circumstances. A large part of my current project depends on this :-)

    PS: Here's what I'm trying to do -- based on a profile ID (which is passed as a parameter), the map/reduce script is going to look up a DB and fetch a bunch of regular expressions from there. Based on these dynamically fetched regular expressions it'll process the input.

    Saurabh.
  • Saurabh Nanda at Aug 4, 2009 at 5:29 pm
    Prasad, my query was more about the concept. Not the syntax. I hope
    this will still work when hive 0.4 is released. I'm basing a lot on
    it.
    On 8/4/09, Prasad Chakka wrote:
    Quotes should work.

    Prasad


    ________________________________
    From: Saurabh Nanda <saurabhnanda@gmail.com>
    Reply-To: <hive-user@hadoop.apache.org>
    Date: Tue, 4 Aug 2009 05:43:34 -0700
    To: <hive-user@hadoop.apache.org>
    Subject: Passing parameters to custom map/reduce scripts

    Can I pass parameters to custom map/reduce scripts like this:

    from try_hits
    select transform ip_address, aid, uid
    using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid

    I've tried it, and it seems to work, but I just want to be sure whether this
    is guaranteed to work under all circumstances. A large part of my current
    project depends on this :-)

    PS: Here's what I'm trying to do -- based on a profile ID (which is passed
    as a parameter), the map/reduce script is going to look up a DB and fetch a
    bunch of regular expressions from there. Based on these dynamically fetched
    regular expressions it'll process the input.

    Saurabh.
    --
    http://nandz.blogspot.com
    http://foodieforlife.blogspot.com

    --
    http://nandz.blogspot.com
    http://foodieforlife.blogspot.com
  • Prasad Chakka at Aug 4, 2009 at 6:21 pm
    Everything inside the quotes will be executed using a shell, similar to that of Hadoop Streaming. Unless Hadoop Streeaming changes, Hive would not. And it is highly unlikely that Hadoop Streaming changes since lot of people are dependent on it.

    Prasad


    ________________________________
    From: Saurabh Nanda <saurabhnanda@gmail.com>
    Reply-To: <hive-user@hadoop.apache.org>
    Date: Tue, 4 Aug 2009 10:28:34 -0700
    To: <hive-user@hadoop.apache.org>
    Subject: Re: Passing parameters to custom map/reduce scripts

    Prasad, my query was more about the concept. Not the syntax. I hope
    this will still work when hive 0.4 is released. I'm basing a lot on
    it.
    On 8/4/09, Prasad Chakka wrote:
    Quotes should work.

    Prasad


    ________________________________
    From: Saurabh Nanda <saurabhnanda@gmail.com>
    Reply-To: <hive-user@hadoop.apache.org>
    Date: Tue, 4 Aug 2009 05:43:34 -0700
    To: <hive-user@hadoop.apache.org>
    Subject: Passing parameters to custom map/reduce scripts

    Can I pass parameters to custom map/reduce scripts like this:

    from try_hits
    select transform ip_address, aid, uid
    using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid

    I've tried it, and it seems to work, but I just want to be sure whether this
    is guaranteed to work under all circumstances. A large part of my current
    project depends on this :-)

    PS: Here's what I'm trying to do -- based on a profile ID (which is passed
    as a parameter), the map/reduce script is going to look up a DB and fetch a
    bunch of regular expressions from there. Based on these dynamically fetched
    regular expressions it'll process the input.

    Saurabh.
    --
    http://nandz.blogspot.com
    http://foodieforlife.blogspot.com

    --
    http://nandz.blogspot.com
    http://foodieforlife.blogspot.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedAug 4, '09 at 12:43p
activeAug 4, '09 at 6:21p
posts4
users2
websitehive.apache.org

2 users in discussion

Prasad Chakka: 2 posts Saurabh Nanda: 2 posts

People

Translate

site design / logo © 2021 Grokbase