Grokbase Groups Pig user January 2009
FAQ
Hi,

This may have already been asked, but I couldn't find anything in old mails
... I did find an old bug report PIG-66 about this but it got closed with no
pointer to what the outcome was.

My question:
Is there any way to get a handle on the HDFS from inside a Pig UDF (in
particular, a StoreFunc)?
(Alternatively, if I can get the hadoop JobConf that would allow me to get
the HDFS by calling FileSystem.get(conf).)

My use case is:
I'm building a StoreFunc that creates a Lucene index, following the rubric
from the hadoop.contrib.index code, in which you first have Lucene create
index files in the local FS, and then copy them to the HDFS.

Thanks!

-Chris

--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research

Search Discussions

  • Craig Macdonald at Jan 9, 2009 at 6:38 pm
    Would it make sense if Pig automatically detected UDFs that implements
    hadoop.conf.Configurable, and automatically passed these the JobConf?

    Craig

    Chris Olston wrote:
    Hi,

    This may have already been asked, but I couldn't find anything in old mails
    ... I did find an old bug report PIG-66 about this but it got closed with no
    pointer to what the outcome was.

    My question:
    Is there any way to get a handle on the HDFS from inside a Pig UDF (in
    particular, a StoreFunc)?
    (Alternatively, if I can get the hadoop JobConf that would allow me to get
    the HDFS by calling FileSystem.get(conf).)

    My use case is:
    I'm building a StoreFunc that creates a Lucene index, following the rubric
    from the hadoop.contrib.index code, in which you first have Lucene create
    index files in the local FS, and then copy them to the HDFS.

    Thanks!

    -Chris

    --
    Christopher Olston, Ph.D.
    Sr. Research Scientist
    Yahoo! Research



  • Ted Dunning at Jan 9, 2009 at 6:42 pm
    Sounds right to me (as a non-implementor).
    On Fri, Jan 9, 2009 at 10:37 AM, Craig Macdonald wrote:

    Would it make sense if Pig automatically detected UDFs that implements
    hadoop.conf.Configurable, and automatically passed these the JobConf?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedJan 9, '09 at 6:02p
activeJan 9, '09 at 6:42p
posts3
users3
websitepig.apache.org

People

Translate

site design / logo © 2022 Grokbase