FAQ
Hi all,

I was reading this:

http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs

It sounded like I can pass some configuration or context to the UDF but I
can't figure out how I would do that after I searched quite a bit on
internet and past discussion.

In my UDF, I can also do this:

UDFContext context = UDFContext.getUDFContext();
Properties properties =
context.getUDFProperties(this.getClass());

so if the context is set on the front end, supposedly, it will be in that
properties object. But how do I set it on the front end or whichever way to
pass it to UDF?

Thanks!
Dexin

Search Discussions

  • Daniel Dai at Nov 24, 2010 at 2:12 am
    The only hook in frontend for a UDF is outputSchema. You can put your
    property into UDFContext in outputSchema, and read back in exec.

    public String exec(Tuple input) throws IOException {
    UDFContext context = UDFContext.getUDFContext();
    String a =
    context.getUDFProperties(this.getClass()).getProperty("Hello");
    return a;
    }

    public Schema outputSchema(Schema input) {
    UDFContext context = UDFContext.getUDFContext();
    context.getUDFProperties(this.getClass()).setProperty("Hello",
    "World");
    return null;
    }

    The other option is to provide a system wide configuration in command
    line (-D), or pig.properties, which can be retrieved in UDF.exec using:
    UDFContext.getUDFContext().getJobConf().get("propertyname")

    Daniel

    Dexin Wang wrote:
    Hi all,

    I was reading this:

    http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs

    It sounded like I can pass some configuration or context to the UDF but I
    can't figure out how I would do that after I searched quite a bit on
    internet and past discussion.

    In my UDF, I can also do this:

    UDFContext context = UDFContext.getUDFContext();
    Properties properties =
    context.getUDFProperties(this.getClass());

    so if the context is set on the front end, supposedly, it will be in that
    properties object. But how do I set it on the front end or whichever way to
    pass it to UDF?

    Thanks!
    Dexin
  • Dexin Wang at Nov 29, 2010 at 8:47 pm
    Thanks Daniel!

    First option won't work for me since the property is not known to UDF
    itself, I need to pass it to UDF.

    Second option, I won't use pig.properties since the property passed to UDF
    are per pig script specific, not a global setting. How do I pass a -D option
    to pig script run (pig -f myscript.pig)?

    Thanks.
    On Tue, Nov 23, 2010 at 6:12 PM, Daniel Dai wrote:

    The only hook in frontend for a UDF is outputSchema. You can put your
    property into UDFContext in outputSchema, and read back in exec.

    public String exec(Tuple input) throws IOException {
    UDFContext context = UDFContext.getUDFContext();
    String a =
    context.getUDFProperties(this.getClass()).getProperty("Hello");
    return a;
    }

    public Schema outputSchema(Schema input) {
    UDFContext context = UDFContext.getUDFContext();
    context.getUDFProperties(this.getClass()).setProperty("Hello",
    "World");
    return null;
    }

    The other option is to provide a system wide configuration in command line
    (-D), or pig.properties, which can be retrieved in UDF.exec using:
    UDFContext.getUDFContext().getJobConf().get("propertyname")

    Daniel


    Dexin Wang wrote:
    Hi all,

    I was reading this:

    http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs

    It sounded like I can pass some configuration or context to the UDF but I
    can't figure out how I would do that after I searched quite a bit on
    internet and past discussion.

    In my UDF, I can also do this:

    UDFContext context = UDFContext.getUDFContext();
    Properties properties =
    context.getUDFProperties(this.getClass());

    so if the context is set on the front end, supposedly, it will be in that
    properties object. But how do I set it on the front end or whichever way
    to
    pass it to UDF?

    Thanks!
    Dexin
  • Daniel Dai at Nov 29, 2010 at 10:50 pm
    You can pass -D using the command line:

    java -Xmx512m -Dxxx=yyy -cp $HADOOP_CONF_DIR:pig.jar org.apache.pig.Main

    Daniel

    Dexin Wang wrote:
    Thanks Daniel!

    First option won't work for me since the property is not known to UDF
    itself, I need to pass it to UDF.

    Second option, I won't use pig.properties since the property passed to UDF
    are per pig script specific, not a global setting. How do I pass a -D option
    to pig script run (pig -f myscript.pig)?

    Thanks.

    On Tue, Nov 23, 2010 at 6:12 PM, Daniel Dai wrote:

    The only hook in frontend for a UDF is outputSchema. You can put your
    property into UDFContext in outputSchema, and read back in exec.

    public String exec(Tuple input) throws IOException {
    UDFContext context = UDFContext.getUDFContext();
    String a =
    context.getUDFProperties(this.getClass()).getProperty("Hello");
    return a;
    }

    public Schema outputSchema(Schema input) {
    UDFContext context = UDFContext.getUDFContext();
    context.getUDFProperties(this.getClass()).setProperty("Hello",
    "World");
    return null;
    }

    The other option is to provide a system wide configuration in command line
    (-D), or pig.properties, which can be retrieved in UDF.exec using:
    UDFContext.getUDFContext().getJobConf().get("propertyname")

    Daniel


    Dexin Wang wrote:

    Hi all,

    I was reading this:

    http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs

    It sounded like I can pass some configuration or context to the UDF but I
    can't figure out how I would do that after I searched quite a bit on
    internet and past discussion.

    In my UDF, I can also do this:

    UDFContext context = UDFContext.getUDFContext();
    Properties properties =
    context.getUDFProperties(this.getClass());

    so if the context is set on the front end, supposedly, it will be in that
    properties object. But how do I set it on the front end or whichever way
    to
    pass it to UDF?

    Thanks!
    Dexin

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedNov 24, '10 at 1:25a
activeNov 29, '10 at 10:50p
posts4
users2
websitepig.apache.org

2 users in discussion

Dexin Wang: 2 posts Daniel Dai: 2 posts

People

Translate

site design / logo © 2021 Grokbase