Grokbase Groups Pig user May 2011
FAQ
Hi,

I have a LoadFunc that loads data using a complex schema. I don't want to have to specify the schema every time. LoadFunc used to have a method "determineSchema". The current docs reference this:
http://wiki.apache.org/pig/UDFManual#line-595
However, the current source does not have this method:
http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup
Why was this removed? What replaces this functionality?

Thanks,
-Nate

Search Discussions

  • Daniel Dai at May 20, 2011 at 11:31 pm
    It is changed to LoadMetadata.getSchema() starting 0.7.

    Daniel
    On 05/20/2011 02:20 PM, Sweet, Nate wrote:
    Hi,

    I have a LoadFunc that loads data using a complex schema. I don't want to have to specify the schema every time. LoadFunc used to have a method "determineSchema". The current docs reference this:
    http://wiki.apache.org/pig/UDFManual#line-595
    However, the current source does not have this method:
    http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup
    Why was this removed? What replaces this functionality?

    Thanks,
    -Nate
  • Sweet, Nate at May 23, 2011 at 7:39 pm
    Thanks. I guess this means the docs have been wrong for two versions? Uhg.

    Also, ResourceFieldSchema getSchema says only a tuple can have a schema, which is wrong because it seems a bag can have a schema just fine. Apparently map cannot have a schema, but I'm just guessing at this point.

    I must say I am very frustrated with the general lack of (and incorrect) documentation for Pig. I understand the project is evolving rapidly, but IMO documentation should not suffer.

    -Nate


    -----Original Message-----
    From: Daniel Dai
    Sent: Friday, May 20, 2011 4:31 PM
    To: user@pig.apache.org
    Cc: Sweet, Nate
    Subject: Re: specifying the schema with a LoadFunc

    It is changed to LoadMetadata.getSchema() starting 0.7.

    Daniel
    On 05/20/2011 02:20 PM, Sweet, Nate wrote:
    Hi,

    I have a LoadFunc that loads data using a complex schema. I don't want to have to specify the schema every time. LoadFunc used to have a method "determineSchema". The current docs reference this:
    http://wiki.apache.org/pig/UDFManual#line-595
    However, the current source does not have this method:
    http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup
    Why was this removed? What replaces this functionality?

    Thanks,
    -Nate
  • Daniel Dai at May 23, 2011 at 7:49 pm
    I think the document is updated:
    http://pig.apache.org/docs/r0.8.1/udf.html. Which document are you
    looking at?

    Also there is a Pig 0.6->0.7 migration guide:
    http://wiki.apache.org/pig/LoadStoreMigrationGuide

    Javadoc for ResourceSchema.getSchema() is wrong, I will change it.

    Daniel
    On 05/23/2011 12:38 PM, Sweet, Nate wrote:
    Thanks. I guess this means the docs have been wrong for two versions? Uhg.

    Also, ResourceFieldSchema getSchema says only a tuple can have a schema, which is wrong because it seems a bag can have a schema just fine. Apparently map cannot have a schema, but I'm just guessing at this point.

    I must say I am very frustrated with the general lack of (and incorrect) documentation for Pig. I understand the project is evolving rapidly, but IMO documentation should not suffer.

    -Nate


    -----Original Message-----
    From: Daniel Dai
    Sent: Friday, May 20, 2011 4:31 PM
    To: user@pig.apache.org
    Cc: Sweet, Nate
    Subject: Re: specifying the schema with a LoadFunc

    It is changed to LoadMetadata.getSchema() starting 0.7.

    Daniel
    On 05/20/2011 02:20 PM, Sweet, Nate wrote:
    Hi,

    I have a LoadFunc that loads data using a complex schema. I don't want to have to specify the schema every time. LoadFunc used to have a method "determineSchema". The current docs reference this:
    http://wiki.apache.org/pig/UDFManual#line-595
    However, the current source does not have this method:
    http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup
    Why was this removed? What replaces this functionality?

    Thanks,
    -Nate
  • Dmitriy Ryaboy at May 23, 2011 at 7:49 pm
    Nate the docs are here (under "documentation" on the main pig site).
    http://pig.apache.org/docs/r0.8.1/
    You are looking specifically for
    http://pig.apache.org/docs/r0.8.1/udf.html which explains the details
    of writing loadfuncs
    .

    Bags only have one schema: bags contain tuples. The tuples a bag
    contains have schemas.
    Until version 0.9, the only schema maps supported was key:chararray,
    value:bytearray. Version 9 added support for typed values (keys are
    still only Strings). I am not sure that there is a way to specify the
    types for map values from a loader.

    D
    On Mon, May 23, 2011 at 12:38 PM, Sweet, Nate wrote:
    Thanks. I guess this means the docs have been wrong for two versions? Uhg.

    Also, ResourceFieldSchema getSchema says only a tuple can have a schema, which is wrong because it seems a bag can have a schema just fine. Apparently map cannot have a schema, but I'm just guessing at this point.

    I must say I am very frustrated with the general lack of (and incorrect) documentation for Pig. I understand the project is evolving rapidly, but IMO documentation should not suffer.

    -Nate


    -----Original Message-----
    From: Daniel Dai
    Sent: Friday, May 20, 2011 4:31 PM
    To: user@pig.apache.org
    Cc: Sweet, Nate
    Subject: Re: specifying the schema with a LoadFunc

    It is changed to LoadMetadata.getSchema() starting 0.7.

    Daniel
    On 05/20/2011 02:20 PM, Sweet, Nate wrote:
    Hi,

    I have a LoadFunc that loads data using a complex schema. I don't want to have to specify the schema every time. LoadFunc used to have a method "determineSchema". The current docs reference this:
    http://wiki.apache.org/pig/UDFManual#line-595
    However, the current source does not have this method:
    http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup
    Why was this removed? What replaces this functionality?

    Thanks,
    -Nate

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 20, '11 at 9:21p
activeMay 23, '11 at 7:49p
posts5
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase