Grokbase Groups Pig dev August 2010
FAQ
I just noticed that even though Utf8StorageConverter implements the various
byte[] toBytes(Obj o) methods, they are not part of the LoadCaster interface
-- and therefore can't be relied on when using modular Casters, like I am
trying to do for the HBaseLoader.

Since we don't want to introduce backwards-incompatible changes, I propose
adding a ByteCaster interface that defines these methods, and extending
Utf8StorageConverter to implement them (without actually changing the
implementation at all).
That way StoreFuncs that need to convert to bytes can use pluggable
converters. Objections?

-D

Search Discussions

  • Dmitriy Ryaboy at Aug 22, 2010 at 8:41 am
    The current HBase patch on PIG-1205 (patch 7) includes this refactoring.
    Please take a look if you have concerns.

    Or just if you feel like reviewing the code... :)

    -D
    On Sat, Aug 21, 2010 at 5:22 PM, Dmitriy Ryaboy wrote:

    I just noticed that even though Utf8StorageConverter implements the various
    byte[] toBytes(Obj o) methods, they are not part of the LoadCaster interface
    -- and therefore can't be relied on when using modular Casters, like I am
    trying to do for the HBaseLoader.

    Since we don't want to introduce backwards-incompatible changes, I propose
    adding a ByteCaster interface that defines these methods, and extending
    Utf8StorageConverter to implement them (without actually changing the
    implementation at all).
    That way StoreFuncs that need to convert to bytes can use pluggable
    converters. Objections?

    -D
  • Alan Gates at Aug 24, 2010 at 4:19 pm
    This seems fine. Is the Pig engine at any point testing to see if the
    interface is implemented and if so calling toBytes, or is this totally
    for use inside the store functions themselves to serialize Pig data
    types?

    Alan.
    On Aug 22, 2010, at 1:40 AM, Dmitriy Ryaboy wrote:

    The current HBase patch on PIG-1205 (patch 7) includes this
    refactoring.
    Please take a look if you have concerns.

    Or just if you feel like reviewing the code... :)

    -D
    On Sat, Aug 21, 2010 at 5:22 PM, Dmitriy Ryaboy wrote:

    I just noticed that even though Utf8StorageConverter implements the
    various
    byte[] toBytes(Obj o) methods, they are not part of the LoadCaster
    interface
    -- and therefore can't be relied on when using modular Casters,
    like I am
    trying to do for the HBaseLoader.

    Since we don't want to introduce backwards-incompatible changes, I
    propose
    adding a ByteCaster interface that defines these methods, and
    extending
    Utf8StorageConverter to implement them (without actually changing the
    implementation at all).
    That way StoreFuncs that need to convert to bytes can use pluggable
    converters. Objections?

    -D
  • Alan Gates at Aug 24, 2010 at 5:02 pm
    One other comment. By making this part of an interface that extends
    LoadCaster you are assuming the implementing class is both a load and
    store function. It makes more sense to have a separate StoreCaster
    interface rather than extending LoadCaster.

    Alan.
    On Aug 24, 2010, at 9:18 AM, Alan Gates wrote:

    This seems fine. Is the Pig engine at any point testing to see if the
    interface is implemented and if so calling toBytes, or is this totally
    for use inside the store functions themselves to serialize Pig data
    types?

    Alan.
    On Aug 22, 2010, at 1:40 AM, Dmitriy Ryaboy wrote:

    The current HBase patch on PIG-1205 (patch 7) includes this
    refactoring.
    Please take a look if you have concerns.

    Or just if you feel like reviewing the code... :)

    -D

    On Sat, Aug 21, 2010 at 5:22 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    I just noticed that even though Utf8StorageConverter implements the
    various
    byte[] toBytes(Obj o) methods, they are not part of the LoadCaster
    interface
    -- and therefore can't be relied on when using modular Casters,
    like I am
    trying to do for the HBaseLoader.

    Since we don't want to introduce backwards-incompatible changes, I
    propose
    adding a ByteCaster interface that defines these methods, and
    extending
    Utf8StorageConverter to implement them (without actually changing
    the
    implementation at all).
    That way StoreFuncs that need to convert to bytes can use pluggable
    converters. Objections?

    -D
  • Dmitriy Ryaboy at Aug 24, 2010 at 8:23 pm
    As far as the toBytes methods -- I am not sure what they were originally
    for. They aren't actually called anywhere that I can find, except my new
    HBase stuff.
    You are right, I could make it two interfaces, but I consolidated them for
    simplicity of use/implementation. Now that I think about it, I can put all
    the methods into StoreCaster and just have a unioning interface for
    simplicity:

    @InterfaceAudience.Public
    @InterfaceStability.Evolving
    public interface LoadStoreCaster extends LoadCaster, StoreCaster {

    }

    Does that seem ok?

    -D
    On Tue, Aug 24, 2010 at 10:01 AM, Alan Gates wrote:

    One other comment. By making this part of an interface that extends
    LoadCaster you are assuming the implementing class is both a load and store
    function. It makes more sense to have a separate StoreCaster interface
    rather than extending LoadCaster.

    Alan.


    On Aug 24, 2010, at 9:18 AM, Alan Gates wrote:

    This seems fine. Is the Pig engine at any point testing to see if the
    interface is implemented and if so calling toBytes, or is this totally
    for use inside the store functions themselves to serialize Pig data
    types?

    Alan.

    On Aug 22, 2010, at 1:40 AM, Dmitriy Ryaboy wrote:

    The current HBase patch on PIG-1205 (patch 7) includes this
    refactoring.
    Please take a look if you have concerns.

    Or just if you feel like reviewing the code... :)

    -D

    On Sat, Aug 21, 2010 at 5:22 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:

    I just noticed that even though Utf8StorageConverter implements the
    various
    byte[] toBytes(Obj o) methods, they are not part of the LoadCaster
    interface
    -- and therefore can't be relied on when using modular Casters, like I
    am
    trying to do for the HBaseLoader.

    Since we don't want to introduce backwards-incompatible changes, I
    propose
    adding a ByteCaster interface that defines these methods, and
    extending
    Utf8StorageConverter to implement them (without actually changing the
    implementation at all).
    That way StoreFuncs that need to convert to bytes can use pluggable
    converters. Objections?

    -D
  • Alan Gates at Aug 24, 2010 at 10:17 pm

    On Aug 24, 2010, at 1:22 PM, Dmitriy Ryaboy wrote:

    As far as the toBytes methods -- I am not sure what they were
    originally
    for. They aren't actually called anywhere that I can find, except my
    new
    HBase stuff.
    You are right, I could make it two interfaces, but I consolidated
    them for
    simplicity of use/implementation. Now that I think about it, I can
    put all
    the methods into StoreCaster and just have a unioning interface for
    simplicity:

    @InterfaceAudience.Public
    @InterfaceStability.Evolving
    public interface LoadStoreCaster extends LoadCaster, StoreCaster {

    }

    Does that seem ok?
    Yeah, makes sense.

    Alan.
    -D
    On Tue, Aug 24, 2010 at 10:01 AM, Alan Gates wrote:

    One other comment. By making this part of an interface that extends
    LoadCaster you are assuming the implementing class is both a load
    and store
    function. It makes more sense to have a separate StoreCaster
    interface
    rather than extending LoadCaster.

    Alan.


    On Aug 24, 2010, at 9:18 AM, Alan Gates wrote:

    This seems fine. Is the Pig engine at any point testing to see if
    the
    interface is implemented and if so calling toBytes, or is this
    totally
    for use inside the store functions themselves to serialize Pig data
    types?

    Alan.

    On Aug 22, 2010, at 1:40 AM, Dmitriy Ryaboy wrote:

    The current HBase patch on PIG-1205 (patch 7) includes this
    refactoring.
    Please take a look if you have concerns.

    Or just if you feel like reviewing the code... :)

    -D

    On Sat, Aug 21, 2010 at 5:22 PM, Dmitriy Ryaboy
    <dvryaboy@gmail.com>
    wrote:

    I just noticed that even though Utf8StorageConverter implements the
    various
    byte[] toBytes(Obj o) methods, they are not part of the LoadCaster
    interface
    -- and therefore can't be relied on when using modular Casters,
    like I
    am
    trying to do for the HBaseLoader.

    Since we don't want to introduce backwards-incompatible changes, I
    propose
    adding a ByteCaster interface that defines these methods, and
    extending
    Utf8StorageConverter to implement them (without actually
    changing the
    implementation at all).
    That way StoreFuncs that need to convert to bytes can use
    pluggable
    converters. Objections?

    -D

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedAug 22, '10 at 12:23a
activeAug 24, '10 at 10:17p
posts6
users2
websitepig.apache.org

2 users in discussion

Alan Gates: 3 posts Dmitriy Ryaboy: 3 posts

People

Translate

site design / logo © 2022 Grokbase