Grokbase Groups Pig user April 2011
FAQ
wondering whether this is a bug or originally designed for. When I
register my python udf file like this:

Register 'a/b/mypyudfs.py' using jython as mypyudfs;

I got an error saying "could not initialize: a/b/mypyudfs.py". looks
like the original file path is sent to backend without any conversion.
It works when registered like this:

Register 'mypyudfs.py' using jython as mypyudfs;

Regards,
Shawn

Search Discussions

  • Richard Ding at Apr 14, 2011 at 10:35 pm
    This should work (see PIG-1653 for a related issue). Can you post the whole stack trace?

    Thanks
    -- Richard




    On 4/14/11 12:59 PM, "Xiaomeng Wan" wrote:

    wondering whether this is a bug or originally designed for. When I
    register my python udf file like this:

    Register 'a/b/mypyudfs.py' using jython as mypyudfs;

    I got an error saying "could not initialize: a/b/mypyudfs.py". looks
    like the original file path is sent to backend without any conversion.
    It works when registered like this:

    Register 'mypyudfs.py' using jython as mypyudfs;

    Regards,
    Shawn
  • Xiaomeng Wan at Apr 14, 2011 at 10:48 pm
    java.io.IOException: Deserialization error: could not instantiate
    'org.apache.pig.scripting.jython.JythonFunction' with arguments
    '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
    at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.setup(PigMapBase.java:151)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:251)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.Child.main(Child.java:245)
    Caused by: java.lang.RuntimeException: could not instantiate
    'org.apache.pig.scripting.jython.JythonFunction' with arguments
    '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
    at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:502)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.instantiateFunc(POUserFunc.java:109)
    at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.readObject(POUserFunc.java:451)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at java.util.ArrayList.readObject(ArrayList.java:593)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at java.util.HashMap.readObject(HashMap.java:1030)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at java.util.ArrayList.readObject(ArrayList.java:593)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at java.util.ArrayList.readObject(ArrayList.java:593)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at java.util.HashMap.readObject(HashMap.java:1030)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
    at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
    ... 9 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:470)
    ... 87 more
    Caused by: java.lang.IllegalStateException: Could not initialize:
    /home/shawn/TESS/code/mypyudfs.py
    at org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:86)
    ... 92 more
    2011-04-14 11:35:50,402 INFO org.apache.hadoop.mapred.Task: Runnning
    cleanup for the task

    and I am using pig-0.8.3. Thanks
    Shawn
    On Thu, Apr 14, 2011 at 4:35 PM, Richard Ding wrote:
    This should work (see PIG-1653 for a related issue). Can you post the whole
    stack trace?

    Thanks
    -- Richard




    On 4/14/11 12:59 PM, "Xiaomeng Wan" wrote:

    wondering whether this is a bug or originally designed for. When I
    register my python udf  file like this:

    Register 'a/b/mypyudfs.py' using jython as mypyudfs;

    I got an error saying "could not initialize: a/b/mypyudfs.py". looks
    like the original file path is sent to backend without any conversion.
    It works when registered like this:

    Register 'mypyudfs.py' using jython as mypyudfs;

    Regards,
    Shawn

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedApr 14, '11 at 7:59p
activeApr 14, '11 at 10:48p
posts3
users2
websitepig.apache.org

2 users in discussion

Xiaomeng Wan: 2 posts Richard Ding: 1 post

People

Translate

site design / logo © 2021 Grokbase