Grokbase Groups Pig user October 2012
FAQ
Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird issue with Pig.. Everytime I try to run a LOAD command, it dies with the following exception:

ERROR 2998: Unhandled internal error. name

java.lang.NoSuchFieldError: name
at org.apache.pig.parser.QueryParserStringStream.(QueryParserDriver.java:194)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

The same script/command worked fine with CDH3U3.. Did any APIs change? Or is this a bug?
Regards,
Dhaval

Search Discussions

  • Cheolsoo Park at Oct 9, 2012 at 10:57 pm
    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.

    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah wrote:

    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Dhaval Shah at Oct 9, 2012 at 11:02 pm
    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.

    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah wrote:

    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Cheolsoo Park at Oct 9, 2012 at 11:15 pm
    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah wrote:

    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.


    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Cheolsoo Park at Oct 9, 2012 at 11:31 pm
    Please note that there was an internal change in the Pig query parser from
    0.8 to 0.9, but this change should be visible to the end user:
    https://issues.apache.org/jira/browse/PIG-1618

    Given your call stack, my best guess is that the old version of pig jar is
    present in classpath and causes trouble. Can you verify that PIG_HOME
    points to where the CDH4.0.0 pig is installed, and the old pig is not
    present in classpath?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah wrote:

    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.


    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Cheolsoo Park at Oct 9, 2012 at 11:31 pm
    Typo: should NOT be visible.
    On Tue, Oct 9, 2012 at 4:31 PM, Cheolsoo Park wrote:

    Please note that there was an internal change in the Pig query parser from
    0.8 to 0.9, but this change should be visible to the end user:
    https://issues.apache.org/jira/browse/PIG-1618

    Given your call stack, my best guess is that the old version of pig jar is
    present in classpath and causes trouble. Can you verify that PIG_HOME
    points to where the CDH4.0.0 pig is installed, and the old pig is not
    present in classpath?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah wrote:

    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.


    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Russell Jurney at Oct 9, 2012 at 11:34 pm
    I've often wondered - as we see a lot of these legacy pig issues in
    CDH on this list - is it hard to upgrade Pig on CDH by downloading the
    latest stable release of pig, unpacking the .tgz and running it? Is
    upgrading Pig on CDH as simple as wget, like on Apache Hadoop and
    others, or is it somehow more complex? Pig is after all a client-side
    tool.

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah wrote:

    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.


    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Cheolsoo Park at Oct 9, 2012 at 11:50 pm
    Hi Russell,

    I am sorry if this was a spam to other Pig users. I will try to redirect
    CDH-specific questions to cdh-user if that's preferred. Please let me know.

    Regarding your question, I guess that the main complexity with upgrading
    from CDH3 to CDH4 come from that CDH4 includes both MR1 and MR2. We provide
    RPMs so that many details can be automatically taken care of, but it can
    sometimes cause subtle problems if the instructions on the Cloudera web
    page are not followed depending on environment.

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:34 PM, Russell Jurney wrote:

    I've often wondered - as we see a lot of these legacy pig issues in
    CDH on this list - is it hard to upgrade Pig on CDH by downloading the
    latest stable release of pig, unpacking the .tgz and running it? Is
    upgrading Pig on CDH as simple as wget, like on Apache Hadoop and
    others, or is it somehow more complex? Pig is after all a client-side
    tool.

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.

    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility
    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <
    prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with
    the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change?
    Or
    is this a bug?
    Regards,
    Dhaval
  • Russell Jurney at Oct 9, 2012 at 11:57 pm
    Cheolsoo, these questions aren't spam. Pig lives in different forests
    :) I've just always wondered why more CDH pig users don't upgrade via
    download, so I finally asked. Your answer makes sense.

    Everyone loves BBQ, pork for everyone! :D

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:50 PM, Cheolsoo Park wrote:

    Hi Russell,

    I am sorry if this was a spam to other Pig users. I will try to redirect
    CDH-specific questions to cdh-user if that's preferred. Please let me know.

    Regarding your question, I guess that the main complexity with upgrading
    from CDH3 to CDH4 come from that CDH4 includes both MR1 and MR2. We provide
    RPMs so that many details can be automatically taken care of, but it can
    sometimes cause subtle problems if the instructions on the Cloudera web
    page are not followed depending on environment.

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:34 PM, Russell Jurney wrote:

    I've often wondered - as we see a lot of these legacy pig issues in
    CDH on this list - is it hard to upgrade Pig on CDH by downloading the
    latest stable release of pig, unpacking the .tgz and running it? Is
    upgrading Pig on CDH as simple as wget, like on Apache Hadoop and
    others, or is it somehow more complex? Pig is after all a client-side
    tool.

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.

    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility
    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <
    prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with
    the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change?
    Or
    is this a bug?
    Regards,
    Dhaval
  • Dhaval Shah at Oct 10, 2012 at 2:58 pm
    Alright I was eventually able to get the issue resolved.. For everyone's benefit, the root issue was that the pig libraries needed to be present in the HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh.. I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those libraries in the shell and then start pig but that did not help either.. Setting that explicitly in hadoop_env.sh did the trick..

    As I see there are 2 issues here:
    1. Somehow the pig executable seems broken.. As in setting HADOOP_CLASSPATH on the shell and then running pig should have the same effect as setting it in hadoop_env.sh.. Also having a * in the PIG_CLASSPATH/HADOOP_CLASSPATH (when set in the shell) makes the pig executable set HADOOP_CLASSPATH to empty before it actually runs the hadoop jar command inside the pig script.. In CDH3, I never needed to set the classpath to include pig libraries.. The pig executable did that for me..(and I am never overriding HADOOP_CLASSPATH.. I only append to it)
    2. The error message is not super helpful.. If libraries are missing, the pig shell/grunt should not open at all.. However, in my case, it did start up and then the error message was in no ways intuitive or pointing to the root issue..

    Also, I don't think this is a legacy pig issue as we tried the newer version 0.10 and it came up with the same issue.. A second issue that did pop up after putting the pig libraries in the classpath was with mapreduce framework being selected as classic and it turned out that the MRv1 libraries needed to be explicitly specified in the HADOOP_CLASSPATH (without that entry Sqoop, Java MR jobs work fine but Pig did not for whatever reason)

    Anyways thanks everyone for the help

    Regards,
    Dhaval


    ----- Original Message -----
    From: Russell Jurney <russell.jurney@gmail.com>
    To: "user@pig.apache.org" <user@pig.apache.org>
    Cc:
    Sent: Tuesday, 9 October 2012 7:34 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    I've often wondered - as we see a lot of these legacy pig issues in
    CDH on this list - is it hard to upgrade Pig on CDH by downloading the
    latest stable release of pig, unpacking the .tgz and running it? Is
    upgrading Pig on CDH as simple as wget, like on Apache Hadoop and
    others, or is it somehow more complex? Pig is after all a client-side
    tool.

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo
    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah wrote:

    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.


    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility

    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change? Or
    is this a bug?
    Regards,
    Dhaval
  • Cheolsoo Park at Oct 10, 2012 at 8:56 pm
    Hi Dhaval,

    Thank you very much for sharing your analysis! Your explanation definitely
    provides more insights. :-)

    If you don't mind, I'd like to clarify a couple of things. I am just trying
    to see if there's something to be fixed in terms of Pig packaging:
    the root issue was that the pig libraries needed to be present in the
    HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh..

    I am not sure if missing dependency libraries is the root cause. Given the
    following error in your call stack, I believe that you have a different
    version of antlr in classpath.

    java.lang.NoSuchFieldError: name
    at org.apache.pig.parser.QueryParserStringStream.<init>
    (QueryParserStringStream.java:32)

    If you look at QueryParserStringStream.java, it extends ANTLRStringStream,
    and Pig 0.9 is compiled against antlr 3.4. Now if antlr is missing, you
    should get ClassNotFoundError not NoSuchFiledError. Only possible reason
    that I can think of is that there is a different version of antlr in
    classpath at run-time.

    In fact, there was a similar discussion on this mailing list a while ago
    (Please note that he is reporting the same call stack as yours):
    http://mail-archives.apache.org/mod_mbox/pig-user/201206.mbox/%3CCAOkn+GjPCgY1i+-zfNWF9cKz+m8feRj=6Rk1shSS293oqECbMg@mail.gmail.com%3E

    Antlr is a common tool, so it's possible that it's installed by other tools
    such as Hive.
    I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those
    libraries in the shell and then start pig but that did not help either..

    Assuming that a wrong version of antlr is present in classpath, this makes
    perfect sense because PIG_CLASSPATH adds libraries to the end of CLASPATH:

    # add user-specified CLASSPATH
    if [ "$PIG_CLASSPATH" != "" ]; then
    CLASSPATH=${CLASSPATH}:${PIG_CLASSPATH}
    fi

    That is, as long as the wrong antlr is present before the correct one, the
    wrong one be picked up at run-time.
    2. The error message is not super helpful.. If libraries are missing,
    the pig shell/grunt should not open at all.. However, in my case, it did
    start up and then the error message was in no ways intuitive or pointing to
    the root issue..

    I agree with you that the error message is not very intuitive here. But
    errors caused by dependency libraries can be only caught when Pig makes
    calls to methods of those libraries at run-time. Furthermore, if there are
    two different versions of the same library in classpath, the root cause can
    be more subtle.

    The challenge is that a Hadoop distribution such as CDH bundles many
    sub-projects, and they often depend on different versions of the same
    libraries. We do our best to harmonize all the versions of dependency
    libraries across the platform, but I admit that it is not always perfect.

    Thanks,
    Cheolsoo
    On Wed, Oct 10, 2012 at 7:57 AM, Dhaval Shah wrote:

    Alright I was eventually able to get the issue resolved.. For everyone's
    benefit, the root issue was that the pig libraries needed to be present in
    the HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh..
    I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those libraries
    in the shell and then start pig but that did not help either.. Setting that
    explicitly in hadoop_env.sh did the trick..

    As I see there are 2 issues here:
    1. Somehow the pig executable seems broken.. As in setting
    HADOOP_CLASSPATH on the shell and then running pig should have the same
    effect as setting it in hadoop_env.sh.. Also having a * in the
    PIG_CLASSPATH/HADOOP_CLASSPATH (when set in the shell) makes the pig
    executable set HADOOP_CLASSPATH to empty before it actually runs the hadoop
    jar command inside the pig script.. In CDH3, I never needed to set the
    classpath to include pig libraries.. The pig executable did that for
    me..(and I am never overriding HADOOP_CLASSPATH.. I only append to it)
    2. The error message is not super helpful.. If libraries are missing, the
    pig shell/grunt should not open at all.. However, in my case, it did start
    up and then the error message was in no ways intuitive or pointing to the
    root issue..

    Also, I don't think this is a legacy pig issue as we tried the newer
    version 0.10 and it came up with the same issue.. A second issue that did
    pop up after putting the pig libraries in the classpath was with mapreduce
    framework being selected as classic and it turned out that the MRv1
    libraries needed to be explicitly specified in the HADOOP_CLASSPATH
    (without that entry Sqoop, Java MR jobs work fine but Pig did not for
    whatever reason)

    Anyways thanks everyone for the help

    Regards,
    Dhaval


    ----- Original Message -----
    From: Russell Jurney <russell.jurney@gmail.com>
    To: "user@pig.apache.org" <user@pig.apache.org>
    Cc:
    Sent: Tuesday, 9 October 2012 7:34 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    I've often wondered - as we see a lot of these legacy pig issues in
    CDH on this list - is it hard to upgrade Pig on CDH by downloading the
    latest stable release of pig, unpacking the .tgz and running it? Is
    upgrading Pig on CDH as simple as wget, like on Apache Hadoop and
    others, or is it somehow more complex? Pig is after all a client-side
    tool.

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.

    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility
    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <
    prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with
    the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change?
    Or
    is this a bug?
    Regards,
    Dhaval
  • Dhaval Shah at Oct 11, 2012 at 12:20 am
    Reply inline!

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Wednesday, 10 October 2012 4:55 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    Thank you very much for sharing your analysis! Your explanation definitely
    provides more insights. :-)

    If you don't mind, I'd like to clarify a couple of things. I am just trying
    to see if there's something to be fixed in terms of Pig packaging:
    the root issue was that the pig libraries needed to be present in the
    HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh..

    I am not sure if missing dependency libraries is the root cause. Given the
    following error in your call stack, I believe that you have a different
    version of antlr in classpath.
    ---Dhaval -> Yes I did have Antlr 2.7.7 and 3.0.1 in the classpath from Mahout 0.6.. However, I still do have it in my classpath (and it appears earlier in the classpath than Pig) and it still works for me.. Not sure what the actual issue is but something to think about.. Also, I saw that pig jars have antlr bundled.. So technically those should take precedence anyways unless its being called in a different way..

    java.lang.NoSuchFieldError: name
    at org.apache.pig.parser.QueryParserStringStream.<init>
    (QueryParserStringStream.java:32)

    If you look at QueryParserStringStream.java, it extends ANTLRStringStream,
    and Pig 0.9 is compiled against antlr 3.4. Now if antlr is missing, you
    should get ClassNotFoundError not NoSuchFiledError. Only possible reason
    that I can think of is that there is a different version of antlr in
    classpath at run-time.

    In fact, there was a similar discussion on this mailing list a while ago
    (Please note that he is reporting the same call stack as yours):
    http://mail-archives.apache.org/mod_mbox/pig-user/201206.mbox/%3CCAOkn+GjPCgY1i+-zfNWF9cKz+m8feRj=6Rk1shSS293oqECbMg@mail.gmail.com%3E

    Antlr is a common tool, so it's possible that it's installed by other tools
    such as Hive.
    I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those
    libraries in the shell and then start pig but that did not help either..

    Assuming that a wrong version of antlr is present in classpath, this makes
    perfect sense because PIG_CLASSPATH adds libraries to the end of CLASPATH:
    ---Dhaval --> I think PIG_CLASSPATH libraries should be added to the front to take precedence since we are calling pig and expressing that we want to use pig and not anything else

    # add user-specified CLASSPATH
    if [ "$PIG_CLASSPATH" != "" ]; then
    CLASSPATH=${CLASSPATH}:${PIG_CLASSPATH}
    fi

    That is, as long as the wrong antlr is present before the correct one, the
    wrong one be picked up at run-time.
    2. The error message is not super helpful.. If libraries are missing,
    the pig shell/grunt should not open at all.. However, in my case, it did
    start up and then the error message was in no ways intuitive or pointing to
    the root issue..

    I agree with you that the error message is not very intuitive here. But
    errors caused by dependency libraries can be only caught when Pig makes
    calls to methods of those libraries at run-time. Furthermore, if there are
    two different versions of the same library in classpath, the root cause can
    be more subtle.

    The challenge is that a Hadoop distribution such as CDH bundles many
    sub-projects, and they often depend on different versions of the same
    libraries. We do our best to harmonize all the versions of dependency
    libraries across the platform, but I admit that it is not always perfect.

    ---Dhaval --> True I understand that issue and I guess the only thing we could do is update the pig executable here to exclude the known conflicts which in my opinion is a reasonable thing to do

    Thanks,
    Cheolsoo
    On Wed, Oct 10, 2012 at 7:57 AM, Dhaval Shah wrote:

    Alright I was eventually able to get the issue resolved.. For everyone's
    benefit, the root issue was that the pig libraries needed to be present in
    the HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh..
    I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those libraries
    in the shell and then start pig but that did not help either.. Setting that
    explicitly in hadoop_env.sh did the trick..

    As I see there are 2 issues here:
    1. Somehow the pig executable seems broken.. As in setting
    HADOOP_CLASSPATH on the shell and then running pig should have the same
    effect as setting it in hadoop_env.sh.. Also having a * in the
    PIG_CLASSPATH/HADOOP_CLASSPATH (when set in the shell) makes the pig
    executable set HADOOP_CLASSPATH to empty before it actually runs the hadoop
    jar command inside the pig script.. In CDH3, I never needed to set the
    classpath to include pig libraries.. The pig executable did that for
    me..(and I am never overriding HADOOP_CLASSPATH.. I only append to it)
    2. The error message is not super helpful.. If libraries are missing, the
    pig shell/grunt should not open at all.. However, in my case, it did start
    up and then the error message was in no ways intuitive or pointing to the
    root issue..

    Also, I don't think this is a legacy pig issue as we tried the newer
    version 0.10 and it came up with the same issue.. A second issue that did
    pop up after putting the pig libraries in the classpath was with mapreduce
    framework being selected as classic and it turned out that the MRv1
    libraries needed to be explicitly specified in the HADOOP_CLASSPATH
    (without that entry Sqoop, Java MR jobs work fine but Pig did not for
    whatever reason)

    Anyways thanks everyone for the help

    Regards,
    Dhaval


    ----- Original Message -----
    From: Russell Jurney <russell.jurney@gmail.com>
    To: "user@pig.apache.org" <user@pig.apache.org>
    Cc:
    Sent: Tuesday, 9 October 2012 7:34 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    I've often wondered - as we see a lot of these legacy pig issues in
    CDH on this list - is it hard to upgrade Pig on CDH by downloading the
    latest stable release of pig, unpacking the .tgz and running it? Is
    upgrading Pig on CDH as simple as wget, like on Apache Hadoop and
    others, or is it somehow more complex? Pig is after all a client-side
    tool.

    Russell Jurney http://datasyndrome.com
    On Oct 9, 2012, at 4:15 PM, Cheolsoo Park wrote:

    Hi Dhaval,

    That certainly works for me.

    How did you upgrade CDH? Did you install it via RPMs? Did you completely
    uninstall CDH3u3 before installing CDH4?

    It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you
    please provide steps that you took to upgrade CDH?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
    wrote:
    Thanks for getting back Cheolsoo..

    All I am trying to do is run the pig shell/grunt and do this:
    p = LOAD 'file_name';

    It comes back with the exception mentioned below..

    Regards,
    Dhaval


    ----- Original Message -----
    From: Cheolsoo Park <cheolsoo@cloudera.com>
    To: user@pig.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
    Cc:
    Sent: Tuesday, 9 October 2012 6:57 PM
    Subject: Re: Error with Pig (CDH4.0.0)

    Hi Dhaval,

    CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were
    some incompatibilities introduced between two version.

    https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility
    To pin down the exact cause, I'd like to reproduce your error. Would
    mind providing an example script that generates the exception?

    Thanks,
    Cheolsoo

    On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah <
    prince_mithibai@yahoo.co.in
    wrote:
    Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird
    issue with Pig.. Everytime I try to run a LOAD command, it dies with
    the
    following exception:

    ERROR 2998: Unhandled internal error. name

    java.lang.NoSuchFieldError: name
    at
    org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32)
    at
    org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194)
    at
    org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162)
    at
    org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
    at
    org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
    at
    org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
    at org.apache.pig.Main.run(Main.java:495)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    The same script/command worked fine with CDH3U3.. Did any APIs change?
    Or
    is this a bug?
    Regards,
    Dhaval

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedOct 9, '12 at 10:31p
activeOct 11, '12 at 12:20a
posts12
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase