FAQ
Hello,

I seem to have broken my Pig install, and I don't know where to look.

If I use directly the script (grunt) everything works ok, but every time I
try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
org.apache.pig.Main script1-hadoop.pig'

I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Undefined parameter : PIGDIR

Obviously I have the PIGDIR var set:
echo $PIGDIR
/home/alex/hadoop/pig
This is something that I did, as I have used 0.5 and 0.6 and a patched
version of 0.6 in parallel, but I can't figure out where to look. Any
version I try to start now, gives the same error.

any help would be greatly appreciated!

alex

Search Discussions

  • Dmitriy Ryaboy at Feb 12, 2010 at 5:38 pm
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to look.

    If I use directly the script (grunt) everything works ok, but every time I
    try to run a pig script:  'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a patched
    version of 0.6 in parallel, but I can't figure out where to look. Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Alex Parvulescu at Feb 16, 2010 at 8:39 am
    Hello,

    sorry for the delay, but I wanted to build from the source again just to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '

    That's it. My guess is that Pig doesn't even start. Do you think I need to
    change something in the properties file? I'm not sure anymore and I don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch and trunk
    at the same time (because they take a very long time) and both test scripts
    failed. I've switched to running them one after the other and they are fine.
    Do you think that is ok?

    thanks for your time,
    alex
    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy wrote:

    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to look.

    If I use directly the script (grunt) everything works ok, but every time I
    try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a patched
    version of 0.6 in parallel, but I can't figure out where to look. Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Dmitriy Ryaboy at Feb 16, 2010 at 8:59 am
    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment because you
    are juggling too many different versions of pig (granted, I have 3 or
    4 in various stages of development on my laptop most of the time, and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never tried
    running multiple ant tests, either. There's a short "sanity" version
    of tests, ant test-commit, that runs in under 10 minutes. You might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source again just to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '

    That's it. My guess is that Pig doesn't even start. Do you think I need to
    change something in the properties file? I'm not sure anymore and I don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch and trunk
    at the same time (because they take a very long time) and both test scripts
    failed. I've switched to running them one after the other and they are fine.
    Do you think that is ok?

    thanks for your time,
    alex
    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy wrote:

    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc).  Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to look.

    If I use directly the script (grunt) everything works ok, but every time I
    try to run a pig script:  'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a patched
    version of 0.6 in parallel, but I can't figure out where to look. Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Alex Parvulescu at Feb 16, 2010 at 11:38 am
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default pig script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not work(Exception
    in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).

    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by command line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is defined port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only INFO and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file bigger than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've made
    # (almost) no changes to the lines above to make merging in from Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production clusters. If
    you
    # want to run pig with a static cluster you will need to remove everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at
    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)
    ================================================================================


    The script: 'A = load '/home/alex/hadoop/test/t.csv' using PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
    anything as scripts. If I go with manual processing in grunt (line by line)
    it's all good on any version.

    thanks,
    alex

    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy wrote:

    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment because you
    are juggling too many different versions of pig (granted, I have 3 or
    4 in various stages of development on my laptop most of the time, and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never tried
    running multiple ant tests, either. There's a short "sanity" version
    of tests, ant test-commit, that runs in under 10 minutes. You might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source again just to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4: int); '
    That's it. My guess is that Pig doesn't even start. Do you think I need to
    change something in the properties file? I'm not sure anymore and I don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch and trunk
    at the same time (because they take a very long time) and both test scripts
    failed. I've switched to running them one after the other and they are fine.
    Do you think that is ok?

    thanks for your time,
    alex
    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy wrote:

    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to look.

    If I use directly the script (grunt) everything works ok, but every
    time
    I
    try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a patched
    version of 0.6 in parallel, but I can't figure out where to look. Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Dmitriy Ryaboy at Feb 16, 2010 at 5:09 pm
    Hm, nothing jumps out. Definitely the error you are getting indicates that
    somehow the preprocessor is trying to substitute a variable called PIGDIR in
    your script. Which is odd. Does the same thing happen if you try bin/pig -f
    test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
    and sending along the output. Why are you using pig-core instead of pig.jar
    that ant should be generating for you when you build? Where did you get it,
    how did you build it, and what's the output of cksum on it?
    The fact that now your 0.5 is broken too makes me think that maybe your
    symlinks are messed up. Surely a change to one jar shouldn't be affecting a
    totally unrelated jar in a directory the first jar doesn't know about.

    Sorry about the barrage of questions, I am just a bit dumbfounded about how
    this could even begin to start happening. Any preprocessor experts around?

    -D


    On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
    wrote:
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default pig
    script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
    work(Exception
    in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).

    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by command line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is defined port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only INFO and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file bigger than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've made
    # (almost) no changes to the lines above to make merging in from Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production clusters. If
    you
    # want to run pig with a static cluster you will need to remove everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at

    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at

    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at

    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)

    ================================================================================


    The script: 'A = load '/home/alex/hadoop/test/t.csv' using PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
    anything as scripts. If I go with manual processing in grunt (line by line)
    it's all good on any version.

    thanks,
    alex

    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy wrote:

    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment because you
    are juggling too many different versions of pig (granted, I have 3 or
    4 in various stages of development on my laptop most of the time, and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never tried
    running multiple ant tests, either. There's a short "sanity" version
    of tests, ant test-commit, that runs in under 10 minutes. You might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source again just
    to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4:
    int);
    '
    That's it. My guess is that Pig doesn't even start. Do you think I need to
    change something in the properties file? I'm not sure anymore and I
    don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch and trunk
    at the same time (because they take a very long time) and both test scripts
    failed. I've switched to running them one after the other and they are fine.
    Do you think that is ok?

    thanks for your time,
    alex

    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to
    look.
    If I use directly the script (grunt) everything works ok, but every
    time
    I
    try to run a pig script: 'java -cp $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a
    patched
    version of 0.6 in parallel, but I can't figure out where to look.
    Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Alex Parvulescu at Feb 17, 2010 at 10:53 am
    Hello

    This is getting tiresome :)

    Today, I tried with a fresh download of pig 0.5 and I get the exact same
    error. Something somewhere just broke down, and it took my machine with it.
    At this point I have no way to run any kind of pig version.
    Needless to say, this is a big problem for me :)

    Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
    I even dropped the sym links.

    Back to our debugging session:
    Fetching external item into 'pig-branch-0.6-take2/test/bin'
    A pig-branch-0.6-take2/test/bin/test-patch.sh
    Checked out external at revision 910901.

    Checked out revision 910900.
    cd pig-branch-0.6-take2/
    ant
    BUILD SUCCESSFUL
    Total time: 26 seconds
    cd bin/
    next is to check if all vars are there
    printenv | grep PIG
    PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
    PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
    printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf
    pig
    Exception in thread "main" java.lang.NoClassDefFoundError:
    jline/ConsoleReaderInputStream
    Caused by: java.lang.ClassNotFoundException: jline.ConsoleReaderInputStream
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    Could not find the main class: org.apache.pig.Main. Program will exit.

    I've sent some email to the list about this problem before. A quick fix is
    to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
    cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
    pig
    2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
    2010-02-17 11:48:53,175 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
    to hadoop file system at: file:///
    grunt>

    Now at least it will start.

    Getting to the initial issue:

    the '-f' option:
    pig -f /home/alex/hadoop/test/test_lite.pig
    2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
    2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log

    the '- secretDebugCmd' option

    pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
    dry run:
    /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
    -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
    -Dpig.log.file=pig.log
    -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
    -Dpig.root.logger=INFO,console,DRFA -classpath
    /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
    org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig

    I did not change the scripts, and I get the same error.

    This is really killing me :)

    thanks,
    alex


    On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy wrote:

    Hm, nothing jumps out. Definitely the error you are getting indicates that
    somehow the preprocessor is trying to substitute a variable called PIGDIR
    in
    your script. Which is odd. Does the same thing happen if you try bin/pig -f
    test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
    and sending along the output. Why are you using pig-core instead of pig.jar
    that ant should be generating for you when you build? Where did you get it,
    how did you build it, and what's the output of cksum on it?
    The fact that now your 0.5 is broken too makes me think that maybe your
    symlinks are messed up. Surely a change to one jar shouldn't be affecting a
    totally unrelated jar in a directory the first jar doesn't know about.

    Sorry about the barrage of questions, I am just a bit dumbfounded about how
    this could even begin to start happening. Any preprocessor experts around?

    -D


    On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
    wrote:
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default pig
    script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
    work(Exception
    in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).

    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by command line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is defined port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only INFO and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file bigger than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've made
    # (almost) no changes to the lines above to make merging in from Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production clusters. If
    you
    # want to run pig with a static cluster you will need to remove
    everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at

    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at

    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at

    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)

    ================================================================================

    The script: 'A = load '/home/alex/hadoop/test/t.csv' using
    PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I can't start
    anything as scripts. If I go with manual processing in grunt (line by line)
    it's all good on any version.

    thanks,
    alex


    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment because you
    are juggling too many different versions of pig (granted, I have 3 or
    4 in various stages of development on my laptop most of the time, and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never tried
    running multiple ant tests, either. There's a short "sanity" version
    of tests, ant test-commit, that runs in under 10 minutes. You might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source again just
    to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t') as (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4:
    int);
    '
    That's it. My guess is that Pig doesn't even start. Do you think I
    need
    to
    change something in the properties file? I'm not sure anymore and I
    don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch and trunk
    at the same time (because they take a very long time) and both test scripts
    failed. I've switched to running them one after the other and they
    are
    fine.
    Do you think that is ok?

    thanks for your time,
    alex

    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to
    look.
    If I use directly the script (grunt) everything works ok, but
    every
    time
    I
    try to run a pig script: 'java -cp
    $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a
    patched
    version of 0.6 in parallel, but I can't figure out where to look.
    Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Dmitriy Ryaboy at Feb 17, 2010 at 9:52 pm
    Probably unrelated, but I always run from the root directory, not from bin/,
    when working with an svn version. Also, never had dev vs core issues. Can
    you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
    pig.jar (not core, not dev) on $PIG_CLASSPATH?

    How about avoiding the pig script and just running

    java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
    <<yourscript>> ?

    I grepped through all the code, including the codegenned stuff, and none of
    it contains the string PIGDIR (except bin/pig of course).

    At this point, assuming none of the above helps (which I have little hope it
    will), the best I can say is set up your eclipse environment and step
    through pig's main with the debugger.

    -D

    On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
    wrote:
    Hello

    This is getting tiresome :)

    Today, I tried with a fresh download of pig 0.5 and I get the exact same
    error. Something somewhere just broke down, and it took my machine with it.
    At this point I have no way to run any kind of pig version.
    Needless to say, this is a big problem for me :)

    Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
    I even dropped the sym links.

    Back to our debugging session:
    svn co
    http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
    A pig-branch-0.6-take2/test/bin/test-patch.sh
    Checked out external at revision 910901.

    Checked out revision 910900.
    cd pig-branch-0.6-take2/
    ant
    BUILD SUCCESSFUL
    Total time: 26 seconds
    cd bin/
    next is to check if all vars are there
    printenv | grep PIG
    PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
    PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
    printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf
    pig
    Exception in thread "main" java.lang.NoClassDefFoundError:
    jline/ConsoleReaderInputStream
    Caused by: java.lang.ClassNotFoundException: jline.ConsoleReaderInputStream
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    Could not find the main class: org.apache.pig.Main. Program will exit.

    I've sent some email to the list about this problem before. A quick fix is
    to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
    cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
    pig
    2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
    2010-02-17 11:48:53,175 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
    to hadoop file system at: file:///
    grunt>

    Now at least it will start.

    Getting to the initial issue:

    the '-f' option:
    pig -f /home/alex/hadoop/test/test_lite.pig
    2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
    2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log

    the '- secretDebugCmd' option

    pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
    dry run:
    /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
    -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
    -Dpig.log.file=pig.log
    -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
    -Dpig.root.logger=INFO,console,DRFA -classpath

    /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
    org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig

    I did not change the scripts, and I get the same error.

    This is really killing me :)

    thanks,
    alex


    On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy wrote:

    Hm, nothing jumps out. Definitely the error you are getting indicates that
    somehow the preprocessor is trying to substitute a variable called PIGDIR
    in
    your script. Which is odd. Does the same thing happen if you try bin/pig -f
    test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
    and sending along the output. Why are you using pig-core instead of pig.jar
    that ant should be generating for you when you build? Where did you get it,
    how did you build it, and what's the output of cksum on it?
    The fact that now your 0.5 is broken too makes me think that maybe your
    symlinks are messed up. Surely a change to one jar shouldn't be affecting a
    totally unrelated jar in a directory the first jar doesn't know about.

    Sorry about the barrage of questions, I am just a bit dumbfounded about how
    this could even begin to start happening. Any preprocessor experts around?
    -D


    On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
    wrote:
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of
    the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default pig
    script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
    work(Exception
    in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/Main).

    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by command line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is defined
    port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only INFO
    and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file bigger
    than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've made
    # (almost) no changes to the lines above to make merging in from Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production clusters. If
    you
    # want to run pig with a static cluster you will need to remove
    everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging
    error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at
    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)
    ================================================================================

    The script: 'A = load '/home/alex/hadoop/test/t.csv' using
    PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I can't
    start
    anything as scripts. If I go with manual processing in grunt (line by line)
    it's all good on any version.

    thanks,
    alex


    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment because
    you
    are juggling too many different versions of pig (granted, I have 3 or
    4 in various stages of development on my laptop most of the time, and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never
    tried
    running multiple ant tests, either. There's a short "sanity" version
    of tests, ant test-commit, that runs in under 10 minutes. You might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source again
    just
    to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using PigStorage('\t')
    as
    (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int, r4:
    int);
    '
    That's it. My guess is that Pig doesn't even start. Do you think I
    need
    to
    change something in the properties file? I'm not sure anymore and I
    don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch
    and
    trunk
    at the same time (because they take a very long time) and both test scripts
    failed. I've switched to running them one after the other and they
    are
    fine.
    Do you think that is ok?

    thanks for your time,
    alex

    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
    dvryaboy@gmail.com>
    wrote:
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where to
    look.
    If I use directly the script (grunt) everything works ok, but
    every
    time
    I
    try to run a pig script: 'java -cp
    $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a
    patched
    version of 0.6 in parallel, but I can't figure out where to
    look.
    Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Ashutosh Chauhan at Feb 18, 2010 at 1:51 am
    Alex,

    I just downloaded the latest Pig 0.6 Release Candidate from
    http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz and
    followed the instructions in "Trying the Release" section as outlined in
    http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt and
    without taking any extra step anywhere in between I am able to get Pig
    running. You may want to do the same thing. If it works that would be great
    else we will be sure that its some environment problem in your setup and we
    can take it from there.

    Ashutosh
    On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy wrote:

    Probably unrelated, but I always run from the root directory, not from
    bin/,
    when working with an svn version. Also, never had dev vs core issues. Can
    you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
    pig.jar (not core, not dev) on $PIG_CLASSPATH?

    How about avoiding the pig script and just running

    java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
    <<yourscript>> ?

    I grepped through all the code, including the codegenned stuff, and none of
    it contains the string PIGDIR (except bin/pig of course).

    At this point, assuming none of the above helps (which I have little hope
    it
    will), the best I can say is set up your eclipse environment and step
    through pig's main with the debugger.

    -D

    On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
    wrote:
    Hello

    This is getting tiresome :)

    Today, I tried with a fresh download of pig 0.5 and I get the exact same
    error. Something somewhere just broke down, and it took my machine with it.
    At this point I have no way to run any kind of pig version.
    Needless to say, this is a big problem for me :)

    Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
    I even dropped the sym links.

    Back to our debugging session:
    svn co
    http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
    A pig-branch-0.6-take2/test/bin/test-patch.sh
    Checked out external at revision 910901.

    Checked out revision 910900.
    cd pig-branch-0.6-take2/
    ant
    BUILD SUCCESSFUL
    Total time: 26 seconds
    cd bin/
    next is to check if all vars are there
    printenv | grep PIG
    PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
    PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
    printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf
    pig
    Exception in thread "main" java.lang.NoClassDefFoundError:
    jline/ConsoleReaderInputStream
    Caused by: java.lang.ClassNotFoundException:
    jline.ConsoleReaderInputStream
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    Could not find the main class: org.apache.pig.Main. Program will exit.

    I've sent some email to the list about this problem before. A quick fix is
    to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
    cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
    pig
    2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
    2010-02-17 11:48:53,175 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to hadoop file system at: file:///
    grunt>

    Now at least it will start.

    Getting to the initial issue:

    the '-f' option:
    pig -f /home/alex/hadoop/test/test_lite.pig
    2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
    2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log

    the '- secretDebugCmd' option

    pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
    dry run:
    /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
    -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
    -Dpig.log.file=pig.log
    -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
    -Dpig.root.logger=INFO,console,DRFA -classpath

    /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
    org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig

    I did not change the scripts, and I get the same error.

    This is really killing me :)

    thanks,
    alex



    On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    Hm, nothing jumps out. Definitely the error you are getting indicates that
    somehow the preprocessor is trying to substitute a variable called
    PIGDIR
    in
    your script. Which is odd. Does the same thing happen if you try
    bin/pig
    -f
    test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
    and sending along the output. Why are you using pig-core instead of pig.jar
    that ant should be generating for you when you build? Where did you get it,
    how did you build it, and what's the output of cksum on it?
    The fact that now your 0.5 is broken too makes me think that maybe your
    symlinks are messed up. Surely a change to one jar shouldn't be
    affecting
    a
    totally unrelated jar in a directory the first jar doesn't know about.

    Sorry about the barrage of questions, I am just a bit dumbfounded about how
    this could even begin to start happening. Any preprocessor experts around?
    -D


    On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
    wrote:
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build of
    the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default pig
    script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
    work(Exception
    in thread "main" java.lang.NoClassDefFoundError:
    org/apache/pig/Main).
    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by command
    line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is defined
    port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only INFO
    and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file bigger
    than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've made
    # (almost) no changes to the lines above to make merging in from
    Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production
    clusters.
    If
    you
    # want to run pig with a static cluster you will need to remove
    everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR
    org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging
    error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at
    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)
    ================================================================================

    The script: 'A = load '/home/alex/hadoop/test/t.csv' using
    PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I can't
    start
    anything as scripts. If I go with manual processing in grunt (line by line)
    it's all good on any version.

    thanks,
    alex


    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment because
    you
    are juggling too many different versions of pig (granted, I have 3
    or
    4 in various stages of development on my laptop most of the time,
    and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never
    tried
    running multiple ant tests, either. There's a short "sanity"
    version
    of tests, ant test-commit, that runs in under 10 minutes. You might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what
    your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source again
    just
    to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using
    PigStorage('\t')
    as
    (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int,
    r4:
    int);
    '
    That's it. My guess is that Pig doesn't even start. Do you think
    I
    need
    to
    change something in the properties file? I'm not sure anymore and
    I
    don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both 0.6-branch
    and
    trunk
    at the same time (because they take a very long time) and both
    test
    scripts
    failed. I've switched to running them one after the other and
    they
    are
    fine.
    Do you think that is ok?

    thanks for your time,
    alex

    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
    dvryaboy@gmail.com>
    wrote:
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor
    can't
    substitute Pig variables (the stuff you specify with -param and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where
    to
    look.
    If I use directly the script (grunt) everything works ok, but
    every
    time
    I
    try to run a pig script: 'java -cp
    $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main -
    ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and a
    patched
    version of 0.6 in parallel, but I can't figure out where to
    look.
    Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Alex Parvulescu at Feb 24, 2010 at 2:16 pm
    Hello and sorry for the absence

    I went through a hard drive crash, and a 2 day recovery process.

    I have incredible news... Pig is working again!!!!!!!!!!

    it felt good to finally say it out loud

    I've found the problem. Check this out:

    --------- This is my test script -----------

    -- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main
    test_lite.pig

    A = load 't.csv' using PigStorage('\t') as (id: long);
    store A into 'output/pig' using PigStorage('\t');

    --------------------

    I just saved the run instruction in this script to I don't forget it.
    Notice the '$PIGDIR' and '$HADOOPDIR'

    It apprears this is the source if the evil
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
    replacing this with
    -- $whatever
    will give you: ERROR 2999: Unexpected internal error. Undefined parameter :
    whatever

    Now that it's working again, I'll be back with more issues in no time :)

    Thanks a million to everybody, especially Dmitriy!

    alex

    On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan wrote:

    Alex,

    I just downloaded the latest Pig 0.6 Release Candidate from
    http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz>and
    followed the instructions in "Trying the Release" section as outlined in
    http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt>and
    without taking any extra step anywhere in between I am able to get Pig
    running. You may want to do the same thing. If it works that would be great
    else we will be sure that its some environment problem in your setup and we
    can take it from there.

    Ashutosh
    On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy wrote:

    Probably unrelated, but I always run from the root directory, not from
    bin/,
    when working with an svn version. Also, never had dev vs core issues. Can
    you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
    pig.jar (not core, not dev) on $PIG_CLASSPATH?

    How about avoiding the pig script and just running

    java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
    <<yourscript>> ?

    I grepped through all the code, including the codegenned stuff, and none of
    it contains the string PIGDIR (except bin/pig of course).

    At this point, assuming none of the above helps (which I have little hope
    it
    will), the best I can say is set up your eclipse environment and step
    through pig's main with the debugger.

    -D

    On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
    wrote:
    Hello

    This is getting tiresome :)

    Today, I tried with a fresh download of pig 0.5 and I get the exact
    same
    error. Something somewhere just broke down, and it took my machine with it.
    At this point I have no way to run any kind of pig version.
    Needless to say, this is a big problem for me :)

    Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
    I even dropped the sym links.

    Back to our debugging session:
    svn co
    http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
    A pig-branch-0.6-take2/test/bin/test-patch.sh
    Checked out external at revision 910901.

    Checked out revision 910900.
    cd pig-branch-0.6-take2/
    ant
    BUILD SUCCESSFUL
    Total time: 26 seconds
    cd bin/
    next is to check if all vars are there
    printenv | grep PIG
    PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
    PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
    printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf
    pig
    Exception in thread "main" java.lang.NoClassDefFoundError:
    jline/ConsoleReaderInputStream
    Caused by: java.lang.ClassNotFoundException:
    jline.ConsoleReaderInputStream
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    Could not find the main class: org.apache.pig.Main. Program will exit.

    I've sent some email to the list about this problem before. A quick fix is
    to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
    cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
    pig
    2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging
    error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
    2010-02-17 11:48:53,175 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to hadoop file system at: file:///
    grunt>

    Now at least it will start.

    Getting to the initial issue:

    the '-f' option:
    pig -f /home/alex/hadoop/test/test_lite.pig
    2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging
    error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
    2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log

    the '- secretDebugCmd' option

    pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
    dry run:
    /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
    -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
    -Dpig.log.file=pig.log
    -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
    -Dpig.root.logger=INFO,console,DRFA -classpath
    /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
    org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig

    I did not change the scripts, and I get the same error.

    This is really killing me :)

    thanks,
    alex



    On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    Hm, nothing jumps out. Definitely the error you are getting indicates that
    somehow the preprocessor is trying to substitute a variable called
    PIGDIR
    in
    your script. Which is odd. Does the same thing happen if you try
    bin/pig
    -f
    test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
    and sending along the output. Why are you using pig-core instead of pig.jar
    that ant should be generating for you when you build? Where did you
    get
    it,
    how did you build it, and what's the output of cksum on it?
    The fact that now your 0.5 is broken too makes me think that maybe
    your
    symlinks are messed up. Surely a change to one jar shouldn't be
    affecting
    a
    totally unrelated jar in a directory the first jar doesn't know
    about.
    Sorry about the barrage of questions, I am just a bit dumbfounded
    about
    how
    this could even begin to start happening. Any preprocessor experts around?
    -D


    On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
    wrote:
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and build
    of
    the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default
    pig
    script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
    work(Exception
    in thread "main" java.lang.NoClassDefFoundError:
    org/apache/pig/Main).
    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by command
    line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is defined
    port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only
    INFO
    and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file
    bigger
    than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've
    made
    # (almost) no changes to the lines above to make merging in from
    Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production
    clusters.
    If
    you
    # want to run pig with a static cluster you will need to remove
    everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR
    org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - Logging
    error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR

    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at
    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)
    ================================================================================

    The script: 'A = load '/home/alex/hadoop/test/t.csv' using
    PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I can't
    start
    anything as scripts. If I go with manual processing in grunt (line
    by
    line)
    it's all good on any version.

    thanks,
    alex


    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <
    dvryaboy@gmail.com>
    wrote:
    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment
    because
    you
    are juggling too many different versions of pig (granted, I have
    3
    or
    4 in various stages of development on my laptop most of the time,
    and
    haven't had your problems. But then neither of them has a hacked
    bin/pig , with the exception of the cloudera one...). I've never
    tried
    running multiple ant tests, either. There's a short "sanity"
    version
    of tests, ant test-commit, that runs in under 10 minutes. You
    might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what
    your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source
    again
    just
    to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using
    PigStorage('\t')
    as
    (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3: int,
    r4:
    int);
    '
    That's it. My guess is that Pig doesn't even start. Do you
    think
    I
    need
    to
    change something in the properties file? I'm not sure anymore
    and
    I
    don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both
    0.6-branch
    and
    trunk
    at the same time (because they take a very long time) and both
    test
    scripts
    failed. I've switched to running them one after the other and
    they
    are
    fine.
    Do you think that is ok?

    thanks for your time,
    alex

    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
    dvryaboy@gmail.com>
    wrote:
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor
    can't
    substitute Pig variables (the stuff you specify with -param
    and
    %default, etc). Do you have $PIGDIR in your script somewhere?

    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know where
    to
    look.
    If I use directly the script (grunt) everything works ok,
    but
    every
    time
    I
    try to run a pig script: 'java -cp
    $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main -
    ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6 and
    a
    patched
    version of 0.6 in parallel, but I can't figure out where to
    look.
    Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex
  • Dmitriy Ryaboy at Feb 24, 2010 at 8:10 pm
    Still not sure how that's happening, but glad it's working for you now :-).

    -D

    On Wed, Feb 24, 2010 at 6:15 AM, Alex Parvulescu
    wrote:
    Hello and sorry for the absence

    I went through a hard drive crash, and a 2 day recovery process.

    I have incredible news... Pig is working again!!!!!!!!!!

    it felt good to finally say it out loud

    I've found the problem. Check this out:

    --------- This is my test script -----------

    -- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main
    test_lite.pig

    A = load 't.csv' using PigStorage('\t') as (id: long);
    store A into 'output/pig' using PigStorage('\t');

    --------------------

    I just saved the run instruction in this script to I don't forget it.
    Notice the '$PIGDIR' and '$HADOOPDIR'

    It apprears this is the source if the evil
    ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR
    replacing this with
    -- $whatever
    will give you: ERROR 2999: Unexpected internal error. Undefined parameter :
    whatever

    Now that it's working again, I'll be back with more issues in no time :)

    Thanks a million to everybody, especially Dmitriy!

    alex


    On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan <
    ashutosh.chauhan@gmail.com> wrote:
    Alex,

    I just downloaded the latest Pig 0.6 Release Candidate from
    http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz>
    <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz
    and
    followed the instructions in "Trying the Release" section as outlined in
    http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt>
    <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt
    and
    without taking any extra step anywhere in between I am able to get Pig
    running. You may want to do the same thing. If it works that would be great
    else we will be sure that its some environment problem in your setup and we
    can take it from there.

    Ashutosh
    On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy wrote:

    Probably unrelated, but I always run from the root directory, not from
    bin/,
    when working with an svn version. Also, never had dev vs core issues.
    Can
    you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put
    pig.jar (not core, not dev) on $PIG_CLASSPATH?

    How about avoiding the pig script and just running

    java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main
    <<yourscript>> ?

    I grepped through all the code, including the codegenned stuff, and
    none
    of
    it contains the string PIGDIR (except bin/pig of course).

    At this point, assuming none of the above helps (which I have little
    hope
    it
    will), the best I can say is set up your eclipse environment and step
    through pig's main with the debugger.

    -D

    On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu
    wrote:
    Hello

    This is getting tiresome :)

    Today, I tried with a fresh download of pig 0.5 and I get the exact
    same
    error. Something somewhere just broke down, and it took my machine
    with
    it.
    At this point I have no way to run any kind of pig version.
    Needless to say, this is a big problem for me :)

    Another thing I tried is to do a fresh build of the 0.6 branch. Same error.
    I even dropped the sym links.

    Back to our debugging session:
    svn co
    http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2
    A pig-branch-0.6-take2/test/bin/test-patch.sh
    Checked out external at revision 910901.

    Checked out revision 910900.
    cd pig-branch-0.6-take2/
    ant
    BUILD SUCCESSFUL
    Total time: 26 seconds
    cd bin/
    next is to check if all vars are there
    printenv | grep PIG
    PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2
    PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2
    printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf
    pig
    Exception in thread "main" java.lang.NoClassDefFoundError:
    jline/ConsoleReaderInputStream
    Caused by: java.lang.ClassNotFoundException:
    jline.ConsoleReaderInputStream
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    Could not find the main class: org.apache.pig.Main. Program will
    exit.
    I've sent some email to the list about this problem before. A quick
    fix
    is
    to copy and rename the build/pig-0.6.1-dev.jar. Here goes:
    cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar
    pig
    2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging
    error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log
    2010-02-17 11:48:53,175 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to hadoop file system at: file:///
    grunt>

    Now at least it will start.

    Getting to the initial issue:

    the '-f' option:
    pig -f /home/alex/hadoop/test/test_lite.pig
    2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging
    error
    messages to:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log
    2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile:
    /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log

    the '- secretDebugCmd' option

    pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig
    dry run:
    /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
    -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs
    -Dpig.log.file=pig.log
    -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/..
    -Dpig.root.logger=INFO,console,DRFA -classpath
    /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar
    org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig

    I did not change the scripts, and I get the same error.

    This is really killing me :)

    thanks,
    alex



    On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>
    wrote:
    Hm, nothing jumps out. Definitely the error you are getting
    indicates
    that
    somehow the preprocessor is trying to substitute a variable called
    PIGDIR
    in
    your script. Which is odd. Does the same thing happen if you try
    bin/pig
    -f
    test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's secret),
    and sending along the output. Why are you using pig-core instead of pig.jar
    that ant should be generating for you when you build? Where did you
    get
    it,
    how did you build it, and what's the output of cksum on it?
    The fact that now your 0.5 is broken too makes me think that maybe
    your
    symlinks are messed up. Surely a change to one jar shouldn't be
    affecting
    a
    totally unrelated jar in a directory the first jar doesn't know
    about.
    Sorry about the barrage of questions, I am just a bit dumbfounded
    about
    how
    this could even begin to start happening. Any preprocessor experts around?
    -D


    On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu
    wrote:
    Hello

    And thanks again for all your help

    I have a symbolic link /home/alex/hadoop/pig which points to
    /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and
    build
    of
    the
    0.6 branch.
    I'm running the script from /home/alex/hadoop/test.

    I didn't touch pig.properties, also I'm running with the default
    pig
    script.

    The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar
    to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not
    work(Exception
    in thread "main" java.lang.NoClassDefFoundError:
    org/apache/pig/Main).
    I'm running with a fresh 0.6 build and still no luck.

    The info you reqested:

    alex@alex-desktop:~/hadoop/test$ pwd
    /home/alex/hadoop/test
    alex@alex-desktop:~/hadoop/test$ printenv | grep JAVA
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    alex@alex-desktop:~/hadoop/test$ printenv | grep PIG
    PIGDIR=/home/alex/hadoop/pig
    alex@alex-desktop:~/hadoop/test$ printenv | grep HADOOP
    HADOOP_HOME=/home/alex/hadoop/hadoop
    HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf
    HADOOPDIR=/home/alex/hadoop/hadoop/conf

    alex@alex-desktop:~/hadoop/test$ cat
    /home/alex/hadoop/pig/conf/pig.properties
    # Pig configuration file. All values can be overwritten by
    command
    line
    arguments.
    # see bin/pig -help

    # log4jconf log4j configuration file
    # log4jconf=./conf/log4j.properties

    # brief logging (no timestamps)
    brief=false

    # clustername, name of the hadoop jobtracker. If no port is
    defined
    port
    50020 will be used.
    #cluster

    #debug level, INFO is default
    debug=INFO

    # a file that contains pig script
    #file=

    # load jarfile, colon separated
    #jar=

    #verbose print all log messages to screen (default to print only
    INFO
    and
    above to screen)
    verbose=false

    #exectype local|mapreduce, mapreduce is default
    #exectype=mapreduce
    # hod realted properties
    #ssh.gateway
    #hod.expect.root
    #hod.expect.uselatest
    #hod.command
    #hod.config.dir
    #hod.param


    #Do not spill temp files smaller than this size (bytes)
    pig.spill.size.threshold=5000000
    #EXPERIMENT: Activate garbage collection when spilling a file
    bigger
    than
    this size (bytes)
    #This should help reduce the number of files being spilled.
    pig.spill.gc.activation.size=40000000


    ######################
    # Everything below this line is Yahoo specific. Note that I've
    made
    # (almost) no changes to the lines above to make merging in from
    Apache
    # easier. Any values I don't want from above I override below.
    #
    # This file is configured for use with HOD on the production
    clusters.
    If
    you
    # want to run pig with a static cluster you will need to remove
    everything
    # below this line and set the cluster value (above) to the
    # hostname and port of your job tracker.

    exectype=mapreduce

    hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf
    hod.server=local

    cluster.domain=inktomisearch.com

    log.file=

    yinst.cluster=kryptonite



    And now boom!

    java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR
    org.apache.pig.Main
    test_lite.pig


    2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main -
    Logging
    error
    messages to: /home/alex/hadoop/test/pig_1266319637282.log
    2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR
    Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log

    The log file:
    alex@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log
    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Undefined parameter :
    PIGDIR
    java.lang.RuntimeException: Undefined parameter : PIGDIR
    at
    org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
    at
    org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
    at org.apache.pig.Main.runParamPreprocessor(Main.java:515)
    at org.apache.pig.Main.main(Main.java:366)
    ================================================================================

    The script: 'A = load '/home/alex/hadoop/test/t.csv' using
    PigStorage('\t')
    as (id: long); '
    The file: '1' .
    Simple enough :)

    I'm all out of ideas. It seems that even 0.5 is broken now. I
    can't
    start
    anything as scripts. If I go with manual processing in grunt
    (line
    by
    line)
    it's all good on any version.

    thanks,
    alex


    On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy <
    dvryaboy@gmail.com>
    wrote:
    Pig starts, the error you are getting is from inside pig.Main .
    I think something is getting messed up in your environment
    because
    you
    are juggling too many different versions of pig (granted, I
    have
    3
    or
    4 in various stages of development on my laptop most of the
    time,
    and
    haven't had your problems. But then neither of them has a
    hacked
    bin/pig , with the exception of the cloudera one...). I've
    never
    tried
    running multiple ant tests, either. There's a short "sanity"
    version
    of tests, ant test-commit, that runs in under 10 minutes. You
    might
    want to try that if you are not doing things like changing join
    implementations or messing with the optimizer.

    Let's do this: send a full trace of what you are doing and what
    your
    environment looks like. Something like

    pwd
    printenv | grep JAVA
    printenv | grep PIG
    printenv | grep HADOOP
    cat conf/pig.properties
    java -cp ........
    <boom!>

    -D


    On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu
    wrote:
    Hello,

    sorry for the delay, but I wanted to build from the source
    again
    just
    to
    make sure.

    The script is like this:

    'A = load '/home/alex/hadoop/reviews/r.csv' using
    PigStorage('\t')
    as
    (id:
    long, hid: long, locale: chararray, r1: int, r2: int, r3:
    int,
    r4:
    int);
    '
    That's it. My guess is that Pig doesn't even start. Do you
    think
    I
    need
    to
    change something in the properties file? I'm not sure anymore
    and
    I
    don't
    have a lot of luck going through the sources.

    And another thing, I tried running 'ant test' for both
    0.6-branch
    and
    trunk
    at the same time (because they take a very long time) and
    both
    test
    scripts
    failed. I've switched to running them one after the other and
    they
    are
    fine.
    Do you think that is ok?

    thanks for your time,
    alex

    On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy <
    dvryaboy@gmail.com>
    wrote:
    what does your script1-hadoop.pig look like?

    The error you are getting happens when the pig preprocessor
    can't
    substitute Pig variables (the stuff you specify with -param
    and
    %default, etc). Do you have $PIGDIR in your script
    somewhere?
    -D

    On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu
    wrote:
    Hello,

    I seem to have broken my Pig install, and I don't know
    where
    to
    look.
    If I use directly the script (grunt) everything works ok,
    but
    every
    time
    I
    try to run a pig script: 'java -cp
    $PIGDIR/pig.jar:$HADOOPSITEPATH
    org.apache.pig.Main script1-hadoop.pig'

    I get this nice error: [main] ERROR org.apache.pig.Main -
    ERROR
    2999:
    Unexpected internal error. Undefined parameter : PIGDIR

    Obviously I have the PIGDIR var set:
    echo $PIGDIR
    /home/alex/hadoop/pig
    This is something that I did, as I have used 0.5 and 0.6
    and
    a
    patched
    version of 0.6 in parallel, but I can't figure out where
    to
    look.
    Any
    version I try to start now, gives the same error.

    any help would be greatly appreciated!

    alex

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedFeb 12, '10 at 2:51p
activeFeb 24, '10 at 8:10p
posts11
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase