Grokbase Groups Pig user May 2011
FAQ
I'm trying to embed pig into java program. I tried two approaches, none of
them works.

Approach 1:

I followed http://pig.apache.org/docs/r0.6.0/setup.html#Embedded+Programs
and then ran into the error: cannot find the hadoop-core.xml nor
site-core.xml. Then I find the solution at
http://mail-archives.apache.org/mod_mbox/pig-user/201104.mbox/%3CBANLkTinx
G7VwS8hDjaMbR=8_2w7AJ3_qhg@mail.gmail.com%3E, which is just add a line of
System.setProperty("sun.boot.class.path", "/usr/local/hadoop/conf"); to
tell the program the location of hadoop configuration files. Then I had
another error:

11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
hadoop file system at: hdfs://jianting-virtual-machine1:54310
11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
map-reduce job tracker at: jianting-virtual-machine1:54311
11/05/11 17:18:32 INFO pigstats.ScriptState: Pig features used in the
script: UNKNOWN
11/05/11 17:18:32 INFO executionengine.HExecutionEngine:
pig.usenewlogicalplan is set to true. New logical plan will be used.
11/05/11 17:18:33 INFO executionengine.HExecutionEngine: (Name: B:
Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
scope-4)
11/05/11 17:18:33 INFO mapReduceLayer.MRCompiler: File concatenation
threshold: 100 optimistic? false
11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
before optimization: 1
11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
after optimization: 1
11/05/11 17:18:33 INFO pigstats.ScriptState: Pig script settings are added
to the job
11/05/11 17:18:33 INFO mapReduceLayer.JobControlCompiler:
mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
11/05/11 17:18:35 INFO mapReduceLayer.JobControlCompiler: Setting up
single store job
11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
job(s) waiting for submission.
11/05/11 17:18:35 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/05/11 17:18:35 INFO input.FileInputFormat: Total input paths to process
: 1
11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths to process : 1
11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths (combined) to
process : 1
11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 0% complete
org.apache.pig.PigException: ERROR 1002: Unable to store alias B

This is where I am stuck.



Then I tried a second approach.

Approach 2:

The code and everything is the same as in the tutorial of approach 1. The
only difference is that instead of using vim, javac and java to edit,
compile and run the example, I uses Netbeans. I added pig.jar file that is
generated by running "ant" in pig folder. This time the error is:

Have no file for
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/modules/jdk.boot.jar
11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
hadoop file system at: hdfs://jianting-virtual-machine1:54310
11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
map-reduce job tracker at: jianting-virtual-machine1:54311
11/05/11 17:37:53 INFO pigstats.ScriptState: Pig features used in the
script: UNKNOWN
11/05/11 17:37:53 INFO executionengine.HExecutionEngine:
pig.usenewlogicalplan is set to true. New logical plan will be used.
11/05/11 17:37:54 INFO executionengine.HExecutionEngine: (Name: B:
Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
scope-4)
11/05/11 17:37:54 INFO mapReduceLayer.MRCompiler: File concatenation
threshold: 100 optimistic? false
11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
before optimization: 1
11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
after optimization: 1
11/05/11 17:37:54 INFO pigstats.ScriptState: Pig script settings are added
to the job
11/05/11 17:37:54 INFO mapReduceLayer.JobControlCompiler:
mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
11/05/11 17:37:56 INFO mapReduceLayer.JobControlCompiler: Setting up
single store job
11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
job(s) waiting for submission.
11/05/11 17:37:57 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/05/11 17:37:57 INFO input.FileInputFormat: Total input paths to process
: 1
11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths to process : 1
11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 0% complete
11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths (combined) to
process : 1
11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 100% complete
11/05/11 17:37:57 ERROR pigstats.PigStatsUtil: 0 map reduce job(s) failed!
11/05/11 17:37:57 INFO pigstats.PigStats: Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.20.2 0.8.1-SNAPSHOT hadoop 2011-05-11 17:37:54 2011-05-11
17:37:57 UNKNOWN

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs

Input(s):

Output(s):

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
null

11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: Success!
java.io.IOException: Couldn't retrieve job.

I'm stuck at here too.

If I run the example line by line in grunt script mode, the program
completes successfully without any error/warning. My guess is the
warning:WARN mapred.JobClient: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same, which appears
in both approaches and does not appear in grunt script mode.

I'm using Pig 0.8.1, hadoop 0.20.2, java 1.6, Ubuntu 10.10.

I searched the mailing list but without any luck. Any help is appreciated.

Jianting Cao

Search Discussions

  • Daniel Dai at May 14, 2011 at 12:51 am
    Sounds like a hadoop job setup exception. Go to job tracker UI, you may
    have chance to locate the job and check what happen in job setup.

    Daniel
    On 05/11/2011 05:45 PM, Jianting Cao wrote:
    I'm trying to embed pig into java program. I tried two approaches, none of
    them works.

    Approach 1:

    I followed http://pig.apache.org/docs/r0.6.0/setup.html#Embedded+Programs
    and then ran into the error: cannot find the hadoop-core.xml nor
    site-core.xml. Then I find the solution at
    http://mail-archives.apache.org/mod_mbox/pig-user/201104.mbox/%3CBANLkTinx
    G7VwS8hDjaMbR=8_2w7AJ3_qhg@mail.gmail.com%3E, which is just add a line of
    System.setProperty("sun.boot.class.path", "/usr/local/hadoop/conf"); to
    tell the program the location of hadoop configuration files. Then I had
    another error:

    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:18:32 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:18:33 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:18:33 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:18:33 INFO pigstats.ScriptState: Pig script settings are added
    to the job
    11/05/11 17:18:33 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:18:35 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:18:35 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:18:35 INFO input.FileInputFormat: Total input paths to process
    : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    org.apache.pig.PigException: ERROR 1002: Unable to store alias B

    This is where I am stuck.



    Then I tried a second approach.

    Approach 2:

    The code and everything is the same as in the tutorial of approach 1. The
    only difference is that instead of using vim, javac and java to edit,
    compile and run the example, I uses Netbeans. I added pig.jar file that is
    generated by running "ant" in pig folder. This time the error is:

    Have no file for
    /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/modules/jdk.boot.jar
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:37:53 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:37:54 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:37:54 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:37:54 INFO pigstats.ScriptState: Pig script settings are added
    to the job
    11/05/11 17:37:54 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:37:56 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:37:57 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:37:57 INFO input.FileInputFormat: Total input paths to process
    : 1
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 100% complete
    11/05/11 17:37:57 ERROR pigstats.PigStatsUtil: 0 map reduce job(s) failed!
    11/05/11 17:37:57 INFO pigstats.PigStats: Script Statistics:

    HadoopVersion PigVersion UserId StartedAt FinishedAt
    Features
    0.20.2 0.8.1-SNAPSHOT hadoop 2011-05-11 17:37:54 2011-05-11
    17:37:57 UNKNOWN

    Failed!

    Failed Jobs:
    JobId Alias Feature Message Outputs

    Input(s):

    Output(s):

    Counters:
    Total records written : 0
    Total bytes written : 0
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    null

    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: Success!
    java.io.IOException: Couldn't retrieve job.

    I'm stuck at here too.

    If I run the example line by line in grunt script mode, the program
    completes successfully without any error/warning. My guess is the
    warning:WARN mapred.JobClient: Use GenericOptionsParser for parsing the
    arguments. Applications should implement Tool for the same, which appears
    in both approaches and does not appear in grunt script mode.

    I'm using Pig 0.8.1, hadoop 0.20.2, java 1.6, Ubuntu 10.10.

    I searched the mailing list but without any luck. Any help is appreciated.

    Jianting Cao

  • Andrew Wells at May 14, 2011 at 5:06 am
    I had a simular problem, you need to add the path to the rootClasspath, not
    just the class path.


    On Fri, May 13, 2011 at 8:49 PM, Daniel Dai wrote:

    Sounds like a hadoop job setup exception. Go to job tracker UI, you may
    have chance to locate the job and check what happen in job setup.

    Daniel

    On 05/11/2011 05:45 PM, Jianting Cao wrote:

    I'm trying to embed pig into java program. I tried two approaches, none of
    them works.

    Approach 1:

    I followed http://pig.apache.org/docs/r0.6.0/setup.html#Embedded+Programs
    and then ran into the error: cannot find the hadoop-core.xml nor
    site-core.xml. Then I find the solution at
    http://mail-archives.apache.org/mod_mbox/pig-user/201104.mbox/%3CBANLkTinx
    G7VwS8hDjaMbR=8_2w7AJ3_qhg@mail.gmail.com%3E, which is just add a line of
    System.setProperty("sun.boot.class.path", "/usr/local/hadoop/conf"); to
    tell the program the location of hadoop configuration files. Then I had
    another error:

    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:18:32 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:18:33 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:18:33 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:18:33 INFO pigstats.ScriptState: Pig script settings are added
    to the job
    11/05/11 17:18:33 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:18:35 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:18:35 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:18:35 INFO input.FileInputFormat: Total input paths to process
    : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    org.apache.pig.PigException: ERROR 1002: Unable to store alias B

    This is where I am stuck.



    Then I tried a second approach.

    Approach 2:

    The code and everything is the same as in the tutorial of approach 1. The
    only difference is that instead of using vim, javac and java to edit,
    compile and run the example, I uses Netbeans. I added pig.jar file that is
    generated by running "ant" in pig folder. This time the error is:

    Have no file for
    /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/modules/jdk.boot.jar
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:37:53 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:37:54 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:37:54 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:37:54 INFO pigstats.ScriptState: Pig script settings are added
    to the job
    11/05/11 17:37:54 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:37:56 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:37:57 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:37:57 INFO input.FileInputFormat: Total input paths to process
    : 1
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 100% complete
    11/05/11 17:37:57 ERROR pigstats.PigStatsUtil: 0 map reduce job(s) failed!
    11/05/11 17:37:57 INFO pigstats.PigStats: Script Statistics:

    HadoopVersion PigVersion UserId StartedAt FinishedAt
    Features
    0.20.2 0.8.1-SNAPSHOT hadoop 2011-05-11 17:37:54 2011-05-11
    17:37:57 UNKNOWN

    Failed!

    Failed Jobs:
    JobId Alias Feature Message Outputs

    Input(s):

    Output(s):

    Counters:
    Total records written : 0
    Total bytes written : 0
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    null

    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: Success!
    java.io.IOException: Couldn't retrieve job.

    I'm stuck at here too.

    If I run the example line by line in grunt script mode, the program
    completes successfully without any error/warning. My guess is the
    warning:WARN mapred.JobClient: Use GenericOptionsParser for parsing the
    arguments. Applications should implement Tool for the same, which appears
    in both approaches and does not appear in grunt script mode.

    I'm using Pig 0.8.1, hadoop 0.20.2, java 1.6, Ubuntu 10.10.

    I searched the mailing list but without any luck. Any help is appreciated.

    Jianting Cao


  • Andrew Wells at May 14, 2011 at 5:07 am
    I should be more clear, append the path the root class path, not the
    classpath


    On Sat, May 14, 2011 at 1:05 AM, Andrew Wells wrote:

    I had a simular problem, you need to add the path to the rootClasspath, not
    just the class path.


    On Fri, May 13, 2011 at 8:49 PM, Daniel Dai wrote:

    Sounds like a hadoop job setup exception. Go to job tracker UI, you may
    have chance to locate the job and check what happen in job setup.

    Daniel

    On 05/11/2011 05:45 PM, Jianting Cao wrote:

    I'm trying to embed pig into java program. I tried two approaches, none
    of
    them works.

    Approach 1:

    I followed
    http://pig.apache.org/docs/r0.6.0/setup.html#Embedded+Programs
    and then ran into the error: cannot find the hadoop-core.xml nor
    site-core.xml. Then I find the solution at

    http://mail-archives.apache.org/mod_mbox/pig-user/201104.mbox/%3CBANLkTinx
    G7VwS8hDjaMbR=8_2w7AJ3_qhg@mail.gmail.com%3E, which is just add a line
    of
    System.setProperty("sun.boot.class.path", "/usr/local/hadoop/conf"); to
    tell the program the location of hadoop configuration files. Then I had
    another error:

    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:18:32 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:18:33 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:18:33 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:18:33 INFO pigstats.ScriptState: Pig script settings are
    added
    to the job
    11/05/11 17:18:33 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:18:35 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:18:35 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:18:35 INFO input.FileInputFormat: Total input paths to
    process
    : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    org.apache.pig.PigException: ERROR 1002: Unable to store alias B

    This is where I am stuck.



    Then I tried a second approach.

    Approach 2:

    The code and everything is the same as in the tutorial of approach 1. The
    only difference is that instead of using vim, javac and java to edit,
    compile and run the example, I uses Netbeans. I added pig.jar file that
    is
    generated by running "ant" in pig folder. This time the error is:

    Have no file for
    /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/modules/jdk.boot.jar
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:37:53 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:37:54 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:37:54 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:37:54 INFO pigstats.ScriptState: Pig script settings are
    added
    to the job
    11/05/11 17:37:54 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:37:56 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:37:57 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:37:57 INFO input.FileInputFormat: Total input paths to
    process
    : 1
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 100% complete
    11/05/11 17:37:57 ERROR pigstats.PigStatsUtil: 0 map reduce job(s)
    failed!
    11/05/11 17:37:57 INFO pigstats.PigStats: Script Statistics:

    HadoopVersion PigVersion UserId StartedAt FinishedAt
    Features
    0.20.2 0.8.1-SNAPSHOT hadoop 2011-05-11 17:37:54 2011-05-11
    17:37:57 UNKNOWN

    Failed!

    Failed Jobs:
    JobId Alias Feature Message Outputs

    Input(s):

    Output(s):

    Counters:
    Total records written : 0
    Total bytes written : 0
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    null

    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: Success!
    java.io.IOException: Couldn't retrieve job.

    I'm stuck at here too.

    If I run the example line by line in grunt script mode, the program
    completes successfully without any error/warning. My guess is the
    warning:WARN mapred.JobClient: Use GenericOptionsParser for parsing the
    arguments. Applications should implement Tool for the same, which appears
    in both approaches and does not appear in grunt script mode.

    I'm using Pig 0.8.1, hadoop 0.20.2, java 1.6, Ubuntu 10.10.

    I searched the mailing list but without any luck. Any help is
    appreciated.

    Jianting Cao


  • Jianting Cao at May 17, 2011 at 7:28 pm
    Thanks very much. I solved the problem by rebuilding pig.jar without modifying anything.

    Jianting

    ----- Original Message -----
    From: "Andrew Wells" <agwells0714@gmail.com>
    To: user@pig.apache.org
    Sent: Friday, May 13, 2011 10:06:12 PM
    Subject: Re: embedded pig error

    I should be more clear, append the path the root class path, not the
    classpath


    On Sat, May 14, 2011 at 1:05 AM, Andrew Wells wrote:

    I had a simular problem, you need to add the path to the rootClasspath, not
    just the class path.


    On Fri, May 13, 2011 at 8:49 PM, Daniel Dai wrote:

    Sounds like a hadoop job setup exception. Go to job tracker UI, you may
    have chance to locate the job and check what happen in job setup.

    Daniel

    On 05/11/2011 05:45 PM, Jianting Cao wrote:

    I'm trying to embed pig into java program. I tried two approaches, none
    of
    them works.

    Approach 1:

    I followed
    http://pig.apache.org/docs/r0.6.0/setup.html#Embedded+Programs
    and then ran into the error: cannot find the hadoop-core.xml nor
    site-core.xml. Then I find the solution at

    http://mail-archives.apache.org/mod_mbox/pig-user/201104.mbox/%3CBANLkTinx
    G7VwS8hDjaMbR=8_2w7AJ3_qhg@mail.gmail.com%3E, which is just add a line
    of
    System.setProperty("sun.boot.class.path", "/usr/local/hadoop/conf"); to
    tell the program the location of hadoop configuration files. Then I had
    another error:

    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:18:32 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:18:32 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:18:33 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:18:33 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:18:33 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:18:33 INFO pigstats.ScriptState: Pig script settings are
    added
    to the job
    11/05/11 17:18:33 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:18:35 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:18:35 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:18:35 INFO input.FileInputFormat: Total input paths to
    process
    : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:18:35 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:18:35 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    org.apache.pig.PigException: ERROR 1002: Unable to store alias B

    This is where I am stuck.



    Then I tried a second approach.

    Approach 2:

    The code and everything is the same as in the tutorial of approach 1. The
    only difference is that instead of using vim, javac and java to edit,
    compile and run the example, I uses Netbeans. I added pig.jar file that
    is
    generated by running "ant" in pig folder. This time the error is:

    Have no file for
    /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/modules/jdk.boot.jar
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    hadoop file system at: hdfs://jianting-virtual-machine1:54310
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine: Connecting to
    map-reduce job tracker at: jianting-virtual-machine1:54311
    11/05/11 17:37:53 INFO pigstats.ScriptState: Pig features used in the
    script: UNKNOWN
    11/05/11 17:37:53 INFO executionengine.HExecutionEngine:
    pig.usenewlogicalplan is set to true. New logical plan will be used.
    11/05/11 17:37:54 INFO executionengine.HExecutionEngine: (Name: B:
    Store(idout:org.apache.pig.builtin.PigStorage) - scope-4 Operator Key:
    scope-4)
    11/05/11 17:37:54 INFO mapReduceLayer.MRCompiler: File concatenation
    threshold: 100 optimistic? false
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    before optimization: 1
    11/05/11 17:37:54 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size
    after optimization: 1
    11/05/11 17:37:54 INFO pigstats.ScriptState: Pig script settings are
    added
    to the job
    11/05/11 17:37:54 INFO mapReduceLayer.JobControlCompiler:
    mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    11/05/11 17:37:56 INFO mapReduceLayer.JobControlCompiler: Setting up
    single store job
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce
    job(s) waiting for submission.
    11/05/11 17:37:57 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    11/05/11 17:37:57 INFO input.FileInputFormat: Total input paths to
    process
    : 1
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths to process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 0% complete
    11/05/11 17:37:57 INFO util.MapRedUtil: Total input paths (combined) to
    process : 1
    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: 100% complete
    11/05/11 17:37:57 ERROR pigstats.PigStatsUtil: 0 map reduce job(s)
    failed!
    11/05/11 17:37:57 INFO pigstats.PigStats: Script Statistics:

    HadoopVersion PigVersion UserId StartedAt FinishedAt
    Features
    0.20.2 0.8.1-SNAPSHOT hadoop 2011-05-11 17:37:54 2011-05-11
    17:37:57 UNKNOWN

    Failed!

    Failed Jobs:
    JobId Alias Feature Message Outputs

    Input(s):

    Output(s):

    Counters:
    Total records written : 0
    Total bytes written : 0
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    null

    11/05/11 17:37:57 INFO mapReduceLayer.MapReduceLauncher: Success!
    java.io.IOException: Couldn't retrieve job.

    I'm stuck at here too.

    If I run the example line by line in grunt script mode, the program
    completes successfully without any error/warning. My guess is the
    warning:WARN mapred.JobClient: Use GenericOptionsParser for parsing the
    arguments. Applications should implement Tool for the same, which appears
    in both approaches and does not appear in grunt script mode.

    I'm using Pig 0.8.1, hadoop 0.20.2, java 1.6, Ubuntu 10.10.

    I searched the mailing list but without any luck. Any help is
    appreciated.

    Jianting Cao


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 12, '11 at 12:46a
activeMay 17, '11 at 7:28p
posts5
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase