Grokbase Groups Pig user August 2010
FAQ
All,



I am running pig-0.7.0 and I have been running into an issue running the
ORDER command. I have attempted to run pig out of the box on 2 separate
LINUX OS (Ubuntu 10.4 and OpenSuse 11.2) and the same issue has
occurred. I run these commands in a script file:



start = LOAD 'inputData' USING PigStorage('|') AS (sip:chararray,
dip:chararray, sport:int, dport:int, protocol:int, packets:int,
bytes:int, flags:chararray, startTime:long, endTime:long);

target = FILTER start BY sip matches '51.37.8.63';

fail = ORDER target BY bytes DESC;

not_reached = LIMIT fail 10;

dump not_reached;



The error is listed below. I then run:



start = LOAD 'inputData' USING PigStorage('|') AS (sip:chararray,
dip:chararray, sport:int, dport:int, protocol:int, packets:int,
bytes:int, flags:chararray, startTime:long, endTime:long);

target = FILTER start BY sip matches '51.37.8.63';

dump target;



This script produces a large list of sips matching the filter. What am
I doing wrong that causes pig to not want to ORDER these properly? I
have been wrestling with this issue for a week now. Any help would be
greatly appreciated.



Best,

Matthew



/ERROR



10/08/18 11:24:15 INFO pig.Main: Logging error messages to:
/home/matt/workspace/pig-0.7.0/cloudComputingPrototypes/pig_128215585580
9.log

2010-08-18 11:24:16,000 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: file:///

2010-08-18 11:24:16,338 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column
pruned for start

2010-08-18 11:24:16,338 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
pruned for start

2010-08-18 11:24:16,396 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=

2010-08-18 11:24:16,470 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
Store(file:/tmp/temp-2049115517/tmp197746350:org.apache.pig.builtin.BinS
torage) - 1-74 Operator Key: 1-74)

2010-08-18 11:24:16,567 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryO
ptimizer - MR plan size before optimization: 3

2010-08-18 11:24:16,567 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryO
ptimizer - MR plan size after optimization: 3

2010-08-18 11:24:16,577 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:16,581 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:16,581 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlC
ompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to
default 0.3

2010-08-18 11:24:17,828 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlC
ompiler - Setting up single store job

2010-08-18 11:24:17,858 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:17,859 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 1 map-reduce job(s) waiting for submission.

2010-08-18 11:24:17,863 [Thread-4] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

2010-08-18 11:24:18,000 [Thread-4] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,010 [Thread-4] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,027 [Thread-4] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
paths to process : 1

2010-08-18 11:24:18,027 [Thread-4] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1

2010-08-18 11:24:18,237 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,241 [Thread-13] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
paths to process : 1

2010-08-18 11:24:18,241 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1

2010-08-18 11:24:18,302 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,307 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,312 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,317 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,360 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - HadoopJobId: job_local_0001

2010-08-18 11:24:18,360 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 0% complete

2010-08-18 11:24:18,900 [Thread-13] INFO
org.apache.hadoop.mapred.TaskRunner - Task:attempt_local_0001_m_000000_0
is done. And is in the process of commiting

2010-08-18 11:24:18,902 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,903 [Thread-13] INFO
org.apache.hadoop.mapred.LocalJobRunner -

2010-08-18 11:24:18,903 [Thread-13] INFO
org.apache.hadoop.mapred.TaskRunner - Task attempt_local_0001_m_000000_0
is allowed to commit now

2010-08-18 11:24:18,904 [Thread-13] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:18,906 [Thread-13] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved
output of task 'attempt_local_0001_m_000000_0' to
file:/tmp/temp-2049115517/tmp-44482827

2010-08-18 11:24:18,906 [Thread-13] INFO
org.apache.hadoop.mapred.LocalJobRunner -

2010-08-18 11:24:18,907 [Thread-13] INFO
org.apache.hadoop.mapred.TaskRunner - Task
'attempt_local_0001_m_000000_0' done.

2010-08-18 11:24:23,370 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 33% complete

2010-08-18 11:24:23,370 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:23,371 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlC
ompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to
default 0.3

2010-08-18 11:24:24,500 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlC
ompiler - Setting up single store job

2010-08-18 11:24:24,526 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:24,526 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 1 map-reduce job(s) waiting for submission.

2010-08-18 11:24:24,527 [Thread-17] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

2010-08-18 11:24:24,630 [Thread-17] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:24,635 [Thread-17] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:24,640 [Thread-17] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
paths to process : 1

2010-08-18 11:24:24,641 [Thread-17] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1

2010-08-18 11:24:24,785 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:24,787 [Thread-26] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
paths to process : 1

2010-08-18 11:24:24,787 [Thread-26] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1

2010-08-18 11:24:24,821 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:24,825 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:24,838 [Thread-26] INFO
org.apache.hadoop.mapred.MapTask - io.sort.mb = 100

2010-08-18 11:24:26,106 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - HadoopJobId: job_local_0002

2010-08-18 11:24:26,168 [Thread-26] INFO
org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720

2010-08-18 11:24:26,168 [Thread-26] INFO
org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680

2010-08-18 11:24:26,362 [Thread-26] INFO
org.apache.hadoop.mapred.MapTask - Starting flush of map output

2010-08-18 11:24:26,657 [Thread-26] INFO
org.apache.hadoop.mapred.MapTask - Finished spill 0

2010-08-18 11:24:26,661 [Thread-26] INFO
org.apache.hadoop.mapred.TaskRunner - Task:attempt_local_0002_m_000000_0
is done. And is in the process of commiting

2010-08-18 11:24:26,661 [Thread-26] INFO
org.apache.hadoop.mapred.LocalJobRunner -

2010-08-18 11:24:26,661 [Thread-26] INFO
org.apache.hadoop.mapred.TaskRunner - Task
'attempt_local_0002_m_000000_0' done.

2010-08-18 11:24:26,669 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:26,674 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:26,675 [Thread-26] INFO
org.apache.hadoop.mapred.LocalJobRunner -

2010-08-18 11:24:26,681 [Thread-26] INFO
org.apache.hadoop.mapred.Merger - Merging 1 sorted segments

2010-08-18 11:24:26,746 [Thread-26] INFO
org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1
segments left of total size: 3202 bytes

2010-08-18 11:24:26,747 [Thread-26] INFO
org.apache.hadoop.mapred.LocalJobRunner -

2010-08-18 11:24:26,752 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:26,955 [Thread-26] INFO
org.apache.hadoop.mapred.TaskRunner - Task:attempt_local_0002_r_000000_0
is done. And is in the process of commiting

2010-08-18 11:24:26,956 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:26,957 [Thread-26] INFO
org.apache.hadoop.mapred.LocalJobRunner -

2010-08-18 11:24:26,957 [Thread-26] INFO
org.apache.hadoop.mapred.TaskRunner - Task attempt_local_0002_r_000000_0
is allowed to commit now

2010-08-18 11:24:26,958 [Thread-26] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:26,959 [Thread-26] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved
output of task 'attempt_local_0002_r_000000_0' to
file:/tmp/temp-2049115517/tmp-2112064820

2010-08-18 11:24:26,961 [Thread-26] INFO
org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce

2010-08-18 11:24:26,961 [Thread-26] INFO
org.apache.hadoop.mapred.TaskRunner - Task
'attempt_local_0002_r_000000_0' done.

2010-08-18 11:24:30,114 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 66% complete

2010-08-18 11:24:30,115 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:30,115 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlC
ompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to
default 0.3

2010-08-18 11:24:31,408 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlC
ompiler - Setting up single store job

2010-08-18 11:24:31,507 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:31,507 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 1 map-reduce job(s) waiting for submission.

2010-08-18 11:24:31,612 [Thread-32] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

2010-08-18 11:24:31,840 [Thread-32] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:31,843 [Thread-32] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:31,845 [Thread-32] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
paths to process : 1

2010-08-18 11:24:31,845 [Thread-32] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1

2010-08-18 11:24:32,071 [Thread-41] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:32,074 [Thread-41] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input
paths to process : 1

2010-08-18 11:24:32,074 [Thread-41] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1

2010-08-18 11:24:32,075 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - HadoopJobId: job_local_0003

2010-08-18 11:24:32,155 [Thread-41] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:32,160 [Thread-41] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:32,160 [Thread-41] INFO
org.apache.hadoop.mapred.MapTask - io.sort.mb = 100

2010-08-18 11:24:32,491 [Thread-41] INFO
org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720

2010-08-18 11:24:32,491 [Thread-41] INFO
org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680

2010-08-18 11:24:32,926 [Thread-41] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:32,948 [Thread-41] WARN
org.apache.hadoop.mapred.LocalJobRunner - job_local_0003

java.lang.RuntimeException:
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path
does not exist: file:/user/matt/pigsample_24118161_1282155871461

at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioner
s.WeightedRangePartitioner.setConf(WeightedRangePartitioner.java:135)

at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)

at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:
117)

at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:
527)

at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)

at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)

at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input path does not exist:
file:/user/matt/pigsample_24118161_1282155871461

at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInp
utFormat.java:224)

at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigFileInpu
tFormat.listStatus(PigFileInputFormat.java:37)

at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInpu
tFormat.java:241)

at
org.apache.pig.impl.io.ReadToEndLoader.init(ReadToEndLoader.java:153)

at
org.apache.pig.impl.io.ReadToEndLoader.<init>(ReadToEndLoader.java:115)

at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioner
s.WeightedRangePartitioner.setConf(WeightedRangePartitioner.java:108)

... 6 more

2010-08-18 11:24:37,089 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 100% complete

2010-08-18 11:24:37,089 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - 1 map reduce job(s) failed!

2010-08-18 11:24:37,090 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - Failed to produce result in:
"file:/tmp/temp-2049115517/tmp197746350"

2010-08-18 11:24:37,090 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
uncher - Some jobs have failed! Stop running all dependent jobs

2010-08-18 11:24:37,127 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

2010-08-18 11:24:37,140 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1066: Unable to open iterator for alias not_reached

Details at logfile:
/home/matt/workspace/pig-0.7.0/cloudComputingPrototypes/pig_128215585580
9.log

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 1 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedAug 18, '10 at 6:32p
activeAug 18, '10 at 6:32p
posts1
users1
websitepig.apache.org

1 user in discussion

Matthew Smith: 1 post

People

Translate

site design / logo © 2021 Grokbase