FAQ
Hello all,

One of my customer's getting the following when increase input size.

same process runs fine with smaller input size.

/tmp has plenty of space,

each data node has 64 Gig memory.

hadoop 0-19.1

Please let me know if you need any additional info.



[root@head attempt_200904181818_0020_r_000002_1]# cat syslog

2009-04-20 00:52:52,898 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=

2009-04-20 00:52:53,005 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736

2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging
on-disk files

2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging in
memory files

2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Thread waiting: Thread for merging
on-disk files

2009-04-20 00:52:53,015 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Need another 168 map output(s) where 0
is already in progress

2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1: Got 168 new map-outputs

2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)

2009-04-20 00:52:53,068 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000002_0

2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000006_0

2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000003_0

2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000016_0

2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000012_0

2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000016_0

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000006_0

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000012_0

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000003_0

2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000002_0

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000012_0 -> (-1, -1) from n2

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000016_0 -> (-1, -1) from n8

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000006_0 -> (-1, -1) from n4

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000002_0 -> (-1, -1) from n5

2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000003_0 -> (-1, -1) from n7

2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000007_0

2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000007_0

2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000007_0 -> (-1, -1) from n3

2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000037_0

2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000037_0

2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000037_0 -> (-1, -1) from n1

2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000033_0

2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000033_0

2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000033_0 -> (-1, -1) from n6

2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
obsolete output of KILLED map-task: 'attempt_200904181818_0020_m_000166_1'

2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1: Got 1 obsolete map-outputs from
tasktracker

2009-04-20 00:52:55,022 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000005_0

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000005_0

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000005_0 -> (-1, -1) from n1

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000014_0

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000004_0

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000014_0

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000004_0

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000014_0 -> (-1, -1) from n4

2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000004_0 -> (-1, -1) from n2

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000019_0

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000018_0

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000019_0

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000018_0

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000019_0 -> (-1, -1) from n7

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000018_0 -> (-1, -1) from n5

2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000008_0

2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000008_0

2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000008_0 -> (-1, -1) from n8

2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000015_0

2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000015_0

2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000015_0 -> (-1, -1) from n3

2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000001_0

2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000001_0

2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000001_0 -> (-1, -1) from n6

2009-04-20 00:52:57,026 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)

2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000029_0

2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000029_0

2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000029_0 -> (-1, -1) from n1

2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000011_0

2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000020_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000026_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000011_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000020_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000011_0 -> (-1, -1) from n7

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000020_0 -> (-1, -1) from n2

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000026_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000047_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000026_0 -> (-1, -1) from n5

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000047_0

2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000047_0 -> (-1, -1) from n3

2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000022_0

2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000022_0

2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000022_0 -> (-1, -1) from n4

2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000049_0

2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000049_0

2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000049_0 -> (-1, -1) from n6

2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000000_0

2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000000_0

2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000000_0 -> (-1, -1) from n8

2009-04-20 00:52:59,029 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)

2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000052_0

2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000052_0

2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000074_0

2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000052_0 -> (-1, -1) from n2

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000074_0

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000074_0 -> (-1, -1) from n5

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000017_0

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000017_0

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000017_0 -> (-1, -1) from n6

2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000038_0

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000056_0

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000038_0

2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000056_0

2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000038_0 -> (-1, -1) from n4

2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000056_0 -> (-1, -1) from n8

2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000023_0

2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000023_0

2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000023_0 -> (-1, -1) from n3

2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000013_0

2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000013_0

2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000013_0 -> (-1, -1) from n1

2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000027_0

2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000027_0

2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000027_0 -> (-1, -1) from n7

2009-04-20 00:53:01,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)

2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000055_0

2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000055_0

2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000055_0 -> (-1, -1) from n3

2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000025_0

2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000025_0

2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000025_0 -> (-1, -1) from n6

2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000035_0

2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000035_0

2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000035_0 -> (-1, -1) from n7

2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000021_0

2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000021_0

2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000021_0 -> (-1, -1) from n1

2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000034_0

2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000034_0

2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000034_0 -> (-1, -1) from n5

2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 copy failed:
attempt_200904181818_0020_m_000044_0 from n2

2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 copy failed:
attempt_200904181818_0020_m_000054_0 from n4

2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 copy failed:
attempt_200904181818_0020_m_000032_0 from n8

2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000044_0&reduce=2

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)

at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
0)

at java.security.AccessController.doPrivileged(Native Method)

at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
ction.java:1354)

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1008)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
eam(ReduceTask.java:1293)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1231)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1144)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1084)

Caused by: java.io.FileNotFoundException:
http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000044_0&reduce=2

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1303)

... 4 more



2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000032_0&reduce=2

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)

at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
0)

at java.security.AccessController.doPrivileged(Native Method)

at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
ction.java:1354)

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1008)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
eam(ReduceTask.java:1293)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1231)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1144)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1084)

Caused by: java.io.FileNotFoundException:
http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000032_0&reduce=2

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1303)

... 4 more



2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000054_0&reduce=2

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)

at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
0)

at java.security.AccessController.doPrivileged(Native Method)

at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
ction.java:1354)

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1008)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
eam(ReduceTask.java:1293)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1231)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1144)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1084)

Caused by: java.io.FileNotFoundException:
http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000054_0&reduce=2

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1303)

... 4 more

Search Discussions

  • Jothi Padmanabhan at Apr 24, 2009 at 8:46 am
    Hi Brady,

    Could you please check the TaskTracker logs (TTs that ran the maps that are
    failing copy) and see what error you are seeing there?

    Jothi
    On 4/24/09 5:15 AM, "Brady Chang" wrote:

    Hello all,

    One of my customer's getting the following when increase input size.

    same process runs fine with smaller input size.

    /tmp has plenty of space,

    each data node has 64 Gig memory.

    hadoop 0-19.1

    Please let me know if you need any additional info.



    [root@head attempt_200904181818_0020_r_000002_1]# cat syslog

    2009-04-20 00:52:52,898 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=

    2009-04-20 00:52:53,005 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736

    2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging
    on-disk files

    2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging in
    memory files

    2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Thread waiting: Thread for merging
    on-disk files

    2009-04-20 00:52:53,015 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Need another 168 map output(s) where 0
    is already in progress

    2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1: Got 168 new map-outputs

    2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
    dup hosts)

    2009-04-20 00:52:53,068 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000002_0

    2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000006_0

    2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000003_0

    2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000016_0

    2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000012_0

    2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000016_0

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000006_0

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000012_0

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000003_0

    2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000002_0

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000012_0 -> (-1, -1) from n2

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000016_0 -> (-1, -1) from n8

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000006_0 -> (-1, -1) from n4

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000002_0 -> (-1, -1) from n5

    2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000003_0 -> (-1, -1) from n7

    2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000007_0

    2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000007_0

    2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000007_0 -> (-1, -1) from n3

    2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000037_0

    2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000037_0

    2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000037_0 -> (-1, -1) from n1

    2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000033_0

    2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000033_0

    2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000033_0 -> (-1, -1) from n6

    2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
    obsolete output of KILLED map-task: 'attempt_200904181818_0020_m_000166_1'

    2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1: Got 1 obsolete map-outputs from
    tasktracker

    2009-04-20 00:52:55,022 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
    dup hosts)

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000005_0

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000005_0

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000005_0 -> (-1, -1) from n1

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000014_0

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000004_0

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000014_0

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000004_0

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000014_0 -> (-1, -1) from n4

    2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000004_0 -> (-1, -1) from n2

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000019_0

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000018_0

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000019_0

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000018_0

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000019_0 -> (-1, -1) from n7

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000018_0 -> (-1, -1) from n5

    2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000008_0

    2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000008_0

    2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000008_0 -> (-1, -1) from n8

    2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000015_0

    2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000015_0

    2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000015_0 -> (-1, -1) from n3

    2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000001_0

    2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000001_0

    2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000001_0 -> (-1, -1) from n6

    2009-04-20 00:52:57,026 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
    dup hosts)

    2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000029_0

    2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000029_0

    2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000029_0 -> (-1, -1) from n1

    2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000011_0

    2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000020_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000026_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000011_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000020_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000011_0 -> (-1, -1) from n7

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000020_0 -> (-1, -1) from n2

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000026_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000047_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000026_0 -> (-1, -1) from n5

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000047_0

    2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000047_0 -> (-1, -1) from n3

    2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000022_0

    2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000022_0

    2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000022_0 -> (-1, -1) from n4

    2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000049_0

    2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000049_0

    2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000049_0 -> (-1, -1) from n6

    2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000000_0

    2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000000_0

    2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000000_0 -> (-1, -1) from n8

    2009-04-20 00:52:59,029 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
    dup hosts)

    2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000052_0

    2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000052_0

    2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000074_0

    2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000052_0 -> (-1, -1) from n2

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000074_0

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000074_0 -> (-1, -1) from n5

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000017_0

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000017_0

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000017_0 -> (-1, -1) from n6

    2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000038_0

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000056_0

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000038_0

    2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000056_0

    2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000038_0 -> (-1, -1) from n4

    2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000056_0 -> (-1, -1) from n8

    2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000023_0

    2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000023_0

    2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000023_0 -> (-1, -1) from n3

    2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000013_0

    2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000013_0

    2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000013_0 -> (-1, -1) from n1

    2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000027_0

    2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000027_0

    2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000027_0 -> (-1, -1) from n7

    2009-04-20 00:53:01,033 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
    dup hosts)

    2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000055_0

    2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000055_0

    2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000055_0 -> (-1, -1) from n3

    2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000025_0

    2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000025_0

    2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000025_0 -> (-1, -1) from n6

    2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000035_0

    2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000035_0

    2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000035_0 -> (-1, -1) from n7

    2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000021_0

    2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000021_0

    2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000021_0 -> (-1, -1) from n1

    2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
    2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000034_0

    2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
    bytes from map-output for attempt_200904181818_0020_m_000034_0

    2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
    from attempt_200904181818_0020_m_000034_0 -> (-1, -1) from n5

    2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 copy failed:
    attempt_200904181818_0020_m_000044_0 from n2

    2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 copy failed:
    attempt_200904181818_0020_m_000054_0 from n4

    2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200904181818_0020_r_000002_1 copy failed:
    attempt_200904181818_0020_m_000032_0 from n8

    2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
    java.io.FileNotFoundException:
    http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
    _0020_m_000044_0&reduce=2

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)

    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
    sorImpl.java:39)

    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
    torAccessorImpl.java:27)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

    at
    sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
    0)

    at java.security.AccessController.doPrivileged(Native Method)

    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
    ction.java:1354)

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
    .java:1008)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
    eam(ReduceTask.java:1293)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
    t(ReduceTask.java:1231)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
    ReduceTask.java:1144)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
    ask.java:1084)

    Caused by: java.io.FileNotFoundException:
    http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
    _0020_m_000044_0&reduce=2

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
    .java:1303)

    ... 4 more



    2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
    java.io.FileNotFoundException:
    http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
    _0020_m_000032_0&reduce=2

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)

    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
    sorImpl.java:39)

    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
    torAccessorImpl.java:27)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

    at
    sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
    0)

    at java.security.AccessController.doPrivileged(Native Method)

    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
    ction.java:1354)

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
    .java:1008)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
    eam(ReduceTask.java:1293)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
    t(ReduceTask.java:1231)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
    ReduceTask.java:1144)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
    ask.java:1084)

    Caused by: java.io.FileNotFoundException:
    http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
    _0020_m_000032_0&reduce=2

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
    .java:1303)

    ... 4 more



    2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
    java.io.FileNotFoundException:
    http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
    _0020_m_000054_0&reduce=2

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)

    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
    sorImpl.java:39)

    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
    torAccessorImpl.java:27)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

    at
    sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
    0)

    at java.security.AccessController.doPrivileged(Native Method)

    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
    ction.java:1354)

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
    .java:1008)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
    eam(ReduceTask.java:1293)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
    t(ReduceTask.java:1231)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
    ReduceTask.java:1144)

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
    ask.java:1084)

    Caused by: java.io.FileNotFoundException:
    http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
    _0020_m_000054_0&reduce=2

    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
    .java:1303)

    ... 4 more

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedApr 24, '09 at 6:07a
activeApr 24, '09 at 8:46a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Jothi Padmanabhan: 1 post Brady Chang: 1 post

People

Translate

site design / logo © 2022 Grokbase