FAQ
I installed hadoop on my workstation today (pseudo distributed mode)
using the instructions:
http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

The installation is straightforward, but then when i try to run the
example, all the map tasks complete,
but the reduce task never completes. It looks like there is some
connection problems??

I made sure that the user 'viking' running both the hadoop java
processes and the job can ssh into the machine
without needed keys/passphrase.


From looking at the code, in ReduceTask.java, it looks like the
reduce task is trying to contact the TaskTracker, but from the syslog
below,
that seems to fail:
URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
"/mapOutput?job=" +
taskId.getJobID() +
"&map=" + taskId +
"&reduce=" + getPartition());

My TaskTracker is listening on port 50060 and i can access the url.

What can be wrong???


Thanks,

S.


# Test ran:
viking@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar
grep input output 'dfs[a-z.]+'
09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to
process : 10
09/10/15 18:50:00 INFO mapred.JobClient: Running job:
job_200910151506_0001
09/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0%
09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0%
09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0%
09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0%
09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0%
09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0%
09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0%
09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0%
09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0%
09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0%
09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0%

# Syslog from the reduce task
syslog logs
2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=668126400,
MaxSingleShuffleLimit=167031600
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread started: Thread for
merging on-disk files
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread started: Thread for
merging in memory files
2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for
merging on-disk files
2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Need another 11 map output(s)
where 0 is already in progress
2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts
and0 dup hosts)
2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200910151506_0001_r_000000_0 copy failed:
attempt_200910151506_0001_m_000000_0 from stephane-desktop
2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.SocketTimeoutException: connect timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun
.reflect
.NativeConstructorAccessorImpl
.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun
.reflect
.DelegatingConstructorAccessorImpl
.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at sun.net.www.protocol.http.HttpURLConnection
$6.run(HttpURLConnection.java:1360)
at java.security.AccessController.doPrivileged(Native Method)
at
sun
.net
.www
.protocol
.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
at
sun
.net
.www
.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:
1008)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.getInputStream(ReduceTask.java:1293)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.copyOutput(ReduceTask.java:1144)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$MapOutputCopier.run(ReduceTask.java:1084)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun
.net
.www
.protocol
.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
at
sun
.net
.www
.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:
793)
at
sun
.net
.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
at
sun
.net
.www
.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:
1041)
... 4 more

Search Discussions

  • Stanislaw Vasiljev at Oct 16, 2009 at 4:09 pm
    Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
    logs. They can contain map-reduce related errors.

    2009/10/16 Stephane Brossier <stephanebrossier@gmail.com>:
    I installed hadoop on my workstation today (pseudo distributed mode) using
    the instructions:
    http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

    The installation is straightforward, but then when i try to run the example,
    all the map tasks complete,
    but the reduce task never completes. It looks like there is some connection
    problems??

    I made sure that the user 'viking' running both the hadoop java processes
    and the job can ssh into the machine
    without needed keys/passphrase.


    From looking at the code, in ReduceTask.java, it looks like the reduce task
    is trying to contact the TaskTracker, but from the syslog below,
    that seems to fail:
    URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
    "/mapOutput?job=" + taskId.getJobID() +
    "&map=" + taskId +
    "&reduce=" + getPartition());

    My TaskTracker is listening on port 50060 and i can access the url.

    What can be wrong???


    Thanks,

    S.


    # Test ran:
    viking@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar grep
    input output 'dfs[a-z.]+'
    09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to process
    : 10
    09/10/15 18:50:00 INFO mapred.JobClient: Running job: job_200910151506_0001
    09/10/15 18:50:01 INFO mapred.JobClient:  map 0% reduce 0%
    09/10/15 18:50:07 INFO mapred.JobClient:  map 9% reduce 0%
    09/10/15 18:50:08 INFO mapred.JobClient:  map 18% reduce 0%
    09/10/15 18:50:09 INFO mapred.JobClient:  map 27% reduce 0%
    09/10/15 18:50:10 INFO mapred.JobClient:  map 36% reduce 0%
    09/10/15 18:50:11 INFO mapred.JobClient:  map 54% reduce 0%
    09/10/15 18:50:12 INFO mapred.JobClient:  map 63% reduce 0%
    09/10/15 18:50:13 INFO mapred.JobClient:  map 72% reduce 0%
    09/10/15 18:50:14 INFO mapred.JobClient:  map 81% reduce 0%
    09/10/15 18:50:15 INFO mapred.JobClient:  map 90% reduce 0%
    09/10/15 18:50:16 INFO mapred.JobClient:  map 100% reduce 0%

    # Syslog from the reduce task
    syslog logs
    2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=167031600
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging
    on-disk files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging in
    memory files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for merging
    on-disk files
    2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Need another 11 map output(s) where
    is already in progress
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
    dup hosts)
    2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 copy failed:
    attempt_200910151506_0001_m_000000_0 from stephane-desktop
    2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New(HttpClient.java:323)
    at
    sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
    at
    sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
    at
    sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
    ... 4 more
  • Huy Phan at Oct 16, 2009 at 4:17 pm
    Can you also post your configuration files here ?
    Stanislaw Vasiljev wrote:
    Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
    logs. They can contain map-reduce related errors.

    2009/10/16 Stephane Brossier <stephanebrossier@gmail.com>:
    I installed hadoop on my workstation today (pseudo distributed mode) using
    the instructions:
    http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

    The installation is straightforward, but then when i try to run the example,
    all the map tasks complete,
    but the reduce task never completes. It looks like there is some connection
    problems??

    I made sure that the user 'viking' running both the hadoop java processes
    and the job can ssh into the machine
    without needed keys/passphrase.


    From looking at the code, in ReduceTask.java, it looks like the reduce task
    is trying to contact the TaskTracker, but from the syslog below,
    that seems to fail:
    URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
    "/mapOutput?job=" + taskId.getJobID() +
    "&map=" + taskId +
    "&reduce=" + getPartition());

    My TaskTracker is listening on port 50060 and i can access the url.

    What can be wrong???


    Thanks,

    S.


    # Test ran:
    viking@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar grep
    input output 'dfs[a-z.]+'
    09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to process
    : 10
    09/10/15 18:50:00 INFO mapred.JobClient: Running job: job_200910151506_0001
    09/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0%
    09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0%
    09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0%
    09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0%
    09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0%
    09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0%
    09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0%
    09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0%
    09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0%
    09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0%
    09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0%

    # Syslog from the reduce task
    syslog logs
    2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=167031600
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging
    on-disk files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging in
    memory files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for merging
    on-disk files
    2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Need another 11 map output(s) where 0
    is already in progress
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
    dup hosts)
    2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 copy failed:
    attempt_200910151506_0001_m_000000_0 from stephane-desktop
    2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1354)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New(HttpClient.java:323)
    at
    sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
    at
    sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
    at
    sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
    ... 4 more

  • Stephane Brossier at Oct 16, 2009 at 9:09 pm
    Stan, Huy,

    Thanks for your answer. Let me know if i can send you more information
    than that. I am bit clueless on why the reduce task tas
    gets this conection exception. Is my understanding correct regarding
    the failure: Is that the 'reduce task' failing to connect to the
    TaskTracker during the copy of the map output? Where do these map
    output live? Are they on disk somewhere?

    Thanks,

    S.


    The configuration i am using is:

    hadoop-site.xml -- default suggested on the web site:

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>


    I did not modify the hadoop-default.xml


    As for the errors in the logs/userlogs, i see the following-- over and
    over:

    2009-10-15 23:54:33,118 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 23:54:33,186 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread started: Thread for
    merging on-disk files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread started: Thread for
    merging in memory files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread waiting: Thread for
    merging on-disk files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Need another 11 map output(s)
    where 0 is already in progress
    2009-10-15 23:54:33,200 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000000_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000001_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000002_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000003_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000004_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000005_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000006_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000007_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000008_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000009_0'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000010_0'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000000_1'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 23 new map-outputs
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 12 obsolete map-outputs from
    tasktracker
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow hosts
    and0 dup hosts)
    2009-10-15 23:55:03,251 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 copy failed:
    attempt_200910152108_0001_m_000001_1 from stephane-desktop
    2009-10-15 23:55:03,252 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
    sun
    .reflect
    .NativeConstructorAccessorImpl
    .newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun
    .reflect
    .DelegatingConstructorAccessorImpl
    .newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:
    513)
    at sun.net.www.protocol.http.HttpURLConnection
    $6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException
    (HttpURLConnection.java:1354)
    at
    sun
    .net
    .www
    .protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:
    1008)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
    at sun.net.www.http.HttpClient.(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New(HttpClient.java:323)
    at
    sun
    .net
    .www
    .protocol
    .http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
    at
    sun
    .net
    .www
    .protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:
    793)
    at
    sun
    .net
    .www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:718)
    at
    sun
    .net
    .www
    .protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:
    1041)
    ... 4 more

    2009-10-15 23:55:04,196 INFO org.apache.hadoop.mapred.ReduceTask: Task
    attempt_200910152108_0001_r_000000_3: Failed fetch #1 from
    attempt_200910152108_0001_m_000001_1
    2009-10-15 23:55:04,196 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 adding host stephane-desktop to
    penalty box, next contact in 4 seconds
    2009-10-15 23:55:05,196 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 1 map-outputs from previous
    failures
    2009-10-15 23:55:10,196 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow hosts
    and0 dup hosts)
    2009-10-15 23:55:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Need another 11 map output(s)
    where 1 is already in progress
    2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 0 new map-outputs
    2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 0 outputs (0 slow hosts
    and9 dup hosts)
    2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 copy failed:
    attempt_200910152108_0001_m_000001_1 from stephane-desktop
    2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    On Oct 16, 2009, at 9:09 AM, Stanislaw Vasiljev wrote:

    Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
    logs. They can contain map-reduce related errors.

    2009/10/16 Stephane Brossier <stephanebrossier@gmail.com>:
    I installed hadoop on my workstation today (pseudo distributed
    mode) using
    the instructions:
    http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

    The installation is straightforward, but then when i try to run the
    example,
    all the map tasks complete,
    but the reduce task never completes. It looks like there is some
    connection
    problems??

    I made sure that the user 'viking' running both the hadoop java
    processes
    and the job can ssh into the machine
    without needed keys/passphrase.


    From looking at the code, in ReduceTask.java, it looks like the
    reduce task
    is trying to contact the TaskTracker, but from the syslog below,
    that seems to fail:
    URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
    "/mapOutput?job=" +
    taskId.getJobID() +
    "&map=" + taskId +
    "&reduce=" + getPartition());

    My TaskTracker is listening on port 50060 and i can access the url.

    What can be wrong???


    Thanks,

    S.


    # Test ran:
    viking@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar
    grep
    input output 'dfs[a-z.]+'
    09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to
    process
    : 10
    09/10/15 18:50:00 INFO mapred.JobClient: Running job:
    job_200910151506_0001
    09/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0%
    09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0%
    09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0%
    09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0%
    09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0%
    09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0%
    09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0%
    09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0%
    09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0%
    09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0%
    09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0%

    # Syslog from the reduce task
    syslog logs
    2009-10-15 18:50:16,212 INFO
    org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=668126400,
    MaxSingleShuffleLimit=167031600
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for
    merging
    on-disk files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for
    merging in
    memory files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for
    merging
    on-disk files
    2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Need another 11 map output(s)
    where 0
    is already in progress
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow
    hosts and0
    dup hosts)
    2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 copy failed:
    attempt_200910151506_0001_m_000000_0 from stephane-desktop
    2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun
    .reflect
    .NativeConstructorAccessorImpl
    .newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun
    .reflect
    .DelegatingConstructorAccessorImpl
    .newInstance(DelegatingConstructorAccessorImpl.java:27)
    at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    sun.net.www.protocol.http.HttpURLConnection
    $6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    sun
    .net
    .www
    .protocol
    .http.HttpURLConnection.getChainedException(HttpURLConnection.java:
    1354)
    at
    sun
    .net
    .www
    .protocol
    .http.HttpURLConnection.getInputStream(HttpURLConnection.java:1008)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
    333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New(HttpClient.java:323)
    at
    sun
    .net
    .www
    .protocol
    .http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:852)
    at
    sun
    .net
    .www
    .protocol
    .http.HttpURLConnection.plainConnect(HttpURLConnection.java:793)
    at
    sun
    .net
    .www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:
    718)
    at
    sun
    .net
    .www
    .protocol
    .http.HttpURLConnection.getInputStream(HttpURLConnection.java:1041)
    ... 4 more
  • Huy Phan at Oct 17, 2009 at 1:13 am
    Hi Stephane,
    Your configuration file is so simple so it cannot cause any problems here :)
    What is see in the log is that you're using 'stephane-desktop' as the
    hostname of your machine, I guess Hadoop was failed to resolve the IP
    from this hostname.

    Regards,
    Huy Phan

    Stephane Brossier wrote:
    Stan, Huy,

    Thanks for your answer. Let me know if i can send you more information
    than that. I am bit clueless on why the reduce task tas
    gets this conection exception. Is my understanding correct regarding
    the failure: Is that the 'reduce task' failing to connect to the
    TaskTracker during the copy of the map output? Where do these map
    output live? Are they on disk somewhere?

    Thanks,

    S.


    The configuration i am using is:

    *hadoop-site.xm*l -- default suggested on the web site:

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>


    I did not modify the hadoop-default.xml


    As for the errors in the logs/userlogs, i see the following-- over and
    over:

    2009-10-15 23:54:33,118 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 23:54:33,186 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread started: Thread for
    merging on-disk files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread started: Thread for
    merging in memory files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread waiting: Thread for
    merging on-disk files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Need another 11 map output(s)
    where 0 is already in progress
    2009-10-15 23:54:33,200 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000000_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000001_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000002_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000003_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000004_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000005_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000006_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000007_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000008_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000009_0'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000010_0'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000000_1'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 23 new map-outputs
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 12 obsolete map-outputs from
    tasktracker
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow hosts
    and0 dup hosts)
    2009-10-15 23:55:03,251 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 copy failed:
    attempt_200910152108_0001_m_000001_1 from stephane-desktop
    2009-10-15 23:55:03,252 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection
    <http://www.protocol.http.HttpURLConnection>$6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException
    <http://www.protocol.http.HttpURLConnection.getChainedException>(HttpURLConnection.java:1354)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream
    <http://www.protocol.http.HttpURLConnection.getInputStream>(HttpURLConnection.java:1008)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer
    <http://www.http.HttpClient.openServer>(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer
    <http://www.http.HttpClient.openServer>(HttpClient.java:529)
    at sun.net.www.http.HttpClient
    <http://www.http.HttpClient>.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New
    <http://www.http.HttpClient.New>(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New
    <http://www.http.HttpClient.New>(HttpClient.java:323)
    at
    sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient
    <http://www.protocol.http.HttpURLConnection.getNewHttpClient>(HttpURLConnection.java:852)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect
    <http://www.protocol.http.HttpURLConnection.plainConnect>(HttpURLConnection.java:793)
    at sun.net.www.protocol.http.HttpURLConnection.connect
    <http://www.protocol.http.HttpURLConnection.connect>(HttpURLConnection.java:718)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream
    <http://www.protocol.http.HttpURLConnection.getInputStream>(HttpURLConnection.java:1041)
    ... 4 more

    2009-10-15 23:55:04,196 INFO org.apache.hadoop.mapred.ReduceTask: Task
    attempt_200910152108_0001_r_000000_3: Failed fetch #1 from
    attempt_200910152108_0001_m_000001_1
    2009-10-15 23:55:04,196 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 adding host stephane-desktop to
    penalty box, next contact in 4 seconds
    2009-10-15 23:55:05,196 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 1 map-outputs from previous
    failures
    2009-10-15 23:55:10,196 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow hosts
    and0 dup hosts)
    2009-10-15 23:55:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Need another 11 map output(s)
    where 1 is already in progress
    2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 0 new map-outputs
    2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 0 outputs (0 slow hosts
    and9 dup hosts)
    2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 copy failed:
    attempt_200910152108_0001_m_000001_1 from stephane-desktop
    2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    On Oct 16, 2009, at 9:09 AM, Stanislaw Vasiljev wrote:

    Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
    logs. They can contain map-reduce related errors.

    2009/10/16 Stephane Brossier <stephanebrossier@gmail.com
    I installed hadoop on my workstation today (pseudo distributed mode)
    using
    the instructions:
    http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

    The installation is straightforward, but then when i try to run the
    example,
    all the map tasks complete,
    but the reduce task never completes. It looks like there is some
    connection
    problems??

    I made sure that the user 'viking' running both the hadoop java
    processes
    and the job can ssh into the machine
    without needed keys/passphrase.


    From looking at the code, in ReduceTask.java, it looks like the
    reduce task
    is trying to contact the TaskTracker, but from the syslog below,
    that seems to fail:
    URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
    "/mapOutput?job=" +
    taskId.getJobID() +
    "&map=" + taskId +
    "&reduce=" + getPartition());

    My TaskTracker is listening on port 50060 and i can access the url.

    What can be wrong???


    Thanks,

    S.


    # Test ran:
    viking@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-examples.jar grep
    input output 'dfs[a-z.]+'
    09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths to
    process
    : 10
    09/10/15 18:50:00 INFO mapred.JobClient: Running job:
    job_200910151506_0001
    09/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0%
    09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0%
    09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0%
    09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0%
    09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0%
    09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0%
    09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0%
    09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0%
    09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0%
    09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0%
    09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0%

    # Syslog from the reduce task
    syslog logs
    2009-10-15 18:50:16,212 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=668126400,
    MaxSingleShuffleLimit=167031600
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for merging
    on-disk files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for
    merging in
    memory files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for merging
    on-disk files
    2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Need another 11 map output(s)
    where 0
    is already in progress
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow
    hosts and0
    dup hosts)
    2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 copy failed:
    attempt_200910151506_0001_m_000000_0 from stephane-desktop
    2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    sun.net.www.protocol.http.HttpURLConnection
    <http://www.protocol.http.HttpURLConnection>$6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException
    <http://www.protocol.http.HttpURLConnection.getChainedException>(HttpURLConnection.java:1354)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream
    <http://www.protocol.http.HttpURLConnection.getInputStream>(HttpURLConnection.java:1008)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer
    <http://www.http.HttpClient.openServer>(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer
    <http://www.http.HttpClient.openServer>(HttpClient.java:529)
    at sun.net.www.http.HttpClient
    <http://www.http.HttpClient>.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New
    <http://www.http.HttpClient.New>(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New
    <http://www.http.HttpClient.New>(HttpClient.java:323)
    at
    sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient
    <http://www.protocol.http.HttpURLConnection.getNewHttpClient>(HttpURLConnection.java:852)
    at
    sun.net.www.protocol.http.HttpURLConnection.plainConnect
    <http://www.protocol.http.HttpURLConnection.plainConnect>(HttpURLConnection.java:793)
    at
    sun.net.www.protocol.http.HttpURLConnection.connect
    <http://www.protocol.http.HttpURLConnection.connect>(HttpURLConnection.java:718)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream
    <http://www.protocol.http.HttpURLConnection.getInputStream>(HttpURLConnection.java:1041)
    ... 4 more
  • Stephane Brossier at Oct 17, 2009 at 2:22 am
    Huy, Stanislaw,

    Thanks a lot for your help. In the meantime i recompiled hadoop to add
    more traces and figured out the issue.
    My workstation IP had changed and the the entry in /etc/passwd' had
    not been updated correctly so the Reduce
    task could not contact the TaskTracker.

    I can now proceed with real MR jobs and start having some fun:-)

    Thanks again for your replies, i appreciate your help.

    S.

    On Oct 16, 2009, at 6:13 PM, Huy Phan wrote:

    Hi Stephane,
    Your configuration file is so simple so it cannot cause any problems
    here :)
    What is see in the log is that you're using 'stephane-desktop' as
    the hostname of your machine, I guess Hadoop was failed to resolve
    the IP from this hostname.

    Regards,
    Huy Phan

    Stephane Brossier wrote:
    Stan, Huy,

    Thanks for your answer. Let me know if i can send you more
    information than that. I am bit clueless on why the reduce task tas
    gets this conection exception. Is my understanding correct
    regarding the failure: Is that the 'reduce task' failing to connect
    to the TaskTracker during the copy of the map output? Where do
    these map output live? Are they on disk somewhere?

    Thanks,

    S.

    The configuration i am using is:

    *hadoop-site.xm*l -- default suggested on the web site:

    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>


    I did not modify the hadoop-default.xml


    As for the errors in the logs/userlogs, i see the following-- over
    and over:

    2009-10-15 23:54:33,118 INFO
    org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics
    with processName=SHUFFLE, sessionId=
    2009-10-15 23:54:33,186 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=130514944,
    MaxSingleShuffleLimit=32628736
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread started: Thread for
    merging on-disk files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread started: Thread for
    merging in memory files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Thread waiting: Thread for
    merging on-disk files
    2009-10-15 23:54:33,192 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Need another 11 map output(s)
    where 0 is already in progress
    2009-10-15 23:54:33,200 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000000_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000001_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000002_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000003_0'
    2009-10-15 23:54:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000004_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000005_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000006_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000007_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000008_0'
    2009-10-15 23:54:33,202 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000009_0'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000010_0'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    Ignoring obsolete output of FAILED map-task:
    'attempt_200910152108_0001_m_000000_1'
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 23 new map-outputs
    2009-10-15 23:54:33,203 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 12 obsolete map-outputs
    from tasktracker 2009-10-15 23:54:33,203 INFO
    org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow
    hosts and0 dup hosts)
    2009-10-15 23:55:03,251 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 copy failed:
    attempt_200910152108_0001_m_000001_1 from stephane-desktop
    2009-10-15 23:55:03,252 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
    sun
    .reflect
    .NativeConstructorAccessorImpl
    .newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun
    .reflect
    .DelegatingConstructorAccessorImpl
    .newInstance(DelegatingConstructorAccessorImpl.java:27)
    at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at sun.net.www.protocol.http.HttpURLConnection <http://www.protocol.http.HttpURLConnection
    $6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException
    <http://
    www
    .protocol
    .http.HttpURLConnection.getChainedException>(HttpURLConnection.java:
    1354)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream
    (HttpURLConnection.java:1008)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
    333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer
    (HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer
    (HttpClient.java:529)
    at sun.net.www.http.HttpClient <http://
    www.http.HttpClient>.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New
    (HttpClient.java:306)
    at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New
    (HttpClient.java:323)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient
    <http://
    www
    .protocol
    .http.HttpURLConnection.getNewHttpClient>(HttpURLConnection.java:852)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect <http://www.protocol.http.HttpURLConnection.plainConnect
    (HttpURLConnection.java:793)
    at sun.net.www.protocol.http.HttpURLConnection.connect <http://www.protocol.http.HttpURLConnection.connect
    (HttpURLConnection.java:718)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream
    (HttpURLConnection.java:1041)
    ... 4 more

    2009-10-15 23:55:04,196 INFO org.apache.hadoop.mapred.ReduceTask:
    Task attempt_200910152108_0001_r_000000_3: Failed fetch #1 from
    attempt_200910152108_0001_m_000001_1
    2009-10-15 23:55:04,196 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 adding host stephane-desktop
    to penalty box, next contact in 4 seconds
    2009-10-15 23:55:05,196 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 1 map-outputs from
    previous failures
    2009-10-15 23:55:10,196 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 1 outputs (0 slow
    hosts and0 dup hosts)
    2009-10-15 23:55:33,201 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Need another 11 map output(s)
    where 1 is already in progress
    2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3: Got 0 new map-outputs
    2009-10-15 23:55:34,200 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 Scheduled 0 outputs (0 slow
    hosts and9 dup hosts)
    2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910152108_0001_r_000000_3 copy failed:
    attempt_200910152108_0001_m_000001_1 from stephane-desktop
    2009-10-15 23:55:40,228 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    On Oct 16, 2009, at 9:09 AM, Stanislaw Vasiljev wrote:

    Can you check $HADOOP_HOME/logs/userlogs/ subdirectories for attempt
    logs. They can contain map-reduce related errors.

    2009/10/16 Stephane Brossier <stephanebrossier@gmail.com >> >>:
    I installed hadoop on my workstation today (pseudo distributed
    mode) using
    the instructions:
    http://hadoop.apache.org/common/docs/r0.19.1/quickstart.html#Download

    The installation is straightforward, but then when i try to run
    the example,
    all the map tasks complete,
    but the reduce task never completes. It looks like there is some
    connection
    problems??

    I made sure that the user 'viking' running both the hadoop java
    processes
    and the job can ssh into the machine
    without needed keys/passphrase.


    From looking at the code, in ReduceTask.java, it looks like the
    reduce task
    is trying to contact the TaskTracker, but from the syslog below,
    that seems to fail:
    URL mapOutputLocation = new URL(event.getTaskTrackerHttp() +
    "/mapOutput?job=" +
    taskId.getJobID() +
    "&map=" + taskId +
    "&reduce=" + getPartition());

    My TaskTracker is listening on port 50060 and i can access the url.

    What can be wrong???


    Thanks,

    S.


    # Test ran:
    viking@stephane-desktop:~/hadoop$ hadoop jar hadoop-*-
    examples.jar grep
    input output 'dfs[a-z.]+'
    09/10/15 18:50:00 INFO mapred.FileInputFormat: Total input paths
    to process
    : 10
    09/10/15 18:50:00 INFO mapred.JobClient: Running job:
    job_200910151506_0001
    09/10/15 18:50:01 INFO mapred.JobClient: map 0% reduce 0%
    09/10/15 18:50:07 INFO mapred.JobClient: map 9% reduce 0%
    09/10/15 18:50:08 INFO mapred.JobClient: map 18% reduce 0%
    09/10/15 18:50:09 INFO mapred.JobClient: map 27% reduce 0%
    09/10/15 18:50:10 INFO mapred.JobClient: map 36% reduce 0%
    09/10/15 18:50:11 INFO mapred.JobClient: map 54% reduce 0%
    09/10/15 18:50:12 INFO mapred.JobClient: map 63% reduce 0%
    09/10/15 18:50:13 INFO mapred.JobClient: map 72% reduce 0%
    09/10/15 18:50:14 INFO mapred.JobClient: map 81% reduce 0%
    09/10/15 18:50:15 INFO mapred.JobClient: map 90% reduce 0%
    09/10/15 18:50:16 INFO mapred.JobClient: map 100% reduce 0%

    # Syslog from the reduce task
    syslog logs
    2009-10-15 18:50:16,212 INFO
    org.apache.hadoop.metrics.jvm.JvmMetrics:
    Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2009-10-15 18:50:16,281 INFO org.apache.hadoop.mapred.ReduceTask:
    ShuffleRamManager: MemoryLimit=668126400,
    MaxSingleShuffleLimit=167031600
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for
    merging
    on-disk files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread started: Thread for
    merging in
    memory files
    2009-10-15 18:50:16,287 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Thread waiting: Thread for
    merging
    on-disk files
    2009-10-15 18:50:16,288 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Need another 11 map
    output(s) where 0
    is already in progress
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0: Got 11 new map-outputs
    2009-10-15 18:50:16,295 INFO org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 Scheduled 1 outputs (0 slow
    hosts and0
    dup hosts)
    2009-10-15 18:50:46,354 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200910151506_0001_r_000000_0 copy failed:
    attempt_200910151506_0001_m_000000_0 from stephane-desktop
    2009-10-15 18:50:46,355 WARN org.apache.hadoop.mapred.ReduceTask:
    java.net.SocketTimeoutException: connect timed out
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun
    .reflect
    .NativeConstructorAccessorImpl
    .newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun
    .reflect
    .DelegatingConstructorAccessorImpl
    .newInstance(DelegatingConstructorAccessorImpl.java:27)
    at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    sun.net.www.protocol.http.HttpURLConnection <http://www.protocol.http.HttpURLConnection
    $6.run(HttpURLConnection.java:1360)
    at java.security.AccessController.doPrivileged(Native Method)
    at
    sun.net.www.protocol.http.HttpURLConnection.getChainedException <http://www.protocol.http.HttpURLConnection.getChainedException
    (HttpURLConnection.java:1354)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream
    (HttpURLConnection.java:1008)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getInputStream(ReduceTask.java:1293)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.getMapOutput(ReduceTask.java:1231)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.copyOutput(ReduceTask.java:1144)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $MapOutputCopier.run(ReduceTask.java:1084)
    Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
    333)
    at
    java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:519)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:158)
    at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer
    (HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer <http://www.http.HttpClient.openServer
    (HttpClient.java:529)
    at sun.net.www.http.HttpClient <http://
    www.http.HttpClient>.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New
    (HttpClient.java:306)
    at sun.net.www.http.HttpClient.New <http://www.http.HttpClient.New
    (HttpClient.java:323)
    at
    sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient <http://www.protocol.http.HttpURLConnection.getNewHttpClient
    (HttpURLConnection.java:852)
    at
    sun.net.www.protocol.http.HttpURLConnection.plainConnect <http://www.protocol.http.HttpURLConnection.plainConnect
    (HttpURLConnection.java:793)
    at
    sun.net.www.protocol.http.HttpURLConnection.connect <http://www.protocol.http.HttpURLConnection.connect
    (HttpURLConnection.java:718)
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.protocol.http.HttpURLConnection.getInputStream
    (HttpURLConnection.java:1041)
    ... 4 more

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 16, '09 at 3:08p
activeOct 17, '09 at 2:22a
posts6
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase