FAQ
hi all:
I uses two computers A and B as a hadoop cluster,A is JobTracker and NameNode,both A and B are slaves.
The input data size is about 80MB,including 100,000records. The job is to read one record a time and find some useful content in it,and transmit it to reduce.
But when I submit it ,it just run map tasks,and the reduce task did not run at all!This is the JobTacker`s log file:
2008-05-13 21:02:00,007 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805132055_0001_m_000000_0: FSError: java.io.IOException: No space left on device
2008-05-13 21:02:11,952 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805132055_0001_m_000001_0: FSError: java.io.IOException: No space left on device
2008-05-13 21:02:11,953 INFO org.apache.hadoop.mapred.TaskInProgress: Error from task_200805132055_0001_m_000006_0: FSError: java.io.IOException: No space left on device
I think it is something wrong with the configuration file,Can you give me some suggestions?
Do you meet the same questions?

Search Discussions

  • Otis Gospodnetic at May 13, 2008 at 3:16 pm
    It appears that your hard disk is full on one of your 2 slaves, that is all. If you are on UNIX/linux, type this at the prompt:
    df

    You should see 100% for the partition where you put HDFS.


    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

    ----- Original Message ----
    From: wangxiaowei <wangxiaowei@software.ict.ac.cn>
    To: core-user@hadoop.apache.org
    Sent: Tuesday, May 13, 2008 11:12:36 AM
    Subject: why it stopped at Reduce phase?

    hi all:
    I uses two computers A and B as a hadoop cluster,A is JobTracker and
    NameNode,both A and B are slaves.
    The input data size is about 80MB,including 100,000records. The job is to read
    one record a time and find some useful content in it,and transmit it to reduce.
    But when I submit it ,it just run map tasks,and the reduce task did not run
    at all!This is the JobTacker`s log file:
    2008-05-13 21:02:00,007 INFO org.apache.hadoop.mapred.TaskInProgress: Error from
    task_200805132055_0001_m_000000_0: FSError: java.io.IOException: No space left
    on device
    2008-05-13 21:02:11,952 INFO org.apache.hadoop.mapred.TaskInProgress: Error from
    task_200805132055_0001_m_000001_0: FSError: java.io.IOException: No space left
    on device
    2008-05-13 21:02:11,953 INFO org.apache.hadoop.mapred.TaskInProgress: Error from
    task_200805132055_0001_m_000006_0: FSError: java.io.IOException: No space left
    on device
    I think it is something wrong with the configuration file,Can you give me
    some suggestions?
    Do you meet the same questions?
  • Arun C Murthy at May 13, 2008 at 3:23 pm
    Wang,
    On May 13, 2008, at 8:12 AM, wangxiaowei wrote:

    hi all:
    I uses two computers A and B as a hadoop cluster,A is JobTracker
    and NameNode,both A and B are slaves.
    The input data size is about 80MB,including 100,000records. The job
    is to read one record a time and find some useful content in it,and
    transmit it to reduce.
    But when I submit it ,it just run map tasks,and the reduce task
    did not run at all!This is the JobTacker`s log file:
    2008-05-13 21:02:00,007 INFO
    org.apache.hadoop.mapred.TaskInProgress: Error from
    task_200805132055_0001_m_000000_0: FSError: java.io.IOException: No
    space left on device
    2008-05-13 21:02:11,952 INFO
    org.apache.hadoop.mapred.TaskInProgress: Error from
    task_200805132055_0001_m_000001_0: FSError: java.io.IOException: No
    space left on device
    2008-05-13 21:02:11,953 INFO
    org.apache.hadoop.mapred.TaskInProgress: Error from
    task_200805132055_0001_m_000006_0: FSError: java.io.IOException: No
    space left on device
    I think it is something wrong with the configuration file,Can
    you give me some suggestions?
    Do you meet the same questions?
    It looks like your maps are failing since there isn't enough
    temporary disk space on your nodes.
    Usually the outputs of maps are stored on the local-disk of the nodes
    on which they were executed, and it looks like your nodes don't have
    enough.
    You can use multiple disks for your intermediate map-outputs by using
    the configuration knob: *mapred.local.dir*
    (http://hadoop.apache.org/core/docs/current/hadoop-
    default.html#mapred.local.dir).

    Arun
  • James Moore at May 13, 2008 at 7:39 pm
    Also, hadoop.tmp.dir defaults to storing files in
    /tmp/hadoop-{username} - it's very common for machines to not have a
    lot of space available in /tmp. You'll probably want to change that
    location.

    --

    James Moore | james@restphone.com
    blog.restphone.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 13, '08 at 3:13p
activeMay 13, '08 at 7:39p
posts4
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase