FAQ
Hi,
I am running a map-reduce program on 6-Node ec2 cluster. and after a
couple of hours all my tasks gets hanged.

so i started digging into the logs....

there were no logs for regionserver
no logs for tasktracker.
However for jobtracker i get the following:

2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 50002, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false, true,
10745) from 10.254.27.79:44222: error: java.io.IOException:
org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
space left on device
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at
org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
at java.io.BufferedWriter.close(BufferedWriter.java:248)
at java.io.PrintWriter.close(PrintWriter.java:295)
at
org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
at
org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
at org.apache.hadoop.mapred.JobInProgress.comp



following are the disk information on dfs UI
domU-12-31-39-00-0C-A1<http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.8321.19391.360.2
94.672353 domU-12-31-39-00-16-F1<http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service413.380.4621.24391.670.11
94.752399 domU-12-31-39-00-45-71<http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service413.380.6421.34391.40.16
94.682303 domU-12-31-39-00-E5-D2<http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.6621.53391.180.16
94.632319 domU-12-31-39-01-64-12<http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service413.380.6421.24391.490.16
94.712264 domU-12-31-39-01-78-D1<http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.4921.24391.650.12
94.741952

I m using hadoop 0.19.0 and hbase 0.19.0

n googling the error i came arcoss the JIRA issue
http://issues.apache.org/jira/browse/HADOOP-4163

which says tht its been fixed in this version. :(

Has anyone else come up with this exception?

how do we check the maximum capacity for usable dfs and non usable dfs.
Thanks
Raakhi,

Search Discussions

  • Desai, Milind B at Apr 16, 2009 at 7:50 am
    From the exception it appears that there is no space left on machine. You can check using 'df'
    Thanks
    Milind

    -----Original Message-----
    From: Rakhi Khatwani
    Sent: Thursday, April 16, 2009 1:15 PM
    To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
    Subject: No space left on device Exception

    Hi,
    I am running a map-reduce program on 6-Node ec2 cluster. and after a
    couple of hours all my tasks gets hanged.

    so i started digging into the logs....

    there were no logs for regionserver
    no logs for tasktracker.
    However for jobtracker i get the following:

    2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 50002, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false, true,
    10745) from 10.254.27.79:44222: error: java.io.IOException:
    org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
    java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
    space left on device
    at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at
    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
    at
    org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
    at java.io.BufferedWriter.close(BufferedWriter.java:248)
    at java.io.PrintWriter.close(PrintWriter.java:295)
    at
    org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
    at
    org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
    at org.apache.hadoop.mapred.JobInProgress.comp



    following are the disk information on dfs UI
    domU-12-31-39-00-0C-A1<http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
    Service413.380.8321.19391.360.2
    94.672353 domU-12-31-39-00-16-F1<http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
    Service413.380.4621.24391.670.11
    94.752399 domU-12-31-39-00-45-71<http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
    Service413.380.6421.34391.40.16
    94.682303 domU-12-31-39-00-E5-D2<http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
    Service413.380.6621.53391.180.16
    94.632319 domU-12-31-39-01-64-12<http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
    Service413.380.6421.24391.490.16
    94.712264 domU-12-31-39-01-78-D1<http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
    Service413.380.4921.24391.650.12
    94.741952

    I m using hadoop 0.19.0 and hbase 0.19.0

    n googling the error i came arcoss the JIRA issue
    http://issues.apache.org/jira/browse/HADOOP-4163

    which says tht its been fixed in this version. :(

    Has anyone else come up with this exception?

    how do we check the maximum capacity for usable dfs and non usable dfs.
    Thanks
    Raakhi,
  • Rakhi Khatwani at Apr 16, 2009 at 8:07 am
    Hi,
    following is the output on the df command
    [root@domU-12-31-39-00-E5-D2 conf]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda1 9.9G 4.2G 5.2G 45% /
    /dev/sdb 414G 924M 392G 1% /mnt

    from the o/p it seems that i have quite an amount of memory available. but i
    still get the exception :(

    Thanks
    Raakhi
    On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B wrote:

    From the exception it appears that there is no space left on machine. You
    can check using 'df'

    Thanks
    Milind

    -----Original Message-----
    From: Rakhi Khatwani
    Sent: Thursday, April 16, 2009 1:15 PM
    To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
    Subject: No space left on device Exception

    Hi,
    I am running a map-reduce program on 6-Node ec2 cluster. and after a
    couple of hours all my tasks gets hanged.

    so i started digging into the logs....

    there were no logs for regionserver
    no logs for tasktracker.
    However for jobtracker i get the following:

    2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 50002, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
    true,
    10745) from 10.254.27.79:44222: error: java.io.IOException:
    org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
    java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
    space left on device
    at

    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at

    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at

    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
    at

    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
    at
    org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
    at

    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
    at java.io.BufferedWriter.close(BufferedWriter.java:248)
    at java.io.PrintWriter.close(PrintWriter.java:295)
    at

    org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
    at
    org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
    at org.apache.hadoop.mapred.JobInProgress.comp



    following are the disk information on dfs UI
    domU-12-31-39-00-0C-A1<
    http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.8321.19391.360.2
    94.672353 domU-12-31-39-00-16-F1<
    http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.4621.24391.670.11
    94.752399 domU-12-31-39-00-45-71<
    http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.6421.34391.40.16
    94.682303 domU-12-31-39-00-E5-D2<
    http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.6621.53391.180.16
    94.632319 domU-12-31-39-01-64-12<
    http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    2In
    Service413.380.6421.24391.490.16
    94.712264 domU-12-31-39-01-78-D1<
    http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.4921.24391.650.12
    94.741952

    I m using hadoop 0.19.0 and hbase 0.19.0

    n googling the error i came arcoss the JIRA issue
    http://issues.apache.org/jira/browse/HADOOP-4163

    which says tht its been fixed in this version. :(

    Has anyone else come up with this exception?

    how do we check the maximum capacity for usable dfs and non usable dfs.
    Thanks
    Raakhi,
  • Miles Osborne at Apr 16, 2009 at 8:12 am
    it may be that intermediate results are filling your disks and when
    the jobs crash, this all gets deleted. so it would look like you have
    spare space when in reality you don't.

    i would check on the file system as your jobs run and see if indeed
    they are filling-up.

    Miles

    2009/4/16 Rakhi Khatwani <rakhi.khatwani@gmail.com>:
    Hi,
    following is the output on the df command
    [root@domU-12-31-39-00-E5-D2 conf]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1             9.9G  4.2G  5.2G  45% /
    /dev/sdb              414G  924M  392G   1% /mnt

    from the o/p it seems that i have quite an amount of memory available. but i
    still get the exception :(

    Thanks
    Raakhi
    On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B wrote:

    From the exception it appears that there is no space left on machine. You
    can check using 'df'

    Thanks
    Milind

    -----Original Message-----
    From: Rakhi Khatwani
    Sent: Thursday, April 16, 2009 1:15 PM
    To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
    Subject: No space left on device Exception

    Hi,
    I am running a map-reduce program on 6-Node ec2 cluster. and after a
    couple of hours all my tasks gets hanged.

    so i started digging into the logs....

    there were no logs for regionserver
    no logs for tasktracker.
    However for jobtracker i get the following:

    2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 50002, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
    true,
    10745) from 10.254.27.79:44222: error: java.io.IOException:
    org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
    java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
    space left on device
    at

    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at

    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at

    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
    at

    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
    at
    org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
    at

    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
    at java.io.BufferedWriter.close(BufferedWriter.java:248)
    at java.io.PrintWriter.close(PrintWriter.java:295)
    at

    org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
    at
    org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
    at org.apache.hadoop.mapred.JobInProgress.comp



    following are the disk information on dfs UI
    domU-12-31-39-00-0C-A1<
    http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.8321.19391.360.2
    94.672353 domU-12-31-39-00-16-F1<
    http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.4621.24391.670.11
    94.752399 domU-12-31-39-00-45-71<
    http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.6421.34391.40.16
    94.682303 domU-12-31-39-00-E5-D2<
    http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.6621.53391.180.16
    94.632319 domU-12-31-39-01-64-12<
    http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    2In
    Service413.380.6421.24391.490.16
    94.712264 domU-12-31-39-01-78-D1<
    http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.4921.24391.650.12
    94.741952

    I m using hadoop 0.19.0 and hbase 0.19.0

    n googling the error i came arcoss the JIRA issue
    http://issues.apache.org/jira/browse/HADOOP-4163

    which says tht its been fixed in this version. :(

    Has anyone else come up with this exception?

    how do we check the maximum capacity for usable dfs and non usable dfs.
    Thanks
    Raakhi,


    --
    The University of Edinburgh is a charitable body, registered in
    Scotland, with registration number SC005336.
  • Rakhi Khatwani at Apr 16, 2009 at 8:26 am
    Thanks,
    I will check tht

    Regards,
    Raakhi
    On Thu, Apr 16, 2009 at 1:42 PM, Miles Osborne wrote:

    it may be that intermediate results are filling your disks and when
    the jobs crash, this all gets deleted. so it would look like you have
    spare space when in reality you don't.

    i would check on the file system as your jobs run and see if indeed
    they are filling-up.

    Miles

    2009/4/16 Rakhi Khatwani <rakhi.khatwani@gmail.com>:
    Hi,
    following is the output on the df command
    [root@domU-12-31-39-00-E5-D2 conf]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda1 9.9G 4.2G 5.2G 45% /
    /dev/sdb 414G 924M 392G 1% /mnt

    from the o/p it seems that i have quite an amount of memory available. but i
    still get the exception :(

    Thanks
    Raakhi

    On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <milind.desai@hp.com
    wrote:
    From the exception it appears that there is no space left on machine.
    You
    can check using 'df'

    Thanks
    Milind

    -----Original Message-----
    From: Rakhi Khatwani
    Sent: Thursday, April 16, 2009 1:15 PM
    To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
    Subject: No space left on device Exception

    Hi,
    I am running a map-reduce program on 6-Node ec2 cluster. and after a
    couple of hours all my tasks gets hanged.

    so i started digging into the logs....

    there were no logs for regionserver
    no logs for tasktracker.
    However for jobtracker i get the following:

    2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 50002, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
    true,
    10745) from 10.254.27.79:44222: error: java.io.IOException:
    org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
    device
    java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException:
    No
    space left on device
    at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at
    java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at
    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
    at
    org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
    at
    org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
    at java.io.BufferedWriter.close(BufferedWriter.java:248)
    at java.io.PrintWriter.close(PrintWriter.java:295)
    at
    org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
    at
    org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
    at org.apache.hadoop.mapred.JobInProgress.comp



    following are the disk information on dfs UI
    domU-12-31-39-00-0C-A1<
    http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.8321.19391.360.2
    94.672353 domU-12-31-39-00-16-F1<
    http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.4621.24391.670.11
    94.752399 domU-12-31-39-00-45-71<
    http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.6421.34391.40.16
    94.682303 domU-12-31-39-00-E5-D2<
    http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.6621.53391.180.16
    94.632319 domU-12-31-39-01-64-12<
    http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    2In
    Service413.380.6421.24391.490.16
    94.712264 domU-12-31-39-01-78-D1<
    http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.4921.24391.650.12
    94.741952

    I m using hadoop 0.19.0 and hbase 0.19.0

    n googling the error i came arcoss the JIRA issue
    http://issues.apache.org/jira/browse/HADOOP-4163

    which says tht its been fixed in this version. :(

    Has anyone else come up with this exception?

    how do we check the maximum capacity for usable dfs and non usable dfs.
    Thanks
    Raakhi,


    --
    The University of Edinburgh is a charitable body, registered in
    Scotland, with registration number SC005336.
  • Pankil Doshi at Apr 16, 2009 at 3:16 pm
    Hey

    what's your input size?

    from the info you gave it seems you have used 4.2GB and so probably if thats
    your input size your intermediate results mostly is less then your input.but
    that too depends on your map function. Make sure about the size of
    intermediate results.

    Pankil
    On Thu, Apr 16, 2009 at 3:25 AM, Rakhi Khatwani wrote:

    Thanks,
    I will check tht

    Regards,
    Raakhi
    On Thu, Apr 16, 2009 at 1:42 PM, Miles Osborne wrote:

    it may be that intermediate results are filling your disks and when
    the jobs crash, this all gets deleted. so it would look like you have
    spare space when in reality you don't.

    i would check on the file system as your jobs run and see if indeed
    they are filling-up.

    Miles

    2009/4/16 Rakhi Khatwani <rakhi.khatwani@gmail.com>:
    Hi,
    following is the output on the df command
    [root@domU-12-31-39-00-E5-D2 conf]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda1 9.9G 4.2G 5.2G 45% /
    /dev/sdb 414G 924M 392G 1% /mnt

    from the o/p it seems that i have quite an amount of memory available. but i
    still get the exception :(

    Thanks
    Raakhi

    On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <milind.desai@hp.com
    wrote:
    From the exception it appears that there is no space left on machine.
    You
    can check using 'df'

    Thanks
    Milind

    -----Original Message-----
    From: Rakhi Khatwani
    Sent: Thursday, April 16, 2009 1:15 PM
    To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
    Subject: No space left on device Exception

    Hi,
    I am running a map-reduce program on 6-Node ec2 cluster. and after
    a
    couple of hours all my tasks gets hanged.

    so i started digging into the logs....

    there were no logs for regionserver
    no logs for tasktracker.
    However for jobtracker i get the following:

    2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 9 on 50002, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
    true,
    10745) from 10.254.27.79:44222: error: java.io.IOException:
    org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
    device
    java.io.IOException: org.apache.hadoop.fs.FSError:
    java.io.IOException:
    No
    space left on device
    at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at
    java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at
    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
    at
    org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
    at
    org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
    at
    org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
    at java.io.DataOutputStream.write(DataOutputStream.java:90)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
    at java.io.BufferedWriter.close(BufferedWriter.java:248)
    at java.io.PrintWriter.close(PrintWriter.java:295)
    at
    org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
    at
    org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
    at org.apache.hadoop.mapred.JobInProgress.comp



    following are the disk information on dfs UI
    domU-12-31-39-00-0C-A1<
    http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.8321.19391.360.2
    94.672353 domU-12-31-39-00-16-F1<
    http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.4621.24391.670.11
    94.752399 domU-12-31-39-00-45-71<
    http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    1In
    Service413.380.6421.34391.40.16
    94.682303 domU-12-31-39-00-E5-D2<
    http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.6621.53391.180.16
    94.632319 domU-12-31-39-01-64-12<
    http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    2In
    Service413.380.6421.24391.490.16
    94.712264 domU-12-31-39-01-78-D1<
    http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
    0In
    Service413.380.4921.24391.650.12
    94.741952

    I m using hadoop 0.19.0 and hbase 0.19.0

    n googling the error i came arcoss the JIRA issue
    http://issues.apache.org/jira/browse/HADOOP-4163

    which says tht its been fixed in this version. :(

    Has anyone else come up with this exception?

    how do we check the maximum capacity for usable dfs and non usable
    dfs.
    Thanks
    Raakhi,


    --
    The University of Edinburgh is a charitable body, registered in
    Scotland, with registration number SC005336.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 16, '09 at 7:45a
activeApr 16, '09 at 3:16p
posts6
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase