FAQ
I don't know why I can't see my emails immediately sent to the group ...
anyways,

I'm sorting a sequenceFile using it's sorter on my local filesystem. The
inputFile size is 1937690478 bytes.

but after 14 minutes of sorting.. I get :

TEST SORTING ..
java.io.FileNotFoundException: File does not exist:
/usr/mark/tmp/mapred/local/SortedOutput.0
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:676)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1353)
at
org.apache.hadoop.io.SequenceFile$Sorter.cloneFileAttributes(SequenceFile.java:2663)
at
org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:2712)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2285)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2324)
at
CrossPartitionSimilarity.TestSorter(CrossPartitionSimilarity.java:164)
at CrossPartitionSimilarity.main(CrossPartitionSimilarity.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


Yet, the file is still there: wc -c SortedOutput.0 ---> 1918661230
../tmp/mapred/local/SortedOutput.0
and if it is because of space, I checked and it can hold up to 209 GB. So,
my question are there restrictions on some JVM configurations that I should
take care of ?

Thank you,
Maha

Search Discussions

  • Stanley Shi at Apr 29, 2011 at 2:39 am
    Hi guys, I have a very strange problem here, hope someone can help.

    I have a virtual machine with 8 GB memory, the os is CentOS 5.5 x64 version; I got this VM from someone else (means they may have done some configuration on this VM)
    I cannot start even 6 easiest jvm at the same time (for example, start the following java code simultaneously 6 times:

    import java.io.IOException;
    public class InputTest {
    public static void main(String[] args) throws IOException, InterruptedException {
    int length = 0;
    long sum=0;
    do {
    sum+=length++;
    System.out.println("Current sum: "+ sum);
    Thread.sleep(1000);

    } while (length <1000000);

    }
    }


    Any one have any comments?

    Thanks,
    Stanley
  • Steve Loughran at May 2, 2011 at 4:52 pm

    On 29/04/2011 03:37, stanley.shi@emc.com wrote:
    Hi guys, I have a very strange problem here, hope someone can help.

    I have a virtual machine with 8 GB memory, the os is CentOS 5.5 x64 version; I got this VM from someone else (means they may have done some configuration on this VM)
    I cannot start even 6 easiest jvm at the same time (for example, start the following java code simultaneously 6 times:

    import java.io.IOException;
    public class InputTest {
    public static void main(String[] args) throws IOException, InterruptedException {
    int length = 0;
    long sum=0;
    do {
    sum+=length++;
    System.out.println("Current sum: "+ sum);
    Thread.sleep(1000);

    } while (length<1000000);

    }
    }


    Any one have any comments?

    1. check swap space
    2. you work for a storage company, so get some more...
  • Brian Bockelman at May 2, 2011 at 4:56 pm
    Check the overcommit VM settings on your kernel. These prevent swap from being used on older JVMs, and cause out-of-memory errors to be given by Java even when there is free memory.

    Brian
    On May 2, 2011, at 11:51 AM, Steve Loughran wrote:
    On 29/04/2011 03:37, stanley.shi@emc.com wrote:
    Hi guys, I have a very strange problem here, hope someone can help.

    I have a virtual machine with 8 GB memory, the os is CentOS 5.5 x64 version; I got this VM from someone else (means they may have done some configuration on this VM)
    I cannot start even 6 easiest jvm at the same time (for example, start the following java code simultaneously 6 times:

    import java.io.IOException;
    public class InputTest {
    public static void main(String[] args) throws IOException, InterruptedException {
    int length = 0;
    long sum=0;
    do {
    sum+=length++;
    System.out.println("Current sum: "+ sum);
    Thread.sleep(1000);

    } while (length<1000000);

    }
    }


    Any one have any comments?

    1. check swap space
    2. you work for a storage company, so get some more...

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 28, '11 at 8:46p
activeMay 2, '11 at 4:56p
posts4
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase