FAQ
I don't know why I can't see my emails immediately sent to the group ...
anyways,

I'm sorting a sequenceFile using it's sorter on my local filesystem. The
inputFile size is 1937690478 bytes.

but after 14 minutes of sorting.. I get :

TEST SORTING ..
java.io.FileNotFoundException: File does not exist:
/usr/mark/tmp/mapred/local/SortedOutput.0
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:676)
at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1353)
at
org.apache.hadoop.io.SequenceFile$Sorter.cloneFileAttributes(SequenceFile.java:2663)
at
org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:2712)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2285)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2324)
at
CrossPartitionSimilarity.TestSorter(CrossPartitionSimilarity.java:164)
at CrossPartitionSimilarity.main(CrossPartitionSimilarity.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


Yet, the file is still there: wc -c SortedOutput.0 ---> 1918661230
../tmp/mapred/local/SortedOutput.0
and if it is because of space, I checked and it can hold up to 209 GB. So,
my question are there restrictions on some JVM configurations that I should
take care of ?

Thank you,
Maha

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 4 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 28, '11 at 8:46p
activeMay 2, '11 at 4:56p
posts4
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase