[Hadoop-common-user] "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1

I suppose I should add an actual question: Is this a bug or are we
relying on a "feature" that has since been fixed?

I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
below), though its an EOFException. Does anyone have any idea about what
it means and how it can be fixed?

2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
MemMerge(ReduceTask.java:2147)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
ReduceTask.java:2078)
Caused by: java.lang.RuntimeException: java.io.EOFException
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
at
org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
at
org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
at
org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
MemMerge(ReduceTask.java:2123)
... 1 more
Caused by: java.io.EOFException
at
java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
at
com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
ey.java:40)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
97)
... 7 more

2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
an exception: java.io.IOException: java.lang.RuntimeException:
java.io.EOFException
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
at
org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
at
org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
va:226)
at
org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
eTask.java:2035)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
at
com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
ey.java:40)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
97)
... 7 more

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
eTask.java:2039)

2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
running child
java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)




My WritableComparable class looks like this -




public class IncrementalURLIndexKey implements WritableComparable {
private UTF8 url;
private long userid;

public IncrementalURLIndexKey() {
}

public IncrementalURLIndexKey(UTF8 url, long userid) {
this.url = url;
this.userid = userid;
}

public UTF8 getUrl() {
return url;
}

public long getUserid() {
return userid;
}

public void write(DataOutput out) throws IOException {
IOUtil.writeUTF8(out, url);
out.writeLong(userid);
}

public void readFields(DataInput in) throws IOException {
url = IOUtil.readUTF8(in, url);
userid = in.readLong();
}

public int compareTo(Object o) {
IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
int result = url.compareTo(other.getUrl());
if (result == 0) result = CUID.compare(userid, other.userid);
return result;
}

/**
* A Comparator optimized for IncrementalURLIndexKey.
*/
public static class GroupingComparator extends WritableComparator {
public GroupingComparator() {
super(IncrementalURLIndexKey.class, true);
}

public int compare(WritableComparable a, WritableComparable b) {
IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;

return key1.getUrl().compareTo(key2.getUrl());
}
}

static {
WritableComparator.define(IncrementalURLIndexKey.class, new
GroupingComparator());
}
}


Thanks,
Deepika

-----Original Message-----
From: Grant Ingersoll
Sent: Monday, October 27, 2008 7:14 PM
To: core-user@hadoop.apache.org
Subject: Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1

I suppose I should add an actual question: Is this a bug or are we
relying on a "feature" that has since been fixed?
http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
n+exception

Quick question (I haven't looked at your comparator code yet) - is this
reproducible/consistent?

Hi Devraj,

It was pretty consistent with my comparator class in my old email(the
one that uses UTF8). While trying to resolve the issue, I changed UTF8
to Text. That made it disappear for a while but then it came back again.
My new Comparator class(with Text) is -

public class IncrementalURLIndexKey implements WritableComparable {
private Text url;
private long userid;

public IncrementalURLIndexKey() {
}

public IncrementalURLIndexKey(Text url, long userid) {
this.url = url;
this.userid = userid;
}

public Text getUrl() {
return url;
}

public long getUserid() {
return userid;
}

public void write(DataOutput out) throws IOException {
url.write(out);
out.writeLong(userid);
}

public void readFields(DataInput in) throws IOException {
url = new Text();
url.readFields(in);
userid = in.readLong();
}

public int compareTo(Object o) {
IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
int result = url.compareTo(other.getUrl());
if (result == 0) result = CUID.compare(userid, other.userid);
return result;
}

/**
* A Comparator optimized for IncrementalURLIndexKey.
*/
public static class GroupingComparator extends WritableComparator {
public GroupingComparator() {
super(IncrementalURLIndexKey.class, true);
}

public int compare(WritableComparable a, WritableComparable b) {
IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;

return key1.getUrl().compareTo(key2.getUrl());
}
}

static {
WritableComparator.define(IncrementalURLIndexKey.class, new
GroupingComparator());
}
}

Thanks,
Deepika


-----Original Message-----
From: Devaraj Das
Sent: Tuesday, October 28, 2008 9:01 PM
To: core-user@hadoop.apache.org
Subject: Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1

Quick question (I haven't looked at your comparator code yet) - is this
reproducible/consistent?

bytes
org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a

If you are sure that this isn't caused by your application-logic, you
could try running with http://issues.apache.org/jira/browse/HADOOP-4277.

That bug caused many a ship to sail in large circles, hopelessly.

Arun

We'll try it out...
--------------------------
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com


Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

So, Philippe reports that the problem goes away with 0.20-dev
(trunk?): http://mahout.markmail.org/message/swmzreg6fnzf6icv We
aren't totally clear on the structure of SVN for Hadoop, but it seems
like it is not fixed by this patch.

Wow, if the issue is fixed with version 0.20, then could we please have
a patch for version 0.18?

Thanks,
Deepika

-----Original Message-----
From: Grant Ingersoll
Sent: Thursday, October 30, 2008 12:19 PM
To: core-user@hadoop.apache.org
Subject: Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1

So, Philippe reports that the problem goes away with 0.20-dev
(trunk?): http://mahout.markmail.org/message/swmzreg6fnzf6icv We
aren't totally clear on the structure of SVN for Hadoop, but it seems
like it is not fixed by this patch.


http://issues.apache.org/jira/browse/HADOOP-4277
http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
n+exception