FAQ
Hi,

Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with some
of our clustering code and Hadoop 0.18.1. The thread in context is
at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

The problem seems to occur when going from 0.17.2 to 0.18.1. In the
user logs, we are seeing the following exception:
2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 2 segments left of total size: 5011 bytes
2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810272112_0011_r_000000_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.run(ReduceTask.java:2078)
Caused by: java.lang.NumberFormatException: For input string: "["
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224)
at java.lang.Double.parseDouble(Double.java:510)
at
org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
at
org
.apache.mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:
256)
at
org
.apache
.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
at
org
.apache
.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
at org.apache.hadoop.mapred.ReduceTask
$ReduceCopier.combineAndSpill(ReduceTask.java:2174)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
$3100(ReduceTask.java:341)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

And in the main output log (from running bin/hadoop jar mahout/
examples/build/apache-mahout-examples-0.1-dev.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
attempt_200810272112_0011_r_000000_0, Status : FAILED
java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:2207)

If I run this exact same job on 0.17.2 it all runs fine. I suppose
either a bug was introduced in 0.18.1 or a bug was fixed that we were
relying on. Looking at the release notes between the fixes, nothing
in particular struck me as related. If it helps, I can provide the
instructions for how to run the example in question (they need to be
written up anyway!)


I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
, but those are older, it seems, so not sure what to make of them.

Thanks,
Grant

Search Discussions

  • Grant Ingersoll at Oct 28, 2008 at 2:14 am
    I suppose I should add an actual question: Is this a bug or are we
    relying on a "feature" that has since been fixed?
    On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In the
    user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011 bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    38)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I suppose
    either a bug was introduced in 0.18.1 or a bug was fixed that we
    were relying on. Looking at the release notes between the fixes,
    nothing in particular struck me as related. If it helps, I can
    provide the instructions for how to run the example in question
    (they need to be written up anyway!)


    I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
  • Deepika Khera at Oct 28, 2008 at 6:25 pm
    I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
    below), though its an EOFException. Does anyone have any idea about what
    it means and how it can be fixed?

    2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
    an exception: java.io.IOException: Intermedate merge failed
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
    MemMerge(ReduceTask.java:2147)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
    ReduceTask.java:2078)
    Caused by: java.lang.RuntimeException: java.io.EOFException
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    103)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
    at
    org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
    at
    org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
    at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
    MemMerge(ReduceTask.java:2123)
    ... 1 more
    Caused by: java.io.EOFException
    at
    java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
    at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
    at
    com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
    ey.java:40)
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    97)
    ... 7 more

    2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
    an exception: java.io.IOException: java.lang.RuntimeException:
    java.io.EOFException
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    103)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
    at
    org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
    at
    org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
    va:226)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
    at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
    eTask.java:2035)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:180)
    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
    at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
    at
    com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
    ey.java:40)
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    97)
    ... 7 more

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
    eTask.java:2039)

    2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
    running child
    java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at
    org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)




    My WritableComparable class looks like this -




    public class IncrementalURLIndexKey implements WritableComparable {
    private UTF8 url;
    private long userid;

    public IncrementalURLIndexKey() {
    }

    public IncrementalURLIndexKey(UTF8 url, long userid) {
    this.url = url;
    this.userid = userid;
    }

    public UTF8 getUrl() {
    return url;
    }

    public long getUserid() {
    return userid;
    }

    public void write(DataOutput out) throws IOException {
    IOUtil.writeUTF8(out, url);
    out.writeLong(userid);
    }

    public void readFields(DataInput in) throws IOException {
    url = IOUtil.readUTF8(in, url);
    userid = in.readLong();
    }

    public int compareTo(Object o) {
    IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
    int result = url.compareTo(other.getUrl());
    if (result == 0) result = CUID.compare(userid, other.userid);
    return result;
    }

    /**
    * A Comparator optimized for IncrementalURLIndexKey.
    */
    public static class GroupingComparator extends WritableComparator {
    public GroupingComparator() {
    super(IncrementalURLIndexKey.class, true);
    }

    public int compare(WritableComparable a, WritableComparable b) {
    IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
    IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;

    return key1.getUrl().compareTo(key2.getUrl());
    }
    }

    static {
    WritableComparator.define(IncrementalURLIndexKey.class, new
    GroupingComparator());
    }
    }


    Thanks,
    Deepika

    -----Original Message-----
    From: Grant Ingersoll
    Sent: Monday, October 27, 2008 7:14 PM
    To: core-user@hadoop.apache.org
    Subject: Re: "Merge of the inmemory files threw an exception" and diffs
    between 0.17.2 and 0.18.1

    I suppose I should add an actual question: Is this a bug or are we
    relying on a "feature" that has since been fixed?
    On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In the
    user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011 bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    38)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I suppose
    either a bug was introduced in 0.18.1 or a bug was fixed that we
    were relying on. Looking at the release notes between the fixes,
    nothing in particular struck me as related. If it helps, I can
    provide the instructions for how to run the example in question
    (they need to be written up anyway!)


    I see some related things at
    http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
    n+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
  • Devaraj Das at Oct 29, 2008 at 4:01 am
    Quick question (I haven't looked at your comparator code yet) - is this
    reproducible/consistent?

    On 10/28/08 11:52 PM, "Deepika Khera" wrote:

    I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
    below), though its an EOFException. Does anyone have any idea about what
    it means and how it can be fixed?

    2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
    an exception: java.io.IOException: Intermedate merge failed
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
    MemMerge(ReduceTask.java:2147)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
    ReduceTask.java:2078)
    Caused by: java.lang.RuntimeException: java.io.EOFException
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    103)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
    at
    org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
    at
    org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
    at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
    MemMerge(ReduceTask.java:2123)
    ... 1 more
    Caused by: java.io.EOFException
    at
    java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
    at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
    at
    com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
    ey.java:40)
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    97)
    ... 7 more

    2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
    an exception: java.io.IOException: java.lang.RuntimeException:
    java.io.EOFException
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    103)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
    at
    org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
    at
    org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
    va:226)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
    at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
    eTask.java:2035)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:180)
    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
    at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
    at
    com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
    ey.java:40)
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    97)
    ... 7 more

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
    eTask.java:2039)

    2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
    running child
    java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at
    org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)




    My WritableComparable class looks like this -




    public class IncrementalURLIndexKey implements WritableComparable {
    private UTF8 url;
    private long userid;

    public IncrementalURLIndexKey() {
    }

    public IncrementalURLIndexKey(UTF8 url, long userid) {
    this.url = url;
    this.userid = userid;
    }

    public UTF8 getUrl() {
    return url;
    }

    public long getUserid() {
    return userid;
    }

    public void write(DataOutput out) throws IOException {
    IOUtil.writeUTF8(out, url);
    out.writeLong(userid);
    }

    public void readFields(DataInput in) throws IOException {
    url = IOUtil.readUTF8(in, url);
    userid = in.readLong();
    }

    public int compareTo(Object o) {
    IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
    int result = url.compareTo(other.getUrl());
    if (result == 0) result = CUID.compare(userid, other.userid);
    return result;
    }

    /**
    * A Comparator optimized for IncrementalURLIndexKey.
    */
    public static class GroupingComparator extends WritableComparator {
    public GroupingComparator() {
    super(IncrementalURLIndexKey.class, true);
    }

    public int compare(WritableComparable a, WritableComparable b) {
    IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
    IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;

    return key1.getUrl().compareTo(key2.getUrl());
    }
    }

    static {
    WritableComparator.define(IncrementalURLIndexKey.class, new
    GroupingComparator());
    }
    }


    Thanks,
    Deepika

    -----Original Message-----
    From: Grant Ingersoll
    Sent: Monday, October 27, 2008 7:14 PM
    To: core-user@hadoop.apache.org
    Subject: Re: "Merge of the inmemory files threw an exception" and diffs
    between 0.17.2 and 0.18.1

    I suppose I should add an actual question: Is this a bug or are we
    relying on a "feature" that has since been fixed?
    On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In the
    user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011 bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    38)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I suppose
    either a bug was introduced in 0.18.1 or a bug was fixed that we
    were relying on. Looking at the release notes between the fixes,
    nothing in particular struck me as related. If it helps, I can
    provide the instructions for how to run the example in question
    (they need to be written up anyway!)


    I see some related things at
    http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
    n+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
  • Deepika Khera at Oct 31, 2008 at 9:15 pm
    Hi Devraj,

    It was pretty consistent with my comparator class in my old email(the
    one that uses UTF8). While trying to resolve the issue, I changed UTF8
    to Text. That made it disappear for a while but then it came back again.
    My new Comparator class(with Text) is -

    public class IncrementalURLIndexKey implements WritableComparable {
    private Text url;
    private long userid;

    public IncrementalURLIndexKey() {
    }

    public IncrementalURLIndexKey(Text url, long userid) {
    this.url = url;
    this.userid = userid;
    }

    public Text getUrl() {
    return url;
    }

    public long getUserid() {
    return userid;
    }

    public void write(DataOutput out) throws IOException {
    url.write(out);
    out.writeLong(userid);
    }

    public void readFields(DataInput in) throws IOException {
    url = new Text();
    url.readFields(in);
    userid = in.readLong();
    }

    public int compareTo(Object o) {
    IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
    int result = url.compareTo(other.getUrl());
    if (result == 0) result = CUID.compare(userid, other.userid);
    return result;
    }

    /**
    * A Comparator optimized for IncrementalURLIndexKey.
    */
    public static class GroupingComparator extends WritableComparator {
    public GroupingComparator() {
    super(IncrementalURLIndexKey.class, true);
    }

    public int compare(WritableComparable a, WritableComparable b) {
    IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
    IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;

    return key1.getUrl().compareTo(key2.getUrl());
    }
    }

    static {
    WritableComparator.define(IncrementalURLIndexKey.class, new
    GroupingComparator());
    }
    }

    Thanks,
    Deepika


    -----Original Message-----
    From: Devaraj Das
    Sent: Tuesday, October 28, 2008 9:01 PM
    To: core-user@hadoop.apache.org
    Subject: Re: "Merge of the inmemory files threw an exception" and diffs
    between 0.17.2 and 0.18.1

    Quick question (I haven't looked at your comparator code yet) - is this
    reproducible/consistent?

    On 10/28/08 11:52 PM, "Deepika Khera" wrote:

    I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
    below), though its an EOFException. Does anyone have any idea about what
    it means and how it can be fixed?

    2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
    an exception: java.io.IOException: Intermedate merge failed
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
    MemMerge(ReduceTask.java:2147)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
    ReduceTask.java:2078)
    Caused by: java.lang.RuntimeException: java.io.EOFException
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    103)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
    at
    org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
    at
    org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
    at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
    MemMerge(ReduceTask.java:2123)
    ... 1 more
    Caused by: java.io.EOFException
    at
    java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
    at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
    at
    com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
    ey.java:40)
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    97)
    ... 7 more

    2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
    an exception: java.io.IOException: java.lang.RuntimeException:
    java.io.EOFException
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    103)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
    at
    org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
    at
    org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
    va:226)
    at
    org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
    at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
    eTask.java:2035)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:180)
    at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
    at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
    at
    com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
    ey.java:40)
    at
    org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
    97)
    ... 7 more

    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
    eTask.java:2039)

    2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
    running child
    java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at
    org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)




    My WritableComparable class looks like this -




    public class IncrementalURLIndexKey implements WritableComparable {
    private UTF8 url;
    private long userid;

    public IncrementalURLIndexKey() {
    }

    public IncrementalURLIndexKey(UTF8 url, long userid) {
    this.url = url;
    this.userid = userid;
    }

    public UTF8 getUrl() {
    return url;
    }

    public long getUserid() {
    return userid;
    }

    public void write(DataOutput out) throws IOException {
    IOUtil.writeUTF8(out, url);
    out.writeLong(userid);
    }

    public void readFields(DataInput in) throws IOException {
    url = IOUtil.readUTF8(in, url);
    userid = in.readLong();
    }

    public int compareTo(Object o) {
    IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
    int result = url.compareTo(other.getUrl());
    if (result == 0) result = CUID.compare(userid, other.userid);
    return result;
    }

    /**
    * A Comparator optimized for IncrementalURLIndexKey.
    */
    public static class GroupingComparator extends WritableComparator {
    public GroupingComparator() {
    super(IncrementalURLIndexKey.class, true);
    }

    public int compare(WritableComparable a, WritableComparable b) {
    IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
    IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;

    return key1.getUrl().compareTo(key2.getUrl());
    }
    }

    static {
    WritableComparator.define(IncrementalURLIndexKey.class, new
    GroupingComparator());
    }
    }


    Thanks,
    Deepika

    -----Original Message-----
    From: Grant Ingersoll
    Sent: Monday, October 27, 2008 7:14 PM
    To: core-user@hadoop.apache.org
    Subject: Re: "Merge of the inmemory files threw an exception" and diffs
    between 0.17.2 and 0.18.1

    I suppose I should add an actual question: Is this a bug or are we
    relying on a "feature" that has since been fixed?
    On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In the
    user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011
    bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    38)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
    copier failed
    at
    org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I suppose
    either a bug was introduced in 0.18.1 or a bug was fixed that we
    were relying on. Looking at the release notes between the fixes,
    nothing in particular struck me as related. If it helps, I can
    provide the instructions for how to run the example in question
    (they need to be written up anyway!)


    I see some related things at
    http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
    n+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
  • Arun C Murthy at Oct 28, 2008 at 7:03 pm

    On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In the
    user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011 bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    If you are sure that this isn't caused by your application-logic, you
    could try running with http://issues.apache.org/jira/browse/HADOOP-4277.

    That bug caused many a ship to sail in large circles, hopelessly.

    Arun
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    38)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I suppose
    either a bug was introduced in 0.18.1 or a bug was fixed that we
    were relying on. Looking at the release notes between the fixes,
    nothing in particular struck me as related. If it helps, I can
    provide the instructions for how to run the example in question
    (they need to be written up anyway!)


    I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
  • Grant Ingersoll at Oct 29, 2008 at 2:29 pm
    We'll try it out...
    On Oct 28, 2008, at 3:00 PM, Arun C Murthy wrote:

    On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In
    the user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011
    bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    If you are sure that this isn't caused by your application-logic,
    you could try running with http://issues.apache.org/jira/browse/HADOOP-4277
    .

    That bug caused many a ship to sail in large circles, hopelessly.

    Arun
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:
    60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    38)
    at
    org
    .apache
    .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
    31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
    copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I suppose
    either a bug was introduced in 0.18.1 or a bug was fixed that we
    were relying on. Looking at the release notes between the fixes,
    nothing in particular struck me as related. If it helps, I can
    provide the instructions for how to run the example in question
    (they need to be written up anyway!)


    I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
    --------------------------
    Grant Ingersoll
    Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
    http://www.lucenebootcamp.com


    Lucene Helpful Hints:
    http://wiki.apache.org/lucene-java/BasicsOfPerformance
    http://wiki.apache.org/lucene-java/LuceneFAQ
  • Grant Ingersoll at Oct 30, 2008 at 7:19 pm
    So, Philippe reports that the problem goes away with 0.20-dev
    (trunk?): http://mahout.markmail.org/message/swmzreg6fnzf6icv We
    aren't totally clear on the structure of SVN for Hadoop, but it seems
    like it is not fixed by this patch.


    On Oct 29, 2008, at 10:28 AM, Grant Ingersoll wrote:

    We'll try it out...
    On Oct 28, 2008, at 3:00 PM, Arun C Murthy wrote:

    On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In
    the user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011
    bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    If you are sure that this isn't caused by your application-logic,
    you could try running with http://issues.apache.org/jira/browse/HADOOP-4277
    .

    That bug caused many a ship to sail in large circles, hopelessly.

    Arun
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:
    60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout
    .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
    at
    org
    .apache
    .mahout
    .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The
    reduce copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I
    suppose either a bug was introduced in 0.18.1 or a bug was fixed
    that we were relying on. Looking at the release notes between the
    fixes, nothing in particular struck me as related. If it helps, I
    can provide the instructions for how to run the example in
    question (they need to be written up anyway!)


    I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
    --------------------------
    Grant Ingersoll
    Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
    http://www.lucenebootcamp.com


    Lucene Helpful Hints:
    http://wiki.apache.org/lucene-java/BasicsOfPerformance
    http://wiki.apache.org/lucene-java/LuceneFAQ




  • Deepika Khera at Nov 1, 2008 at 12:24 am
    Wow, if the issue is fixed with version 0.20, then could we please have
    a patch for version 0.18?

    Thanks,
    Deepika

    -----Original Message-----
    From: Grant Ingersoll
    Sent: Thursday, October 30, 2008 12:19 PM
    To: core-user@hadoop.apache.org
    Subject: Re: "Merge of the inmemory files threw an exception" and diffs
    between 0.17.2 and 0.18.1

    So, Philippe reports that the problem goes away with 0.20-dev
    (trunk?): http://mahout.markmail.org/message/swmzreg6fnzf6icv We
    aren't totally clear on the structure of SVN for Hadoop, but it seems
    like it is not fixed by this patch.


    On Oct 29, 2008, at 10:28 AM, Grant Ingersoll wrote:

    We'll try it out...
    On Oct 28, 2008, at 3:00 PM, Arun C Murthy wrote:

    On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:

    Hi,

    Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
    some of our clustering code and Hadoop 0.18.1. The thread in
    context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr

    The problem seems to occur when going from 0.17.2 to 0.18.1. In
    the user logs, we are seeing the following exception:
    2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
    to the last merge-pass, with 2 segments left of total size: 5011
    bytes
    2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
    attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
    threw an exception: java.io.IOException: Intermedate merge failed
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.run(ReduceTask.java:2078)
    Caused by: java.lang.NumberFormatException: For input string: "["
    If you are sure that this isn't caused by your application-logic,
    you could try running with
    http://issues.apache.org/jira/browse/HADOOP-4277
    .

    That bug caused many a ship to sail in large circles, hopelessly.

    Arun
    at
    sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
    1224)
    at java.lang.Double.parseDouble(Double.java:510)
    at
    org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:
    60)
    at
    org
    .apache
    .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
    at
    org
    .apache
    .mahout
    .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
    at
    org
    .apache
    .mahout
    .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
    at org.apache.hadoop.mapred.ReduceTask
    $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
    $3100(ReduceTask.java:341)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
    $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)

    And in the main output log (from running bin/hadoop jar mahout/
    examples/build/apache-mahout-examples-0.1-dev.job
    org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
    08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
    attempt_200810272112_0011_r_000000_0, Status : FAILED
    java.io.IOException: attempt_200810272112_0011_r_000000_0The
    reduce copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker
    $Child.main(TaskTracker.java:2207)

    If I run this exact same job on 0.17.2 it all runs fine. I
    suppose either a bug was introduced in 0.18.1 or a bug was fixed
    that we were relying on. Looking at the release notes between the
    fixes, nothing in particular struck me as related. If it helps, I
    can provide the instructions for how to run the example in
    question (they need to be written up anyway!)


    I see some related things at
    http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
    n+exception
    , but those are older, it seems, so not sure what to make of them.

    Thanks,
    Grant
    --------------------------
    Grant Ingersoll
    Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
    http://www.lucenebootcamp.com


    Lucene Helpful Hints:
    http://wiki.apache.org/lucene-java/BasicsOfPerformance
    http://wiki.apache.org/lucene-java/LuceneFAQ




Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 28, '08 at 2:06a
activeNov 1, '08 at 12:24a
posts9
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase