FAQ
hello, list,

Recently I have upgraded our Hadoop cluster from 0.20.2 to
0.21.0, and I found there is something wrong introduced by the upgrade.

In the setConf method of
org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would
be configured twice, and this may cause trouble.
For example, in 0.21.0, KeyFieldBasedPartitioner implements
the Configurable interface. When configured twice, it get two KeyDescription
and gives out wrong partition number.

I have created a ticket for this:
https://issues.apache.org/jira/browse/HADOOP-7425


Paste the source code in 0.21.0 (and 0.20.2 too) below:

public static void setConf(Object theObject, Configuration conf) {
if (conf != null) {
if (theObject instanceof Configurable) {
((Configurable) theObject).setConf(conf);
}
setJobConf(theObject, conf);
}
}



--
best wishes.
steven

Search Discussions

  • Steven zhuang at Jun 29, 2011 at 7:05 am
    Anyone have the same partition problem using Streaming in 0.21.0?


    On Fri, Jun 24, 2011 at 10:22 AM, steven zhuang wrote:

    hello, list,

    Recently I have upgraded our Hadoop cluster from 0.20.2 to
    0.21.0, and I found there is something wrong introduced by the upgrade.

    In the setConf method of
    org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would
    be configured twice, and this may cause trouble.
    For example, in 0.21.0, KeyFieldBasedPartitioner implements
    the Configurable interface. When configured twice, it get two KeyDescription
    and gives out wrong partition number.

    I have created a ticket for this:
    https://issues.apache.org/jira/browse/HADOOP-7425


    Paste the source code in 0.21.0 (and 0.20.2 too) below:

    public static void setConf(Object theObject, Configuration conf) {
    if (conf != null) {
    if (theObject instanceof Configurable) {
    ((Configurable) theObject).setConf(conf);
    }
    setJobConf(theObject, conf);
    }
    }



    --
    best wishes.
    steven

    --
    best wishes.
    steven

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 24, '11 at 2:23a
activeJun 29, '11 at 7:05a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Steven zhuang: 2 posts

People

Translate

site design / logo © 2022 Grokbase