FAQ
The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
the 0.20 release series when the "new" (Context Objects) MapReduce API
was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
was not complete in 0.20 and most users stayed with the old API. This
has led to the confusing situation where the old API is generally
recommended, even though it is deprecated.

To remedy this situation I suggest that we remove deprecations from
the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
MAPREDUCE-1623 for the latter). This would mean a few things:

* The next 0.20 release would have a non-deprecated old API.
* The forthcoming 0.21 release would have a "Stable" (non-deprecated)
old API, and a "Evolving" new API.
* For some pre-1.0 release (perhaps 0.22), the old API could be
deprecated again, and the new API marked as "Stable".
* In the 1.0 release it would be possible to remove the old API.

Thoughts?

Tom

Search Discussions

  • Eric Sammer at Apr 22, 2010 at 6:05 am
    +1. Currently there is almost no way to write a moderately complex MR
    job that doesn't spew deprecation warnings. It serves to endlessly
    confuse newcomers to Hadoop.
    On Wed, Apr 21, 2010 at 5:24 PM, Tom White wrote:
    The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
    the 0.20 release series when the "new" (Context Objects) MapReduce API
    was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
    was not complete in 0.20 and most users stayed with the old API. This
    has led to the confusing situation where the old API is generally
    recommended, even though it is deprecated.

    To remedy this situation I suggest that we remove deprecations from
    the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
    MAPREDUCE-1623 for the latter). This would mean a few things:

    * The next 0.20 release would have a non-deprecated old API.
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API.
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    Thoughts?

    Tom


    --
    Eric Sammer
    phone: +1-917-287-2675
    twitter: esammer
    data: www.cloudera.com
  • Owen O'Malley at Apr 22, 2010 at 1:55 pm
    On the various pieces, I think:

    0.20: -0 for removing the deprecation, +1 for improving the
    deprecation message with links to the corresponding class.

    0.21: new core api should be stable except for Job and Cluster
    new library code should be evolving
    -1 for removing the deprecation, we need to

    0.22: all of the new api should be stable and the old api deprecated.
    Currently there is almost no way to write a moderately complex MR job that doesn't spew deprecation warnings.
    That is false in 0.21.

    -- Owen
  • Doug Cutting at Apr 22, 2010 at 4:15 pm

    Owen O'Malley wrote:
    0.21: new core api should be stable except for Job and Cluster
    new library code should be evolving
    -1 for removing the deprecation, we need to
    The Job API is central to the new API, no? Should we encourage
    applications to move to the new API if it's not entirely stable? Can
    you provide a more detailed rationale for deprecating the only
    fully-stable API?

    Thanks,

    Doug
  • Amr Awadallah at Apr 22, 2010 at 6:06 pm
    +1 (mainly to avoid confusing new comers as Eric Sammer correctly
    indicated).
    On 4/22/2010 9:15 AM, Doug Cutting wrote:
    Owen O'Malley wrote:
    0.21: new core api should be stable except for Job and Cluster
    new library code should be evolving
    -1 for removing the deprecation, we need to
    The Job API is central to the new API, no? Should we encourage
    applications to move to the new API if it's not entirely stable? Can
    you provide a more detailed rationale for deprecating the only
    fully-stable API?

    Thanks,

    Doug
  • Amareshwari Sri Ramadasu at Apr 23, 2010 at 4:47 am
    +1 for removing deprecation in 0.20.

    +0 for removing deprecation in 0.21

    Thanks
    Amareshwari


    On 4/22/10 7:25 PM, "Owen O'Malley" wrote:

    On the various pieces, I think:

    0.20: -0 for removing the deprecation, +1 for improving the
    deprecation message with links to the corresponding class.

    0.21: new core api should be stable except for Job and Cluster
    new library code should be evolving
    -1 for removing the deprecation, we need to

    0.22: all of the new api should be stable and the old api deprecated.
    Currently there is almost no way to write a moderately complex MR job that doesn't spew deprecation warnings.
    That is false in 0.21.

    -- Owen
  • Anthony Urso at Apr 22, 2010 at 3:26 pm
    +1
    On Wed, Apr 21, 2010 at 2:24 PM, Tom White wrote:
    The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
    the 0.20 release series when the "new" (Context Objects) MapReduce API
    was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
    was not complete in 0.20 and most users stayed with the old API. This
    has led to the confusing situation where the old API is generally
    recommended, even though it is deprecated.

    To remedy this situation I suggest that we remove deprecations from
    the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
    MAPREDUCE-1623 for the latter). This would mean a few things:

    * The next 0.20 release would have a non-deprecated old API.
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API.
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    Thoughts?

    Tom
  • Arun C Murthy at Apr 22, 2010 at 6:08 pm
    +1 for un-deprecating. I'm just being pragmatic.

    Arun
    On Apr 21, 2010, at 2:24 PM, Tom White wrote:

    The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
    the 0.20 release series when the "new" (Context Objects) MapReduce API
    was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
    was not complete in 0.20 and most users stayed with the old API. This
    has led to the confusing situation where the old API is generally
    recommended, even though it is deprecated.

    To remedy this situation I suggest that we remove deprecations from
    the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
    MAPREDUCE-1623 for the latter). This would mean a few things:

    * The next 0.20 release would have a non-deprecated old API.
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API.
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    Thoughts?

    Tom
  • Arun C Murthy at Apr 22, 2010 at 6:12 pm
    +1 for undeprecating. I think it's just being pragmatic.

    Arun
    On Apr 21, 2010, at 2:26 PM, "Tom White" wrote:

    The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
    the 0.20 release series when the "new" (Context Objects) MapReduce API
    was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
    was not complete in 0.20 and most users stayed with the old API. This
    has led to the confusing situation where the old API is generally
    recommended, even though it is deprecated.

    To remedy this situation I suggest that we remove deprecations from
    the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
    MAPREDUCE-1623 for the latter). This would mean a few things:

    * The next 0.20 release would have a non-deprecated old API.
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API.
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    Thoughts?

    Tom
  • Alan Gates at Apr 22, 2010 at 7:13 pm
    Speaking for one power user (Pig) that did move to the new APIs,
    moving that interface to evolving is a little unsettling. Is there a
    feel for how much the new API is going to change?

    Alan.
    On Apr 21, 2010, at 2:24 PM, Tom White wrote:

    The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
    the 0.20 release series when the "new" (Context Objects) MapReduce API
    was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
    was not complete in 0.20 and most users stayed with the old API. This
    has led to the confusing situation where the old API is generally
    recommended, even though it is deprecated.

    To remedy this situation I suggest that we remove deprecations from
    the old API in 0.20 and trunk, and mark the new API as "Evolving" (see
    MAPREDUCE-1623 for the latter). This would mean a few things:

    * The next 0.20 release would have a non-deprecated old API.
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API.
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    Thoughts?

    Tom
  • Arun C Murthy at Apr 23, 2010 at 6:32 am
    Alan,
    On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:

    Speaking for one power user (Pig) that did move to the new APIs,
    moving that interface to evolving is a little unsettling. Is there
    a feel for how much the new API is going to change?
    The intent isn't to mark the 'new' apis as 'Evolving' to change them
    willy-nilly... please don't read it so!

    This is just a pragmatic proposal to reflect that the 'old' apis
    will, for lack of stabilization of new apis, be supported.

    Given that, the new apis could mostly be stable, but for Job and
    Cluster - is that reasonable? This will ensure we send the right
    message all concerned regarding stability of o.a.h.mapreduce.{Mapper|
    Reducer|...}. Thoughts?

    Arun
    Alan.
    On Apr 21, 2010, at 2:24 PM, Tom White wrote:

    The "old" MapReduce API in org.apache.hadoop.mapred was deprecated in
    the 0.20 release series when the "new" (Context Objects) MapReduce
    API
    was added in org.apache.hadoop.mapreduce. Unfortunately, the new API
    was not complete in 0.20 and most users stayed with the old API. This
    has led to the confusing situation where the old API is generally
    recommended, even though it is deprecated.

    To remedy this situation I suggest that we remove deprecations from
    the old API in 0.20 and trunk, and mark the new API as
    "Evolving" (see
    MAPREDUCE-1623 for the latter). This would mean a few things:

    * The next 0.20 release would have a non-deprecated old API.
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API.
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    Thoughts?

    Tom
  • Alan Gates at Apr 23, 2010 at 5:22 pm
    I don't have any issue with un-deprecating the old APIs. I agree if
    changes are needed it's better to mark the new APIs to reflect it. I
    just hope those changes can be kept as backward compatible as
    possible. In particular with Job, Pig uses that in some of it's APIs
    that it has declared stable (LoadFunc, StoreFunc).

    Alan.
    On Apr 22, 2010, at 11:30 PM, Arun C Murthy wrote:

    Alan,
    On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:

    Speaking for one power user (Pig) that did move to the new APIs,
    moving that interface to evolving is a little unsettling. Is there
    a feel for how much the new API is going to change?
    The intent isn't to mark the 'new' apis as 'Evolving' to change them
    willy-nilly... please don't read it so!

    This is just a pragmatic proposal to reflect that the 'old' apis
    will, for lack of stabilization of new apis, be supported.

    Given that, the new apis could mostly be stable, but for Job and
    Cluster - is that reasonable? This will ensure we send the right
    message all concerned regarding stability of o.a.h.mapreduce.{Mapper|
    Reducer|...}. Thoughts?

    Arun
    Alan.
  • Tom White at Apr 28, 2010 at 5:03 am
    It sounds like there's no strong objection to un-deprecating the old
    API in 0.20 - I'll create a patch for this (see
    https://issues.apache.org/jira/browse/MAPREDUCE-1734).

    0.21 is less clear cut. However, if the new API were marked as
    Evolving, then it's odd, to say the least, if the old API were
    deprecated since it would send a confusing message to users. There
    seems to be consensus that the new API is Evolving (please comment on
    https://issues.apache.org/jira/browse/MAPREDUCE-1623 to discuss
    whether all of the new API should be marked Evolving, which the latest
    patch does). Indeed, the new API hasn't seen widespread use yet, so it
    still seems premature to deprecate the old API in 0.21. I've opened
    https://issues.apache.org/jira/browse/MAPREDUCE-1735 where we can
    discuss this particular case further.

    Cheers,
    Tom
    On Fri, Apr 23, 2010 at 9:21 AM, Alan Gates wrote:
    I don't have any issue with un-deprecating the old APIs.  I agree if changes
    are needed it's better to mark the new APIs to reflect it.  I just hope
    those changes can be kept as backward compatible as possible.  In particular
    with Job, Pig uses that in some of it's APIs that it has declared stable
    (LoadFunc, StoreFunc).

    Alan.
    On Apr 22, 2010, at 11:30 PM, Arun C Murthy wrote:

    Alan,
    On Apr 22, 2010, at 12:12 PM, Alan Gates wrote:

    Speaking for one power user (Pig) that did move to the new APIs, moving
    that interface to evolving is a little unsettling.  Is there a feel for how
    much the new API is going to change?
    The intent isn't to mark the 'new' apis as 'Evolving' to change them
    willy-nilly... please don't read it so!

    This is just a pragmatic proposal to reflect that the 'old' apis will, for
    lack of stabilization of new apis, be supported.

    Given that, the new apis could mostly be stable, but for Job and Cluster -
    is that reasonable? This will ensure we send the right message all concerned
    regarding stability of o.a.h.mapreduce.{Mapper|Reducer|...}. Thoughts?

    Arun
    Alan.
  • Chris K Wensel at Apr 22, 2010 at 7:35 pm

    * The next 0.20 release would have a non-deprecated old API. +1
    * The forthcoming 0.21 release would have a "Stable" (non-deprecated)
    old API, and a "Evolving" new API. +1
    * For some pre-1.0 release (perhaps 0.22), the old API could be
    deprecated again, and the new API marked as "Stable".
    * In the 1.0 release it would be possible to remove the old API.

    thinking this is a different discussion.

    but my preference is that 0.20 become 1.0

    buying time to evolve the evolving apis and build/dependency system over the next year(s) into a 2.0

    this could bite me as I do like some of the changes i had to effect in my cascading wip 2.0 api to support the 'new' apis.

    but a 1.0 will send a very useful message to potential users

    chris

    --
    Chris K Wensel
    chris@concurrentinc.com
    http://www.concurrentinc.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-dev @
categorieshadoop
postedApr 21, '10 at 9:25p
activeApr 28, '10 at 5:03a
posts14
users11
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase