Grokbase Groups HBase dev July 2012
FAQ
Hi,

I'm trying to get a feel to see how affected folks would be if we
potentially only had hbase support a hadoop 1.0 build and a hadoop 2.0
build profile (and perhaps a hadoop 3.0-SNAPSHOT proflile). Specifically,
does anyone use hbase on top of hadoop 0.21.x, 0.22.x, or 0.23.x (which
became hadoop 2.0.x recently)

Please speak up if so!

Thanks,
Jon.

--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

Search Discussions

  • Ted Yu at Jul 18, 2012 at 8:12 pm
    We use hadoop 0.22

    Cheers
    On Wed, Jul 18, 2012 at 12:58 PM, Jonathan Hsieh wrote:

    Hi,

    I'm trying to get a feel to see how affected folks would be if we
    potentially only had hbase support a hadoop 1.0 build and a hadoop 2.0
    build profile (and perhaps a hadoop 3.0-SNAPSHOT proflile). Specifically,
    does anyone use hbase on top of hadoop 0.21.x, 0.22.x, or 0.23.x (which
    became hadoop 2.0.x recently)

    Please speak up if so!

    Thanks,
    Jon.

    --
    // Jonathan Hsieh (shay)
    // Software Engineer, Cloudera
    // jon@cloudera.com
  • Ramkrishna.S.Vasudevan at Jul 19, 2012 at 4:03 am
    We work on hadoop 2.0.

    Regards
    Ram
    -----Original Message-----
    From: Ted Yu
    Sent: Thursday, July 19, 2012 1:42 AM
    To: dev@hbase.apache.org
    Cc: user@hbase.apache.org
    Subject: Re: [poll] Does anyone run or test against hadoop 0.21, 0.22,
    0.23 under HBase 0.92.0+/0.94.0?

    We use hadoop 0.22

    Cheers
    On Wed, Jul 18, 2012 at 12:58 PM, Jonathan Hsieh wrote:

    Hi,

    I'm trying to get a feel to see how affected folks would be if we
    potentially only had hbase support a hadoop 1.0 build and a hadoop 2.0
    build profile (and perhaps a hadoop 3.0-SNAPSHOT proflile).
    Specifically,
    does anyone use hbase on top of hadoop 0.21.x, 0.22.x, or 0.23.x (which
    became hadoop 2.0.x recently)

    Please speak up if so!

    Thanks,
    Jon.

    --
    // Jonathan Hsieh (shay)
    // Software Engineer, Cloudera
    // jon@cloudera.com
  • Andrew Purtell at Jul 18, 2012 at 8:46 pm
    Works for me, 1.0 and 2.0 only. Not 3.0.
    On Wed, Jul 18, 2012 at 12:58 PM, Jonathan Hsieh wrote:
    Hi,

    I'm trying to get a feel to see how affected folks would be if we
    potentially only had hbase support a hadoop 1.0 build and a hadoop 2.0
    build profile (and perhaps a hadoop 3.0-SNAPSHOT proflile). Specifically,
    does anyone use hbase on top of hadoop 0.21.x, 0.22.x, or 0.23.x (which
    became hadoop 2.0.x recently)

    Please speak up if so!

    Thanks,
    Jon.

    --
    // Jonathan Hsieh (shay)
    // Software Engineer, Cloudera
    // jon@cloudera.com


    --
    Best regards,

    - Andy

    Problems worthy of attack prove their worth by hitting back. - Piet
    Hein (via Tom White)
  • Michael Stack at Jul 19, 2012 at 12:09 am

    On Wed, Jul 18, 2012 at 9:58 PM, Jonathan Hsieh wrote:
    Hi,

    I'm trying to get a feel to see how affected folks would be if we
    potentially only had hbase support a hadoop 1.0 build and a hadoop 2.0
    build profile (and perhaps a hadoop 3.0-SNAPSHOT proflile). Specifically,
    does anyone use hbase on top of hadoop 0.21.x, 0.22.x, or 0.23.x (which
    became hadoop 2.0.x recently)
    Jon, what versions of HBase are you asking about? A while back we
    said that HBase 0.96.0 would be the first release that could assume
    for a minimum a 1.0.x API [1]. Previous releases we postulated
    (though probably haven't tested much) could be run on versions as old
    as branch-0.20-append. We might want to reexamine this "decision" if
    we we are talking HBase 0.94 (or even HBase 0.92)?

    +1 on hadoop 1.0 and 2.0 versions/APIs only. I'd be fine w/ a 3.0 snapshot.

    Any more versions than this and we'll be spending all dev effort on a
    shim layer that looks like the Pompidou Center to make the various
    HBase versions run (slowly) over all 57 varieties of Hadoop.

    St.Ack

    1. http://search-hadoop.com/m/5qFYe24EsUb2/hbase+0.96+hadoop+1.0&subj=Re+DISCUSS+Have+hbase+require+at+least+hadoop+1+0+0+in+hbase+0+96+0+
  • Jonathan Hsieh at Jul 19, 2012 at 12:24 am
    inline
    On Wed, Jul 18, 2012 at 5:09 PM, Stack wrote:
    On Wed, Jul 18, 2012 at 9:58 PM, Jonathan Hsieh wrote:
    Hi,

    I'm trying to get a feel to see how affected folks would be if we
    potentially only had hbase support a hadoop 1.0 build and a hadoop 2.0
    build profile (and perhaps a hadoop 3.0-SNAPSHOT proflile).
    Specifically,
    does anyone use hbase on top of hadoop 0.21.x, 0.22.x, or 0.23.x (which
    became hadoop 2.0.x recently)
    Jon, what versions of HBase are you asking about? A while back we
    said that HBase 0.96.0 would be the first release that could assume
    for a minimum a 1.0.x API [1]. Previous releases we postulated
    (though probably haven't tested much) could be run on versions as old
    as branch-0.20-append. We might want to reexamine this "decision" if
    we we are talking HBase 0.94 (or even HBase 0.92)?

    Its in the the title -- I was asking about specifically about hbase
    0.92.0+/0.94.0+. I should have also included 0.20-append in the list of
    hadoops.

    I don't think this affects the decision you refer to. I'm not trying to age
    off older versions -- I'm trying find out from this discussion which
    "newer" versions we should have HBase work/compile against, and if we can
    just say 1.x and 2.x and not include the 0.21, 0.22, and now 0.23 branches.


    We've found out along the way that while different Hadoop versions are
    "source-compatible" (no source change but requires recompile), they are not
    "binary-compatible" (one hbase jar for all hadoop versions).

    As examples, an hbase compiled against a hadoop 1.0 hdfs doesn't work ontop
    of a hadoop 2.0 hdfs, and that a hbase compiled against hadoop 2.0 hdfs
    doesn't work on a hadoop 1.0 cluster. To add to the mix, today we learned
    that an hbase compiled against a hadoop 23 doesn't work on a hadoop 2.0
    hdfs cluster.

    +1 on hadoop 1.0 and 2.0 versions/APIs only. I'd be fine w/ a 3.0
    snapshot.

    Any more versions than this and we'll be spending all dev effort on a
    shim layer that looks like the Pompidou Center to make the various
    HBase versions run (slowly) over all 57 varieties of Hadoop.

    St.Ack

    1.
    http://search-hadoop.com/m/5qFYe24EsUb2/hbase+0.96+hadoop+1.0&subj=Re+DISCUSS+Have+hbase+require+at+least+hadoop+1+0+0+in+hbase+0+96+0+


    --
    // Jonathan Hsieh (shay)
    // Software Engineer, Cloudera
    // jon@cloudera.com
  • Michael Stack at Jul 19, 2012 at 12:35 am

    On Thu, Jul 19, 2012 at 2:23 AM, Jonathan Hsieh wrote:
    I don't think this affects the decision you refer to. I'm not trying to age
    off older versions -- I'm trying find out from this discussion which
    "newer" versions we should have HBase work/compile against, and if we can
    just say 1.x and 2.x and not include the 0.21, 0.22, and now 0.23 branches.
    When you say HBase 0.92/0.94 above, do you mean the next releases from
    the 0.92 and 0.94 branches (0.92.2 and 0.94.2?) would be tested
    against hadoop 1.x and 2.x only (If the result of this discussion is a
    decision that hadoop 1.0 and 2.0 are all we support)?

    St.Ack
  • Jonathan Hsieh at Jul 19, 2012 at 1:11 am

    On Wed, Jul 18, 2012 at 5:35 PM, Stack wrote:
    On Thu, Jul 19, 2012 at 2:23 AM, Jonathan Hsieh wrote:
    I don't think this affects the decision you refer to. I'm not trying to age
    off older versions -- I'm trying find out from this discussion which
    "newer" versions we should have HBase work/compile against, and if we can
    just say 1.x and 2.x and not include the 0.21, 0.22, and now 0.23 branches.
    When you say HBase 0.92/0.94 above, do you mean the next releases from
    the 0.92 and 0.94 branches (0.92.2 and 0.94.2?) would be tested
    against hadoop 1.x and 2.x only (If the result of this discussion is a
    decision that hadoop 1.0 and 2.0 are all we support)?

    I'd +1 that. I believe the decision regarding hadoop 1.0 as the min
    version in 0.96 really means that we could remove shim code currently
    present to support the older 0.20-append branch then (hflush vs hsync,
    etc). The fact that our pom doesn't contain the 0.20-append branch on
    0.92+ basically says to me that we aren't really testing against that at
    all these days.

    Today we only really run unit tests on hadoop 1.0 -- we have a hadoop 23
    compile check that is done and if that build fails on jenkins no alarms are
    raised (that said, the builds against 1.0.0 are better but still in pretty
    rough shape). Once we fix the last of the hadoop 2.0 unit tests
    uptream I'd like to turn on the hudson builds to build and test trunk
    against hadoop 2.0. As it stands today, trunk needs HBASE-5876 -- which
    needs to be renamed to be hadoop 2.0; 0.94 would needs
    +HBASE-5966,HBASE-5985 which are already on trunk, and 0.92 need 5966, 5985
    and two I believe 2 others but does not need HBASE-5876.

    Jon.


    --
    // Jonathan Hsieh (shay)
    // Software Engineer, Cloudera
    // jon@cloudera.com
  • Michael Stack at Jul 19, 2012 at 9:23 am

    On Thu, Jul 19, 2012 at 3:09 AM, Jonathan Hsieh wrote:
    I'd +1 that. I believe the decision regarding hadoop 1.0 as the min
    version in 0.96 really means that we could remove shim code currently
    present to support the older 0.20-append branch then (hflush vs hsync,
    etc).
    Let that be our answer going forward; customization's go into the
    hadoop compatibility modules in 0.96 forward.
    The fact that our pom doesn't contain the 0.20-append branch on
    0.92+ basically says to me that we aren't really testing against that at
    all these days.
    Agreed. But if someone showed up w/ a patch to fix a bug in
    branch-0.20-append, we'd take it on.
    Today we only really run unit tests on hadoop 1.0 -- we have a hadoop 23
    compile check that is done and if that build fails on jenkins no alarms are
    raised (that said, the builds against 1.0.0 are better but still in pretty
    rough shape). Once we fix the last of the hadoop 2.0 unit tests
    uptream I'd like to turn on the hudson builds to build and test trunk
    against hadoop 2.0. As it stands today, trunk needs HBASE-5876 -- which
    needs to be renamed to be hadoop 2.0; 0.94 would needs
    +HBASE-5966,HBASE-5985 which are already on trunk, and 0.92 need 5966, 5985
    and two I believe 2 others but does not need HBASE-5876.
    Thanks for keeping an account Jon. Lets see how this discussion
    finishes up then write up how we go forward.

    St.Ack

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedJul 18, '12 at 7:59p
activeJul 19, '12 at 9:23a
posts9
users5
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase