FAQ

On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson wrote:
Hey,

There is actually only 1 active branch of hbase, that being the 0.89
release, which is based on 'trunk'.  We have snapshotted a series of
0.89 "developer releases" in hopes that people would try them our and
start thinking about the next major version.  One of these is what SU
is running prod on.

At this point tracking 0.89 and which ones are the 'best' peach sets
to run is a bit of a contact sport, but if you are serious about not
losing data it is worthwhile.  SU is based on the most recent DR with
a few minor patches of our own concoction brought in.  If current
works, but some Master ops are slow, and there are a few patches on
top of that.  I'll poke about and see if its possible to publish to a
github branch or something.
This is why I kind of want to release the latest 0.89 dev release as
0.90, and push off the new master stuff as 0.92 :)

-Todd

--
Todd Lipcon
Software Engineer, Cloudera

Search Discussions

  • Jonathan Gray at Sep 21, 2010 at 11:49 am
    Deep within my soul I do not want to do this.

    But it might be practical.

    FB is going into prod w/ the old master and we've been doing work stabilizing it in ways that do not apply at all to the new one (especially around zk). That'll make having two different active branches a bit of a nightmare but right now some of these patches are not even available as we've been operating under the assumption that the release would be on the new master (and the patches don't apply on trunk).

    I guess SU is also putting an 0.89 old master release into prod.

    What's a little unfortunate is that the 0.89 releases include the unnecessary move of some transition communication into ZK. That was put in as a first step towards new master before we decided to branch it off.

    The question for me would be whether we think it's at all feasible to get the new master 0.90 released in time for HW. If not, maybe we should consider taking an 0.89 branch, making it an 0.90 branch, and focus on stabilizing for release in time for HW, as Todd suggests.

    There are other ramifications of this. It would be very difficult for me to get my flush/compact/split improvements in to the old master as the new implementation I've been working on relies completely on the new stuff. But maybe better to punt that for 0.92 as well so can really nail it?

    The other factor is the enormous amount of ZK improvements that were done in the new master branch. It's a real fuckin mess in 0.89 releases, though I've done a bit of cleanup already towards making it at least tenable for production.

    My main concern is that this move will push back any release of the new master significantly. There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions. But doing a production release in time for HW is probably the most important thing.

    JG
    -----Original Message-----
    From: Todd Lipcon
    Sent: Tuesday, September 21, 2010 8:10 AM
    To: dev
    Subject: Re: Millions of photos into Hbase
    On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson wrote:
    Hey,

    There is actually only 1 active branch of hbase, that being the 0.89
    release, which is based on 'trunk'.  We have snapshotted a series of
    0.89 "developer releases" in hopes that people would try them our and
    start thinking about the next major version.  One of these is what SU
    is running prod on.

    At this point tracking 0.89 and which ones are the 'best' peach sets
    to run is a bit of a contact sport, but if you are serious about not
    losing data it is worthwhile.  SU is based on the most recent DR with
    a few minor patches of our own concoction brought in.  If current
    works, but some Master ops are slow, and there are a few patches on
    top of that.  I'll poke about and see if its possible to publish to a
    github branch or something.
    This is why I kind of want to release the latest 0.89 dev release as
    0.90, and push off the new master stuff as 0.92 :)

    -Todd

    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Jean-Daniel Cryans at Sep 21, 2010 at 4:29 pm
    So while you were in the plane a few things were discussed :)

    In the thread "[VOTE] Release 'development release' HBase 0.89.2010830
    rc2", it was decided that we sink RC2 in favor of a new RC with a
    rollback of the zk-based assignment. This is what we're running here,
    and Ryan published our repo yesterday
    http://github.com/stumbleupon/hbase (see the top of the CHANGES file
    for what we added)

    Also some of your arguments correlate those of Todd, expressed in the
    thread "Next release" that was posted on 09/15. Stack and Andrew
    expressed their opinions in favor of Todd's option #1 (although Stack
    said that he would decide tomorrow if he changes his opinion after
    some more work on the master).

    My personal opinion leans towards Todd's option #2. 0.20.0 is now more
    than a year old, we are able to release almost production-ready code
    that is durable, so why not move forward with it (this is what we did
    here). There are a few issues to fix, but it's small compared to
    getting the new master in shape plus getting the rest working (like
    the replication code's interaction with ZooKeeper that needs to be
    redone).

    J-D
    On Tue, Sep 21, 2010 at 4:49 AM, Jonathan Gray wrote:
    Deep within my soul I do not want to do this.

    But it might be practical.

    FB is going into prod w/ the old master and we've been doing work stabilizing it in ways that do not apply at all to the new one (especially around zk).  That'll make having two different active branches a bit of a nightmare but right now some of these patches are not even available as we've been operating under the assumption that the release would be on the new master (and the patches don't apply on trunk).

    I guess SU is also putting an 0.89 old master release into prod.

    What's a little unfortunate is that the 0.89 releases include the unnecessary move of some transition communication into ZK.  That was put in as a first step towards new master before we decided to branch it off.

    The question for me would be whether we think it's at all feasible to get the new master 0.90 released in time for HW.  If not, maybe we should consider taking an 0.89 branch, making it an 0.90 branch, and focus on stabilizing for release in time for HW, as Todd suggests.

    There are other ramifications of this.  It would be very difficult for me to get my flush/compact/split improvements in to the old master as the new implementation I've been working on relies completely on the new stuff.  But maybe better to punt that for 0.92 as well so can really nail it?

    The other factor is the enormous amount of ZK improvements that were done in the new master branch.  It's a real fuckin mess in 0.89 releases, though I've done a bit of cleanup already towards making it at least tenable for production.

    My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.

    JG
    -----Original Message-----
    From: Todd Lipcon
    Sent: Tuesday, September 21, 2010 8:10 AM
    To: dev
    Subject: Re: Millions of photos into Hbase

    On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson <ryanobjc@gmail.com>
    wrote:
    Hey,

    There is actually only 1 active branch of hbase, that being the 0.89
    release, which is based on 'trunk'.  We have snapshotted a series of
    0.89 "developer releases" in hopes that people would try them our and
    start thinking about the next major version.  One of these is what SU
    is running prod on.

    At this point tracking 0.89 and which ones are the 'best' peach sets
    to run is a bit of a contact sport, but if you are serious about not
    losing data it is worthwhile.  SU is based on the most recent DR with
    a few minor patches of our own concoction brought in.  If current
    works, but some Master ops are slow, and there are a few patches on
    top of that.  I'll poke about and see if its possible to publish to a
    github branch or something.
    This is why I kind of want to release the latest 0.89 dev release as
    0.90, and push off the new master stuff as 0.92 :)

    -Todd

    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Stack at Sep 21, 2010 at 5:12 pm

    On Tue, Sep 21, 2010 at 4:49 AM, Jonathan Gray wrote:
    My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.
    IMO, we're almost there w/ the new master. I'd like to wait till
    Thursday or Friday before making a call.

    Regards HW, we might consider pressing on with new master even if it
    means we don't exactly line up a new master 0.89.x (or a 0.90.xRC)
    release w/ HW. You fellas talking at HW can fudge it some if it comes
    to that (We won't be the only ones. CDH3 will not be ready for HW
    apparently). I'm thinking we can afford to take such a position
    because if someone wants durable hbase now, they can run with the
    SU-prod 0.89.x that J-D is about to put up.

    Putting off new master till 0.92 means it'll be maybe 6 months before
    it appears. During this time we'll be paying a high price keeping up
    two disparate branches -- TRUNK w/ new master and the release branch
    -- shoe-horning patches to fit both.

    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.

    St.Ack
  • Todd Lipcon at Sep 21, 2010 at 5:20 pm

    On Tue, Sep 21, 2010 at 10:11 AM, Stack wrote:
    On Tue, Sep 21, 2010 at 4:49 AM, Jonathan Gray wrote:
    My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.
    IMO, we're almost there w/ the new master.  I'd like to wait till
    Thursday or Friday before making a call.

    Regards HW, we might consider pressing on with new master even if it
    means we don't exactly line up a new master 0.89.x (or a 0.90.xRC)
    release w/ HW.  You fellas talking at HW can fudge it some if it comes
    to that (We won't be the only ones.  CDH3 will not be ready for HW
    apparently).
    CDH3b3 will be ready for Hadoop world, and we'd kind of like to freeze
    component versions at this point in the beta cycle. So if 0.90 is out,
    that would be great. We can certainly work with what we've got
    (20100830 minus ZK assignment) but if a production-worth new master
    isn't ready by the end of October or so we'll probably push that out
    til CDH4.
    I'm thinking we can afford to take such a position
    because if someone wants durable hbase now, they can run with the
    SU-prod 0.89.x that J-D is about to put up.
    This is going to be an official Apache release, right? I guess my
    question is: if it's stable and usable for production, why call it a
    "development release"? If we're recommending it for all new users over
    and above 0.20.6, then it seems like this should be deemed stable (ie
    even release number).
    Putting off new master till 0.92 means it'll be maybe 6 months before
    it appears.  During this time we'll be paying a high price keeping up
    two disparate branches -- TRUNK w/ new master and the release branch
    -- shoe-horning patches to fit both.
    If you guys are running 20100830 in production, won't you be doing
    that anyway? Assumedly we'd treat this 0.90 as "no new features" and
    put the new features into 0.91.x leading up to 0.92?
    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.
    True. The question is whether we prefer to slip time or slip scope. In
    my opinion slipping scope is better - it's open source and people
    understand that schedules slip. Keeping strong release momentum up
    helps adoption and will get people off 0.20.6 which no one really
    wants to support anymore.

    -Todd

    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Stack at Sep 21, 2010 at 6:08 pm

    On Tue, Sep 21, 2010 at 10:19 AM, Todd Lipcon wrote:
    CDH3b3 will be ready for Hadoop world, and we'd kind of like to freeze
    component versions at this point in the beta cycle. So if 0.90 is out,
    that would be great. We can certainly work with what we've got
    (20100830 minus ZK assignment) but if a production-worth new master
    isn't ready by the end of October or so we'll probably push that out
    til CDH4.
    Ok.

    Its my thought that we'd have a production ready master way before end
    of October.

    I'm thinking we can afford to take such a position
    because if someone wants durable hbase now, they can run with the
    SU-prod 0.89.x that J-D is about to put up.
    This is going to be an official Apache release, right? I guess my
    question is: if it's stable and usable for production, why call it a
    "development release"? If we're recommending it for all new users over
    and above 0.20.6, then it seems like this should be deemed stable (ie
    even release number).
    I was not talking of recommending the next 0.89.x to all new users
    over 0.20.6. I thought it plain that running a release in production
    at SU did not mean the release 'stable and useable' by others. SU has
    3 hbase committers aboard to fix and hand-hold the software over rough
    patches.

    I was thinking that the next 0.89.x comes w/ the same caveats as
    previous 0.89.xs, that its a 'developer release' for those willing to
    put up w/ some rough edges and that its just another step on the way
    to 0.90.0.

    Putting off new master till 0.92 means it'll be maybe 6 months before
    it appears.  During this time we'll be paying a high price keeping up
    two disparate branches -- TRUNK w/ new master and the release branch
    -- shoe-horning patches to fit both.
    If you guys are running 20100830 in production, won't you be doing
    that anyway? Assumedly we'd treat this 0.90 as "no new features" and
    put the new features into 0.91.x leading up to 0.92?
    Nope. We'd move to new master release. If it don't work for us, we'd
    feel a little awkward recommending it to others.

    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.
    True. The question is whether we prefer to slip time or slip scope. In
    my opinion slipping scope is better - it's open source and people
    understand that schedules slip. Keeping strong release momentum up
    helps adoption and will get people off 0.20.6 which no one really
    wants to support anymore.
    This is the question. I'm just suggesting that new master MAY not be
    that far out. I want to do another couple of days work on it and then
    have us make a call; i.e. vote that we press on to get new master into
    0.90 or punt on new master for 0.90.

    St.Ack
  • Lars Francke at Sep 21, 2010 at 11:55 pm

    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.
    This might be a little naïve but what about calling the "SU"-build
    0.88 and release it (with a small disclaimer). It would make sense in
    that the 0.89-dev releases have more stuff in them than 0.88 and it
    would still allow for a 0.90 release with all the announced features.

    As a side note: I plan to finish work on the Maven changes in time for
    HW but tests keep on failing (Hudson has the same problem...) which
    doesn't make it easy to completely test the build. I don't know if
    I'll be able to get Thrift done though, sorry. I'm doing this in my
    spare time and I didn't have a lot of that :( but with a bit of luck I
    can work on HBase in my new job. But I'll upload updated Thrift files
    in the next few days if someone wants to check what I've done (modeled
    it after Jeff's Avro work).

    Cheers,
    Lars
  • Stack at Sep 22, 2010 at 5:22 am

    On Tue, Sep 21, 2010 at 4:54 PM, Lars Francke wrote:
    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.
    This might be a little naïve but what about calling the "SU"-build
    0.88 and release it (with a small disclaimer). It would make sense in
    that the 0.89-dev releases have more stuff in them than 0.88 and it
    would still allow for a 0.90 release with all the announced features.
    Thats an idea.
    As a side note: I plan to finish work on the Maven changes in time for
    HW but tests keep on failing (Hudson has the same problem...) which
    doesn't make it easy to completely test the build.
    Sorry about that. I've been picking away at them from time to time
    but some require more than a glance to fix (For me at least, most of
    what fails on hudson does not fail for me here below).

    I don't know if
    I'll be able to get Thrift done though, sorry. I'm doing this in my
    spare time and I didn't have a lot of that :( but with a bit of luck I
    can work on HBase in my new job. But I'll upload updated Thrift files
    in the next few days if someone wants to check what I've done (modeled
    it after Jeff's Avro work).
    Thanks for the update and good luck in the new Job.

    St.Ack
  • Jonathan Gray at Sep 22, 2010 at 12:48 pm

    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.
    This might be a little naïve but what about calling the "SU"-build
    0.88 and release it (with a small disclaimer). It would make sense in
    that the 0.89-dev releases have more stuff in them than 0.88 and it
    would still allow for a 0.90 release with all the announced features.
    I like this too... if we go the direction of an old master release.

    It addresses most of the issues, besides slowing release of new master. So not necessarily +1 on doing it but this would be better than an old master 0.90 release imo.

    Good idea Lars.

    JG
  • Todd Lipcon at Sep 22, 2010 at 5:12 pm

    On Wed, Sep 22, 2010 at 5:47 AM, Jonathan Gray wrote:

    We also confuse the 0.90 'message' given we've been talking about new
    master at HUGs and here on the lists with a good while now.
    This might be a little naïve but what about calling the "SU"-build
    0.88 and release it (with a small disclaimer). It would make sense in
    that the 0.89-dev releases have more stuff in them than 0.88 and it
    would still allow for a 0.90 release with all the announced features.
    I like this too... if we go the direction of an old master release.
    I'm -1 on an out-of-order release like this. I think it will really confuse
    people, please won't work correctly with upgrades on packaging systems.

    I know we've been promising "new master for 0.90" at meetups and stuff, but
    outside the dev team, does anyone really care about the implementation
    details? The feature here isn't a new master, it's more stable master
    failover, better locality, etc. Whether we have that or say that it's going
    to be in the next version, I don't think people will be too upset.

    The argument that it's hell to maintain two branches is a fair one, but
    unless we can get trunk usable very soon now, we're already going to be in
    that situation. People are migrating production instances to 0.89.x now, and
    once people go to production they are very hesitant to upgrade, regardless
    of what we tell them.

    -Todd

    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Andrew Purtell at Sep 22, 2010 at 10:11 pm
    If JG is apprehensive about putting the new master into 0.90, that's good enough for me. But I'll vote +0. I'm too unfamiliar with the details.

    Best regards,

    - Andy

    Why is this email five sentences or less?
    http://five.sentenc.es/
  • Ryan Rawson at Sep 22, 2010 at 10:15 pm
    I've been using the new master a bit, and the improvements are just
    too dramatic to ignore or delay. Instant table transitions, clusters
    that actually shut down, etc, etc.


    On Wed, Sep 22, 2010 at 3:11 PM, Andrew Purtell wrote:
    If JG is apprehensive about putting the new master into 0.90, that's good enough for me. But I'll vote +0. I'm too unfamiliar with the details.

    Best regards,

    - Andy

    Why is this email five sentences or less?
    http://five.sentenc.es/




  • Andrew Purtell at Sep 22, 2010 at 10:41 pm
    Thanks. I'm going to try it out right away.
    From: Ryan Rawson
    I've been using the new master a bit, and the improvements are just
    too dramatic to ignore or delay.  Instant table transitions, clusters
    that actually shut down, etc, etc.

    On Wed, Sep 22, 2010 at 3:11 PM, Andrew Purtell wrote:
    If JG is apprehensive about putting the new master
    into 0.90, that's good enough for me. But I'll vote +0. I'm
    too unfamiliar with the details.
  • Jonathan Gray at Sep 23, 2010 at 6:51 pm
    Yes, there are way more changes/improvements made beyond the initial scope of working master failover / zk-based region transitions. Lots of cleanup, removal of many of the nasty bits we've always hated, and the groundwork is laid for a lot of future improvements.

    I'm not apprehensive. I think many of the points raised are valid, but my concerns have mostly been addressed at this point. I'm going to be spending the next 10 days or so helping stabilize and test. We'll see where we get by then.

    JG
    -----Original Message-----
    From: Andrew Purtell
    Sent: Thursday, September 23, 2010 3:47 AM
    To: dev@hbase.apache.org
    Subject: Re: old master to 0.90 and new master to 0.92? (was RE:
    Millions of photos into Hbase)

    Thanks. I'm going to try it out right away.
    From: Ryan Rawson
    I've been using the new master a bit, and the improvements are just
    too dramatic to ignore or delay.  Instant table transitions, clusters
    that actually shut down, etc, etc.


    On Wed, Sep 22, 2010 at 3:11 PM, Andrew Purtell <apurtell@apache.org>
    wrote:
    If JG is apprehensive about putting the new master
    into 0.90, that's good enough for me. But I'll vote +0. I'm
    too unfamiliar with the details.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedSep 21, '10 at 2:41a
activeSep 23, '10 at 6:51p
posts14
users7
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase