[HBase-dev] Re: Millions of photos into Hbase

Deep within my soul I do not want to do this.

But it might be practical.

FB is going into prod w/ the old master and we've been doing work stabilizing it in ways that do not apply at all to the new one (especially around zk). That'll make having two different active branches a bit of a nightmare but right now some of these patches are not even available as we've been operating under the assumption that the release would be on the new master (and the patches don't apply on trunk).

I guess SU is also putting an 0.89 old master release into prod.

What's a little unfortunate is that the 0.89 releases include the unnecessary move of some transition communication into ZK. That was put in as a first step towards new master before we decided to branch it off.

The question for me would be whether we think it's at all feasible to get the new master 0.90 released in time for HW. If not, maybe we should consider taking an 0.89 branch, making it an 0.90 branch, and focus on stabilizing for release in time for HW, as Todd suggests.

There are other ramifications of this. It would be very difficult for me to get my flush/compact/split improvements in to the old master as the new implementation I've been working on relies completely on the new stuff. But maybe better to punt that for 0.92 as well so can really nail it?

The other factor is the enormous amount of ZK improvements that were done in the new master branch. It's a real fuckin mess in 0.89 releases, though I've done a bit of cleanup already towards making it at least tenable for production.

My main concern is that this move will push back any release of the new master significantly. There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions. But doing a production release in time for HW is probably the most important thing.

JG

So while you were in the plane a few things were discussed :)

In the thread "[VOTE] Release 'development release' HBase 0.89.2010830
rc2", it was decided that we sink RC2 in favor of a new RC with a
rollback of the zk-based assignment. This is what we're running here,
and Ryan published our repo yesterday
http://github.com/stumbleupon/hbase (see the top of the CHANGES file
for what we added)

Also some of your arguments correlate those of Todd, expressed in the
thread "Next release" that was posted on 09/15. Stack and Andrew
expressed their opinions in favor of Todd's option #1 (although Stack
said that he would decide tomorrow if he changes his opinion after
some more work on the master).

My personal opinion leans towards Todd's option #2. 0.20.0 is now more
than a year old, we are able to release almost production-ready code
that is durable, so why not move forward with it (this is what we did
here). There are a few issues to fix, but it's small compared to
getting the new master in shape plus getting the rest working (like
the replication code's interaction with ZooKeeper that needs to be
redone).

J-D

IMO, we're almost there w/ the new master. I'd like to wait till
Thursday or Friday before making a call.

Regards HW, we might consider pressing on with new master even if it
means we don't exactly line up a new master 0.89.x (or a 0.90.xRC)
release w/ HW. You fellas talking at HW can fudge it some if it comes
to that (We won't be the only ones. CDH3 will not be ready for HW
apparently). I'm thinking we can afford to take such a position
because if someone wants durable hbase now, they can run with the
SU-prod 0.89.x that J-D is about to put up.

Putting off new master till 0.92 means it'll be maybe 6 months before
it appears. During this time we'll be paying a high price keeping up
two disparate branches -- TRUNK w/ new master and the release branch
-- shoe-horning patches to fit both.

We also confuse the 0.90 'message' given we've been talking about new
master at HUGs and here on the lists with a good while now.

St.Ack

CDH3b3 will be ready for Hadoop world, and we'd kind of like to freeze
component versions at this point in the beta cycle. So if 0.90 is out,
that would be great. We can certainly work with what we've got
(20100830 minus ZK assignment) but if a production-worth new master
isn't ready by the end of October or so we'll probably push that out
til CDH4.
This is going to be an official Apache release, right? I guess my
question is: if it's stable and usable for production, why call it a
"development release"? If we're recommending it for all new users over
and above 0.20.6, then it seems like this should be deemed stable (ie
even release number).
If you guys are running 20100830 in production, won't you be doing
that anyway? Assumedly we'd treat this 0.90 as "no new features" and
put the new features into 0.91.x leading up to 0.92?
True. The question is whether we prefer to slip time or slip scope. In
my opinion slipping scope is better - it's open source and people
understand that schedules slip. Keeping strong release momentum up
helps adoption and will get people off 0.20.6 which no one really
wants to support anymore.

-Todd

Ok.

Its my thought that we'd have a production ready master way before end
of October.

I was not talking of recommending the next 0.89.x to all new users
over 0.20.6. I thought it plain that running a release in production
at SU did not mean the release 'stable and useable' by others. SU has
3 hbase committers aboard to fix and hand-hold the software over rough
patches.

I was thinking that the next 0.89.x comes w/ the same caveats as
previous 0.89.xs, that its a 'developer release' for those willing to
put up w/ some rough edges and that its just another step on the way
to 0.90.0.

Nope. We'd move to new master release. If it don't work for us, we'd
feel a little awkward recommending it to others.

This is the question. I'm just suggesting that new master MAY not be
that far out. I want to do another couple of days work on it and then
have us make a call; i.e. vote that we press on to get new master into
0.90 or punt on new master for 0.90.

St.Ack

This might be a little naïve but what about calling the "SU"-build
0.88 and release it (with a small disclaimer). It would make sense in
that the 0.89-dev releases have more stuff in them than 0.88 and it
would still allow for a 0.90 release with all the announced features.

As a side note: I plan to finish work on the Maven changes in time for
HW but tests keep on failing (Hudson has the same problem...) which
doesn't make it easy to completely test the build. I don't know if
I'll be able to get Thrift done though, sorry. I'm doing this in my
spare time and I didn't have a lot of that :( but with a bit of luck I
can work on HBase in my new job. But I'll upload updated Thrift files
in the next few days if someone wants to check what I've done (modeled
it after Jeff's Avro work).

Cheers,
Lars

Sorry about that. I've been picking away at them from time to time
but some require more than a glance to fix (For me at least, most of
what fails on hudson does not fail for me here below).

I don't know if
Thanks for the update and good luck in the new Job.

St.Ack

I like this too... if we go the direction of an old master release.

It addresses most of the issues, besides slowing release of new master. So not necessarily +1 on doing it but this would be better than an old master 0.90 release imo.

Good idea Lars.

JG

I'm -1 on an out-of-order release like this. I think it will really confuse
people, please won't work correctly with upgrades on packaging systems.

I know we've been promising "new master for 0.90" at meetups and stuff, but
outside the dev team, does anyone really care about the implementation
details? The feature here isn't a new master, it's more stable master
failover, better locality, etc. Whether we have that or say that it's going
to be in the next version, I don't think people will be too upset.

The argument that it's hell to maintain two branches is a fair one, but
unless we can get trunk usable very soon now, we're already going to be in
that situation. People are migrating production instances to 0.89.x now, and
once people go to production they are very hesitant to upgrade, regardless
of what we tell them.

-Todd

If JG is apprehensive about putting the new master into 0.90, that's good enough for me. But I'll vote +0. I'm too unfamiliar with the details.

Best regards,

- Andy

Why is this email five sentences or less?
http://five.sentenc.es/

I've been using the new master a bit, and the improvements are just
too dramatic to ignore or delay. Instant table transitions, clusters
that actually shut down, etc, etc.

Thanks. I'm going to try it out right away.

Yes, there are way more changes/improvements made beyond the initial scope of working master failover / zk-based region transitions. Lots of cleanup, removal of many of the nasty bits we've always hated, and the groundwork is laid for a lot of future improvements.

I'm not apprehensive. I think many of the points raised are valid, but my concerns have mostly been addressed at this point. I'm going to be spending the next 10 days or so helping stabilize and test. We'll see where we get by then.

JG