[HBase-dev] provide a 0.20-append tarball?

Hi Andrew,

Just to make sure I'm clear, are you saying that HBase 0.90.0 is
incompatible with CDH3b3 due to the security changes?

We're just getting going with HBase and have been running 0.90.0rc1 on
an un-patched version of Hadoop in dev. We were planning on upgrading
to CDH3b3 to get the sync patches.

thanks,
Bill

Bill,

I believe using CDH3B*2* will get you what you want.

ASF HBase 0.90 will not be compatible with secure versions of Hadoop, which includes CDH3B3.

Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back.
- Piet Hein (via Tom White)


--- On Tue, 12/21/10, Bill Graham wrote:

If you do not enable any of the security features in CDH3b3 or another flavor of secure Hadoop, it should just work, right?

Friso

Hi all,

The only thing you need to do to run an 0.90 release is to sub out the
hadoop-core jar with the one from your version of CDH. The issue isn't the
security stuff (which we now shim on the hbase side, thanks to the Trend
guys!) but rather a different protocol version number for the data transfer
protocol.

Of course once a final 0.90 release is out, we'll have a CDH release off
this tree which "just works"

-Todd

(I moved this topic to [email protected] from user).
Well, the 'preferred' is the Apache distribution. Thats what we
should bundle, etc.

That said, I get that you were trying to talk more to our current
predicament where our setup is more involved than it should be for new
users because the 'preferred' Hadoop is a source only distribution and
many will be running other than the Apache distribution anyways. In
general we need to do work -- documentation, introspecting shims, etc
-- to make it so HBase is more 'universal' and can more easily be
deployed atop any Hadoop whether Cloudera's CDH, newer versions of
Hadoop (hadoop 0.21, 0.22, 0.20s), or vendor X's implementation of
HDFS, Ceph, etc.

I like the idea of linking off to Cloudera doc. -- or any other
vendor's doc -- from ours for install (Todd?). Or, maybe better would
be linking to a wiki page that vendors can edit as they wish?

Let me ask Dhruba what he thinks about making a 0.20-append release
(He's the release manager). Will also sound out the hadoop pmc since
they'll have an opinion.

St.Ack

I asked Dhruba. He's fine w/ a release off tip of branch--0.20-append.

I just wrote a message to general up on hadoop to gauge what hadoopers
think of the idea.

St.Ack

Looks like the fight does not go well. A lot of hdfs developers are
concerned that it would detract resources. I'm not sure who's
resources.

I hope my 13-15 month commented helped... I've heard "wait for the
next" version before and I am not interested in it. If that indeed
worked, a year ago we'd have a stable working sync/hlog recovery
support.

-ryan

If you have a PMC member who is willing to be release manager, what is the
beef?

Were the comments of actual HDFS developers the majority of them?

I probably helped less but I have larger concerns about the long term viability of Hadoop of which the situation with -append is only one symptom. Somebody needs to say something.


Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back.
- Piet Hein (via Tom White)


--- On Thu, 12/23/10, Ryan Rawson wrote:

For CDH specific compatibility issues beyond general stuff like does it work, I think users should be pointed to Cloudera for support.

For security issues, they probably won't find much help here so would need to go to Cloudera or some stuff might be answerable on the HDFS lists since security is in Apache 0.22 branch.

We also want to support HDFS 0.22 at some point so being aware of potential issues is at least somewhat relevant for the community at large even if not using CDH.


The line that I believe should be drawn, and I think most have agreed, is that we should be compatible with and ship with an official Apache Hadoop release. Having something Apache to ship with was one of the motivations of the 20-append branch.

So yeah, the plan is to do an official 20-append Apache release. Some effort needs to go into it, so when is an open question. Don't know if there's a release manager yet, I think Dhruba might have stated so when we made the branch but don't exactly recall. However there is one very common theme in the 20-append open JIRAs :)

https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&resolution=-1&pid=12310942&fixfor=12315103

There are currently 15 open issues listed. The blocker, at least, is already committed to 20-append and is just open because it's not in trunk yet, so the number may be off a bit.

JG

There is nothing in the ASF that requires us to depend solely on ASF
releases. As long as we are good with the licensing guidelines. But
that is the legal situation, not the community situation which is of
course more complex.

My stand on this is the one I have always taken. I need a HBase that
works, well, and with no data loss.

-ryan

Agreed it's not an ASF requirement. Just stating my opinion and sense from others.

An additional motivation and benefit of the Apache 20-append repository and release is a de facto record of what patches are actually required to achieve the requirement you just laid out (works well with HBase and no data loss). CDH goes beyond that.

What I'd recommend to someone today for production is a different story. There has not been sufficient testing on the 20-append branch at this point so I would personally recommend CDH3b2 to most users.

JG

Yep, I wholeheartedly agree with Jonathan. If something is obviously
CDH-specific, HBase devs should feel free to punt to the
[email protected] list. That's what we've got that list for :)

But Jonathan's other point is also quite valid - 90+% of what's in CDH is
going to be in 0.22. So spending a few minutes to look into an issue will
save us time later, too.

What you thinking Andrew? I was thinking we should push for a release
on the 0.20-append branch. I'm not so sure how well that would go
over in hadoop land. We could take a few soundings. Were you
thinking something else?
St.Ack

Yes, this is what I was thinking.

- Andy