Your feedback is well taken. However, in defence of the team at FB - we have tried to keep the communication as open as possible through the JIRA. Most of the thoughts and design issues are documented there. It is also used a lot as a conversational tool around specific issues. If you are critically blocked on an issue, do mention it on the JIRA and you can be assured of a good response.
It is true that there are many times we work on things that are a priority for FB but that is not always the case. We have tried to be very community oriented. We have always included contributions from the community in our internal deployments and we have taken contributions in core aspects of the code from the community as well even though they may not be important for FBs environment. I would like you to think of us also as part of the Hive community as opposed to FB.
Regarding code quality, that is always a work in progress. Trunk is a development branch and some times things break in trunk. Our code reviews minimize that but cannot eliminate that. Code review is done fairly thoroughly, which you can gauge from the quality of code review comments at large. Having said that this is a process of continuous improvements for everyone and it is never a perfect process.
Regarding roadmap, the current roadmap is published on the wiki. We can be more frequent in our updates but essentially the roadmap is a way of summarizing what core features or performance improvements every one is working on in the JIRA. It is on the wiki and it owned by the Hive community and is built bottom up rather than top down. You will get full support from the Hive community in terms of patches being incorporated even if they are not on the roadmap. The roadmap is more of a guidance rather than a dictat.
I hope that answers a lot of questions/concerns that you have. I would also encourage a more positive tone in terms of how you can improve things rather than how FB can improve things for you. For example, if you feel a certain code review is not appropriate or has missed things, do jump in and mention your concerns on the JIRA and put your suggestions there. If you feel that some code is badly written, do jump in to fix it or suggest changes. If you feel some issue is taking too much time, chime in and take it on. Most successful movements are bottom up rather than top down and that is what we encourage in Hive as well.
From: Edward Capriolo
Sent: Thursday, July 09, 2009 8:12 AM
Subject: Re: Some wish after serious consideration
I am also waiting for HIVE-438 to do some work with the Hive authentication system. I chose to sideline my work until a solid thrift release.
I personally am very impressed by the open source contribution at facebook. Hadoop-Hive, Thrift, Scribe, Cassandra probably more stuff I do not know about.
Also the Hive crew is very amicable! Do you want to see a horror story? Here is a patch that I use for (SSH public keys in LDAP support)!http://www.nabble.com/IMPORTANT:-change-of-project-hosting-notice-td17271488.html
...However, their efforts to
find a way to get this kind of functionality into OpenSSH have met with absolutely no reaction whatsoever from the OpenSSH developers. ...
Software has to developed based on usage. Since thrift has to be very very interoperable and standardized just ram-rodding it with developers may not help since there may need to be numerous cycles between users and developers.
I believe Hive was/is waiting for an official thrift release before updating.
2009/7/9 Min Zhou <firstname.lastname@example.org>:
I have been watching HIVE-438 for so long a time, you know that's a
critical change almost impact the whole hive source tree, a quick resolve is need.
It's understandable human resources of facebook are very nervous,
developers there always join several projects at the same time.
Therefore, we should use the power of the open source community to
speed up the development of it. But right now, my feeling is that we
only care about their own affairs, regardless of what other people do.
This is not the pattern of the open source community, but we still immersed in this pattern.
2009/7/9 He Yongqiang <email@example.com>
"But I think no conflicts can be guaranteed, since conflicts are not
raised by one patch. If no conflict appears to this patch, then there
will be conflicts for other patches."
My meaning was not refuse conflicts, there should be a way to avoid
the frequency of them if we work together.
But I think no conflicts can NOT be guaranteed, since conflicts are
not raised by one patch. If no conflict appears to this patch, then
there will be conflicts for other patches.
On 09-7-9 下午2:43, "He Yongqiang" wrote:
On 09-7-9 下午2:14, "Min Zhou" wrote:
Having focused on hive for several month, here is some wish of me
after serious consideration
1. All auto-gen code for hive was under the facebook commercial
of thrift, which is older than the open source one, would lead
compatible problems and stop from all helps from the open
We need to remove them as soon as possible, but it seems the
this issue has stopped.
See Hive-438. I think it will be committed by this weekend?
2. Please give us a clear roadmap. We also have a plan
but our patches would probably be uncared-for, because it's not
schedule of facebook. If go on like this, there should be a lot of
compatible problems brought by other commits, we were surfing
conflicts again and again.
I think the hive roadmap on hive wiki page has just been updated.
Please send out request for code review if you think the patch is ready.
But I think no conflicts can be guaranteed, since conflicts are not raised
by one patch. If no conflict appears to this patch, then there will
be conflicts for other patches.
3. Please don't commit code so rashly. Code from Ashish could
committed by others without a strict examination, that caused
problems when using it here, bugs and incondite code hard to
extend it. Perhaps the main reason is that Ashish is the leader
Another person, Namit, always committed buggy or ugly code. I have a
suggestion, just more discussion and tests with the helps of
community. Code quality would be raised if so. (I don't intend
personal attacks here)
Code review is a kind of really hard and boring work. And we can
only say that the code is much likely with no error. A patch is
committed with at least two persons' work, the patch submitter and the code reviewer.
Sometimes the code is really hard to find errors either by eyes or
please be more patient. The bugs can be fixed soon after observing.
And I agree with you suggestion on more discussion, so please
comment on the
jira pages for issues you think need more discussion and tests.
BTW, I think as the hive community grows, there could be more
So the first priority issue should be how to enlarge the hive
let more people involved in the discussion of the hive mail-list or jira.
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.