FAQ
For those people who aren't on the Apache Incubator mailing lists, Tom
White has proposed "BigTop", which is the tooling to integrate the build
and testing of the Apache Hadoop technologies, perhaps eventually to
have coordinated releases.

http://wiki.apache.org/incubator/BigtopProposal


For this to work, it needs support from all the hadoop-related projects
in the ASF, and, in an ideal world, integration with those bits from
outside (JUnit, Jetty) as well as downstream code (Cascading, etc). What
is key is that the projects it builds and tests will need to care about
the test failure reports, because BigTop should not be applying any
patches to private branches of the specific projects (the way Cloudera
and Ubuntu have done with the Hadoop and Linux source trees,
respectively). Some kind of synchronized release schedule would be nice
too, but I view that as a feature creep once the basic set up "build and
test everything" is done.

This could also be the place to put the functional tests against live
clusters, tests which could eventually become the definition of
compatibility that we've discussed before.

For these reasons
-I think everyone in the Hadoop-* projects dev teams ought to get
involved, initially in reviewing the proposal, then hopefully on the
mailing lists.

-It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c
committers to all have access to this, so that they can help evolve the
packaging and functional tests.

thoughts?

-steve

Search Discussions

  • Konstantin Boudnik at Jun 14, 2011 at 6:38 pm
    Steve, 'functional' or rather system & integration tests are the core of this
    idea. If you're interested in a little history of this project feel free to
    check

    https://docs.google.com/present/edit?id=0AVSlqgtwzvr9ZGdtNGM0OTJfMTRoYmRkMzhndA&hl=en_US&authkey=CLjMt_IN

    that's basically the original idea/initial implementation (based on the work
    done by Yahoo and known as HIT) which essentially lead to this proposal, I
    believe (anyone please correct me if you found this suggestion wrong).

    With regards,
    Cos
    On Tue, Jun 14, 2011 at 01:33PM, Steve Loughran wrote:
    For those people who aren't on the Apache Incubator mailing lists, Tom
    White has proposed "BigTop", which is the tooling to integrate the build
    and testing of the Apache Hadoop technologies, perhaps eventually to
    have coordinated releases.

    http://wiki.apache.org/incubator/BigtopProposal

    For this to work, it needs support from all the hadoop-related projects
    in the ASF, and, in an ideal world, integration with those bits from
    outside (JUnit, Jetty) as well as downstream code (Cascading, etc). What
    is key is that the projects it builds and tests will need to care about
    the test failure reports, because BigTop should not be applying any
    patches to private branches of the specific projects (the way Cloudera
    and Ubuntu have done with the Hadoop and Linux source trees,
    respectively). Some kind of synchronized release schedule would be nice
    too, but I view that as a feature creep once the basic set up "build and
    test everything" is done.

    This could also be the place to put the functional tests against live
    clusters, tests which could eventually become the definition of
    compatibility that we've discussed before.

    For these reasons
    -I think everyone in the Hadoop-* projects dev teams ought to get
    involved, initially in reviewing the proposal, then hopefully on the
    mailing lists.

    -It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c
    committers to all have access to this, so that they can help evolve the
    packaging and functional tests.

    thoughts?

    -steve
  • Ted Dunning at Jun 15, 2011 at 6:24 pm

    On Tue, Jun 14, 2011 at 2:33 PM, Steve Loughran wrote:

    For those people who aren't on the Apache Incubator mailing lists, Tom
    White has proposed "BigTop", which is the tooling to integrate the build and
    testing of the Apache Hadoop technologies, perhaps eventually to have
    coordinated releases.
    I will comment on the proposal directly as well, but coordinated releases
    would be a disaster. Mahout, for one, should not have to wait for Hadoop to
    have a release. We are already unhappy that we only get two releases a year
    and want to up that to four.

    Much better is to have a directed acyclic dependency graph with coherent
    versioning. That allows projects to stay independent, but still express
    version requirements.


    -It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c
    committers to all have access to this, so that they can help evolve the
    packaging and functional tests.
    Multi-project access sounds right.
  • Tom White at Jun 15, 2011 at 7:02 pm

    On Wed, Jun 15, 2011 at 11:23 AM, Ted Dunning wrote:
    On Tue, Jun 14, 2011 at 2:33 PM, Steve Loughran wrote:

    For those people who aren't on the Apache Incubator mailing lists, Tom
    White has proposed "BigTop", which is the tooling to integrate the build and
    testing of the Apache Hadoop technologies, perhaps eventually to have
    coordinated releases.
    I will comment on the proposal directly as well, but coordinated releases
    would be a disaster.  Mahout, for one, should not have to wait for Hadoop to
    have a release.  We are already unhappy that we only get two releases a year
    and want to up that to four.

    Much better is to have a directed acyclic dependency graph with coherent
    versioning.  That allows projects to stay independent, but still express
    version requirements.
    I agree. Bigtop would use the releases that the upstream projects have
    made - it cannot impose new requirements on them like coordinated
    releases. Any bugs that are uncovered would be submitted to the
    upstream project, for release there.

    I'm going to clarify the text in the proposal on this. Thanks for
    pointing it out Ted.

    Cheers,
    Tom

    -It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c
    committers to all have access to this, so that they can help evolve the
    packaging and functional tests.
    Multi-project access sounds right.
  • Steve Loughran at Jun 16, 2011 at 11:00 am

    On 15/06/11 19:23, Ted Dunning wrote:
    On Tue, Jun 14, 2011 at 2:33 PM, Steve Loughranwrote:
    For those people who aren't on the Apache Incubator mailing lists, Tom
    White has proposed "BigTop", which is the tooling to integrate the build and
    testing of the Apache Hadoop technologies, perhaps eventually to have
    coordinated releases.
    I will comment on the proposal directly as well, but coordinated releases
    would be a disaster. Mahout, for one, should not have to wait for Hadoop to
    have a release. We are already unhappy that we only get two releases a year
    and want to up that to four.
    Much better is to have a directed acyclic dependency graph with coherent
    versioning. That allows projects to stay independent, but still express

    you could do synchronised factored releases: if Hadoop ships every 12
    months, other things could come out every 4 months against the last
    major release, things like Hama could come out monthly while in the
    rapidly evolving state.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgeneral @
categorieshadoop
postedJun 14, '11 at 12:34p
activeJun 16, '11 at 11:00a
posts5
users4
websitehadoop.apache.org
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase