FAQ
We're getting ready to contribute our FileSystem implementation for the
Sector DFS (sector.sf.net). Up to now our development and testing has been
against 0.18.3, so our intention was to first integrate with that release
and then work on integrating with the trunk for the next release. A couple
of questions:
* Should we integrate with the 0.18 branch, or just put our changes into the
0.18.3 release? We're not sure if there are plans for further releases on
the 0.18 branch.

* Should we create separate Jira's for the two releases, or can we create a
single Jira to cover the 0.18 and trunk releases?

Thanks for your help. We'd like to make sure we're facilitating the process,
so any suggestions are appreciated.

--
Jonathan Seidman
Open Data Group

Search Discussions

  • Todd Lipcon at Aug 10, 2009 at 8:36 pm
    Hi Jonathan,

    Responses inline below:
    On Mon, Aug 10, 2009 at 1:28 PM, Jonathan Seidman wrote:

    We're getting ready to contribute our FileSystem implementation for the
    Sector DFS (sector.sf.net). Up to now our development and testing has been
    against 0.18.3, so our intention was to first integrate with that release
    and then work on integrating with the trunk for the next release. A couple
    of questions:
    * Should we integrate with the 0.18 branch, or just put our changes into
    the
    0.18.3 release? We're not sure if there are plans for further releases on
    the 0.18 branch.
    I believe there are plans for an 0.18.4 release eventually, though a new
    feature like a new DFS is unlikely to be included. I would concentrate on
    integrating against trunk rather than a past release. If trunk is
    inconvenient since it's a moving target, you could integrate against
    branch-20. While a lot of people are currently using 0.18, I anticipate that
    most users will be switching over to 0.20 in the next couple of months.

    * Should we create separate Jira's for the two releases, or can we create a
    single Jira to cover the 0.18 and trunk releases?
    Single JIRA should be fine - just post multiple patches.

    Thanks for your help. We'd like to make sure we're facilitating the
    process,
    so any suggestions are appreciated.
    In general you should be aware that you should integrate as a new project in
    the contrib/ dir. If all goes well you shouldn't need to make any patches
    that touch Hadoop itself.

    -Todd
  • Chris Douglas at Aug 10, 2009 at 11:37 pm

    * Should we integrate with the 0.18 branch, or just put our changes into
    the
    0.18.3 release? We're not sure if there are plans for further releases on
    the 0.18  branch.
    This will not be committed to the 0.18 branch, even if there is an
    0.18.4 release. If you wanted to post an 0.18 compatible patch on the
    same JIRA for reference, that's fine, but only patches to trunk will
    be considered for inclusion.
    In general you should be aware that you should integrate as a new project in
    the contrib/ dir. If all goes well you shouldn't need to make any patches
    that touch Hadoop itself.
    If this is a shim, it should probably live in o.a.h.fs with
    S3FileSystem, KosmosFileSystem, etc. rather than being added to
    contrib. -C
  • Jonathan Seidman at Aug 11, 2009 at 3:09 pm
    Thanks for the replies. We'll create a patch for trunk and then include a
    0.18 compatible patch with the Jira, as you suggest.

    Based on other contributed FileSystem implementations, we were assuming this
    should go in o.a.h.fs and not contrib, so thanks for the clarification.

    Jonathan
    On Mon, Aug 10, 2009 at 6:37 PM, Chris Douglas wrote:

    * Should we integrate with the 0.18 branch, or just put our changes into
    the
    0.18.3 release? We're not sure if there are plans for further releases
    on
    the 0.18 branch.
    This will not be committed to the 0.18 branch, even if there is an
    0.18.4 release. If you wanted to post an 0.18 compatible patch on the
    same JIRA for reference, that's fine, but only patches to trunk will
    be considered for inclusion.
    In general you should be aware that you should integrate as a new project in
    the contrib/ dir. If all goes well you shouldn't need to make any patches
    that touch Hadoop itself.
    If this is a shim, it should probably live in o.a.h.fs with
    S3FileSystem, KosmosFileSystem, etc. rather than being added to
    contrib. -C


    --
    Jonathan Seidman
    Open Data Group
  • Steve Loughran at Aug 11, 2009 at 3:32 pm

    Jonathan Seidman wrote:
    Thanks for the replies. We'll create a patch for trunk and then include a
    0.18 compatible patch with the Jira, as you suggest.

    Based on other contributed FileSystem implementations, we were assuming this
    should go in o.a.h.fs and not contrib, so thanks for the clarification.
    what's the test plan going to be for a new FS?
  • Konstantin Boudnik at Aug 11, 2009 at 4:43 pm
    Jonathan,

    in case you need to take a look at a common testplan template you can find one
    in HDFS-265 or more generic in HADOOP-5587

    Cos
    On 8/11/09 8:32 AM, Steve Loughran wrote:
    Jonathan Seidman wrote:
    Thanks for the replies. We'll create a patch for trunk and then include a
    0.18 compatible patch with the Jira, as you suggest.

    Based on other contributed FileSystem implementations, we were assuming this
    should go in o.a.h.fs and not contrib, so thanks for the clarification.
    what's the test plan going to be for a new FS?
  • Tom White at Aug 12, 2009 at 11:58 am
    Hi Jonathan,

    For testing you can subclass FileSystemContractBaseTest and create an
    instance of the filesystem to test in the set up method, to check that
    the new implementation conforms to the contract of the FileSystem
    interface (although see also
    https://issues.apache.org/jira/browse/HDFS-303).

    See TestHDFSFileSystemContract for an example.

    Cheers,
    Tom

    On Tue, Aug 11, 2009 at 5:13 PM, Konstantin Boudnikwrote:
    Jonathan,

    in case you need to take a look at a common testplan template you can find
    one in HDFS-265 or more generic in HADOOP-5587

    Cos
    On 8/11/09 8:32 AM, Steve Loughran wrote:

    Jonathan Seidman wrote:
    Thanks for the replies. We'll create a patch for trunk and then include a
    0.18 compatible patch with the Jira, as you suggest.

    Based on other contributed FileSystem implementations, we were assuming
    this
    should go in o.a.h.fs  and not contrib, so thanks for the clarification.
    what's the test plan going to be for a new FS?
  • Jonathan Seidman at Aug 12, 2009 at 9:38 pm
    Thanks for the pointers Tom and Cos, this is helpful. We'll take a look at
    these as we firm up our test plan.
    On Wed, Aug 12, 2009 at 6:58 AM, Tom White wrote:

    Hi Jonathan,

    For testing you can subclass FileSystemContractBaseTest and create an
    instance of the filesystem to test in the set up method, to check that
    the new implementation conforms to the contract of the FileSystem
    interface (although see also
    https://issues.apache.org/jira/browse/HDFS-303).

    See TestHDFSFileSystemContract for an example.

    Cheers,
    Tom

    On Tue, Aug 11, 2009 at 5:13 PM, Konstantin Boudnikwrote:
    Jonathan,

    in case you need to take a look at a common testplan template you can find
    one in HDFS-265 or more generic in HADOOP-5587

    Cos
    On 8/11/09 8:32 AM, Steve Loughran wrote:

    Jonathan Seidman wrote:
    Thanks for the replies. We'll create a patch for trunk and then include
    a
    0.18 compatible patch with the Jira, as you suggest.

    Based on other contributed FileSystem implementations, we were assuming
    this
    should go in o.a.h.fs and not contrib, so thanks for the
    clarification.
    what's the test plan going to be for a new FS?


    --
    Jonathan Seidman
    Open Data Group

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 10, '09 at 8:29p
activeAug 12, '09 at 9:38p
posts8
users6
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase