FAQ
Hi Folks,

Seems like we are adding more and more options and commands that take
a file or URL argument, where the file can be one of a number of
places. Perhaps we can unify things a bit by agreeing to use URL
conventions in these places?

Where:

file:// is a file on the current machine
http is supported everywhere
hdfs://host[:port]/ is the used to refer to HDFS files
hdfs:/// is used to use the default hdfs?
rsync, ftp and https are supported too (eventually)
torrent:// is used to support bittorent someday...

Should we standardize on such a conventions and start digging to put
a class together to make this all seamless?

e14

Search Discussions

  • Andrzej Bialecki at Aug 21, 2006 at 9:51 pm

    Eric Baldeschwieler wrote:
    Hi Folks,

    Seems like we are adding more and more options and commands that take
    a file or URL argument, where the file can be one of a number of
    places. Perhaps we can unify things a bit by agreeing to use URL
    conventions in these places?

    Where:

    file:// is a file on the current machine
    http is supported everywhere
    hdfs://host[:port]/ is the used to refer to HDFS files
    hdfs:/// is used to use the default hdfs?
    rsync, ftp and https are supported too (eventually)
    torrent:// is used to support bittorent someday...

    Should we standardize on such a conventions and start digging to put a
    class together to make this all seamless?
    Wouldn't we end up with another java.net.URL in disguise? java.net.URL
    already supports pluggable protocols, we could create one for hdfs ...

    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Doug Cutting at Aug 21, 2006 at 10:02 pm

    Andrzej Bialecki wrote:
    Wouldn't we end up with another java.net.URL in disguise? java.net.URL
    already supports pluggable protocols, we could create one for hdfs ...
    FYI, we already use java.net.URI for HDFS in CopyFiles.

    Doug
  • Eric Baldeschwieler at Aug 22, 2006 at 4:03 am
    I'm all for not reinventing the wheel. This is probably the right
    implementation direction.

    I mainly wanted to get to agreement that we would start taking URIs
    instead of HDFS paths in most API/UIs and generalize this behavior.
    On Aug 21, 2006, at 2:49 PM, Andrzej Bialecki wrote:

    Eric Baldeschwieler wrote:
    Hi Folks,

    Seems like we are adding more and more options and commands that
    take a file or URL argument, where the file can be one of a number
    of places. Perhaps we can unify things a bit by agreeing to use
    URL conventions in these places?

    Where:

    file:// is a file on the current machine
    http is supported everywhere
    hdfs://host[:port]/ is the used to refer to HDFS files
    hdfs:/// is used to use the default hdfs?
    rsync, ftp and https are supported too (eventually)
    torrent:// is used to support bittorent someday...

    Should we standardize on such a conventions and start digging to
    put a class together to make this all seamless?
    Wouldn't we end up with another java.net.URL in disguise?
    java.net.URL already supports pluggable protocols, we could create
    one for hdfs ...

    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 21, '06 at 9:21p
activeAug 22, '06 at 4:03a
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase