FAQ
Folks,

I heard that there was a plans to porting HDFS to Zookeeper to get a high
availability.
Is this true?
If so, could anyone let me know about the status of this effort?

Cheers,

-sangmin

Search Discussions

  • Doug Cutting at Jun 30, 2008 at 8:48 pm

    Sangmin Lee wrote:
    I heard that there was a plans to porting HDFS to Zookeeper to get a high
    availability.
    Is this true?
    If so, could anyone let me know about the status of this effort?
    I have heard folks talk of this, but have heard of no concrete plans
    yet. The radical approach would be to replace the namenode entirely
    with Zookeeper.

    Doug
  • Sangmin Lee at Jun 30, 2008 at 9:44 pm
    Was that from HDFS developers? and could you let me know how serious that
    plans was?
    Thanks,

    -sangmin
    On Mon, Jun 30, 2008 at 3:46 PM, Doug Cutting wrote:

    Sangmin Lee wrote:
    I heard that there was a plans to porting HDFS to Zookeeper to get a high
    availability.
    Is this true?
    If so, could anyone let me know about the status of this effort?
    I have heard folks talk of this, but have heard of no concrete plans yet.
    The radical approach would be to replace the namenode entirely with
    Zookeeper.

    Doug
  • Mahadev Konar at Jun 30, 2008 at 11:03 pm
    The radical approach was from zookeeper dev team :). Mostly the
    discussion around using zookeeper is to be able to mount different
    namespaces on a single tree.

    Mahadev
    -----Original Message-----
    From: Sangmin Lee
    Sent: Monday, June 30, 2008 2:44 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: porting HDFS to Zookeeper??

    Was that from HDFS developers? and could you let me know how serious that
    plans was?
    Thanks,

    -sangmin
    On Mon, Jun 30, 2008 at 3:46 PM, Doug Cutting wrote:

    Sangmin Lee wrote:
    I heard that there was a plans to porting HDFS to Zookeeper to get
    a
    high
    availability.
    Is this true?
    If so, could anyone let me know about the status of this effort?
    I have heard folks talk of this, but have heard of no concrete plans yet.
    The radical approach would be to replace the namenode entirely with
    Zookeeper.

    Doug
  • Sanjay Radia at Jun 30, 2008 at 11:40 pm

    Sangmin Lee wrote:
    Folks,

    I heard that there was a plans to porting HDFS to Zookeeper to get a high
    availability.
    Is this true?
    If so, could anyone let me know about the status of this effort?

    Cheers,

    -sangmin
    Do you literally mean port or simply use Zookeeper.
    One could store the name space and data in Zookeeper - an interesting
    exercise.

    Alternatively, one could use Zookeeper to elect a primary Namenode for
    HA; this would be a good use of Zookeeper.
    I can see this happening once we have replicated Namenodes.

    sanjay
  • Ted Dunning at Jul 1, 2008 at 12:07 am
    The idea of using zookeeper to coordinate multiple namenodes which handle
    different parts of the name-space would be nice as well.

    Using zookeeper as it currently is to replace the name node is probably a
    non-starter because zookeeper keeps everything in memory which means that
    there would be no win in terms of number of files.
    On Mon, Jun 30, 2008 at 4:38 PM, Sanjay Radia wrote:

    Sangmin Lee wrote:
    Folks,

    I heard that there was a plans to porting HDFS to Zookeeper to get a high
    availability.
    Is this true?
    If so, could anyone let me know about the status of this effort?

    Cheers,

    -sangmin

    Do you literally mean port or simply use Zookeeper.
    One could store the name space and data in Zookeeper - an interesting
    exercise.

    Alternatively, one could use Zookeeper to elect a primary Namenode for HA;
    this would be a good use of Zookeeper.
    I can see this happening once we have replicated Namenodes.

    sanjay


    --
    ted
  • Sangmin Lee at Jul 1, 2008 at 8:42 am
    Thank you for the replies.
    I meant porting HDFS to Zookeeper to make namenode fault-tolerant and highly
    available rather than the simple usage of Zookeeper. Basically by storing
    namespace metadata in Zookeeper, we may be able to get highly available
    replicated namenodes. So I meant using Zookeeper as frontend of HDFS. Anyway
    it seems to me that there has been no actual work going on regarding this,
    isn't it? :-)

    -sangmin

    On Mon, Jun 30, 2008 at 6:38 PM, Sanjay Radia wrote:

    Sangmin Lee wrote:
    Folks,

    I heard that there was a plans to porting HDFS to Zookeeper to get a high
    availability.
    Is this true?
    If so, could anyone let me know about the status of this effort?

    Cheers,

    -sangmin

    Do you literally mean port or simply use Zookeeper.
    One could store the name space and data in Zookeeper - an interesting
    exercise.

    Alternatively, one could use Zookeeper to elect a primary Namenode for HA;
    this would be a good use of Zookeeper.
    I can see this happening once we have replicated Namenodes.

    sanjay
  • Steve Loughran at Jul 2, 2008 at 12:50 pm

    Sangmin Lee wrote:
    Thank you for the replies.
    I meant porting HDFS to Zookeeper to make namenode fault-tolerant and highly
    available rather than the simple usage of Zookeeper. Basically by storing
    namespace metadata in Zookeeper, we may be able to get highly available
    replicated namenodes. So I meant using Zookeeper as frontend of HDFS. Anyway
    it seems to me that there has been no actual work going on regarding this,
    isn't it? :-)

    -sangmin

    I think you've just volunteered.

    @work we use our own tuple space thing, Anubis:
    http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/anubis/doc/

    it lets you do the election/voting stuff to see who is in charge in an
    HA cluster, though to deal with a partition event robustly is hard; you
    need somebody that really understands the maths behind HA distributed
    computing. I am not in that category, so we are lucky there are others
    nearby who do, but it implies there is a block of code I am very
    reluctant to approach.

    In fact, its a bit like security. just like it is easy to do an insecure
    system, but very hard to do a secure one, its easy to be
    low-availability, really, really hard to be HA.

    -steve

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 30, '08 at 8:36p
activeJul 2, '08 at 12:50p
posts8
users6
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase