FAQ
Hi All,

I've searched for a while on the possibility to configure multiple masters to govern large number of slaves, but I barely found any except for multi-masters for backup in HBase. Is one master enough to support thousands of slaves? Or in real we just configure separate Hadoop instances in a data center(I mean independent master-slaves sets on thousands of hosts)?

I really appreciate that someone here could enlighten me on those questions!

Thanks and regards
Chengwei

Search Discussions

  • Allen Wittenauer at Oct 27, 2010 at 10:08 pm

    On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:

    Hi All,

    I've searched for a while on the possibility to configure multiple masters to govern large number of slaves, but I barely found any except for multi-masters for backup in HBase. Is one master enough to support thousands of slaves? Or in real we just configure separate Hadoop instances in a data center(I mean independent master-slaves sets on thousands of hosts)?

    I really appreciate that someone here could enlighten me on those questions!
    This is question #3 on the Hadoop FAQ.

    http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
  • Wang, Chengwei at Oct 27, 2010 at 10:29 pm
    Thanks for pointing it out!

    Because it did not mentioned how may masters in the practice of 4000 nodes, is there only one master managing 4000 slaves?

    Thanks
    Chengwei

    ----- Original Message -----
    From: "Allen Wittenauer" <awittenauer@linkedin.com>
    To: "<common-dev@hadoop.apache.org>" <common-dev@hadoop.apache.org>
    Sent: Wednesday, October 27, 2010 6:08:10 PM
    Subject: Re: Multi-Master Hadoop Configuration

    On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:

    Hi All,

    I've searched for a while on the possibility to configure multiple masters to govern large number of slaves, but I barely found any except for multi-masters for backup in HBase. Is one master enough to support thousands of slaves? Or in real we just configure separate Hadoop instances in a data center(I mean independent master-slaves sets on thousands of hosts)?

    I really appreciate that someone here could enlighten me on those questions!
    This is question #3 on the Hadoop FAQ.

    http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
  • Allen Wittenauer at Oct 27, 2010 at 10:32 pm

    On Oct 27, 2010, at 3:28 PM, Wang, Chengwei wrote:

    Thanks for pointing it out!

    Because it did not mentioned how may masters in the practice of 4000 nodes, is there only one master managing 4000 slaves?
    Hadoop only supports one master, so yes.

    [At least until the AvatarNode gets checked into trunk... altho it might be there now.]
  • Konstantin Shvachko at Oct 28, 2010 at 9:09 pm
    Yes. Only one master, called name-node, is managing HDFS, and only one
    master, called job tracker, is managing MapReduce cluster.
    You can also read some online documentation, may be even publications.
    Thanks,
    --Konstantin
    On Wed, Oct 27, 2010 at 3:28 PM, Wang, Chengwei wrote:

    Thanks for pointing it out!

    Because it did not mentioned how may masters in the practice of 4000 nodes,
    is there only one master managing 4000 slaves?

    Thanks
    Chengwei

    ----- Original Message -----
    From: "Allen Wittenauer" <awittenauer@linkedin.com>
    To: "<common-dev@hadoop.apache.org>" <common-dev@hadoop.apache.org>
    Sent: Wednesday, October 27, 2010 6:08:10 PM
    Subject: Re: Multi-Master Hadoop Configuration

    On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:

    Hi All,

    I've searched for a while on the possibility to configure multiple
    masters to govern large number of slaves, but I barely found any except for
    multi-masters for backup in HBase. Is one master enough to support thousands
    of slaves? Or in real we just configure separate Hadoop instances in a data
    center(I mean independent master-slaves sets on thousands of hosts)?
    I really appreciate that someone here could enlighten me on those
    questions!

    This is question #3 on the Hadoop FAQ.

    http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
  • Wang, Chengwei at Oct 28, 2010 at 9:12 pm
    Thanks a bunch!

    ----- Original Message -----
    From: "Konstantin Shvachko" <shv.hadoop@gmail.com>
    To: common-dev@hadoop.apache.org
    Sent: Thursday, October 28, 2010 5:09:27 PM
    Subject: Re: Multi-Master Hadoop Configuration

    Yes. Only one master, called name-node, is managing HDFS, and only one
    master, called job tracker, is managing MapReduce cluster.
    You can also read some online documentation, may be even publications.
    Thanks,
    --Konstantin
    On Wed, Oct 27, 2010 at 3:28 PM, Wang, Chengwei wrote:

    Thanks for pointing it out!

    Because it did not mentioned how may masters in the practice of 4000 nodes,
    is there only one master managing 4000 slaves?

    Thanks
    Chengwei

    ----- Original Message -----
    From: "Allen Wittenauer" <awittenauer@linkedin.com>
    To: "<common-dev@hadoop.apache.org>" <common-dev@hadoop.apache.org>
    Sent: Wednesday, October 27, 2010 6:08:10 PM
    Subject: Re: Multi-Master Hadoop Configuration

    On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:

    Hi All,

    I've searched for a while on the possibility to configure multiple
    masters to govern large number of slaves, but I barely found any except for
    multi-masters for backup in HBase. Is one master enough to support thousands
    of slaves? Or in real we just configure separate Hadoop instances in a data
    center(I mean independent master-slaves sets on thousands of hosts)?
    I really appreciate that someone here could enlighten me on those
    questions!

    This is question #3 on the Hadoop FAQ.

    http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
  • Steve Loughran at Oct 29, 2010 at 1:07 pm

    On 27/10/10 23:28, Wang, Chengwei wrote:
    Thanks for pointing it out!

    Because it did not mentioned how may masters in the practice of 4000 nodes, is there only one master managing 4000 slaves?
    Given that you can now get 24 TB worth of HDD in a single server, I
    wouldn't worry that much about running out of worker nodes unless you
    are feeling very ambitious, as you will have to be well funded to hit
    the node limits, and you will probably hit other namenode scalability
    limits first, mainly those related to the number of files.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedOct 27, '10 at 9:56p
activeOct 29, '10 at 1:07p
posts7
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase