FAQ
Hi,

Am trying to bring high availability in namenode(in hadoop cluster) for
small setup.

So the idea is to keep one extra namenode and use terracotta to replicate
the log(in memory details-jvm) from primary datacode to standby namenode, if
primary goes down secondary can take the primary role.

Now am bit confused how to start second namenode while the primary name node
up and running(with datanode connected) with keep relying the info from
primary thro terracotta. Is there a way to achieve this without restarting
namenode or datanode, any patch?

Please help and suggest me.

Regards,
Gokul

Search Discussions

  • Konstantin Boudnik at Sep 23, 2011 at 6:28 pm
    Gokul,

    Terracotta is a fascinating framework which might be a bit of an overshot for a
    small setup as well as adding another level of complexity in the form
    Terracotta framework.

    Quite interesting approach for NN's HA is Jini (aka ASF River) along with
    JavaSpaces for the distributed storage of the meta-data. Using Jini also gives
    you for free such benefits as auto-discovering making hot-swap of NN a non-issue.

    I've been musing over this idea for some time and am trying to find some time
    to actually experiment with it.

    Cos
    On Fri, Sep 23, 2011 at 11:36PM, Gokul Prasad wrote:
    Hi,
    Am trying to bring high availability in namenode(in hadoop cluster) for
    small setup.
    So the idea is to keep one extra namenode and use terracotta to replicate
    the log(in memory details-jvm) from primary datacode to standby namenode,
    if primary goes down secondary can take the primary role.
    Now am bit confused how to start second namenode while the primary name
    node up and running(with datanode connected) with keep relying the info
    from primary thro terracotta. Is there a way to achieve this without
    restarting namenode or datanode, any patch?
    Please help and suggest me.
    Regards,
    Gokul
  • Gokul Prasad at Sep 23, 2011 at 7:06 pm
    Cos,

    Am planning to implement this idea in bigger cluster but first i want
    to get confidence in small cluster. I do read about terracotta and
    inspired by this idea.

    -gokul
    On Fri, Sep 23, 2011 at 11:57 PM, Konstantin Boudnik wrote:
    Gokul,

    Terracotta is a fascinating framework which might be a bit of an overshot for a
    small setup as well as adding another level of complexity in the form
    Terracotta framework.

    Quite interesting approach for NN's HA is Jini (aka ASF River) along with
    JavaSpaces for the distributed storage of the meta-data. Using Jini also gives
    you for free such benefits as auto-discovering making hot-swap of NN a non-issue.

    I've been musing over this idea for some time and am trying to find some time
    to actually experiment with it.

    Cos
    On Fri, Sep 23, 2011 at 11:36PM, Gokul Prasad wrote:
    Hi,
    Am trying to bring high availability in namenode(in hadoop cluster) for
    small setup.
    So the idea is to keep one extra namenode and use terracotta to replicate
    the log(in memory details-jvm) from primary datacode to standby namenode,
    if primary goes down secondary can take the primary role.
    Now am bit confused how to start second namenode while the primary name
    node up and running(with datanode connected) with keep relying the info
    from primary thro terracotta. Is there a way to achieve this without
    restarting namenode or datanode, any patch?
    Please help and suggest me.
    Regards,
    Gokul
  • GOEKE, MATTHEW (AG/1000) at Sep 23, 2011 at 7:30 pm
    Is there any reason not to just use something as simple as drbd (which is baked into vanilla linux)? Unless you are already paying terracotta for their software there are plenty of HA setups that are supported through the use of drbd and a virtual ip.

    Matt

    -----Original Message-----
    From: Gokul Prasad
    Sent: Friday, September 23, 2011 2:06 PM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: hadoop high availability using terracotta

    Cos,

    Am planning to implement this idea in bigger cluster but first i want
    to get confidence in small cluster. I do read about terracotta and
    inspired by this idea.

    -gokul
    On Fri, Sep 23, 2011 at 11:57 PM, Konstantin Boudnik wrote:
    Gokul,

    Terracotta is a fascinating framework which might be a bit of an overshot for a
    small setup as well as adding another level of complexity in the form
    Terracotta framework.

    Quite interesting approach for NN's HA is Jini (aka ASF River) along with
    JavaSpaces for the distributed storage of the meta-data. Using Jini also gives
    you for free such benefits as auto-discovering making hot-swap of NN a non-issue.

    I've been musing over this idea for some time and am trying to find some time
    to actually experiment with it.

    Cos
    On Fri, Sep 23, 2011 at 11:36PM, Gokul Prasad wrote:
    Hi,
    Am trying to bring high availability in namenode(in hadoop cluster) for
    small setup.
    So the idea is to keep one extra namenode and use terracotta to replicate
    the log(in memory details-jvm) from primary datacode to standby namenode,
    if primary goes down secondary can take the primary role.
    Now am bit confused how to start second namenode while the primary name
    node up and running(with datanode connected) with keep relying the info
    from primary thro terracotta. Is there a way to achieve this without
    restarting namenode or datanode, any patch?
    Please help and suggest me.
    Regards,
    Gokul
    This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
    to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
    all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

    All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
    subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
    Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
    this e-mail or any attachment.


    The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
    including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
    Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
    applicable U.S. export laws and regulations.
  • Gokul Prasad at Sep 24, 2011 at 4:33 pm
    afaik..drbd approach will take longer to bring standby namenode to
    live..am trying to minimise the downtime..like standby namenode will
    be relayed with the necessary info thro terracotta and if primary
    namenode goes down standby will take its role.

    -gokul

    On Sat, Sep 24, 2011 at 1:00 AM, GOEKE, MATTHEW (AG/1000)
    wrote:
    Is there any reason not to just use something as simple as drbd (which is baked into vanilla linux)? Unless you are already paying terracotta for their software there are plenty of HA setups that are supported through the use of drbd and a virtual ip.

    Matt

    -----Original Message-----
    From: Gokul Prasad
    Sent: Friday, September 23, 2011 2:06 PM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: hadoop high availability using terracotta

    Cos,

    Am planning to implement this idea in bigger cluster but first i want
    to get confidence in small cluster. I do read about terracotta and
    inspired by this idea.

    -gokul
    On Fri, Sep 23, 2011 at 11:57 PM, Konstantin Boudnik wrote:
    Gokul,

    Terracotta is a fascinating framework which might be a bit of an overshot for a
    small setup as well as adding another level of complexity in the form
    Terracotta framework.

    Quite interesting approach for NN's HA is Jini (aka ASF River) along with
    JavaSpaces for the distributed storage of the meta-data. Using Jini also gives
    you for free such benefits as auto-discovering making hot-swap of NN a non-issue.

    I've been musing over this idea for some time and am trying to find some time
    to actually experiment with it.

    Cos
    On Fri, Sep 23, 2011 at 11:36PM, Gokul Prasad wrote:
    Hi,
    Am trying to bring high availability in namenode(in hadoop cluster) for
    small setup.
    So the idea is to keep one extra namenode and use terracotta to replicate
    the log(in memory details-jvm) from primary datacode to standby namenode,
    if primary goes down secondary can take the primary role.
    Now am bit confused how to start second namenode while the primary name
    node up and running(with datanode connected) with keep relying the info
    from primary thro terracotta. Is there a way to achieve this without
    restarting namenode or datanode, any patch?
    Please help and suggest me.
    Regards,
    Gokul
    This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
    to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
    all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

    All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
    subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
    Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
    this e-mail or any attachment.


    The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
    including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
    Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this information you are obligated to comply with all
    applicable U.S. export laws and regulations.
  • GOEKE, MATTHEW (AG/1000) at Sep 26, 2011 at 2:43 am
    I don't have much terracotta experience but which solution of theirs are you referring to? After looking at their product page I am having a hard time figuring out what architecture you are planning to build that would have a faster turnaround on failure.

    Matt

    -----Original Message-----
    From: Gokul Prasad
    Sent: Saturday, September 24, 2011 11:33 AM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: hadoop high availability using terracotta

    afaik..drbd approach will take longer to bring standby namenode to
    live..am trying to minimise the downtime..like standby namenode will
    be relayed with the necessary info thro terracotta and if primary
    namenode goes down standby will take its role.

    -gokul

    On Sat, Sep 24, 2011 at 1:00 AM, GOEKE, MATTHEW (AG/1000)
    wrote:
    Is there any reason not to just use something as simple as drbd (which is baked into vanilla linux)? Unless you are already paying terracotta for their software there are plenty of HA setups that are supported through the use of drbd and a virtual ip.

    Matt

    -----Original Message-----
    From: Gokul Prasad
    Sent: Friday, September 23, 2011 2:06 PM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: hadoop high availability using terracotta

    Cos,

    Am planning to implement this idea in bigger cluster but first i want
    to get confidence in small cluster. I do read about terracotta and
    inspired by this idea.

    -gokul
    On Fri, Sep 23, 2011 at 11:57 PM, Konstantin Boudnik wrote:
    Gokul,

    Terracotta is a fascinating framework which might be a bit of an overshot for a
    small setup as well as adding another level of complexity in the form
    Terracotta framework.

    Quite interesting approach for NN's HA is Jini (aka ASF River) along with
    JavaSpaces for the distributed storage of the meta-data. Using Jini also gives
    you for free such benefits as auto-discovering making hot-swap of NN a non-issue.

    I've been musing over this idea for some time and am trying to find some time
    to actually experiment with it.

    Cos
    On Fri, Sep 23, 2011 at 11:36PM, Gokul Prasad wrote:
    Hi,
    Am trying to bring high availability in namenode(in hadoop cluster) for
    small setup.
    So the idea is to keep one extra namenode and use terracotta to replicate
    the log(in memory details-jvm) from primary datacode to standby namenode,
    if primary goes down secondary can take the primary role.
    Now am bit confused how to start second namenode while the primary name
    node up and running(with datanode connected) with keep relying the info
    from primary thro terracotta. Is there a way to achieve this without
    restarting namenode or datanode, any patch?
    Please help and suggest me.
    Regards,
    Gokul
    This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
    to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
    all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

    All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
    subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
    Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
    this e-mail or any attachment.


    The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
    including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
    Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this information you are obligated to comply with all
    applicable U.S. export laws and regulations.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-user @
categorieshadoop
postedSep 23, '11 at 6:07p
activeSep 26, '11 at 2:43a
posts6
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase