FAQ
Hi All,

While discussing about Hadoop backup & restore plan with my team I thought
about a scenario I wanted to ask you about:

I wonder what will happen following the steps below:

1. Backup the namenode's metadata (fsimage & edits log).

2. Adding files to the Hadoop clutser.

3. Deleting files from the Hadoop cluster.

4. Corruption in the namenode's metadata or whatever.

5. Restoring the namenode's metadata which I backed up (step 1).

How the namenode handle the blocks written or deleted before the corruption
?

Will I have access to them ? Is there any procedure I need to do in order to
"fix" the gap created in the time since the last backup?



Thanks.



Avi

Search Discussions

  • Ravi Prakash at Aug 30, 2011 at 1:01 pm
    Hi Avi,

    If you restored the metadata from Step 1, it would have no memory of what
    happened after that point. I guess you could try using the Offline Image
    Viewer and the OfflineEditsViewer tools, to read the corrupted metadata and
    see if you can recover the blocks from there.

    Cheers
    Ravi
    On Tue, Aug 30, 2011 at 7:05 AM, Avi Vaknin wrote:

    Hi All,

    While discussing about Hadoop backup & restore plan with my team I thought
    about a scenario I wanted to ask you about:

    I wonder what will happen following the steps below:

    1. Backup the namenode's metadata (fsimage & edits log).

    2. Adding files to the Hadoop clutser.

    3. Deleting files from the Hadoop cluster.

    4. Corruption in the namenode's metadata or whatever.

    5. Restoring the namenode's metadata which I backed up (step 1).

    How the namenode handle the blocks written or deleted before the corruption
    ?

    Will I have access to them ? Is there any procedure I need to do in order
    to
    "fix" the gap created in the time since the last backup?



    Thanks.



    Avi
  • Avi Vaknin at Aug 30, 2011 at 1:21 pm
    Thanks.
    I actually wanted to ask what will happen if I restore the fsimage and the
    edits log and try to bring up the Hadoop cluster?
    I mentioned corruption in the metadata just to demonstrate some kind of
    problem in the metadata which I can't overcome.
    I assume there are tools to correct corruption in those files.



    -----Original Message-----
    From: Ravi Prakash
    Sent: Tuesday, August 30, 2011 4:01 PM
    To: common-user@hadoop.apache.org
    Subject: Re: Hadoop Backup & Restore

    Hi Avi,

    If you restored the metadata from Step 1, it would have no memory of what
    happened after that point. I guess you could try using the Offline Image
    Viewer and the OfflineEditsViewer tools, to read the corrupted metadata and
    see if you can recover the blocks from there.

    Cheers
    Ravi
    On Tue, Aug 30, 2011 at 7:05 AM, Avi Vaknin wrote:

    Hi All,

    While discussing about Hadoop backup & restore plan with my team I thought
    about a scenario I wanted to ask you about:

    I wonder what will happen following the steps below:

    1. Backup the namenode's metadata (fsimage & edits log).

    2. Adding files to the Hadoop clutser.

    3. Deleting files from the Hadoop cluster.

    4. Corruption in the namenode's metadata or whatever.

    5. Restoring the namenode's metadata which I backed up (step 1).

    How the namenode handle the blocks written or deleted before the
    corruption
    ?

    Will I have access to them ? Is there any procedure I need to do in order
    to
    "fix" the gap created in the time since the last backup?



    Thanks.



    Avi
    -----
    No virus found in this message.
    Checked by AVG - www.avg.com
    Version: 10.0.1392 / Virus Database: 1520/3866 - Release Date: 08/29/11
  • GOEKE, MATTHEW (AG/1000) at Aug 30, 2011 at 4:54 pm
    All,

    We were discussing how we would backup our data from the various environments we will have and I was hoping someone could chime in with previous experience in this. My primary concern about our cluster is that we would like to be able to recover anything within the last 60 days so having full backups both on tape and through distcp is preferred.

    Out initial thoughts can be seen in the jpeg attached but just in case any of you are weary of attachments it can also be summarized below:

    Prod Cluster --DistCp--> On-site Backup cluster with Fuse mount point running NetBackup daemon --NetBackup--> Media Server --> Tape

    One of our biggest grey areas so far is how do most people accomplish incremental backups? Our thought was to tie this into our NetBackup configuration as this can be done for other connectors but we do not see anything for HDFS yet.

    Thanks,
    Matt
    This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
    to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
    all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

    All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
    subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
    Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
    this e-mail or any attachment.


    The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
    including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
    Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
    applicable U.S. export laws and regulations.
  • Michael Segel at Aug 30, 2011 at 8:13 pm
    Matthew, the short answer is hire a consultant to work with you on your DR/BCP strategy. :-)

    Short of that... you have a couple of things...

    Your back-up cluster, is it in the same site? (What happens when site goes down?)

    Are you planning to make your back up cluster and main cluster homogenous? By this I mean if your main cluster has 1PB of disk w 4x2TB or 4x3TB drives, will your backup cluster have the same configuration?
    (You may want to consider asymmetry in designing your clusters) So your backup cluster has fewer nodes but more drives per node.

    You also have to look at your data. Are your data sets small and discrete? If so, you could probably back them up to tape, (snapshots) , just in case of human error and you didn't catch it in time and the error gets propagated to your backup cluster.

    I haven't played with fuse, so I don't know if there are any performance issues, but on a back up cluster, I don't think its much of an issue.
    From: matthew.goeke@monsanto.com
    To: common-user@hadoop.apache.org; cdh-user@cloudera.org
    Subject: Hadoop multi tier backup
    Date: Tue, 30 Aug 2011 16:54:07 +0000

    All,

    We were discussing how we would backup our data from the various environments we will have and I was hoping someone could chime in with previous experience in this. My primary concern about our cluster is that we would like to be able to recover anything within the last 60 days so having full backups both on tape and through distcp is preferred.

    Out initial thoughts can be seen in the jpeg attached but just in case any of you are weary of attachments it can also be summarized below:

    Prod Cluster --DistCp--> On-site Backup cluster with Fuse mount point running NetBackup daemon --NetBackup--> Media Server --> Tape

    One of our biggest grey areas so far is how do most people accomplish incremental backups? Our thought was to tie this into our NetBackup configuration as this can be done for other connectors but we do not see anything for HDFS yet.

    Thanks,
    Matt
    This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
    to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
    all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

    All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
    subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
    Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
    this e-mail or any attachment.


    The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
    including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
    Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
    applicable U.S. export laws and regulations.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 30, '11 at 12:06p
activeAug 30, '11 at 8:13p
posts5
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase