FAQ
I have the same problem. How do I dump the in-memory image to disk?
On Thursday, February 3, 2011 8:14:33 PM UTC+1, Patrick Angeles wrote:

Dan,

You should *NOT* take down the Namenode before you get this done. The
Namenode does not create a new fsimage when you shut it down. It does try
to reconstitute the fsimage by applying the edits file to the old image on
startup. This is risky, esp with a 1GB edits file, as you may have
corruption at some point in that edits file and you will not be able to
recover fully.

If you leave the Namenode running and let the SNN do the checkpoint, you
have one last resort if the SNN fails (because of a corrupt edits log or
whatever). You can basically have the NN dump it's in-memory image to disk
to create an fsimage. This won't work if you shut down the NN.

- P

On Thu, Feb 3, 2011 at 2:02 PM, djones <d...@lijit.com <javascript:>>wrote:
Actually, I have one more related question. If we were to take the
master namenode down before we perform this checkpoint, will there be
any problem with the namenode when it restarts. That is, will it first
apply the edits file to the fsimage, and then apply the edits.new
file? If that is the case, then would another option to my original
question be to backup the namenode files as you mentioned above, and
then stop the namenode, and then restart it. I assume this will result
in creating a new merged version of the fsimage file and a new empty
edits file. Then we could just start the SNN and have it go through
its normal checkpointing process on the already merged fsimage and
edits files.

Thanks,
Dan
On Feb 3, 11:53 am, djones wrote:
Thanks Eric, that's very helpful. We'll give this a try and let you
know how it goes.

On Feb 3, 12:25 am, Eric Sammer wrote:






Dan:
Good question. The short answer is:
1. Back up ${dfs.name.dirs}/current/{fsimage,edits,edits.new} files.
2. Make sure you did #1.
3. Restart the SNN and monitor the logs on both it and the NN to
ensure it
checkpoints properly.
The SNN should account for this case cleanly. It may need to copy the
edits
file again and perform compaction rather than simply send back the new
fsimage it has already built, but that shouldn't be an issue. You
will want
to make sure you have enough memory allocated to both the NN and the
SNN as,
like you mentioned, you'll need to make sure you can hold a fair bit
in RAM.
You can check this by looking at the variables in hadoop-env.sh for
each
process (HADOOP_NAMENODE_OPTS and HADOOP_SECONDARYNAMENODE_OPTS); they
should contain a -XmxNg where N is an appropriate amount of RAM. All
of that
said, a ~1GB edits file isn't that bad.
It definitely makes sense to monitor for this kind of a situation.
Easy
things to check for with Nagios would be file presence + mtime and
alert if
edits.new falls out of a normal range (> 1 hour to be generous). I
believe
last checkpoint time is also exposed as a metric by the NN but don't
quote
me on that.
Let us know how it goes.
Thanks (and don't forget rule #1 when messing around with the NN /
SNN)!
On Tue, Feb 1, 2011 at 6:51 PM, djones wrote:
We have a Hadoop cluster running CDH2 and we have a secondary
namenode
configured to do checkpoints. The checkpoints were working for a
long
time and then a couple of months ago, there appears to have been a
failure of the secondary namenode in the middle of a checkpoint. It
looks like the master namenode rolled the edits file to edits.new
(because this file continues to grow in size but the edits file does
not), and the secondary namenode copied the edits and fsimage file
over (since they are the same size on the master and secondary
namenodes). However, the logs show continuous errors that indicate
that the secondary couldn't connect to the master, so the new merged
file could not be copied back.
The cluster ran in this state for several months, so now the
edits.new
file is very large. We were able to fix the configuration so that
the
secondary namenode should be able to connect to the master, but we
are
a little concerned about whether it is safe to simply turn the
secondary namenode back on. Our concerns are:
1. Will the secondary namenode realize that it had already copied
over the edits and fsimage file, and just continue with the merge
process and then the copy of the merged file back? Or is it possible
that it could start over and tell the master to rotate out the edits
file to edits.new which would result in overwriting the existing
edits.new that the master is currently writing new edits to?
2. Since the checkpoint process has been failing for about 3 months,
the edits.new file has grown to be over 1.05 GB. Are there any
possible issues that might arise the next time that this edits file
has to get merged into the fsimage file? Things I'm thinking about
are
possible Out of Memory errors, etc.
--
Eric Sammer
twitter: esammer
data:www.cloudera.com
--

Search Discussions

  • Todd Lipcon at Sep 5, 2012 at 4:00 pm
    Hi Wouter,

    hadoop dfsadmin -safemode enter
    hadoop dfsadmin -savenamespace

    should be all you need.

    -Todd
    On Wed, Sep 5, 2012 at 8:12 AM, Wouter de Bie wrote:
    I have the same problem. How do I dump the in-memory image to disk?

    On Thursday, February 3, 2011 8:14:33 PM UTC+1, Patrick Angeles wrote:

    Dan,

    You should *NOT* take down the Namenode before you get this done. The
    Namenode does not create a new fsimage when you shut it down. It does try to
    reconstitute the fsimage by applying the edits file to the old image on
    startup. This is risky, esp with a 1GB edits file, as you may have
    corruption at some point in that edits file and you will not be able to
    recover fully.

    If you leave the Namenode running and let the SNN do the checkpoint, you
    have one last resort if the SNN fails (because of a corrupt edits log or
    whatever). You can basically have the NN dump it's in-memory image to disk
    to create an fsimage. This won't work if you shut down the NN.

    - P
    On Thu, Feb 3, 2011 at 2:02 PM, djones wrote:

    Actually, I have one more related question. If we were to take the
    master namenode down before we perform this checkpoint, will there be
    any problem with the namenode when it restarts. That is, will it first
    apply the edits file to the fsimage, and then apply the edits.new
    file? If that is the case, then would another option to my original
    question be to backup the namenode files as you mentioned above, and
    then stop the namenode, and then restart it. I assume this will result
    in creating a new merged version of the fsimage file and a new empty
    edits file. Then we could just start the SNN and have it go through
    its normal checkpointing process on the already merged fsimage and
    edits files.

    Thanks,
    Dan
    On Feb 3, 11:53 am, djones wrote:
    Thanks Eric, that's very helpful. We'll give this a try and let you
    know how it goes.

    On Feb 3, 12:25 am, Eric Sammer wrote:






    Dan:
    Good question. The short answer is:
    1. Back up ${dfs.name.dirs}/current/{fsimage,edits,edits.new} files.
    2. Make sure you did #1.
    3. Restart the SNN and monitor the logs on both it and the NN to
    ensure it
    checkpoints properly.
    The SNN should account for this case cleanly. It may need to copy the
    edits
    file again and perform compaction rather than simply send back the
    new
    fsimage it has already built, but that shouldn't be an issue. You
    will want
    to make sure you have enough memory allocated to both the NN and the
    SNN as,
    like you mentioned, you'll need to make sure you can hold a fair bit
    in RAM.
    You can check this by looking at the variables in hadoop-env.sh for
    each
    process (HADOOP_NAMENODE_OPTS and HADOOP_SECONDARYNAMENODE_OPTS);
    they
    should contain a -XmxNg where N is an appropriate amount of RAM. All
    of that
    said, a ~1GB edits file isn't that bad.
    It definitely makes sense to monitor for this kind of a situation.
    Easy
    things to check for with Nagios would be file presence + mtime and
    alert if
    edits.new falls out of a normal range (> 1 hour to be generous). I
    believe
    last checkpoint time is also exposed as a metric by the NN but don't
    quote
    me on that.
    Let us know how it goes.
    Thanks (and don't forget rule #1 when messing around with the NN /
    SNN)!
    On Tue, Feb 1, 2011 at 6:51 PM, djones wrote:
    We have a Hadoop cluster running CDH2 and we have a secondary
    namenode
    configured to do checkpoints. The checkpoints were working for a
    long
    time and then a couple of months ago, there appears to have been a
    failure of the secondary namenode in the middle of a checkpoint. It
    looks like the master namenode rolled the edits file to edits.new
    (because this file continues to grow in size but the edits file
    does
    not), and the secondary namenode copied the edits and fsimage file
    over (since they are the same size on the master and secondary
    namenodes). However, the logs show continuous errors that indicate
    that the secondary couldn't connect to the master, so the new
    merged
    file could not be copied back.
    The cluster ran in this state for several months, so now the
    edits.new
    file is very large. We were able to fix the configuration so that
    the
    secondary namenode should be able to connect to the master, but we
    are
    a little concerned about whether it is safe to simply turn the
    secondary namenode back on. Our concerns are:
    1. Will the secondary namenode realize that it had already copied
    over the edits and fsimage file, and just continue with the merge
    process and then the copy of the merged file back? Or is it
    possible
    that it could start over and tell the master to rotate out the
    edits
    file to edits.new which would result in overwriting the existing
    edits.new that the master is currently writing new edits to?
    2. Since the checkpoint process has been failing for about 3
    months,
    the edits.new file has grown to be over 1.05 GB. Are there any
    possible issues that might arise the next time that this edits file
    has to get merged into the fsimage file? Things I'm thinking about
    are
    possible Out of Memory errors, etc.
    --
    Eric Sammer
    twitter: esammer
    data:www.cloudera.com
    --



    --
    Todd Lipcon
    Software Engineer, Cloudera

    --
  • Wouter de Bie at Sep 5, 2012 at 4:01 pm
    Great,

    Thanks!

    --
    Wouter de Bie
    On Sep 5, 2012, at 6:00 PM, Todd Lipcon wrote:

    Hi Wouter,

    hadoop dfsadmin -safemode enter
    hadoop dfsadmin -savenamespace

    should be all you need.

    -Todd
    On Wed, Sep 5, 2012 at 8:12 AM, Wouter de Bie wrote:
    I have the same problem. How do I dump the in-memory image to disk?

    On Thursday, February 3, 2011 8:14:33 PM UTC+1, Patrick Angeles wrote:

    Dan,

    You should *NOT* take down the Namenode before you get this done. The
    Namenode does not create a new fsimage when you shut it down. It does try to
    reconstitute the fsimage by applying the edits file to the old image on
    startup. This is risky, esp with a 1GB edits file, as you may have
    corruption at some point in that edits file and you will not be able to
    recover fully.

    If you leave the Namenode running and let the SNN do the checkpoint, you
    have one last resort if the SNN fails (because of a corrupt edits log or
    whatever). You can basically have the NN dump it's in-memory image to disk
    to create an fsimage. This won't work if you shut down the NN.

    - P
    On Thu, Feb 3, 2011 at 2:02 PM, djones wrote:

    Actually, I have one more related question. If we were to take the
    master namenode down before we perform this checkpoint, will there be
    any problem with the namenode when it restarts. That is, will it first
    apply the edits file to the fsimage, and then apply the edits.new
    file? If that is the case, then would another option to my original
    question be to backup the namenode files as you mentioned above, and
    then stop the namenode, and then restart it. I assume this will result
    in creating a new merged version of the fsimage file and a new empty
    edits file. Then we could just start the SNN and have it go through
    its normal checkpointing process on the already merged fsimage and
    edits files.

    Thanks,
    Dan
    On Feb 3, 11:53 am, djones wrote:
    Thanks Eric, that's very helpful. We'll give this a try and let you
    know how it goes.

    On Feb 3, 12:25 am, Eric Sammer wrote:






    Dan:
    Good question. The short answer is:
    1. Back up ${dfs.name.dirs}/current/{fsimage,edits,edits.new} files.
    2. Make sure you did #1.
    3. Restart the SNN and monitor the logs on both it and the NN to
    ensure it
    checkpoints properly.
    The SNN should account for this case cleanly. It may need to copy the
    edits
    file again and perform compaction rather than simply send back the
    new
    fsimage it has already built, but that shouldn't be an issue. You
    will want
    to make sure you have enough memory allocated to both the NN and the
    SNN as,
    like you mentioned, you'll need to make sure you can hold a fair bit
    in RAM.
    You can check this by looking at the variables in hadoop-env.sh for
    each
    process (HADOOP_NAMENODE_OPTS and HADOOP_SECONDARYNAMENODE_OPTS);
    they
    should contain a -XmxNg where N is an appropriate amount of RAM. All
    of that
    said, a ~1GB edits file isn't that bad.
    It definitely makes sense to monitor for this kind of a situation.
    Easy
    things to check for with Nagios would be file presence + mtime and
    alert if
    edits.new falls out of a normal range (> 1 hour to be generous). I
    believe
    last checkpoint time is also exposed as a metric by the NN but don't
    quote
    me on that.
    Let us know how it goes.
    Thanks (and don't forget rule #1 when messing around with the NN /
    SNN)!
    On Tue, Feb 1, 2011 at 6:51 PM, djones wrote:
    We have a Hadoop cluster running CDH2 and we have a secondary
    namenode
    configured to do checkpoints. The checkpoints were working for a
    long
    time and then a couple of months ago, there appears to have been a
    failure of the secondary namenode in the middle of a checkpoint. It
    looks like the master namenode rolled the edits file to edits.new
    (because this file continues to grow in size but the edits file
    does
    not), and the secondary namenode copied the edits and fsimage file
    over (since they are the same size on the master and secondary
    namenodes). However, the logs show continuous errors that indicate
    that the secondary couldn't connect to the master, so the new
    merged
    file could not be copied back.
    The cluster ran in this state for several months, so now the
    edits.new
    file is very large. We were able to fix the configuration so that
    the
    secondary namenode should be able to connect to the master, but we
    are
    a little concerned about whether it is safe to simply turn the
    secondary namenode back on. Our concerns are:
    1. Will the secondary namenode realize that it had already copied
    over the edits and fsimage file, and just continue with the merge
    process and then the copy of the merged file back? Or is it
    possible
    that it could start over and tell the master to rotate out the
    edits
    file to edits.new which would result in overwriting the existing
    edits.new that the master is currently writing new edits to?
    2. Since the checkpoint process has been failing for about 3
    months,
    the edits.new file has grown to be over 1.05 GB. Are there any
    possible issues that might arise the next time that this edits file
    has to get merged into the fsimage file? Things I'm thinking about
    are
    possible Out of Memory errors, etc.
    --
    Eric Sammer
    twitter: esammer
    data:www.cloudera.com
    --



    --
    Todd Lipcon
    Software Engineer, Cloudera

    --

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedSep 5, '12 at 3:12p
activeSep 5, '12 at 4:01p
posts3
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Wouter de Bie: 2 posts Todd Lipcon: 1 post

People

Translate

site design / logo © 2018 Grokbase