FAQ
Hi,
I'm trying to implement NameNode failover (or at least NameNode local
data backup), but it is hard since there is no official documentation.
Pages on this subject are created, but still empty:

http://wiki.apache.org/hadoop/NameNodeFailover
http://wiki.apache.org/hadoop/SecondaryNameNode

I have been browsing the web and hadoop mailing list to see how this
should be implemented, but I got even more confused. People are asking
do we even need SecondaryNameNode etc. (since NameNode can write local
data to multiple locations, so one of those locations can be a mounted
disk from other machine). I think I understand the motivation for
SecondaryNameNode (to create a snapshoot of NameNode data every n
seconds/hours), but setting (deploying and running) SecondaryNameNode on
different machine than NameNode is not as trivial as I expected. First I
found that if I need to run SecondaryNameNode on other machine than
NameNode I should change masters file on NameNode (change localhost to
SecondaryNameNode host) and set some properties in hadoop-site.xml on
SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
fs.checkpoint.period etc.)

This was enough to start SecondaryNameNode when starting NameNode with
bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then
I found that I need to set dfs.http.address on NameNode address (so now
I have NameNode address in both fs.default.name and dfs.http.address).

Now I get following exception:

2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
doCheckpoint:
2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
java.net.SocketException: Unexpected end of file from server

My questions are following:
How to resolve this problem (this exception)?
Do I need additional property in SecondaryNameNode's hadoop-site.xml or
NameNode's hadoop-site.xml?

How should NameNode failover work ideally? Is it like this:

SecondaryNameNode runs on separate machine than NameNode and stores
NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
When NameNode machine crashes, we start NameNode on machine where
SecondaryNameNode was running and we set dfs.name.dir to
fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
hostname (change from the primary to the secondary).

Is this correct ?

Tomislav

Search Discussions

  • Jean-Daniel Cryans at Oct 29, 2008 at 12:15 am
    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode

    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode on
    different machine than NameNode is not as trivial as I expected. First I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Otis Gospodnetic at Oct 29, 2008 at 6:14 am
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with Hadoop?
    From what I can tell, I think one has to do the following two things:
    1) configure primary NN to save namespace and xa logs to multiple dirs, one of which is actually on a remotely mounted disk, so that the data actually lives on a separate disk on a separate box. This saves namespace and xa logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create the fsimage checkpoint. This really is a second NN process running on another box. It sounds like this secondary NN has to somehow have access to fsimage & edits files from the primary NN server. http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode does not describe the best practise around that - the recommended way to give secondary NN access to primary NN's fsimage and edits files. Should one mount a disk from the primary NN box to the secondary NN box to get access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and again is there in case the box with the primary NN dies. That's what's described on http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode more or less.

    Is this sufficient, or are there other things one has to do to eliminate NN SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans <jdcryans@apache.org>
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode

    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode on
    different machine than NameNode is not as trivial as I expected. First I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Jean-Daniel Cryans at Oct 29, 2008 at 1:08 pm
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D
    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic wrote:

    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs, one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less.

    Is this sufficient, or are there other things one has to do to eliminate NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans <jdcryans@apache.org>
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Tomislav Poljak at Oct 29, 2008 at 10:11 pm
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs, one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less.

    Is this sufficient, or are there other things one has to do to eliminate NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans <jdcryans@apache.org>
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Konstantin Shvachko at Oct 29, 2008 at 11:11 pm
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of the primary.

    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs, one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less.

    Is this sufficient, or are there other things one has to do to eliminate NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans <jdcryans@apache.org>
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Tomislav Poljak at Oct 30, 2008 at 5:53 pm
    Hi,
    can you, please, explain the difference between fs.default.name and
    dfs.http.address (like how and when is SecondaryNameNode using
    fs.default.name and how/when dfs.http.address). I have set them both to
    same (namenode's) hostname:port. Is this correct (or dfs.http.address
    needs some other port)?

    Thanks,

    Tomislav
    On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote:
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of the primary.

    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs, one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less.

    Is this sufficient, or are there other things one has to do to eliminate NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans <jdcryans@apache.org>
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Otis Gospodnetic at Oct 31, 2008 at 6:10 am
    Konstantin & Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN.

    The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. "hdfs://") in this URL.

    Is this correct?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Tomislav Poljak <tpoljak@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Thursday, October 30, 2008 1:52:18 PM
    Subject: Re: SecondaryNameNode on separate machine

    Hi,
    can you, please, explain the difference between fs.default.name and
    dfs.http.address (like how and when is SecondaryNameNode using
    fs.default.name and how/when dfs.http.address). I have set them both to
    same (namenode's) hostname:port. Is this correct (or dfs.http.address
    needs some other port)?

    Thanks,

    Tomislav
    On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote:
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of
    the primary.
    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs,
    one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access
    to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes
    not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore
    or less.
    Is this sufficient, or are there other things one has to do to eliminate
    NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Doug Cutting at Oct 31, 2008 at 4:40 pm

    Otis Gospodnetic wrote:
    Konstantin & Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN.

    The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. "hdfs://") in this URL.

    Is this correct?
    Yes. The default port for an HDFS URI is 8020 (NameNode.DEFAULT_PORT).
    The value of fs.default.name is used by HDFS. When starting the
    namenode or datanodes, this must be an HDFS URI. If this names an
    explicit port, then that will be used, otherwise the default, 8020 will
    be used.

    The default port for HTTP URIs is 80, but the namenode typically runs
    its web UI on 50070 (the default for dfs.http.address).

    Doug
  • Konstantin Shvachko at Oct 31, 2008 at 6:39 pm
    True, dfs.http.address is the NN Web UI address.
    This where the NN http server runs. Besides the Web UI there also
    a servlet running on that server which is used to transfer image
    and edits from NN to the secondary using http get.
    So SNN uses both addresses fs.default.name and dfs.http.address.

    When SNN finishes the checkpoint the primary needs to transfer the
    resulting image back. This is done via the http server running on SNN.

    Answering Tomislav's question:
    The difference between fs.default.name and dfs.http.address is that
    fs.default.name is the name-node's PRC address, where clients and
    data-nodes connect to, while dfs.http.address is the NN's http server
    address where our browsers connect to, but it is also used for
    transferring image and edits files.

    --Konstantin

    Otis Gospodnetic wrote:
    Konstantin & Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN.

    The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. "hdfs://") in this URL.

    Is this correct?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Tomislav Poljak <tpoljak@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Thursday, October 30, 2008 1:52:18 PM
    Subject: Re: SecondaryNameNode on separate machine

    Hi,
    can you, please, explain the difference between fs.default.name and
    dfs.http.address (like how and when is SecondaryNameNode using
    fs.default.name and how/when dfs.http.address). I have set them both to
    same (namenode's) hostname:port. Is this correct (or dfs.http.address
    needs some other port)?

    Thanks,

    Tomislav
    On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote:
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of
    the primary.
    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs,
    one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access
    to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes
    not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore
    or less.
    Is this sufficient, or are there other things one has to do to eliminate
    NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Tomislav Poljak at Nov 3, 2008 at 1:11 pm
    Hi,
    Thank you all for your time and your answers!

    Now SecondaryNameNode connects to the NameNode (after I configured
    dfs.http.address to the NN's http server -> NN hostname on port 50070)
    and creates(transfers) edits and fsimage from NameNode.

    Can you explain me a little bit more how NameNode failover should work
    now?

    For example, SecondaryNameNode now stores fsimage and edits to (SNN's)
    dirX and let's say NameNode goes down (disk becomes unreadable). Now I
    create/dedicate a new machine for NameNode (also change DNS to point to
    this new NameNode machine as nameNode host) and take the data dirX from
    SNN and copy it to new NameNode. How do I configure new NameNode to use
    data from dirX (do I configure dfs.name.dir to point to dirX and start
    new NameNode)?

    Thanks,
    Tomislav


    On Fri, 2008-10-31 at 11:38 -0700, Konstantin Shvachko wrote:
    True, dfs.http.address is the NN Web UI address.
    This where the NN http server runs. Besides the Web UI there also
    a servlet running on that server which is used to transfer image
    and edits from NN to the secondary using http get.
    So SNN uses both addresses fs.default.name and dfs.http.address.

    When SNN finishes the checkpoint the primary needs to transfer the
    resulting image back. This is done via the http server running on SNN.

    Answering Tomislav's question:
    The difference between fs.default.name and dfs.http.address is that
    fs.default.name is the name-node's PRC address, where clients and
    data-nodes connect to, while dfs.http.address is the NN's http server
    address where our browsers connect to, but it is also used for
    transferring image and edits files.

    --Konstantin

    Otis Gospodnetic wrote:
    Konstantin & Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN.

    The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. "hdfs://") in this URL.

    Is this correct?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Tomislav Poljak <tpoljak@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Thursday, October 30, 2008 1:52:18 PM
    Subject: Re: SecondaryNameNode on separate machine

    Hi,
    can you, please, explain the difference between fs.default.name and
    dfs.http.address (like how and when is SecondaryNameNode using
    fs.default.name and how/when dfs.http.address). I have set them both to
    same (namenode's) hostname:port. Is this correct (or dfs.http.address
    needs some other port)?

    Thanks,

    Tomislav
    On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote:
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of
    the primary.
    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs,
    one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access
    to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes
    not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore
    or less.
    Is this sufficient, or are there other things one has to do to eliminate
    NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Konstantin Shvachko at Nov 3, 2008 at 7:17 pm
    You can either do what you just described with dfs.name.dir = dirX
    or you can start name-node with -importCheckpoint option.
    This is an automation for copying image files from secondary to primary.

    See here:
    http://hadoop.apache.org/core/docs/current/commands_manual.html#namenode
    http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+NameNode
    http://issues.apache.org/jira/browse/HADOOP-2585#action_12584755

    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    Thank you all for your time and your answers!

    Now SecondaryNameNode connects to the NameNode (after I configured
    dfs.http.address to the NN's http server -> NN hostname on port 50070)
    and creates(transfers) edits and fsimage from NameNode.

    Can you explain me a little bit more how NameNode failover should work
    now?

    For example, SecondaryNameNode now stores fsimage and edits to (SNN's)
    dirX and let's say NameNode goes down (disk becomes unreadable). Now I
    create/dedicate a new machine for NameNode (also change DNS to point to
    this new NameNode machine as nameNode host) and take the data dirX from
    SNN and copy it to new NameNode. How do I configure new NameNode to use
    data from dirX (do I configure dfs.name.dir to point to dirX and start
    new NameNode)?

    Thanks,
    Tomislav


    On Fri, 2008-10-31 at 11:38 -0700, Konstantin Shvachko wrote:
    True, dfs.http.address is the NN Web UI address.
    This where the NN http server runs. Besides the Web UI there also
    a servlet running on that server which is used to transfer image
    and edits from NN to the secondary using http get.
    So SNN uses both addresses fs.default.name and dfs.http.address.

    When SNN finishes the checkpoint the primary needs to transfer the
    resulting image back. This is done via the http server running on SNN.

    Answering Tomislav's question:
    The difference between fs.default.name and dfs.http.address is that
    fs.default.name is the name-node's PRC address, where clients and
    data-nodes connect to, while dfs.http.address is the NN's http server
    address where our browsers connect to, but it is also used for
    transferring image and edits files.

    --Konstantin

    Otis Gospodnetic wrote:
    Konstantin & Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN.

    The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. "hdfs://") in this URL.

    Is this correct?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Tomislav Poljak <tpoljak@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Thursday, October 30, 2008 1:52:18 PM
    Subject: Re: SecondaryNameNode on separate machine

    Hi,
    can you, please, explain the difference between fs.default.name and
    dfs.http.address (like how and when is SecondaryNameNode using
    fs.default.name and how/when dfs.http.address). I have set them both to
    same (namenode's) hostname:port. Is this correct (or dfs.http.address
    needs some other port)?

    Thanks,

    Tomislav
    On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote:
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of
    the primary.
    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs,
    one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access
    to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes
    not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore
    or less.
    Is this sufficient, or are there other things one has to do to eliminate
    NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Tomislav Poljak at Nov 4, 2008 at 1:06 pm
    Konstantin,

    it works, thanks a lot!

    Tomislav

    On Mon, 2008-11-03 at 11:13 -0800, Konstantin Shvachko wrote:
    You can either do what you just described with dfs.name.dir = dirX
    or you can start name-node with -importCheckpoint option.
    This is an automation for copying image files from secondary to primary.

    See here:
    http://hadoop.apache.org/core/docs/current/commands_manual.html#namenode
    http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+NameNode
    http://issues.apache.org/jira/browse/HADOOP-2585#action_12584755

    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    Thank you all for your time and your answers!

    Now SecondaryNameNode connects to the NameNode (after I configured
    dfs.http.address to the NN's http server -> NN hostname on port 50070)
    and creates(transfers) edits and fsimage from NameNode.

    Can you explain me a little bit more how NameNode failover should work
    now?

    For example, SecondaryNameNode now stores fsimage and edits to (SNN's)
    dirX and let's say NameNode goes down (disk becomes unreadable). Now I
    create/dedicate a new machine for NameNode (also change DNS to point to
    this new NameNode machine as nameNode host) and take the data dirX from
    SNN and copy it to new NameNode. How do I configure new NameNode to use
    data from dirX (do I configure dfs.name.dir to point to dirX and start
    new NameNode)?

    Thanks,
    Tomislav


    On Fri, 2008-10-31 at 11:38 -0700, Konstantin Shvachko wrote:
    True, dfs.http.address is the NN Web UI address.
    This where the NN http server runs. Besides the Web UI there also
    a servlet running on that server which is used to transfer image
    and edits from NN to the secondary using http get.
    So SNN uses both addresses fs.default.name and dfs.http.address.

    When SNN finishes the checkpoint the primary needs to transfer the
    resulting image back. This is done via the http server running on SNN.

    Answering Tomislav's question:
    The difference between fs.default.name and dfs.http.address is that
    fs.default.name is the name-node's PRC address, where clients and
    data-nodes connect to, while dfs.http.address is the NN's http server
    address where our browsers connect to, but it is also used for
    transferring image and edits files.

    --Konstantin

    Otis Gospodnetic wrote:
    Konstantin & Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN.

    The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. "hdfs://") in this URL.

    Is this correct?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Tomislav Poljak <tpoljak@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Thursday, October 30, 2008 1:52:18 PM
    Subject: Re: SecondaryNameNode on separate machine

    Hi,
    can you, please, explain the difference between fs.default.name and
    dfs.http.address (like how and when is SecondaryNameNode using
    fs.default.name and how/when dfs.http.address). I have set them both to
    same (namenode's) hostname:port. Is this correct (or dfs.http.address
    needs some other port)?

    Thanks,

    Tomislav
    On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote:
    SecondaryNameNode uses http protocol to transfer the image and the edits
    from the primary name-node and vise versa.
    So the secondary does not access local files on the primary directly.
    The primary NN should know the secondary's http address.
    And the secondary NN need to know both fs.default.name and dfs.http.address of
    the primary.
    In general we usually create one configuration file hadoop-site.xml
    and copy it to all other machines. So you don't need to set up different
    values for all servers.

    Regards,
    --Konstantin

    Tomislav Poljak wrote:
    Hi,
    I'm not clear on how does SecondaryNameNode communicates with NameNode
    (if deployed on separate machine). Does SecondaryNameNode uses direct
    connection (over some port and protocol) or is it enough for
    SecondaryNameNode to have access to data which NameNode writes locally
    on disk?

    Tomislav
    On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote:
    I think a lot of the confusion comes from this thread :
    http://www.nabble.com/NameNode-failover-procedure-td11711842.html

    Particularly because the wiki was updated with wrong information, not
    maliciously I'm sure. This information is now gone for good.

    Otis, your solution is pretty much like the one given by Dhruba Borthakur
    and augmented by Konstantin Shvachko later in the thread but I never did it
    myself.

    One thing should be clear though, the NN is and will remain a SPOF (just
    like HBase's Master) as long as a distributed manager service (like
    Zookeeper) is not plugged into Hadoop to help with failover.

    J-D

    On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic <
    otis_gospodnetic@yahoo.com> wrote:
    Hi,
    So what is the "recipe" for avoiding NN SPOF using only what comes with
    Hadoop?

    From what I can tell, I think one has to do the following two things:

    1) configure primary NN to save namespace and xa logs to multiple dirs,
    one
    of which is actually on a remotely mounted disk, so that the data actually
    lives on a separate disk on a separate box. This saves namespace and xa
    logs on multiple boxes in case of primary NN hardware failure.

    2) configure secondary NN to periodically merge fsimage+edits and create
    the fsimage checkpoint. This really is a second NN process running on
    another box. It sounds like this secondary NN has to somehow have access
    to
    fsimage & edits files from the primary NN server.
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes
    not describe the best practise around that - the recommended way to
    give secondary NN access to primary NN's fsimage and edits files. Should
    one mount a disk from the primary NN box to the secondary NN box to get
    access to those files? Or is there a simpler way?
    In any case, this checkpoint is just a merge of fsimage+edits files and
    again is there in case the box with the primary NN dies. That's what's
    described on
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore
    or less.
    Is this sufficient, or are there other things one has to do to eliminate
    NN
    SPOF?


    Thanks,
    Otis
    --
    Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch


    ----- Original Message ----
    From: Jean-Daniel Cryans
    To: core-user@hadoop.apache.org
    Sent: Tuesday, October 28, 2008 8:14:44 PM
    Subject: Re: SecondaryNameNode on separate machine

    Tomislav.

    Contrary to popular belief the secondary namenode does not provide failover,
    it's only used to do what is described here :
    http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode
    So the term "secondary" does not mean "a second one" but is more like "a
    second part of".

    J-D
    On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote:

    Hi,
    I'm trying to implement NameNode failover (or at least NameNode local
    data backup), but it is hard since there is no official documentation.
    Pages on this subject are created, but still empty:

    http://wiki.apache.org/hadoop/NameNodeFailover
    http://wiki.apache.org/hadoop/SecondaryNameNode

    I have been browsing the web and hadoop mailing list to see how this
    should be implemented, but I got even more confused. People are asking
    do we even need SecondaryNameNode etc. (since NameNode can write local
    data to multiple locations, so one of those locations can be a mounted
    disk from other machine). I think I understand the motivation for
    SecondaryNameNode (to create a snapshoot of NameNode data every n
    seconds/hours), but setting (deploying and running) SecondaryNameNode
    on
    different machine than NameNode is not as trivial as I expected. First
    I
    found that if I need to run SecondaryNameNode on other machine than
    NameNode I should change masters file on NameNode (change localhost to
    SecondaryNameNode host) and set some properties in hadoop-site.xml on
    SecondaryNameNode (fs.default.name, fs.checkpoint.dir,
    fs.checkpoint.period etc.)

    This was enough to start SecondaryNameNode when starting NameNode with
    bin/start-dfs.sh , but it didn't create image on SecondaryNameNode.
    Then
    I found that I need to set dfs.http.address on NameNode address (so now
    I have NameNode address in both fs.default.name and dfs.http.address).

    Now I get following exception:

    2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in
    doCheckpoint:
    2008-10-28 09:18:00,098 ERROR NameNode.Secondary -
    java.net.SocketException: Unexpected end of file from server

    My questions are following:
    How to resolve this problem (this exception)?
    Do I need additional property in SecondaryNameNode's hadoop-site.xml or
    NameNode's hadoop-site.xml?

    How should NameNode failover work ideally? Is it like this:

    SecondaryNameNode runs on separate machine than NameNode and stores
    NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir.
    When NameNode machine crashes, we start NameNode on machine where
    SecondaryNameNode was running and we set dfs.name.dir to
    fs.checkpoint.dir. Also we need to change how DNS resolves NameNode
    hostname (change from the primary to the secondary).

    Is this correct ?

    Tomislav

  • Yossale at Nov 12, 2008 at 4:09 pm

    Now SecondaryNameNode connects to the NameNode (after I configured
    dfs.http.address to the NN's http server -> NN hostname on port 50070)
    and creates(transfers) edits and fsimage from NameNode.
    It didn't work for me - I get an error:
    java.io.FileNotFoundException:
    http://192.168.30.5:50070/getimage?putimage=1&port=50090&machine=127.0.0.1&token=-16:1173009257:0:1226503705000:1226503705207
    at
    sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1168)
    at
    org.apache.hadoop.dfs.TransferFsImage.getFileClient(TransferFsImage.java:150)
    at
    org.apache.hadoop.dfs.SecondaryNameNode.putFSImage(SecondaryNameNode.java:271)
    at
    org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:311)
    at
    org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:216)
    at java.lang.Thread.run(Thread.java:595)

    And when I run the http request directly (in the browser) , I receive this :
    GetImage failed. java.io.IOException: Namenode is not expecting an new image
    UPLOAD_START
    at
    org.apache.hadoop.dfs.FSImage.validateCheckpointUpload(FSImage.java:1193)
    at org.apache.hadoop.dfs.GetImageServlet.doGet(GetImageServlet.java:57)
    ......

    If it is a mundane thing (i.e "not need to check point now" ) why does it
    throw an Error? What is the "UPLOAD_START" at the end of the message? (if it
    failed , how come it starts?) - but more importantly - how do I get rid of
    it?

    Thanks!


    --
    View this message in context: http://www.nabble.com/SecondaryNameNode-on-separate-machine-tp20207482p20463349.html
    Sent from the Hadoop core-user mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 28, '08 at 1:44p
activeNov 12, '08 at 4:09p
posts14
users6
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase