FAQ
Hi all ,

In HDFS-903 - ( md5 verification of fsimage ) , with this change ,
Backupnode is downloading the image & edit files from namenode everytime
since the difference in checkpoint time is always maintined b/w Namenode and
Backupnode . This happens since Namenode is resetting its checkpoint time
everytime since we are ignoring renewCheckpointTime and passing true
explicitly to rollFsimage during endcheckpoint ..

Also , though a proposal for using md5 to decide whether to download image
is also mentioned , it doesnt seeem to be implemented ..

Isn't this downloading of image everytime a problem or am I missing
something ?

HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo


Solitaire
Domlur
Bangalore
www.huawei.com
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!

Search Discussions

  • Todd Lipcon at May 25, 2011 at 5:45 pm
    Hi Sreehari,

    The BackupNode isn't really production-ized. I don't think anyone uses it,
    so it's not surprising that there are a lot of broken parts.

    If you're interested in contributing in this area, I would encourage you to
    look at the HDFS-1073 branch. The BN and CheckpointNode will be undergoing
    various surgery during the course of this project, so it's a good
    opportunity to fix it up, add test coverage, etc.

    -Todd
    On Wed, May 25, 2011 at 1:08 AM, Sreehari G wrote:

    Hi all ,

    In HDFS-903 - ( md5 verification of fsimage ) , with this change ,
    Backupnode is downloading the image & edit files from namenode everytime
    since the difference in checkpoint time is always maintined b/w Namenode and
    Backupnode . This happens since Namenode is resetting its checkpoint time
    everytime since we are ignoring renewCheckpointTime and passing true
    explicitly to rollFsimage during endcheckpoint ..

    Also , though a proposal for using md5 to decide whether to download image
    is also mentioned , it doesnt seeem to be implemented ..

    Isn't this downloading of image everytime a problem or am I missing
    something ?

    HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]


    Solitaire
    Domlur
    Bangalore
    www.huawei.com

    -------------------------------------------------------------------------------------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Konstantin Shvachko at May 25, 2011 at 7:34 pm
    Hi Sreehari,

    I actually also see this problem. It would be good to fix it.
    You can simply create a jira describing the issue and we can discuss
    there how to fix it. If you have a patch - even better.
    Once fixed in trunk it can be ported to other branches,
    including 0.22 or HDFS-1073-branch if desired.

    Thanks,
    --Konstantin
    On Wed, May 25, 2011 at 10:45 AM, Todd Lipcon wrote:

    Hi Sreehari,

    The BackupNode isn't really production-ized. I don't think anyone uses it,
    so it's not surprising that there are a lot of broken parts.

    If you're interested in contributing in this area, I would encourage you to
    look at the HDFS-1073 branch. The BN and CheckpointNode will be undergoing
    various surgery during the course of this project, so it's a good
    opportunity to fix it up, add test coverage, etc.

    -Todd
    On Wed, May 25, 2011 at 1:08 AM, Sreehari G wrote:

    Hi all ,

    In HDFS-903 - ( md5 verification of fsimage ) , with this change ,
    Backupnode is downloading the image & edit files from namenode everytime
    since the difference in checkpoint time is always maintined b/w Namenode and
    Backupnode . This happens since Namenode is resetting its checkpoint time
    everytime since we are ignoring renewCheckpointTime and passing true
    explicitly to rollFsimage during endcheckpoint ..

    Also , though a proposal for using md5 to decide whether to download image
    is also mentioned , it doesnt seeem to be implemented ..

    Isn't this downloading of image everytime a problem or am I missing
    something ?

    HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]


    Solitaire
    Domlur
    Bangalore
    www.huawei.com

    -------------------------------------------------------------------------------------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Sreehari G at May 27, 2011 at 10:36 am
    Hi all ,

    Thnx Todd & Konstantin .. I've raised HDFS-2007 for addressing this issue ..


    Regards ,
    Sreehari

    HUAWEI TECHNOLOGIES CO.,LTD.
    Solitaire
    Domlur
    Bangalore
    www.huawei.com
    ----------------------------------------------------------------------------
    ---------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above. Any
    use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!



    _____

    From: Konstantin Shvachko
    Sent: Thursday, May 26, 2011 1:04 AM
    To: hdfs-dev@hadoop.apache.org
    Cc: sreehari.g@huawei.com
    Subject: Re: HDFS-903 - Backupnode always downloading image from Namenode
    after this change


    Hi Sreehari,

    I actually also see this problem. It would be good to fix it.
    You can simply create a jira describing the issue and we can discuss
    there how to fix it. If you have a patch - even better.
    Once fixed in trunk it can be ported to other branches,
    including 0.22 or HDFS-1073-branch if desired.

    Thanks,
    --Konstantin


    On Wed, May 25, 2011 at 10:45 AM, Todd Lipcon wrote:


    Hi Sreehari,

    The BackupNode isn't really production-ized. I don't think anyone uses it,
    so it's not surprising that there are a lot of broken parts.

    If you're interested in contributing in this area, I would encourage you to
    look at the HDFS-1073 branch. The BN and CheckpointNode will be undergoing
    various surgery during the course of this project, so it's a good
    opportunity to fix it up, add test coverage, etc.

    -Todd
    On Wed, May 25, 2011 at 1:08 AM, Sreehari G wrote:

    Hi all ,

    In HDFS-903 - ( md5 verification of fsimage ) , with this change ,
    Backupnode is downloading the image & edit files from namenode everytime
    since the difference in checkpoint time is always maintined b/w Namenode and
    Backupnode . This happens since Namenode is resetting its checkpoint time
    everytime since we are ignoring renewCheckpointTime and passing true
    explicitly to rollFsimage during endcheckpoint ..

    Also , though a proposal for using md5 to decide whether to download image
    is also mentioned , it doesnt seeem to be implemented ..

    Isn't this downloading of image everytime a problem or am I missing
    something ?

    HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]


    Solitaire
    Domlur
    Bangalore
    www.huawei.com
    ----------------------------------------------------------------------------
    ---------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!


    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedMay 25, '11 at 8:16a
activeMay 27, '11 at 10:36a
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase