FAQ
Hi all ,

What is the expectation in the following scenario :

Namenode has 3 name dirs configured ..
1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded
(2nd and 3rd dir is left unchanged ..)
2) Namenode starts
3) Namenode shutdown and rollbacked

Are the new changes done meant to be visible ? With current implementation ,
after a rollback new changes are visible .. Is this expected ??

As per my observation , since Namenode is saving the latest image dir(the
upgraded 1st dir since checkpointtime is incremented during upgrade for this
dir) will be loaded and saved to all dirs during loadfsimage ..

But if a ROLLBACK is done , the 1st dir will be rolled back (the older copy
becomes current and its checkpointtime is now LESS than other dirs ..) and
others left behind since they dont contain previous .. Now during
loadfsimage , the 2nd dir will be selected since it has the highest
checkpoint time and saved to all dirs (including 1st ) .. Now due to this ,
the new changes b/w UPGRADE and ROLLBACK DELTA present in 2nd dir gets
reflected even after ROLLBACK ..

Is this expected ? I ask this since :

1) this is not the case with a SUCCESSFULL Upgrade/Rollback (New changes are
lost after rollback).. &
2) In Hadoop-702 (DFS Upgrade Proposal) , FSStateTransition7.htm says { r0.
if previous directory does not exist then fail rollback } , but it continues
in the current implementation ..



Regards ,

Sreehari

HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo


Solitaire
Domlur
Bangalore
www.huawei.com
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!

Search Discussions

  • Todd Lipcon at May 30, 2011 at 4:26 pm
    Hi Sreehari,

    Which versions are you using for this test?

    In step 1 below, how does the upgrade fail? Can you provide specific steps
    to reproduce?

    Smells like a bug to me, if I understand the scenario correctly.

    -Todd
    On Mon, May 30, 2011 at 5:29 AM, Sreehari G wrote:

    Hi all ,

    What is the expectation in the following scenario :

    Namenode has 3 name dirs configured ..
    1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded
    (2nd and 3rd dir is left unchanged ..)
    2) Namenode starts
    3) Namenode shutdown and rollbacked

    Are the new changes done meant to be visible ? With current implementation
    , after a rollback new changes are visible .. Is this expected ??

    As per my observation , since Namenode is saving the latest image dir(the
    upgraded 1st dir since checkpointtime is incremented during upgrade for this
    dir) will be loaded and saved to all dirs during loadfsimage ..

    But if a ROLLBACK is done , the 1st dir will be rolled back (the older copy
    becomes current and its checkpointtime is now LESS than other dirs ..) and
    others left behind since they dont contain previous .. Now during
    loadfsimage , the 2nd dir will be selected since it has the highest
    checkpoint time and saved to all dirs (including 1st ) .. Now due to this ,
    the new changes b/w UPGRADE and ROLLBACK DELTA present in 2nd dir gets
    reflected even after ROLLBACK ..

    Is this expected ? I ask this since :

    1) this is not the case with a SUCCESSFULL Upgrade/Rollback (New changesare lost
    after rollback).. &
    2) In Hadoop-702 (DFS Upgrade Proposal) , FSStateTransition7.htm says {
    r0. if previous directory does not exist then fail rollback } , but it
    continues in the current implementation ..



    Regards ,

    Sreehari

    HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]


    Solitaire
    Domlur
    Bangalore
    www.huawei.com

    -------------------------------------------------------------------------------------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Sreehari G at May 31, 2011 at 9:18 am
    Hi Todd ,

    Steps to reproduce :

    1) Configure n>1 ,say 3 name dirs
    2) Upgrade Namenode - After upgrade is over for 1st dir , stop/kill the
    namenode
    3) Start namenode and write some files (delta) & then stop
    4) Rollback namenode

    Now , after (4) , namenode will start normally , but it will contain the
    delta , since the unrollbacked 2nd dir will be the latest one (with the
    greatest checkpointtime ) & during savenamespace during loadFSImage() , the
    2nd dir image will be saved to all the 3 dirs ..

    The system could 've gone back to the state before rollback - a FAILED
    Rolback which i think would 've been fine rather than the current behaviour
    ..

    I'am using 20.1 , but I verified in latest trunk also ..


    -- Sreehari

    HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo


    Solitaire
    Domlur
    Bangalore
    www.huawei.com
    ----------------------------------------------------------------------------
    ---------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above. Any
    use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!



    _____

    From: Todd Lipcon
    Sent: Monday, May 30, 2011 9:56 PM
    To: hdfs-dev@hadoop.apache.org; sreehari.g@huawei.com
    Subject: Re: Expectation during Rollback after a partially failed Upgrade ..
    Should DELTA be visible ?


    Hi Sreehari,

    Which versions are you using for this test?

    In step 1 below, how does the upgrade fail? Can you provide specific steps
    to reproduce?

    Smells like a bug to me, if I understand the scenario correctly.

    -Todd


    On Mon, May 30, 2011 at 5:29 AM, Sreehari G wrote:


    Hi all ,

    What is the expectation in the following scenario :

    Namenode has 3 name dirs configured ..
    1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded
    (2nd and 3rd dir is left unchanged ..)
    2) Namenode starts
    3) Namenode shutdown and rollbacked

    Are the new changes done meant to be visible ? With current implementation ,
    after a rollback new changes are visible .. Is this expected ??

    As per my observation , since Namenode is saving the latest image dir(the
    upgraded 1st dir since checkpointtime is incremented during upgrade for this
    dir) will be loaded and saved to all dirs during loadfsimage ..

    But if a ROLLBACK is done , the 1st dir will be rolled back (the older copy
    becomes current and its checkpointtime is now LESS than other dirs ..) and
    others left behind since they dont contain previous .. Now during
    loadfsimage , the 2nd dir will be selected since it has the highest
    checkpoint time and saved to all dirs (including 1st ) .. Now due to this ,
    the new changes b/w UPGRADE and ROLLBACK DELTA present in 2nd dir gets
    reflected even after ROLLBACK ..

    Is this expected ? I ask this since :

    1) this is not the case with a SUCCESSFULL Upgrade/Rollback (New changes are
    lost after rollback).. &
    2) In Hadoop-702 (DFS Upgrade Proposal) , FSStateTransition7.htm says { r0.
    if previous directory does not exist then fail rollback } , but it continues
    in the current implementation ..



    Regards ,

    Sreehari





    HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo




    Solitaire
    Domlur
    Bangalore
    www.huawei.com
    ----------------------------------------------------------------------------
    ---------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above. Any
    use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!






    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Todd Lipcon at May 31, 2011 at 3:25 pm

    On Tue, May 31, 2011 at 2:15 AM, Sreehari G wrote:

    Hi Todd ,

    Steps to reproduce :

    1) Configure n>1 ,say 3 name dirs
    2) Upgrade Namenode - After upgrade is over for 1st dir , stop/kill
    the namenode
    Are you running the "upgrade" command with 0.21?

    3) Start namenode and write some files (delta) & then stop
    Which version do you restart with? 0.21 again or 0.20?

    4) Rollback namenode
    Which version do you run the rollback command with?

    Now , after (4) , namenode will start normally , but it will contain the
    delta , since the unrollbacked 2nd dir will be the latest one (with the
    greatest checkpointtime ) & during savenamespace during loadFSImage() , the
    2nd dir image will be saved to all the 3 dirs ..

    The system could 've gone back to the state before rollback - a FAILED
    Rolback which i think would 've been fine rather than the current behaviour
    ..

    I'am using 20.1 , but I verified in latest trunk also ..


    -- Sreehari

    HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]


    Solitaire
    Domlur
    Bangalore
    www.huawei.com

    -------------------------------------------------------------------------------------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!


    ------------------------------
    *From:* Todd Lipcon
    *Sent:* Monday, May 30, 2011 9:56 PM
    *To:* hdfs-dev@hadoop.apache.org; sreehari.g@huawei.com
    *Subject:* Re: Expectation during Rollback after a partially failed
    Upgrade .. Should DELTA be visible ?

    Hi Sreehari,

    Which versions are you using for this test?

    In step 1 below, how does the upgrade fail? Can you provide specific steps
    to reproduce?

    Smells like a bug to me, if I understand the scenario correctly.

    -Todd
    On Mon, May 30, 2011 at 5:29 AM, Sreehari G wrote:

    Hi all ,

    What is the expectation in the following scenario :

    Namenode has 3 name dirs configured ..
    1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded
    (2nd and 3rd dir is left unchanged ..)
    2) Namenode starts
    3) Namenode shutdown and rollbacked

    Are the new changes done meant to be visible ? With current implementation
    , after a rollback new changes are visible .. Is this expected ??

    As per my observation , since Namenode is saving the latest image dir(the
    upgraded 1st dir since checkpointtime is incremented during upgrade for this
    dir) will be loaded and saved to all dirs during loadfsimage ..

    But if a ROLLBACK is done , the 1st dir will be rolled back (the older
    copy becomes current and its checkpointtime is now LESS than other dirs ..)
    and others left behind since they dont contain previous .. Now during
    loadfsimage , the 2nd dir will be selected since it has the highest
    checkpoint time and saved to all dirs (including 1st ) .. Now due to this ,
    the new changes b/w UPGRADE and ROLLBACK DELTA present in 2nd dir gets
    reflected even after ROLLBACK ..

    Is this expected ? I ask this since :

    1) this is not the case with a SUCCESSFULL Upgrade/Rollback (New changesare lost
    after rollback).. &
    2) In Hadoop-702 (DFS Upgrade Proposal) , FSStateTransition7.htm says {
    r0. if previous directory does not exist then fail rollback } , but it
    continues in the current implementation ..



    Regards ,

    Sreehari

    HUAWEI TECHNOLOGIES CO.,LTD. [image: huawei_logo]


    Solitaire
    Domlur
    Bangalore
    www.huawei.com

    -------------------------------------------------------------------------------------------------------------------------------------
    This e-mail and its attachments contain confidential information from
    HUAWEI, which
    is intended only for the person or entity whose address is listed above.
    Any use of the
    information contained herein in any way (including, but not limited to,
    total or partial
    disclosure, reproduction, or dissemination) by persons other than the
    intended
    recipient(s) is prohibited. If you receive this e-mail in error, please
    notify the sender by
    phone or email immediately and delete it!


    --
    Todd Lipcon
    Software Engineer, Cloudera


    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedMay 30, '11 at 12:32p
activeMay 31, '11 at 3:25p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Todd Lipcon: 2 posts Sreehari G: 2 posts

People

Translate

site design / logo © 2022 Grokbase