FAQ
Transfer of image from secondary name node should not interrupt service
-----------------------------------------------------------------------

Key: HADOOP-3631
URL: https://issues.apache.org/jira/browse/HADOOP-3631
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.17.0
Reporter: Robert Chansler
Priority: Critical
Fix For: 0.19.0


The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)

Maybe the new image does not have to be transfered; it could be fetched when needed.

Maybe the priority of the transfer task can be reduced so that the transfer is not observed.

Maybe a different transfer protocol is more appropriate.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Lohit Vijayarenu (JIRA) at Jul 24, 2008 at 8:57 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616652#action_12616652 ]

    Lohit Vijayarenu commented on HADOOP-3631:
    ------------------------------------------

    On trunk this the sequence of events that happen during checkpoint.
    - SecondaryNameNode request rollEditLog() and NameNode would close edits and starts writing to edits.new. (This should not take much time)
    - SecondaryNameNode gets edits from NameNode via HTTP (This might take time depending on the size of edits)
    - SecondaryNameNode merges edits with image creating fsimage.ckpt (This wont affect NameNode)
    - SecondaryNameNode transfers the image to NameNode via HTTP (This again takes time depending on the size of new image)
    - SecondaryNameNode request rollFSImage() and NameNode renames edits.new to edits and fsimage.ckpt to fsimage. (This should not take much time)

    Most of the time is spent in transferring the Image over HTTP. One way out of this is to not do the transfer. Configure a shared directory and SecondaryNameNode reads and writes to that directory requesting NameNode to read from in there. This could be a special startup option for SecondarNameNode.
    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical
    Fix For: 0.19.0


    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Lohit Vijayarenu (JIRA) at Aug 6, 2008 at 4:33 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620305#action_12620305 ]

    Lohit Vijayarenu commented on HADOOP-3631:
    ------------------------------------------

    After talking to Konstantin realized that having a shared directory would only get rid of transferring the image back and forth between NameNode and SecondaryNameNode. Since we have a requirement of maintaining the updated image consistent on all dfs.name.dir configured directories, we would still have to copy the image from shared directory to those directories. This does not solve the contention to the disk since we save image and edits on same disk. Many ppl already have suggested having edits and image in separate disks which would help us in this case. So, the above [approach|https://issues.apache.org/jira/browse/HADOOP-3631?focusedCommentId=12616652#action_12616652] wont solve the actual problem.
    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical
    Fix For: 0.19.0


    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Aug 6, 2008 at 5:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620335#action_12620335 ]

    dhruba borthakur commented on HADOOP-3631:
    ------------------------------------------

    The problem arises because the secondary namenode reads/writes to/from the same disk device on which the primary namenode is writing transactions. One approach would be to allow the namenode to use three separate physicl disk devices, D1, D2 D3.

    Initial : D1 has the fsimage
    D2 has edits log
    D3 none

    RollEditLog does the following: D1 : fsimage
    D2: edits
    D3 edits.new

    Now, the Secondary fetches fsimage and edits, merged them and uploads them back as follows:
    D1 fsimage, fsimage.ckpt
    D2: edits
    D3 edits.new

    This download and upload of files by the secondary does not affect namenode performance because namenode perfrmance is dependent on performance of transaction log that is occuring on disk device D3. This is not touched by Secondary namenode.

    The, the primary namenode atomically switches fsimage/edits (moves fsimage.ckpt to fsimage, removes edits, renames edits.mew to edits) and we are lft with the following:
    D1 : fsimage
    D2: none
    D3: edits

    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical
    Fix For: 0.19.0


    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Lohit Vijayarenu (JIRA) at Aug 6, 2008 at 9:57 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620442#action_12620442 ]

    Lohit Vijayarenu commented on HADOOP-3631:
    ------------------------------------------

    Thanks Dhruba. The idea of 3 separate disks looks very good, but if we are maintaining multiple copies in dfs.name.dir, then number of disks might be a problem.
    we might also have the option of storing image and edits in two separate disks, but then again there would be contention for writing of edits while it is being read for transfer. I ran namenode and secondarynamenode to monitor the heap, there was not much of a problem regarding heap space while this transfer happened. Koji also helped me point to a case where we saw the client timeout happened, I could see that sync time and write time was high on the disk causing namenode to slow down.
    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical
    Fix For: 0.19.0


    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Aug 8, 2008 at 5:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620842#action_12620842 ]

    dhruba borthakur commented on HADOOP-3631:
    ------------------------------------------

    My proposal actually needs three disks irrespective of the number of namedirs. For starters, the fsimage on all the namedirs will use D1, the edits file on all the namedirs will be on D2 and the edits.new on all namedirs will be on D3.

    Also, since edits and edits.new are on different disks, it is never the case that the Secondary will read/write to the same disk on which the namenode is currently logging transactions.
    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical
    Fix For: 0.19.0


    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Robert Chansler (JIRA) at Sep 15, 2008 at 11:28 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-3631:
    ------------------------------------

    Fix Version/s: (was: 0.19.0)

    I've unassigned this from 0.19 as no additional work is contemplated. Since the measured transfer times are small, and clients no longer time out, the impact on users is minimal in 0.19.
    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical

    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Robert Chansler (JIRA) at Nov 15, 2008 at 12:51 am
    [ https://issues.apache.org/jira/browse/HADOOP-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler resolved HADOOP-3631.
    -------------------------------------

    Resolution: Won't Fix

    Seems to no longer be a problem. In any case, no work is contemplated.
    Transfer of image from secondary name node should not interrupt service
    -----------------------------------------------------------------------

    Key: HADOOP-3631
    URL: https://issues.apache.org/jira/browse/HADOOP-3631
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.0
    Reporter: Robert Chansler
    Priority: Critical

    The transfer of the new image prepared by the secondary name node can interfere with client services. Clients observe delays in completing RPCs. In general, administrative activities should not be observed by the clients. For large clusters, administrators are reluctant to run the secondary name node leading to excessive edit logs. (Excessive in the sense that if the cluster must be restarted, a long time is required to process the log.)
    Maybe the new image does not have to be transfered; it could be fetched when needed.
    Maybe the priority of the transfer task can be reduced so that the transfer is not observed.
    Maybe a different transfer protocol is more appropriate.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 24, '08 at 7:45p
activeNov 15, '08 at 12:51a
posts8
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Robert Chansler (JIRA): 8 posts

People

Translate

site design / logo © 2022 Grokbase