FAQ
Hadoop DFS to report more replication metrics
---------------------------------------------

Key: HADOOP-3058
URL: https://issues.apache.org/jira/browse/HADOOP-3058
Project: Hadoop Core
Issue Type: Improvement
Components: metrics
Reporter: Marco Nicosia
Priority: Minor


Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'

We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Robert Chansler (JIRA) at Apr 3, 2008 at 1:08 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-3058:
    ------------------------------------

    Component/s: dfs
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Priority: Minor

    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Marco Nicosia (JIRA) at Apr 14, 2008 at 10:00 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588813#action_12588813 ]

    Marco Nicosia commented on HADOOP-3058:
    ---------------------------------------

    Now that this has been scheduled for a release, I realize that there were a few metrics that I should also have asked for. Hopefully these are not significant changes in scope.

    In addition to the above, I forgot to ask for the most basic stats. It's very important that the NN send metrics on the number of files and blocks in the system, so that we can trend these over time. Including the number of directories would be a bonus.

    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Priority: Minor

    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at Apr 25, 2008 at 9:15 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu reassigned HADOOP-3058:
    ----------------------------------------

    Assignee: lohit vijayarenu
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor

    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 2, 2008 at 7:08 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Attachment: HADOOP-3058.patch

    Attaching a patch which addes FSNamesystem status metrics.
    Since these are not timevarying int or timevaying rate. I use MetricsLongValue similar to MetricsIntValue using their set and get methods. The metrics are recorded as FSNamesystem record and list these
    - FilesTotal
    - BlocksTotal
    - CapacityTotal
    - CapacityUsed
    - CapacityRemaining
    - TotalLoad
    - PendingReplicationBlocks
    - UnderReplicatedBlocks
    - ScheduledReplicationBlocks

    Tested this using FileContext to log these while FSNamesystem was reporting and I could see the values being updated
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 2, 2008 at 7:10 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Release Note: This JIRA adds new FSNamesystem status metrics.
    Status: Patch Available (was: Open)
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 2, 2008 at 10:18 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593915#action_12593915 ]

    Raghu Angadi commented on HADOOP-3058:
    --------------------------------------

    +1 Patch looks fine. I have no idea if this has real penalty on namenode performance. Mostly it does not affect. Multiple simple integer and long operation in the critical paths are replaced by methods that synchronize on a different object.

    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 3, 2008 at 1:18 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593956#action_12593956 ]

    Hadoop QA commented on HADOOP-3058:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12381301/HADOOP-3058.patch
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included -1. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    core tests -1. The patch failed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2365/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2365/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2365/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2365/console

    This message is automatically generated.
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 3, 2008 at 7:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Status: Open (was: Patch Available)
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 3, 2008 at 7:34 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Status: Patch Available (was: Open)
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 3, 2008 at 7:34 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Attachment: HADOOP-3058-2.patch

    FSNameSystem has 2 constructors and registerMBean was being called only one place. This was causing NPE, fixed it in the updated patch
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 4, 2008 at 4:11 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594058#action_12594058 ]

    Hadoop QA commented on HADOOP-3058:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12381365/HADOOP-3058-2.patch
    against trunk revision 645773.

    @author +1. The patch does not contain any @author tags.

    tests included -1. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    javadoc +1. The javadoc tool did not generate any warning messages.

    javac +1. The applied patch does not generate any new javac compiler warnings.

    findbugs +1. The patch does not introduce any new Findbugs warnings.

    release audit +1. The applied patch does not generate any new release audit warnings.

    core tests +1. The patch passed core unit tests.

    contrib tests +1. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2387/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2387/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2387/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2387/console

    This message is automatically generated.
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 5, 2008 at 6:52 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594299#action_12594299 ]

    lohit vijayarenu commented on HADOOP-3058:
    ------------------------------------------

    Yes, I also agree that these adds more operations. The metrics frequently updated are
    - filesTotal which are updated whenever we add/delete new files.
    - blocksTotal which are updated whenever we add/delete new blocks
    I guess it should be fine in the above case.

    Few metrics are replaced by updating a global variables regarding the DFS capacity. These were updated on each heart beat once, which should be fine.
    Another set of operations are done by ReplicationMonitor in ComputeDatanodeWork(), which should also be fine.
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 5, 2008 at 7:22 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594314#action_12594314 ]

    Raghu Angadi commented on HADOOP-3058:
    --------------------------------------


    I would think the heartBeat variables will be updated thousands of times every second. I think better approach would be to update the heavy weight metric variables only inside {{FSNamesystemMetrics.doUpdates()}} that gets called every 5 seconds or so. This way these stats become pretty much free and also sets good precedence for new metrics.
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 7, 2008 at 9:12 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Status: Open (was: Patch Available)
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 7, 2008 at 9:12 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Attachment: HADOOP-3058-3.patch

    Thanks Rahgu, I have attached an updated patch in which we maintain local counters in FSNamesystem as earlier. FSNamesystemMetrics object is updated only during invoking of doUpdates()
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • lohit vijayarenu (JIRA) at May 7, 2008 at 9:12 am
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    lohit vijayarenu updated HADOOP-3058:
    -------------------------------------

    Status: Patch Available (was: Open)
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 7, 2008 at 7:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595014#action_12595014 ]

    Hadoop QA commented on HADOOP-3058:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12381578/HADOOP-3058-3.patch
    against trunk revision 654128.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    +1 contrib tests. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2418/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2418/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2418/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2418/console

    This message is automatically generated.
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 7, 2008 at 8:32 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595029#action_12595029 ]

    Raghu Angadi commented on HADOOP-3058:
    --------------------------------------

    I just committed this. Thanks Lohit!
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 7, 2008 at 8:40 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Raghu Angadi updated HADOOP-3058:
    ---------------------------------

    Resolution: Fixed
    Fix Version/s: 0.18.0
    Hadoop Flags: [Reviewed]
    Status: Resolved (was: Patch Available)
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Fix For: 0.18.0

    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at May 8, 2008 at 12:25 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595211#action_12595211 ]

    Hudson commented on HADOOP-3058:
    --------------------------------

    Integrated in Hadoop-trunk #484 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/484/])
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: lohit vijayarenu
    Priority: Minor
    Fix For: 0.18.0

    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Robert Chansler (JIRA) at Jun 27, 2008 at 9:06 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-3058:
    ------------------------------------

    Release Note: Added FSNamesystem status metrics. (was: This JIRA adds new FSNamesystem status metrics. )
    Hadoop DFS to report more replication metrics
    ---------------------------------------------

    Key: HADOOP-3058
    URL: https://issues.apache.org/jira/browse/HADOOP-3058
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs, metrics
    Reporter: Marco Nicosia
    Assignee: Lohit Vijayarenu
    Priority: Minor
    Fix For: 0.18.0

    Attachments: HADOOP-3058-2.patch, HADOOP-3058-3.patch, HADOOP-3058.patch


    Currently, the namenode and each datanode reports 'blocksreplicatedpersec.'
    We'd like to be able to graph pending replications, vs number of under replicated blocks, vs. replications per second, so that we can get a better idea of the replication activity within the DFS.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 20, '08 at 5:46a
activeJun 27, '08 at 9:06p
posts22
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Robert Chansler (JIRA): 22 posts

People

Translate

site design / logo © 2022 Grokbase