FAQ
Slow namenode webUI (Taking 1 minute to load on 2000 nodes cluster.)
--------------------------------------------------------------------

Key: HADOOP-3909
URL: https://issues.apache.org/jira/browse/HADOOP-3909
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.17.1
Reporter: Koji Noguchi


Showing the namenode webui takes a long time with large cluster
and also adds high cpu load to the namenode.
Is this expected?

WHILE pulling the namenode webUI, top showed

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20957 hadoop 16 0 20.3g 19g 4732 S 131 63.7 9767:05 java

AFTER the pull

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20957 hadoop 16 0 20.3g 19g 4732 S 24 63.7 9768:12 java

It would be nice if we can improve this especially since we tend to hit the webUI when namenode is having trouble.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Koji Noguchi (JIRA) at Aug 6, 2008 at 5:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620354#action_12620354 ]

    Koji Noguchi commented on HADOOP-3909:
    --------------------------------------

    I should have quoted the top output.

    WHILE pulling the namenode webUI, top showed cpu 131%
    {noformat}
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    20957 hdfs 16 0 20.3g 19g 4732 S 131 63.7 9767:05 java
    {noformat}

    AFTER the pull cpu 24%

    {noformat}
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    20957 hdfs 16 0 20.3g 19g 4732 S 24 63.7 9768:12 java
    {noformat}
    Slow namenode webUI (Taking 1 minute to load on 2000 nodes cluster.)
    --------------------------------------------------------------------

    Key: HADOOP-3909
    URL: https://issues.apache.org/jira/browse/HADOOP-3909
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.1
    Reporter: Koji Noguchi

    Showing the namenode webui takes a long time with large cluster
    and also adds high cpu load to the namenode.
    Is this expected?
    WHILE pulling the namenode webUI, top showed
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    20957 hadoop 16 0 20.3g 19g 4732 S 131 63.7 9767:05 java
    AFTER the pull
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    20957 hadoop 16 0 20.3g 19g 4732 S 24 63.7 9768:12 java
    It would be nice if we can improve this especially since we tend to hit the webUI when namenode is having trouble.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at Aug 6, 2008 at 6:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620356#action_12620356 ]

    Raghu Angadi commented on HADOOP-3909:
    --------------------------------------

    This is because of DatanodeDescriptor.numBlocks(). This iterates over *all* the blocks for a node to count them. So the
    the call iterates over all the blocks (including the replicas) every time front page is loaded.

    We either need to make block count optional on the front page or keep a count of blocks with each datanode descriptor.

    Slow namenode webUI (Taking 1 minute to load on 2000 nodes cluster.)
    --------------------------------------------------------------------

    Key: HADOOP-3909
    URL: https://issues.apache.org/jira/browse/HADOOP-3909
    Project: Hadoop Core
    Issue Type: Improvement
    Components: dfs
    Affects Versions: 0.17.1
    Reporter: Koji Noguchi

    Showing the namenode webui takes a long time with large cluster
    and also adds high cpu load to the namenode.
    Is this expected?
    WHILE pulling the namenode webUI, top showed
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    20957 hadoop 16 0 20.3g 19g 4732 S 131 63.7 9767:05 java
    AFTER the pull
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    20957 hadoop 16 0 20.3g 19g 4732 S 24 63.7 9768:12 java
    It would be nice if we can improve this especially since we tend to hit the webUI when namenode is having trouble.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 6, '08 at 5:57p
activeAug 6, '08 at 6:09p
posts3
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Raghu Angadi (JIRA): 3 posts

People

Translate

site design / logo © 2022 Grokbase