FAQ
Hi all,
I am not able to understand cleary what *Non-DFS Used *in
Hadoop-Namenode Web UI.
I think it is the extra space that occured by temporary map-reduce local
files.
Can anyone Please tell me how to change that parameter and what it is
comprised of.

Thanks in Advance.

Search Discussions

  • Michael Thomas at Aug 17, 2010 at 3:37 pm

    On 08/17/2010 05:23 AM, Adarsh Sharma wrote:
    Hi all,
    I am not able to understand cleary what *Non-DFS Used *in
    Hadoop-Namenode Web UI.
    I think it is the extra space that occured by temporary map-reduce local
    files.
    Can anyone Please tell me how to change that parameter and what it is
    comprised of.
    "Non-DFS used" is any space on the datanode's ${dfs.data.dir}
    _partition_ that is not part of the ${dfs.data.dir}/current _directory_.
    If your log files or OS share the same partition, then the non-HDFS
    space will be reported as "Non-DFS used". If you have extra files in MR
    may also make use of this partition, but I can't comment on what it
    might place there. In the past we have seen non-dfs files in this
    partition, including:

    * files in ${dfs.data.dir}/tmp that were stale in-transit blocks left
    behind after a node crash

    * Hadoop log files that we misconfigured to write to the same partition
    as ${dfs.data.dir}

    * Other non-Hadoop batch system files that were sharing the same
    partition, before we moved HDFS and our batch system to separate partitions.

    If you have a persistent large amount of "Non-DFS Used", then you might
    start by running df and du on your ${dfs.data.dir} partition and
    directory structure to track down what these non-dfs files might be.

    --Mike

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 17, '10 at 3:28p
activeAug 17, '10 at 3:37p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Michael Thomas: 1 post Adarsh Sharma: 1 post

People

Translate

site design / logo © 2022 Grokbase