FAQ
Hi,

One of my data nodes is practically under utilized in the cluster of 4
datanodes and one namenode.
I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
files in the HDFS.

All the nodes are balancer except one. Any clues? Shall i run hadoop
balancer after the rep factor is reset.

Thanks,
Usman


-------------------------------------------------
Datanodes available: 4

Name: 1
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 217963111201(202.99 GB)
Used raw bytes: 52500185088 (48.89 GB)
% used: 17.87%
Last contact: Sun Apr 26 15:49:17 UTC 2009


Name: 2
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 227810456463(212.17 GB)
Used raw bytes: 44199026688 (41.16 GB)
% used: 15.04%
Last contact: Sun Apr 26 15:49:19 UTC 2009


Name: 3
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 207735568331(193.47 GB)
Used raw bytes: 63878131712 (59.49 GB)
% used: 21.74%
Last contact: Sun Apr 26 15:49:19 UTC 2009


Name: 4
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 270775018072(252.18 GB)
Used raw bytes: 1916928 (1.83 MB)
% used: 0%
Last contact: Sun Apr 26 15:49:17 UTC 2009


--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

Search Discussions

  • Mithila Nagendra at Apr 26, 2009 at 4:18 pm
    How do I list all the datanodes like you ve shown in your example?
    Thanks
    Mithila
    On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed wrote:

    Hi,

    One of my data nodes is practically under utilized in the cluster of 4
    datanodes and one namenode.
    I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
    files in the HDFS.

    All the nodes are balancer except one. Any clues? Shall i run hadoop
    balancer after the rep factor is reset.

    Thanks,
    Usman


    -------------------------------------------------
    Datanodes available: 4

    Name: 1
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 217963111201(202.99 GB)
    Used raw bytes: 52500185088 (48.89 GB)
    % used: 17.87%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    Name: 2
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 227810456463(212.17 GB)
    Used raw bytes: 44199026688 (41.16 GB)
    % used: 15.04%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 3
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 207735568331(193.47 GB)
    Used raw bytes: 63878131712 (59.49 GB)
    % used: 21.74%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 4
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 270775018072(252.18 GB)
    Used raw bytes: 1916928 (1.83 MB)
    % used: 0%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    --
    Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
  • Jason hadoop at Apr 26, 2009 at 4:38 pm
    hadoop dfsadmin -report
    On Sun, Apr 26, 2009 at 9:18 AM, Mithila Nagendra wrote:

    How do I list all the datanodes like you ve shown in your example?
    Thanks
    Mithila
    On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed wrote:

    Hi,

    One of my data nodes is practically under utilized in the cluster of 4
    datanodes and one namenode.
    I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
    files in the HDFS.

    All the nodes are balancer except one. Any clues? Shall i run hadoop
    balancer after the rep factor is reset.

    Thanks,
    Usman


    -------------------------------------------------
    Datanodes available: 4

    Name: 1
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 217963111201(202.99 GB)
    Used raw bytes: 52500185088 (48.89 GB)
    % used: 17.87%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    Name: 2
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 227810456463(212.17 GB)
    Used raw bytes: 44199026688 (41.16 GB)
    % used: 15.04%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 3
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 207735568331(193.47 GB)
    Used raw bytes: 63878131712 (59.49 GB)
    % used: 21.74%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 4
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 270775018072(252.18 GB)
    Used raw bytes: 1916928 (1.83 MB)
    % used: 0%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    --
    Using Opera's revolutionary e-mail client: http://www.opera.com/mail/


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • Jason hadoop at Apr 26, 2009 at 4:40 pm
    you need to add the -R parameter to setrep to force recursively setting the
    replication, other wise it just affects the single file that is the last
    argument.

    Is there some issue with the one datanode, where it is unable to accept or
    store blocks?
    On Sun, Apr 26, 2009 at 9:37 AM, jason hadoop wrote:

    hadoop dfsadmin -report

    On Sun, Apr 26, 2009 at 9:18 AM, Mithila Nagendra wrote:

    How do I list all the datanodes like you ve shown in your example?
    Thanks
    Mithila
    On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed wrote:

    Hi,

    One of my data nodes is practically under utilized in the cluster of 4
    datanodes and one namenode.
    I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
    files in the HDFS.

    All the nodes are balancer except one. Any clues? Shall i run hadoop
    balancer after the rep factor is reset.

    Thanks,
    Usman


    -------------------------------------------------
    Datanodes available: 4

    Name: 1
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 217963111201(202.99 GB)
    Used raw bytes: 52500185088 (48.89 GB)
    % used: 17.87%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    Name: 2
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 227810456463(212.17 GB)
    Used raw bytes: 44199026688 (41.16 GB)
    % used: 15.04%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 3
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 207735568331(193.47 GB)
    Used raw bytes: 63878131712 (59.49 GB)
    % used: 21.74%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 4
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 270775018072(252.18 GB)
    Used raw bytes: 1916928 (1.83 MB)
    % used: 0%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    --
    Using Opera's revolutionary e-mail client: http://www.opera.com/mail/


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422
  • Usman Waheed at Apr 26, 2009 at 5:27 pm
    Hi,
    I did run the -setrep -w 2 -R /mydir and it basically emptied out one of
    my nodes.
    I just ran the hadoop balancer and now the datanode that was empty is
    picking up blocks.
    So lets see what happens once it is done and i will write back here.
    Thanks,
    Usman
    you need to add the -R parameter to setrep to force recursively setting
    the
    replication, other wise it just affects the single file that is the last
    argument.

    Is there some issue with the one datanode, where it is unable to accept
    or
    store blocks?

    On Sun, Apr 26, 2009 at 9:37 AM, jason hadoop
    wrote:
    hadoop dfsadmin -report


    On Sun, Apr 26, 2009 at 9:18 AM, Mithila Nagendra
    wrote:
    How do I list all the datanodes like you ve shown in your example?
    Thanks
    Mithila
    On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed wrote:

    Hi,

    One of my data nodes is practically under utilized in the cluster of 4
    datanodes and one namenode.
    I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
    files in the HDFS.

    All the nodes are balancer except one. Any clues? Shall i run hadoop
    balancer after the rep factor is reset.

    Thanks,
    Usman


    -------------------------------------------------
    Datanodes available: 4

    Name: 1
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 217963111201(202.99 GB)
    Used raw bytes: 52500185088 (48.89 GB)
    % used: 17.87%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    Name: 2
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 227810456463(212.17 GB)
    Used raw bytes: 44199026688 (41.16 GB)
    % used: 15.04%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 3
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 207735568331(193.47 GB)
    Used raw bytes: 63878131712 (59.49 GB)
    % used: 21.74%
    Last contact: Sun Apr 26 15:49:19 UTC 2009


    Name: 4
    State : In Service
    Total raw bytes: 293778976768 (273.6 GB)
    Remaining raw bytes: 270775018072(252.18 GB)
    Used raw bytes: 1916928 (1.83 MB)
    % used: 0%
    Last contact: Sun Apr 26 15:49:17 UTC 2009


    --
    Using Opera's revolutionary e-mail client: http://www.opera.com/mail/


    --
    Alpha Chapters of my book on Hadoop are available
    http://www.apress.com/book/view/9781430219422


    --
    Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedApr 26, '09 at 3:54p
activeApr 26, '09 at 5:27p
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase