FAQ
Hi,

Is it possible move all the data blocks off a cluster node and then decommision the node?

I'm asking because, now that my MR job is working, I'd like see how things scale. I.e.,
less processing nodes, amount of data (number & size of files, etc.). I currently have 8 nodes,
and am processing 5GB spread across 2000 files.

Alan

Search Discussions

  • Ken Goodhope at Jul 6, 2010 at 3:35 pm
    Inside the hdfs conf,

    <property>
    <name>dfs.hosts.exclude</name>
    <value></value>
    <description>Names a file that contains a list of hosts that are
    not permitted to connect to the namenode. The full pathname of the
    file must be specified. If the value is empty, no hosts are
    excluded.</description>
    </property>

    Point this property at a file containing a list of nodes you want to
    decommision. From there, use the command line "hadoop dfsadmin
    -refreshNodes".

    On Tue, Jul 6, 2010 at 7:31 AM, Some Body wrote:

    Hi,

    Is it possible move all the data blocks off a cluster node and then
    decommision the node?

    I'm asking because, now that my MR job is working, I'd like see how things
    scale. I.e.,
    less processing nodes, amount of data (number & size of files, etc.). I
    currently have 8 nodes,
    and am processing 5GB spread across 2000 files.

    Alan
  • Michael Segel at Jul 6, 2010 at 3:36 pm
    Alan,

    You shouldn't need to worry about moving the data blocks off your node, assuming you're replicating your blocks 3x times. (I think its the default...)

    You can bring down your node, and within 15 mins, Hadoop will recognize that node as down...

    I think if you do a $> hadoop fsck / your system will recognize that those blocks are under replicated and will replicate the blocks on another machine.

    If you're decommissioning ?sp? a node, then you're taking the machine out of the cluster permanently.

    I'm also not sure how dropping a node will test the scalability. You would be testing resilience.

    HTH

    -Mike

    Date: Tue, 6 Jul 2010 16:31:58 +0200
    To: common-user@hadoop.apache.org
    From: somebody@squareplanet.de
    Subject: decomission a node

    Hi,

    Is it possible move all the data blocks off a cluster node and then decommision the node?

    I'm asking because, now that my MR job is working, I'd like see how things scale. I.e.,
    less processing nodes, amount of data (number & size of files, etc.). I currently have 8 nodes,
    and am processing 5GB spread across 2000 files.

    Alan
    _________________________________________________________________
    Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
    http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
  • Allen Wittenauer at Jul 6, 2010 at 6:07 pm

    On Jul 6, 2010, at 8:35 AM, Michael Segel wrote:
    I'm also not sure how dropping a node will test the scalability. You would be testing resilience.
    He's testing scale down, not scale up (which is the way we normally think of things... I was confused by the wording too).

    In other words, "if I drop a node, how much of a performance hit is my job going to take?"

    Also, for this type of timing/testing, I'd probably make sure speculative execution is off. It will likely throw some curve balls into the time.
  • Alan Miller at Jul 7, 2010 at 7:07 am
    Yes the effect of "scaling down" was the first thing I wanted to look at.
    To process X GB it currently takes Y seconds with Z nodes.
    If I process X GB with Z/2 nodes, does it take Y/2 seconds?
    How about Z-1,Z-2,Z-3,.... nodes?

    Right now my MR job process alot of small files (2000 files, @2.5MB each)
    individually, so the next test would involve changing my MR job to combine
    the small files into bigger pieces (closer to hdfs block size) and see
    if that
    is more effective.

    Each line of my small files has a timestamp column and 55 columns with
    numerical data and my reducer needs to calc the column averages for
    certain time periods (last day, last hour,etc.) based on the timestamp.

    Alan
    On 07/06/2010 08:06 PM, Allen Wittenauer wrote:
    On Jul 6, 2010, at 8:35 AM, Michael Segel wrote:

    I'm also not sure how dropping a node will test the scalability. You would be testing resilience.
    He's testing scale down, not scale up (which is the way we normally think of things... I was confused by the wording too).

    In other words, "if I drop a node, how much of a performance hit is my job going to take?"

    Also, for this type of timing/testing, I'd probably make sure speculative execution is off. It will likely throw some curve balls into the time.
  • Michael Segel at Jul 7, 2010 at 2:01 pm
    This is pretty predictable.
    Determine the average time it takes to process a m/r task.
    If you can process 100 m/r tasks simultaneously, and then cut that to 50 m/r tasks you can handle simultaneously, your job will take twice as long to run.

    Granted this will give you a rough estimate of how long it will take your job to run.

    HTH

    -Mike
    Date: Wed, 7 Jul 2010 09:00:47 +0200
    From: somebody@squareplanet.de
    To: common-user@hadoop.apache.org
    Subject: Re: decomission a node

    Yes the effect of "scaling down" was the first thing I wanted to look at.
    To process X GB it currently takes Y seconds with Z nodes.
    If I process X GB with Z/2 nodes, does it take Y/2 seconds?
    How about Z-1,Z-2,Z-3,.... nodes?

    Right now my MR job process alot of small files (2000 files, @2.5MB each)
    individually, so the next test would involve changing my MR job to combine
    the small files into bigger pieces (closer to hdfs block size) and see
    if that
    is more effective.

    Each line of my small files has a timestamp column and 55 columns with
    numerical data and my reducer needs to calc the column averages for
    certain time periods (last day, last hour,etc.) based on the timestamp.

    Alan
    On 07/06/2010 08:06 PM, Allen Wittenauer wrote:
    On Jul 6, 2010, at 8:35 AM, Michael Segel wrote:

    I'm also not sure how dropping a node will test the scalability. You would be testing resilience.
    He's testing scale down, not scale up (which is the way we normally think of things... I was confused by the wording too).

    In other words, "if I drop a node, how much of a performance hit is my job going to take?"

    Also, for this type of timing/testing, I'd probably make sure speculative execution is off. It will likely throw some curve balls into the time.
    _________________________________________________________________
    The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
    http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 6, '10 at 2:36p
activeJul 7, '10 at 2:01p
posts6
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase