FAQ
Hi list,

How does the NN place blocks on the disks within a single node? Does it spread out adjecent blocks of a single file horizontally over the disks? For example, lets say I have four DN's and each has 4 disks. (And forget about replication.) If I copy a file existing of 16 blocks of 128MB each to the cluster, will each disk have exactly one block of the file?

If I run some job over this file with its sixteen blocks this is important, since the cluster would use its maximum I/O capabilities.

This leads me to another question (which might be better of on mapred-user). Does the JT schedule its tasks to maximally use I/O capabilities? Would it try to process blocks that reside on a disk that is not currently being read from or written to? Or does it just use a randomized strategy?

Cheers,
Evert

Search Discussions

  • Harsh J at Jun 30, 2011 at 3:00 pm
    Evert,

    With the default behavior, every block request is handed a storage
    device in a round robin fashion. So yes, parallel writes should be
    well spread over the configured amount of disks. But that's not to say
    that the file is perfectly distributed across the DNs as you describe.
    The DNs are chosen randomly for writes (if no local one is available).

    Regd. MR, I do not believe it does any such optimization right now (in
    fact, the MR code is quite FS-agnostic). Right now, tasks are run on
    nodes where blocks are located but metadata about which disk the block
    may reside on is not maintained by the NN, so MR can't naturally know
    this to do anything about. This would be good to discuss, however
    (Search or file a new JIRA?)
    On Thu, Jun 30, 2011 at 6:33 PM, Evert Lammerts wrote:
    Hi list,

    How does the NN place blocks on the disks within a single node? Does it spread out adjecent blocks of a single file horizontally over the disks? For example, lets say I have four DN's and each has 4 disks. (And forget about replication.) If I copy a file existing of 16 blocks of 128MB each to the cluster, will each disk have exactly one block of the file?

    If I run some job over this file with its sixteen blocks this is important, since the cluster would use its maximum I/O capabilities.

    This leads me to another question (which might be better of on mapred-user). Does the JT schedule its tasks to maximally use I/O capabilities? Would it try to process blocks that reside on a disk that is not currently being read from or written to? Or does it just use a randomized strategy?

    Cheers,
    Evert


    --
    Harsh J
  • Evert Lammerts at Jul 1, 2011 at 8:30 am
    Well, here's my first Hadoop Jira :-)

    https://issues.apache.org/jira/browse/MAPREDUCE-2636

    Cheers,
    Evert

    ________________________________________
    From: Harsh J [harsh@cloudera.com]
    Sent: Thursday, June 30, 2011 4:59 PM
    To: hdfs-user@hadoop.apache.org
    Subject: Re: block placement

    Evert,

    With the default behavior, every block request is handed a storage
    device in a round robin fashion. So yes, parallel writes should be
    well spread over the configured amount of disks. But that's not to say
    that the file is perfectly distributed across the DNs as you describe.
    The DNs are chosen randomly for writes (if no local one is available).

    Regd. MR, I do not believe it does any such optimization right now (in
    fact, the MR code is quite FS-agnostic). Right now, tasks are run on
    nodes where blocks are located but metadata about which disk the block
    may reside on is not maintained by the NN, so MR can't naturally know
    this to do anything about. This would be good to discuss, however
    (Search or file a new JIRA?)
    On Thu, Jun 30, 2011 at 6:33 PM, Evert Lammerts wrote:
    Hi list,

    How does the NN place blocks on the disks within a single node? Does it spread out adjecent blocks of a single file horizontally over the disks? For example, lets say I have four DN's and each has 4 disks. (And forget about replication.) If I copy a file existing of 16 blocks of 128MB each to the cluster, will each disk have exactly one block of the file?

    If I run some job over this file with its sixteen blocks this is important, since the cluster would use its maximum I/O capabilities.

    This leads me to another question (which might be better of on mapred-user). Does the JT schedule its tasks to maximally use I/O capabilities? Would it try to process blocks that reside on a disk that is not currently being read from or written to? Or does it just use a randomized strategy?

    Cheers,
    Evert


    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-user @
categorieshadoop
postedJun 30, '11 at 1:04p
activeJul 1, '11 at 8:30a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Evert Lammerts: 2 posts Harsh J: 1 post

People

Translate

site design / logo © 2022 Grokbase