FAQ
Hello all,

I want to get datanode information (related to block placement) that is
kept at a namenode from a jobtracker. As far as I understand, the
jobtracker uses the locality-of-data for job scheduling, so I believe the
jobtracker is keeping the information somewhere in the source code.
However, I could not find the location. Can anyone give me a starting point
(source code) where the jobtracker has access to block placement
information? Thanks.

Search Discussions

  • Harsh J at Jul 21, 2012 at 3:55 am
    Hi,

    Any HDFS client can request a list of block locations for a given file
    path (node-level detail of where blocks are placed for a file), via
    the FileSystem#getFileBlockLocations API:
    http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.FileStatus,%20long,%20long)

    MR too gets this info via the user's InputFormat#getSplits method, and
    schedules with these locations.
    On Sat, Jul 21, 2012 at 2:06 AM, Kyungyong Lee wrote:
    Hello all,

    I want to get datanode information (related to block placement) that is kept
    at a namenode from a jobtracker. As far as I understand, the jobtracker uses
    the locality-of-data for job scheduling, so I believe the jobtracker is
    keeping the information somewhere in the source code. However, I could not
    find the location. Can anyone give me a starting point (source code) where
    the jobtracker has access to block placement information? Thanks.


    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-user @
categorieshadoop
postedJul 20, '12 at 8:37p
activeJul 21, '12 at 3:55a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Kyungyong Lee: 1 post Harsh J: 1 post

People

Translate

site design / logo © 2021 Grokbase