FAQ
Hi,

I've hadoop installed in a cluster and I would like that JT could
guess in the network topology what are the input files in HDFS that
are closer to him, and further.
So, how can a JT know if an input file is located on local-level, on
rack-level, or on the other level?

Thanks,
--
Pedro

Search Discussions

  • Harsh J at Jan 14, 2011 at 6:58 am
    All splits sent for processing in a Job carry a list of locations
    where their blocks reside -- this plus the network hierarchy details
    held by the JT is used to determine the locality level.

    Have a look at JobInProgress.getLocalityLevel(), which takes a given
    TaskInProgress object, and a TaskTrackerStatus (got via Heartbeats
    from TT) and determines the level of locality that can be obtained if
    the task were scheduled to that particular TT. You can dig down or up
    from here :)
    On Fri, Jan 14, 2011 at 4:18 AM, Pedro Costa wrote:
    Hi,

    I've hadoop installed in a cluster and I would like that JT could
    guess in the network topology what are the input files in HDFS that
    are closer to him, and further.
    So, how can a JT know if an input file is located on local-level, on
    rack-level, or on the other level?

    Thanks,
    --
    Pedro


    --
    Harsh J
    www.harshj.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJan 13, '11 at 10:49p
activeJan 14, '11 at 6:58a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Harsh J: 1 post Pedro Costa: 1 post

People

Translate

site design / logo © 2022 Grokbase