FAQ
Hi,

I might be misunderstanding how scheduling is supposed to work, or I might
have something misconfigured, but my Map/Reduce jobs don't seem to run where
my data is located.

I get a bunch of these messages:
INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201106062049_0001_
m_000021 has split on node:/rack1/rack1node1.local
... indicating it has correctly found the source data at my node
/rack1/rack1node1 (the only copy of the data - for the purpose of this
experiment I have set dfs.replication = dfs.replication.min =
dfs.replication.max = 1 so I only have 1 replica).

However, it then goes on to run the JOB_SETUP, MAP, REDUCE, JOB_CLEANUP
tasks on abitrary tasktrackers, usually not where the data is located, so
the first thing they have to do is pull it over the network from another
node.

Did I miss something - or hopefully configure something wrong? :)

Ian

Search Discussions

  • Harsh J at Jun 7, 2011 at 4:00 am
    Ian,

    The reducer needn't run on the same TaskTracker as it does not work on
    local data. The mapper 000021, however, should've run on
    node:/rack1/rack1node1.local.
    On Tue, Jun 7, 2011 at 5:09 AM, Ian Halperin wrote:
    Hi,

    I might be misunderstanding how scheduling is supposed to work, or I might
    have something misconfigured, but my Map/Reduce jobs don't seem to run where
    my data is located.

    I get a bunch of these messages:
    INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201106062049_0001_
    m_000021 has split on node:/rack1/rack1node1.local
    ... indicating it has correctly found the source data at my node
    /rack1/rack1node1 (the only copy of the data - for the purpose of this
    experiment I have set dfs.replication = dfs.replication.min =
    dfs.replication.max = 1 so I only have 1 replica).

    However, it then goes on to run the JOB_SETUP, MAP, REDUCE, JOB_CLEANUP
    tasks on abitrary tasktrackers, usually not where the data is located, so
    the first thing they have to do is pull it over the network from another
    node.

    Did I miss something - or hopefully configure something wrong? :)

    Ian


    --
    Harsh J
  • Ian Halperin at Jun 7, 2011 at 11:42 am
    Harsh, thanks for the clarification.

    But my mappers always seem to run elsewhere. Here's an example with 2
    splits, both on rack1node1, but the 2 mappers get started on other nodes.
    Could the "choosing a non-local task" message be significant?

    I have actually read through the JobTracker source, but didn't get too far
    in understanding the logic. :)

    The only other odd-looking thing is that trackers have "localhost/127.0.0.1"
    in their names as well as their real hostnames - not sure if that could be
    related.

    ...
    mapred.JobClient: Running job: job_201106071125_0004
    org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with
    users keys in /mapred/system/job_201106071125_0004/jobToken
    org.apache.hadoop.mapred.JobInProgress: Input size for job
    job_201106071125_0004 = 100086130. Number of splits = 2
    org.apache.hadoop.mapred.JobInProgress: tip:task_201106071125_0004_m_000000
    has split on node:/rack1/rack1node1.local
    org.apache.hadoop.mapred.JobInProgress: tip:task_201106071125_0004_m_000001
    has split on node:/rack1/rack1node1.local
    org.apache.hadoop.mapred.JobInProgress: job_201106071125_0004
    LOCALITY_WAIT_FACTOR=0.16666667
    org.apache.hadoop.mapred.JobInProgress: Job job_201106071125_0004
    initialized successfully with 2 map tasks and 1 reduce tasks.
    org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP)
    'attempt_201106071125_0004_m_000003_0' to tip
    task_201106071125_0004_m_000003, for tracker
    'tracker_rack3node1.local:localhost/127.0.0.1:55842'
    mapred.JobClient: map 0% reduce 0%
    org.apache.hadoop.mapred.JobInProgress: Task
    'attempt_201106071125_0004_m_000003_0' has completed
    task_201106071125_0004_m_000003 successfully.
    org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task
    task_201106071125_0004_m_000000
    org.apache.hadoop.mapred.JobTracker: Adding task (MAP)
    'attempt_201106071125_0004_m_000000_0' to tip
    task_201106071125_0004_m_000000, for tracker
    'tracker_rack3node1.local:localhost/127.0.0.1:55842'
    org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task
    task_201106071125_0004_m_000001
    org.apache.hadoop.mapred.JobTracker: Adding task (MAP)
    'attempt_201106071125_0004_m_000001_0' to tip
    task_201106071125_0004_m_000001, for tracker
    'tracker_rack2node1.local:localhost/127.0.0.1:37910'
    ...
    On 7 June 2011 04:59, Harsh J wrote:

    Ian,

    The reducer needn't run on the same TaskTracker as it does not work on
    local data. The mapper 000021, however, should've run on
    node:/rack1/rack1node1.local.
    On Tue, Jun 7, 2011 at 5:09 AM, Ian Halperin wrote:
    Hi,

    I might be misunderstanding how scheduling is supposed to work, or I might
    have something misconfigured, but my Map/Reduce jobs don't seem to run where
    my data is located.

    I get a bunch of these messages:
    INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201106062049_0001_
    m_000021 has split on node:/rack1/rack1node1.local
    ... indicating it has correctly found the source data at my node
    /rack1/rack1node1 (the only copy of the data - for the purpose of this
    experiment I have set dfs.replication = dfs.replication.min =
    dfs.replication.max = 1 so I only have 1 replica).

    However, it then goes on to run the JOB_SETUP, MAP, REDUCE, JOB_CLEANUP
    tasks on abitrary tasktrackers, usually not where the data is located, so
    the first thing they have to do is pull it over the network from another
    node.

    Did I miss something - or hopefully configure something wrong? :)

    Ian


    --
    Harsh J
  • Ian Halperin at Jun 7, 2011 at 8:01 pm
    I found from Googling around that I should probably be seeing messages like
    "Choosing data-local task" and "Choosing rack-local task" - from
    JobInProgress::addRunningTaskToTIP(). (e.g. here:
    http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201012.mbox/%3C2120373776.44711293532894724.JavaMail.hudson@aegis%3E).

    I'm digging through the code, but any idea why mine's always choosing
    non-local trackers?

    On 7 June 2011 12:41, Ian Halperin wrote:

    Harsh, thanks for the clarification.

    But my mappers always seem to run elsewhere. Here's an example with 2
    splits, both on rack1node1, but the 2 mappers get started on other nodes.
    Could the "choosing a non-local task" message be significant?

    I have actually read through the JobTracker source, but didn't get too far
    in understanding the logic. :)

    The only other odd-looking thing is that trackers have "localhost/
    127.0.0.1" in their names as well as their real hostnames - not sure if
    that could be related.

    ...
    mapred.JobClient: Running job: job_201106071125_0004
    org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with
    users keys in /mapred/system/job_201106071125_0004/jobToken
    org.apache.hadoop.mapred.JobInProgress: Input size for job
    job_201106071125_0004 = 100086130. Number of splits = 2
    org.apache.hadoop.mapred.JobInProgress: tip:task_201106071125_0004_m_000000
    has split on node:/rack1/rack1node1.local
    org.apache.hadoop.mapred.JobInProgress: tip:task_201106071125_0004_m_000001
    has split on node:/rack1/rack1node1.local
    org.apache.hadoop.mapred.JobInProgress: job_201106071125_0004
    LOCALITY_WAIT_FACTOR=0.16666667
    org.apache.hadoop.mapred.JobInProgress: Job job_201106071125_0004
    initialized successfully with 2 map tasks and 1 reduce tasks.
    org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP)
    'attempt_201106071125_0004_m_000003_0' to tip
    task_201106071125_0004_m_000003, for tracker
    'tracker_rack3node1.local:localhost/127.0.0.1:55842'
    mapred.JobClient: map 0% reduce 0%
    org.apache.hadoop.mapred.JobInProgress: Task
    'attempt_201106071125_0004_m_000003_0' has completed
    task_201106071125_0004_m_000003 successfully.
    org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task
    task_201106071125_0004_m_000000
    org.apache.hadoop.mapred.JobTracker: Adding task (MAP)
    'attempt_201106071125_0004_m_000000_0' to tip
    task_201106071125_0004_m_000000, for tracker
    'tracker_rack3node1.local:localhost/127.0.0.1:55842'
    org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task
    task_201106071125_0004_m_000001
    org.apache.hadoop.mapred.JobTracker: Adding task (MAP)
    'attempt_201106071125_0004_m_000001_0' to tip
    task_201106071125_0004_m_000001, for tracker
    'tracker_rack2node1.local:localhost/127.0.0.1:37910'
    ...

    On 7 June 2011 04:59, Harsh J wrote:

    Ian,

    The reducer needn't run on the same TaskTracker as it does not work on
    local data. The mapper 000021, however, should've run on
    node:/rack1/rack1node1.local.
    On Tue, Jun 7, 2011 at 5:09 AM, Ian Halperin wrote:
    Hi,

    I might be misunderstanding how scheduling is supposed to work, or I might
    have something misconfigured, but my Map/Reduce jobs don't seem to run where
    my data is located.

    I get a bunch of these messages:
    INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201106062049_0001_
    m_000021 has split on node:/rack1/rack1node1.local
    ... indicating it has correctly found the source data at my node
    /rack1/rack1node1 (the only copy of the data - for the purpose of this
    experiment I have set dfs.replication = dfs.replication.min =
    dfs.replication.max = 1 so I only have 1 replica).

    However, it then goes on to run the JOB_SETUP, MAP, REDUCE, JOB_CLEANUP
    tasks on abitrary tasktrackers, usually not where the data is located, so
    the first thing they have to do is pull it over the network from another
    node.

    Did I miss something - or hopefully configure something wrong? :)

    Ian


    --
    Harsh J
  • Ian Halperin at Jun 8, 2011 at 12:56 am
    For the record, I figured this out.

    The default task scheduler just assigns tasks to trackers on a first come
    first served basis as trackers' heartbeats are received; although this might
    tend to favour data-local and then rack-local trackers at a large enough
    scale.

    I switched over to using FairScheduler, which seems to make a real attempt
    to schedule tasks near the data first, and only falls back to other nodes
    when there are no other options.

    On 7 June 2011 21:00, Ian Halperin wrote:

    I found from Googling around that I should probably be seeing messages like
    "Choosing data-local task" and "Choosing rack-local task" - from
    JobInProgress::addRunningTaskToTIP(). (e.g. here:
    http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201012.mbox/%3C2120373776.44711293532894724.JavaMail.hudson@aegis%3E).

    I'm digging through the code, but any idea why mine's always choosing
    non-local trackers?


    On 7 June 2011 12:41, Ian Halperin wrote:

    Harsh, thanks for the clarification.

    But my mappers always seem to run elsewhere. Here's an example with 2
    splits, both on rack1node1, but the 2 mappers get started on other nodes.
    Could the "choosing a non-local task" message be significant?

    I have actually read through the JobTracker source, but didn't get too far
    in understanding the logic. :)

    The only other odd-looking thing is that trackers have "localhost/
    127.0.0.1" in their names as well as their real hostnames - not sure if
    that could be related.

    ...
    mapred.JobClient: Running job: job_201106071125_0004
    org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with
    users keys in /mapred/system/job_201106071125_0004/jobToken
    org.apache.hadoop.mapred.JobInProgress: Input size for job
    job_201106071125_0004 = 100086130. Number of splits = 2
    org.apache.hadoop.mapred.JobInProgress:
    tip:task_201106071125_0004_m_000000 has split on
    node:/rack1/rack1node1.local
    org.apache.hadoop.mapred.JobInProgress:
    tip:task_201106071125_0004_m_000001 has split on
    node:/rack1/rack1node1.local
    org.apache.hadoop.mapred.JobInProgress: job_201106071125_0004
    LOCALITY_WAIT_FACTOR=0.16666667
    org.apache.hadoop.mapred.JobInProgress: Job job_201106071125_0004
    initialized successfully with 2 map tasks and 1 reduce tasks.
    org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP)
    'attempt_201106071125_0004_m_000003_0' to tip
    task_201106071125_0004_m_000003, for tracker
    'tracker_rack3node1.local:localhost/127.0.0.1:55842'
    mapred.JobClient: map 0% reduce 0%
    org.apache.hadoop.mapred.JobInProgress: Task
    'attempt_201106071125_0004_m_000003_0' has completed
    task_201106071125_0004_m_000003 successfully.
    org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task
    task_201106071125_0004_m_000000
    org.apache.hadoop.mapred.JobTracker: Adding task (MAP)
    'attempt_201106071125_0004_m_000000_0' to tip
    task_201106071125_0004_m_000000, for tracker
    'tracker_rack3node1.local:localhost/127.0.0.1:55842'
    org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task
    task_201106071125_0004_m_000001
    org.apache.hadoop.mapred.JobTracker: Adding task (MAP)
    'attempt_201106071125_0004_m_000001_0' to tip
    task_201106071125_0004_m_000001, for tracker
    'tracker_rack2node1.local:localhost/127.0.0.1:37910'
    ...

    On 7 June 2011 04:59, Harsh J wrote:

    Ian,

    The reducer needn't run on the same TaskTracker as it does not work on
    local data. The mapper 000021, however, should've run on
    node:/rack1/rack1node1.local.

    On Tue, Jun 7, 2011 at 5:09 AM, Ian Halperin <lomax0000@gmail.com>
    wrote:
    Hi,

    I might be misunderstanding how scheduling is supposed to work, or I might
    have something misconfigured, but my Map/Reduce jobs don't seem to run where
    my data is located.

    I get a bunch of these messages:
    INFO org.apache.hadoop.mapred.JobInProgress:
    tip:task_201106062049_0001_
    m_000021 has split on node:/rack1/rack1node1.local
    ... indicating it has correctly found the source data at my node
    /rack1/rack1node1 (the only copy of the data - for the purpose of this
    experiment I have set dfs.replication = dfs.replication.min =
    dfs.replication.max = 1 so I only have 1 replica).

    However, it then goes on to run the JOB_SETUP, MAP, REDUCE, JOB_CLEANUP
    tasks on abitrary tasktrackers, usually not where the data is located, so
    the first thing they have to do is pull it over the network from another
    node.

    Did I miss something - or hopefully configure something wrong? :)

    Ian


    --
    Harsh J

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJun 6, '11 at 11:39p
activeJun 8, '11 at 12:56a
posts5
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Ian Halperin: 4 posts Harsh J: 1 post

People

Translate

site design / logo © 2021 Grokbase