FAQ
JobTracker and TaskTrackers fail with a misleading error if one of the mapreduce.cluster.dir has unusable permissions / is unavailable.
---------------------------------------------------------------------------------------------------------------------------------------

Key: MAPREDUCE-2390
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2390
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker, tasktracker
Affects Versions: 0.20.2
Environment: CDH3 and Apache 0.20 || Linux
Reporter: Harsh J Chouraria
Assignee: Harsh J Chouraria


To reproduce, have a mapred.local.dir property set to a few directories. Before starting up the JT, set one of these directories' permission as 'd---------', and then start the JT/TT. The JT, although it tries to ignore this directory, fails with an odd and misleading message claiming that its configured address in use.

Fixing the permission clears this issue!

This was also reported in the mailing lists by Ted Yu, quite a few months ago. But I had forgotten about filing a bug for it here. Still seems to happen. A log is attached below.

{code}
2011-03-17 00:40:32,321 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: java.io.IOException: Cannot create toBeDeleted in /home/hack/.tmplocalz/2
at org.apache.hadoop.util.MRAsyncDiskService.(JobTracker.java:2189)
at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:276)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:268)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4712)

2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-03-17 00:40:33,322 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2011-03-17 00:40:33,322 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2011-03-17 00:40:33,350 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as hack
2011-03-17 00:40:33,351 FATAL org.apache.hadoop.mapred.JobTracker: java.net.BindException: Problem binding to localhost/127.0.0.1:8021 : Address already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:227)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:1411)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:471)
at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2022)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:276)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:268)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4712)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:225)
... 9 more

2011-03-17 00:40:33,352 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at QDuo/127.0.0.1
************************************************************/
{code}

The list conversation in context, at {{search-hadoop.com}}:
http://search-hadoop.com/m/FzN7iqreL/problem+starting+cdh3b2+jobtracker&subj=problem+starting+cdh3b2+jobtracker

I'll try to investigate and post the exact problem / solution soon.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Harsh J (JIRA) at Jun 11, 2011 at 6:40 pm
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Harsh J resolved MAPREDUCE-2390.
    --------------------------------

    Resolution: Duplicate
    Fix Version/s: 0.22.0
    Release Note: (was: Ignore bad locations in mapreduce.cluster.dir / mapred.local.dir directories in a proper fashion.)

    Resolving as Duplicate of MAPREDUCE-1382 (Which is fixed in 0.22)
    JobTracker and TaskTrackers fail with a misleading error if one of the mapreduce.cluster.dir has unusable permissions / is unavailable.
    ---------------------------------------------------------------------------------------------------------------------------------------

    Key: MAPREDUCE-2390
    URL: https://issues.apache.org/jira/browse/MAPREDUCE-2390
    Project: Hadoop Map/Reduce
    Issue Type: Bug
    Components: jobtracker, tasktracker
    Affects Versions: 0.20.2
    Environment: CDH3 and Apache 0.20 || Linux
    Reporter: Harsh J
    Assignee: Harsh J
    Labels: configuration, directory-permissions, mapreduce
    Fix For: 0.22.0


    To reproduce, have a mapred.local.dir property set to a few directories. Before starting up the JT, set one of these directories' permission as 'd---------', and then start the JT/TT. The JT, although it tries to ignore this directory, fails with an odd and misleading message claiming that its configured address in use.
    Fixing the permission clears this issue!
    This was also reported in the mailing lists by Ted Yu, quite a few months ago. But I had forgotten about filing a bug for it here. Still seems to happen. A log is attached below.
    {code}
    2011-03-17 00:40:32,321 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: java.io.IOException: Cannot create toBeDeleted in /home/hack/.tmplocalz/2
    at org.apache.hadoop.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:86)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2189)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2022)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:276)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:268)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4712)
    2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
    2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2011-03-17 00:40:33,322 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-03-17 00:40:33,322 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
    2011-03-17 00:40:33,350 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as hack
    2011-03-17 00:40:33,351 FATAL org.apache.hadoop.mapred.JobTracker: java.net.BindException: Problem binding to localhost/127.0.0.1:8021 : Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:227)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:314)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1411)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:510)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:471)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2112)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2022)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:276)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:268)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4712)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:225)
    ... 9 more
    2011-03-17 00:40:33,352 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at QDuo/127.0.0.1
    ************************************************************/
    {code}
    The list conversation in context, at {{search-hadoop.com}}:
    http://search-hadoop.com/m/FzN7iqreL/problem+starting+cdh3b2+jobtracker&subj=problem+starting+cdh3b2+jobtracker
    I'll try to investigate and post the exact problem / solution soon.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-dev @
categorieshadoop
postedMar 16, '11 at 7:46p
activeJun 11, '11 at 6:40p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Harsh J (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase