FAQ
Hi All,

i am not sure why i start getting this error. can anybody let me know?

2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:230)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:1510)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:500)
at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2043)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:228)
... 9 more


Thanks

--
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

Search Discussions

  • Harsh J at Jun 10, 2011 at 8:33 pm
    On face value, it appears to look like (1) A JT instance is already up
    and bound to that port (2) Something else is using 8021 and has bound
    it to itself.
    On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman wrote:
    Hi All,

    i am not sure why i start getting this error. can  anybody let me know?

    2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more


    Thanks

    --
    Regards
    Shuja-ur-Rehman Baig



    --
    Harsh J
  • Shuja Rehman at Jun 10, 2011 at 8:45 pm
    i have checked using netstat. the port is not used by any process. more over
    i change from 8021 to 8022 in mapred.site.xml. but the error still
    remains.
    the complete stack trace is as follow,
    ************************************************************/
    2011-06-10 13:34:53,415 INFO org.apache.hadoop.mapred.JobTracker:
    STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting JobTracker
    STARTUP_MSG: host = server1/10.0.3.85
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
    compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
    ************************************************************/
    2011-06-10 13:34:54,125 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,127 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
    configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:54,128 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,204 INFO org.apache.hadoop.mapred.JobTracker: Starting
    jobtracker with owner as mapred
    2011-06-10 13:34:54,233 INFO org.apache.hadoop.ipc.Server: Starting Socket
    Reader #1 for port 8022
    2011-06-10 13:34:54,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
    Initializing RPC Metrics with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,236 INFO
    org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
    with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,312 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2011-06-10 13:34:54,366 INFO org.apache.hadoop.http.HttpServer: Added global
    filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-06-10 13:34:54,403 WARN org.apache.hadoop.util.NativeCodeLoader: Unable
    to load native-hadoop library for your platform... using builtin-java
    classes where applicable
    2011-06-10 13:34:54,418 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: org.apache.hadoop.util.Shell$ExitCodeException: chmod:
    cannot access `/var/log/hadoop-0.20/history': No such file or directory

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2171)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)

    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
    set up for Hadoop, not re-installing.
    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
    configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:55,447 INFO org.apache.hadoop.mapred.JobTracker: Starting
    jobtracker with owner as mapred
    2011-06-10 13:34:55,447 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8022 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.(RPC.java:500)
    at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,448 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at server1/10.0.3.85
    ************************************************************/


    On Sat, Jun 11, 2011 at 1:33 AM, Harsh J wrote:

    On face value, it appears to look like (1) A JT instance is already up
    and bound to that port (2) Something else is using 8021 and has bound
    it to itself.
    On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman wrote:
    Hi All,

    i am not sure why i start getting this error. can anybody let me know?

    2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more


    Thanks

    --
    Regards
    Shuja-ur-Rehman Baig



    --
    Harsh J


    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>
  • Alex Kozlov at Jun 10, 2011 at 9:18 pm
    1. Check the /etc/hosts file that it translates the local machine name
    (server1) into local IP (or just contains loopback interface record
    "localhost"). What does ping server1 returns with?
    2. Try `telnet server1 8022` as well as `telnet 10.0.3.85 8022`: does it
    connect?

    There is another unrelated error about /var/log/hadoop-0.20/history: make
    sure your have x permissions along the whole path.

    Alex K
    On Fri, Jun 10, 2011 at 1:44 PM, Shuja Rehman wrote:

    i have checked using netstat. the port is not used by any process. more
    over i change from 8021 to 8022 in mapred.site.xml. but the error still
    remains.
    the complete stack trace is as follow,
    ************************************************************/
    2011-06-10 13:34:53,415 INFO org.apache.hadoop.mapred.JobTracker:
    STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting JobTracker
    STARTUP_MSG: host = server1/10.0.3.85
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
    compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
    ************************************************************/
    2011-06-10 13:34:54,125 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,127 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
    configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:54,128 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,204 INFO org.apache.hadoop.mapred.JobTracker: Starting
    jobtracker with owner as mapred
    2011-06-10 13:34:54,233 INFO org.apache.hadoop.ipc.Server: Starting Socket
    Reader #1 for port 8022
    2011-06-10 13:34:54,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
    Initializing RPC Metrics with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,236 INFO
    org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
    with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,312 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2011-06-10 13:34:54,366 INFO org.apache.hadoop.http.HttpServer: Added
    global filtersafety
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-06-10 13:34:54,403 WARN org.apache.hadoop.util.NativeCodeLoader:
    Unable to load native-hadoop library for your platform... using builtin-java
    classes where applicable
    2011-06-10 13:34:54,418 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: org.apache.hadoop.util.Shell$ExitCodeException: chmod:
    cannot access `/var/log/hadoop-0.20/history': No such file or directory

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2171)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2168)

    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)

    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
    set up for Hadoop, not re-installing.
    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
    configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:55,447 INFO org.apache.hadoop.mapred.JobTracker: Starting
    jobtracker with owner as mapred
    2011-06-10 13:34:55,447 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8022 :
    Address already in use

    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,448 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at server1/10.0.3.85
    ************************************************************/



    On Sat, Jun 11, 2011 at 1:33 AM, Harsh J wrote:

    On face value, it appears to look like (1) A JT instance is already up
    and bound to that port (2) Something else is using 8021 and has bound
    it to itself.

    On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman <shujamughal@gmail.com>
    wrote:
    Hi All,

    i am not sure why i start getting this error. can anybody let me know?

    2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at
    org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more


    Thanks

    --
    Regards
    Shuja-ur-Rehman Baig



    --
    Harsh J


    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>
  • Shuja Rehman at Jun 10, 2011 at 9:33 pm
    Thanks Alex,
    the problem was due to history folder.
    On Sat, Jun 11, 2011 at 2:18 AM, Alex Kozlov wrote:

    1. Check the /etc/hosts file that it translates the local machine name
    (server1) into local IP (or just contains loopback interface record
    "localhost"). What does ping server1 returns with?
    2. Try `telnet server1 8022` as well as `telnet 10.0.3.85 8022`: does it
    connect?

    There is another unrelated error about /var/log/hadoop-0.20/history: make
    sure your have x permissions along the whole path.

    Alex K

    On Fri, Jun 10, 2011 at 1:44 PM, Shuja Rehman wrote:

    i have checked using netstat. the port is not used by any process. more
    over i change from 8021 to 8022 in mapred.site.xml. but the error still
    remains.
    the complete stack trace is as follow,
    ************************************************************/
    2011-06-10 13:34:53,415 INFO org.apache.hadoop.mapred.JobTracker:
    STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting JobTracker
    STARTUP_MSG: host = server1/10.0.3.85
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
    compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
    ************************************************************/
    2011-06-10 13:34:54,125 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,127 INFO org.apache.hadoop.mapred.JobTracker:
    Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:54,128 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,204 INFO org.apache.hadoop.mapred.JobTracker: Starting
    jobtracker with owner as mapred
    2011-06-10 13:34:54,233 INFO org.apache.hadoop.ipc.Server: Starting Socket
    Reader #1 for port 8022
    2011-06-10 13:34:54,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
    Initializing RPC Metrics with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,236 INFO
    org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
    with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,312 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2011-06-10 13:34:54,366 INFO org.apache.hadoop.http.HttpServer: Added
    global filtersafety
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-06-10 13:34:54,403 WARN org.apache.hadoop.util.NativeCodeLoader:
    Unable to load native-hadoop library for your platform... using builtin-java
    classes where applicable
    2011-06-10 13:34:54,418 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: org.apache.hadoop.util.Shell$ExitCodeException: chmod:
    cannot access `/var/log/hadoop-0.20/history': No such file or directory

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2171)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2168)

    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)

    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
    set up for Hadoop, not re-installing.
    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.mapred.JobTracker:
    Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:55,447 INFO org.apache.hadoop.mapred.JobTracker: Starting
    jobtracker with owner as mapred
    2011-06-10 13:34:55,447 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8022 :
    Address already in use

    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,448 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at server1/10.0.3.85
    ************************************************************/



    On Sat, Jun 11, 2011 at 1:33 AM, Harsh J wrote:

    On face value, it appears to look like (1) A JT instance is already up
    and bound to that port (2) Something else is using 8021 and has bound
    it to itself.

    On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman <shujamughal@gmail.com>
    wrote:
    Hi All,

    i am not sure why i start getting this error. can anybody let me know?

    2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at
    org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at
    org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more


    Thanks

    --
    Regards
    Shuja-ur-Rehman Baig



    --
    Harsh J


    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>

    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>
  • Alex Kozlov at Jun 10, 2011 at 9:43 pm
    Can you be more explicit for the benefit of others? How can a history
    folder result in the "Address already in use" exception?
    On Fri, Jun 10, 2011 at 2:33 PM, Shuja Rehman wrote:

    Thanks Alex,
    the problem was due to history folder.
    On Sat, Jun 11, 2011 at 2:18 AM, Alex Kozlov wrote:

    1. Check the /etc/hosts file that it translates the local machine name
    (server1) into local IP (or just contains loopback interface record
    "localhost"). What does ping server1 returns with?
    2. Try `telnet server1 8022` as well as `telnet 10.0.3.85 8022`: does it
    connect?

    There is another unrelated error about /var/log/hadoop-0.20/history: make
    sure you have x permissions along the whole path.

    Alex K

    On Fri, Jun 10, 2011 at 1:44 PM, Shuja Rehman wrote:

    i have checked using netstat. the port is not used by any process. more
    over i change from 8021 to 8022 in mapred.site.xml. but the error still
    remains.
    the complete stack trace is as follow,
    ************************************************************/
    2011-06-10 13:34:53,415 INFO org.apache.hadoop.mapred.JobTracker:
    STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting JobTracker
    STARTUP_MSG: host = server1/10.0.3.85
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
    compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
    ************************************************************/
    2011-06-10 13:34:54,125 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,127 INFO org.apache.hadoop.mapred.JobTracker:
    Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:54,128 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,204 INFO org.apache.hadoop.mapred.JobTracker:
    Starting jobtracker with owner as mapred
    2011-06-10 13:34:54,233 INFO org.apache.hadoop.ipc.Server: Starting
    Socket Reader #1 for port 8022
    2011-06-10 13:34:54,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
    Initializing RPC Metrics with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,236 INFO
    org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
    with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,312 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2011-06-10 13:34:54,366 INFO org.apache.hadoop.http.HttpServer: Added
    global filtersafety
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-06-10 13:34:54,403 WARN org.apache.hadoop.util.NativeCodeLoader:
    Unable to load native-hadoop library for your platform... using builtin-java
    classes where applicable
    2011-06-10 13:34:54,418 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: org.apache.hadoop.util.Shell$ExitCodeException: chmod:
    cannot access `/var/log/hadoop-0.20/history': No such file or directory

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at
    org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2171)
    at
    org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2168)

    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)

    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
    set up for Hadoop, not re-installing.
    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.mapred.JobTracker:
    Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:55,447 INFO org.apache.hadoop.mapred.JobTracker:
    Starting jobtracker with owner as mapred
    2011-06-10 13:34:55,447 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8022 :
    Address already in use

    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,448 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at server1/10.0.3.85
    ************************************************************/



    On Sat, Jun 11, 2011 at 1:33 AM, Harsh J wrote:

    On face value, it appears to look like (1) A JT instance is already up
    and bound to that port (2) Something else is using 8021 and has bound
    it to itself.

    On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman <shujamughal@gmail.com>
    wrote:
    Hi All,

    i am not sure why i start getting this error. can anybody let me know?
    2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at
    org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at
    org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more


    Thanks

    --
    Regards
    Shuja-ur-Rehman Baig



    --
    Harsh J


    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>

    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>
  • Shuja Rehman at Jun 11, 2011 at 9:40 am
    The history folder was deleted by some how and I have created this folder
    and its internal folders manually and then restarted the job tracker and
    then it works. I am not sure why it shows the "Address already in use"
    exception on missing any folders.
    On Sat, Jun 11, 2011 at 2:42 AM, Alex Kozlov wrote:

    Can you be more explicit for the benefit of others? How can a history
    folder result in the "Address already in use" exception?
    On Fri, Jun 10, 2011 at 2:33 PM, Shuja Rehman wrote:

    Thanks Alex,
    the problem was due to history folder.
    On Sat, Jun 11, 2011 at 2:18 AM, Alex Kozlov wrote:

    1. Check the /etc/hosts file that it translates the local machine name
    (server1) into local IP (or just contains loopback interface record
    "localhost"). What does ping server1 returns with?
    2. Try `telnet server1 8022` as well as `telnet 10.0.3.85 8022`: does it
    connect?

    There is another unrelated error about /var/log/hadoop-0.20/history: make
    sure you have x permissions along the whole path.


    Alex K

    On Fri, Jun 10, 2011 at 1:44 PM, Shuja Rehman wrote:

    i have checked using netstat. the port is not used by any process. more
    over i change from 8021 to 8022 in mapred.site.xml. but the error still
    remains.
    the complete stack trace is as follow,
    ************************************************************/
    2011-06-10 13:34:53,415 INFO org.apache.hadoop.mapred.JobTracker:
    STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting JobTracker
    STARTUP_MSG: host = server1/10.0.3.85
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
    compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
    ************************************************************/
    2011-06-10 13:34:54,125 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,127 INFO org.apache.hadoop.mapred.JobTracker:
    Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:54,128 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:54,159 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:54,204 INFO org.apache.hadoop.mapred.JobTracker:
    Starting jobtracker with owner as mapred
    2011-06-10 13:34:54,233 INFO org.apache.hadoop.ipc.Server: Starting
    Socket Reader #1 for port 8022
    2011-06-10 13:34:54,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
    Initializing RPC Metrics with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,236 INFO
    org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
    with hostName=JobTracker, port=8022
    2011-06-10 13:34:54,312 INFO org.mortbay.log: Logging to
    org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
    org.mortbay.log.Slf4jLog
    2011-06-10 13:34:54,366 INFO org.apache.hadoop.http.HttpServer: Added
    global filtersafety
    (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-06-10 13:34:54,403 WARN org.apache.hadoop.util.NativeCodeLoader:
    Unable to load native-hadoop library for your platform... using builtin-java
    classes where applicable
    2011-06-10 13:34:54,418 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: org.apache.hadoop.util.Shell$ExitCodeException: chmod:
    cannot access `/var/log/hadoop-0.20/history': No such file or directory

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at
    org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2171)
    at
    org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2168)

    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at
    org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)

    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
    set up for Hadoop, not re-installing.
    2011-06-10 13:34:55,423 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.mapred.JobTracker:
    Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
    limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
    2011-06-10 13:34:55,424 INFO org.apache.hadoop.util.HostsFileReader:
    Refreshing hosts (include/exclude) list
    2011-06-10 13:34:55,447 INFO org.apache.hadoop.mapred.JobTracker:
    Starting jobtracker with owner as mapred
    2011-06-10 13:34:55,447 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8022 :
    Address already in use

    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at
    org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Starting expired delegation token remover thread,
    tokenRemoverScanInterval=60 min(s)
    2011-06-10 13:34:55,448 INFO
    org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
    Updating the current master key for generating delegation tokens
    2011-06-10 13:34:55,448 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at server1/10.0.3.85
    ************************************************************/



    On Sat, Jun 11, 2011 at 1:33 AM, Harsh J wrote:

    On face value, it appears to look like (1) A JT instance is already up
    and bound to that port (2) Something else is using 8021 and has bound
    it to itself.

    On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman <shujamughal@gmail.com>
    wrote:
    Hi All,

    i am not sure why i start getting this error. can anybody let me know?
    2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address
    already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at
    org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
    at
    org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more


    Thanks

    --
    Regards
    Shuja-ur-Rehman Baig



    --
    Harsh J


    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>

    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>

    --
    Regards
    Shuja-ur-Rehman Baig
    <http://pk.linkedin.com/in/shujamughal>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJun 10, '11 at 8:27p
activeJun 11, '11 at 9:40a
posts7
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase