FAQ
I have been in this mess for a couple of days, when I try to solve a
problem another occurs, don't know why.

now when I start the zookeeper i get the following error although all hosts
are up.

Service did not start successfully; not all of the required roles started:
No ZooKeeper servers are running.


S any help plz ???

Search Discussions

  • bc Wong at Dec 2, 2012 at 10:50 am

    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles started:
    No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I looked
    through the questions that you've posted in this mailing list. Most threads
    have recommendations but not the final resolution. But from what I can
    tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not starting
    right. Could you attach the ZK server log (go to CM -> ZK -> Server ->
    Processes -> Role log)? And make sure that the agents are heartbeating (go
    to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc
  • Shouvanik Haldar at Dec 2, 2012 at 11:01 am
    Hi,

    Can you please help?
    The previous error that "hosts are not showing up" is resolved!
    Now, I am facing a weird issue.

    I have installed and started services 1) hdfs 2) hbase 3) mapreduce and
    4)zookeeper
    Of these 4, 3 are running successfully.

    Only mapreduce is not starting properly. task trackers are running in
    distributed mode. But Job tracker is not starting at all!

    2012-12-02 03:47:02,994 INFO
    [CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
    command (128) has finished unsuccessfully on service DbService{id=10,
    name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=5} role id:28
    name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
    roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:3 status:NA
    service:mapreduce1.

    Please help!

    Regards,
    Shouvanik
    On Sun, Dec 2, 2012 at 4:20 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles
    started: No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I looked
    through the questions that you've posted in this mailing list. Most threads
    have recommendations but not the final resolution. But from what I can
    tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not starting
    right. Could you attach the ZK server log (go to CM -> ZK -> Server ->
    Processes -> Role log)? And make sure that the agents are heartbeating (go
    to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc


    --
    Thanks,
    *Shouvanik*
  • Shouvanik Haldar at Dec 2, 2012 at 11:46 am
    In addition to my previous mail,
    below are the log messages, when job tracker does not start

    2012-12-02 06:36:26,190 INFO
    [CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
    command (155) has finished unsuccessfully on service DbService{id=10,
    name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=8} role id:28
    name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
    roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:6 status:NA
    service:mapreduce1.


    2012-12-02 00:02:53,680 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2178)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2175)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)



    2012-12-02 00:02:54,739 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to ip-10-40-222-77.ec2.internal/
    10.40.222.77:8021 : Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.(Server.java:1529)
    at org.apache.hadoop.ipc.RPC$Server.(RPC.java:500)
    at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2012-12-02 00:02:54,757 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at ip-10-40-222-77.ec2.internal/
    10.40.222.77


    Please help.

    Regards
    Shouvanik


    On Sun, Dec 2, 2012 at 4:20 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles
    started: No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I looked
    through the questions that you've posted in this mailing list. Most threads
    have recommendations but not the final resolution. But from what I can
    tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not starting
    right. Could you attach the ZK server log (go to CM -> ZK -> Server ->
    Processes -> Role log)? And make sure that the agents are heartbeating (go
    to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc


    --
    Thanks,
    *Shouvanik*
  • Deepak Tiwari at Dec 2, 2012 at 4:16 pm
    I would voice Marco's opinion... but did you analyzed the error when it
    says "java.net.BindException: Problem binding to
    ip-10-40-222-77.ec2.internal/10.40.222.77:8021 : Address already in use"

    Either you change the port 8021 in the Cloudera Manager or see what is
    running on port 8021 via "sudo netstat -alpn | grep 8021" and resolve it
    On Sun, Dec 2, 2012 at 3:41 AM, Shouvanik Haldar wrote:

    In addition to my previous mail,
    below are the log messages, when job tracker does not start

    2012-12-02 06:36:26,190 INFO
    [CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
    command (155) has finished unsuccessfully on service DbService{id=10,
    name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=8} role id:28
    name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
    roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:6 status:NA
    service:mapreduce1.


    2012-12-02 00:02:53,680 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2178)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2175)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2175)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)



    2012-12-02 00:02:54,739 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to ip-10-40-222-77.ec2.internal/
    10.40.222.77:8021 : Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1529)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2143)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2012-12-02 00:02:54,757 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at ip-10-40-222-77.ec2.internal/
    10.40.222.77


    Please help.

    Regards
    Shouvanik


    On Sun, Dec 2, 2012 at 4:20 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles
    started: No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I
    looked through the questions that you've posted in this mailing list. Most
    threads have recommendations but not the final resolution. But from what I
    can tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not starting
    right. Could you attach the ZK server log (go to CM -> ZK -> Server ->
    Processes -> Role log)? And make sure that the agents are heartbeating (go
    to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc


    --
    Thanks,
    *Shouvanik*
  • bc Wong at Dec 2, 2012 at 8:37 pm
    Shouvanik,

    Do you have another JobTracker running from the initscript? Cloudera
    Manager does not know about the daemon instances that are started from the
    initscripts, and will run into these kind of port conflicts. You can go to
    CM -> Hosts -> Host Inspector -> View results, which will check it for you.

    If not, Deepak's `netstat' will tell you what's using that port.

    Cheers,
    bc

    On Sun, Dec 2, 2012 at 8:16 AM, Deepak Tiwari wrote:

    I would voice Marco's opinion... but did you analyzed the error when it
    says "java.net.BindException: Problem binding to
    ip-10-40-222-77.ec2.internal/10.40.222.77:8021 : Address already in use"

    Either you change the port 8021 in the Cloudera Manager or see what is
    running on port 8021 via "sudo netstat -alpn | grep 8021" and resolve it


    On Sun, Dec 2, 2012 at 3:41 AM, Shouvanik Haldar <
    shouvanik.haldar@gmail.com> wrote:
    In addition to my previous mail,
    below are the log messages, when job tracker does not start

    2012-12-02 06:36:26,190 INFO
    [CommandPusher:service.AbstractBringUpBringDownCommands@505] BringUp
    command (155) has finished unsuccessfully on service DbService{id=10,
    name=mapreduce1, serviceType=MAPREDUCE, optimisticLockVersion=8} role id:28
    name: mapreduce1-JOBTRACKER-2 hostId:ip-10-40-222-77.ec2.internal
    roleType:JOBTRACKER configuredStatus:STOPPED configGeneration:6 status:NA
    service:mapreduce1.


    2012-12-02 00:02:53,680 WARN org.apache.hadoop.mapred.JobTracker: Error
    starting tracker: ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
    at
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
    org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2178)
    at org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2175)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2175)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)



    2012-12-02 00:02:54,739 FATAL org.apache.hadoop.mapred.JobTracker:
    java.net.BindException: Problem binding to ip-10-40-222-77.ec2.internal/
    10.40.222.77:8021 : Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:230)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1529)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2143)
    at
    org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2050)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:296)
    at
    org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:288)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at
    sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:228)
    ... 9 more

    2012-12-02 00:02:54,757 INFO org.apache.hadoop.mapred.JobTracker:
    SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down JobTracker at ip-10-40-222-77.ec2.internal/
    10.40.222.77


    Please help.

    Regards
    Shouvanik


    On Sun, Dec 2, 2012 at 4:20 PM, bc Wong wrote:
    On Sun, Dec 2, 2012 at 12:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a
    problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all
    hosts are up.

    Service did not start successfully; not all of the required roles
    started: No ZooKeeper servers are running.
    Dalia,

    I noticed your frustration from your choice of subject title. And I
    looked through the questions that you've posted in this mailing list. Most
    threads have recommendations but not the final resolution. But from what I
    can tell, you're having a fair amount of environmental issues (like DNS),
    especially if you're running in a cloud. While Cloudera Manager tries to
    eliminate the need to have special Hadoop knowledge and sysadmin expertise,
    we're not there yet. Running a distributed system is fundamentally hard.
    Everyone on the CM team works day and night to build solid user-friendly
    features. The software can cover more error cases, and can give most useful
    error messages, and we will make that happen.

    For this specific problem, it seems that your ZK servers are not
    starting right. Could you attach the ZK server log (go to CM -> ZK ->
    Server -> Processes -> Role log)? And make sure that the agents are
    heartbeating (go to CM -> Hosts).

    Thanks for your patience.

    Cheers,
    bc


    --
    Thanks,
    *Shouvanik*
  • Marco Shaw at Dec 2, 2012 at 11:08 am
    I think coming along saying something like this is inappropriate no matter your level of frustration. Are you experienced? Do you have any training? Have you seeked paid professional services?

    You have to appreciate that some products can be complicated and looking for free support from a mailing list can come with its own challenges.

    Hopefully you figure it out, but come to appreciate that vendors can't account for *everything*.

    Marco
    On 2012-12-02, at 4:05 AM, Dalia Hassan wrote:

    I have been in this mess for a couple of days, when I try to solve a problem another occurs, don't know why.

    now when I start the zookeeper i get the following error although all hosts are up.

    Service did not start successfully; not all of the required roles started: No ZooKeeper servers are running.


    S any help plz ???

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedDec 2, '12 at 8:12a
activeDec 2, '12 at 8:37p
posts7
users5
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase