FAQ
+scm-users@
bcc: cdh-user@

Hi Nicolas,

Sending this to scm-users@, which might be more helpful with this issue.

I do have one comment/observation down below.

--
Aaron T. Myers
Software Engineer, Cloudera


On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard wrote:

Hi everyone

Follow up on my previous message.
I played around a bit tried supressing and changing what machine had the
jobtracker in cloudera manager 4.1.1.
But on start everything looks fine the manager says started and good to go.
However if I look in the jobtracker log and tasktracker logs I see errors:
and a frightening nullpointer.

Have I missed a step not installed a package, is unbuntu 12.04 not
supported for mapreduce.
If anyone has an idea what I am doing wrong.

JobYTacker
2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.JobTracker: Lost
tracker 'tracker_ks25937.kimsufi.com:localhost/127.0.0.1:47149'
2012-11-09 15:25:59,772 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:mapred (auth:SIMPLE) cause:java.io.IOException:
java.lang.NullPointerException
2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 13 on 8021, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@12fcf657, true,
true, true, -1), rpc version=2, client version=32,
methodsFingerPrint=-159967141 from 37.59.44.188:45329: error:
java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2735)
at
org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2715)
at
org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3211)
at
org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.ScriptBasedMapping:
Exception running
/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py
91.121.69.14
java.io.IOException: Cannot run program
"/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py" (in
directory "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER"):
java.io.IOException: error=13, Permission denied
This looks to me to be the root cause. The question is why can't the JT run
the topology script. Though it's a shame that it results in an NPE, I'm not
entirely surprised that a properly configured topology script that is
present but can't be executed for some reason would result in an error on
the JT.

at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
at org.apache.hadoop.util.Shell.run(Shell.java:188)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
at
org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at
org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2734)
at
org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2715)
at
org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3211)
at
org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
Caused by: java.io.IOException: java.io.IOException: error=13, Permission
denied
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 21 more


TASKTracker:2012-11-06 10:29:36,641 INFO
org.apache.hadoop.mapred.TaskTracker: Resending 'status' to '
ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.TaskTracker:
Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


--


Search Discussions

  • Philip Zeyliger at Nov 10, 2012 at 3:29 am

    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER/topology.py" (in directory
    "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied

    As Aaron points out below, this is unusual and is the cause of your errors
    in the log. Could you run "ls -ld / /run /run/cloudera-scm-agent
    /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py".
    Perhaps a permission is wrong along one of those directories.

    Alternately, does "/usr/bin/env" exist and does "python" exist in your
    path? If the interpreter for a shell script doesn't exist, I could see an
    error occurring. (Unfortunately I'm not currently able to double-check
    manually.)

    Finally, if your /run file system is mounted noexec, I could see that. Are
    there any odd options in the output of 'mount'?

    -- Philip
    On Fri, Nov 9, 2012 at 2:29 PM, Aaron T. Myers wrote:

    +scm-users@
    bcc: cdh-user@

    Hi Nicolas,

    Sending this to scm-users@, which might be more helpful with this issue.

    I do have one comment/observation down below.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard <
    nicolas.maillard@fifty-five.com> wrote:
    Hi everyone

    Follow up on my previous message.
    I played around a bit tried supressing and changing what machine had the
    jobtracker in cloudera manager 4.1.1.
    But on start everything looks fine the manager says started and good to
    go.
    However if I look in the jobtracker log and tasktracker logs I see
    errors: and a frightening nullpointer.

    Have I missed a step not installed a package, is unbuntu 12.04 not
    supported for mapreduce.
    If anyone has an idea what I am doing wrong.

    JobYTacker
    2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.JobTracker: Lost
    tracker 'tracker_ks25937.kimsufi.com:localhost/127.0.0.1:47149'
    2012-11-09 15:25:59,772 ERROR
    org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
    as:mapred (auth:SIMPLE) cause:java.io.IOException:
    java.lang.NullPointerException
    2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 13 on 8021, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@12fcf657, true,
    true, true, -1), rpc version=2, client version=32,
    methodsFingerPrint=-159967141 from 37.59.44.188:45329: error:
    java.io.IOException: java.lang.NullPointerException
    java.io.IOException: java.lang.NullPointerException
    at
    org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2735)
    at
    org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2715)
    at
    org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3211)
    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
    2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.ScriptBasedMapping:
    Exception running
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py
    91.121.69.14
    java.io.IOException: Cannot run program
    "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py" (in
    directory "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied
    This looks to me to be the root cause. The question is why can't the JT
    run the topology script. Though it's a shame that it results in an NPE, I'm
    not entirely surprised that a properly configured topology script that is
    present but can't be executed for some reason would result in an error on
    the JT.

    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.run(Shell.java:188)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
    at
    org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
    at
    org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
    at
    org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
    at
    org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2734)
    at
    org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2715)
    at
    org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3211)
    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
    Caused by: java.io.IOException: java.io.IOException: error=13, Permission
    denied
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 21 more


    TASKTracker:2012-11-06 10:29:36,641 INFO
    org.apache.hadoop.mapred.TaskTracker: Resending 'status' to '
    ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


    --


  • Nick at Nov 13, 2012 at 9:46 am
    Hi everyone and thanks for the help

    So I have gone ahead and checked my directories and files the rights and
    ownserhsip seem good. To be sure I have chmod 775 on all of them.
    *
    *
    *ls -ld / /run /run/cloudera-scm-agent /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *drwxr-xr-x 25 root root 4096 Nov 2 18:57 /*
    *drwxr-xr-x 23 root root 760 Nov 13 09:50 /run*
    *drwxrwxr-x 4 root root 100 Nov 5 17:56 /run/cloudera-scm-agent*
    *drwxrwxr-x 259 root root 5180 Nov 9 15:18
    /run/cloudera-scm-agent/process*
    *drwxrwxr-x 3 mapred hadoop 360 Nov 9 15:17
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER*
    *-rwxrwxr-x 1 mapred hadoop 1382 Nov 9 15:17
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *
    *
    I was not aware python was needed for cloudera manager, I stuck to the
    installation guide. However python is intalled an in the path.
    I made sure "/usr/bin/env" exists:
    *ls -ld /usr/bin/env*
    *-rwxr-xr-x 1 root root 27040 Apr 1 2012 /usr/bin/env*
    *
    *
    *
    *
    The mount seems ok to me:

    mount
    /dev/root on / type ext4
    (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=1,data=ordered)
    none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc
    (rw,nosuid,nodev,noexec,relatime)
    none on /sys/fs/fuse/connections type fusectl (rw,relatime)
    /dev on /dev type devtmpfs
    (rw,relatime,size=12338840k,nr_inodes=3084710,mode=755)
    none on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
    none on /run type tmpfs (rw,nosuid,noexec,relatime,size=2474428k,mode=755)
    none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
    none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
    /dev/sda2 on /home type ext4
    (rw,relatime,user_xattr,acl,barrier=1,data=ordered)
    *
    *
    *
    *
    I hope all of this info can help, it has been really complicated to get a
    hadoop/hbase cluster running.

    thanks for all the help

    regards
    On Saturday, November 10, 2012 4:29:22 AM UTC+1, Philip Zeyliger wrote:

    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER/topology.py" (in directory
    "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied

    As Aaron points out below, this is unusual and is the cause of your errors
    in the log. Could you run "ls -ld / /run /run/cloudera-scm-agent
    /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py".
    Perhaps a permission is wrong along one of those directories.

    Alternately, does "/usr/bin/env" exist and does "python" exist in your
    path? If the interpreter for a shell script doesn't exist, I could see an
    error occurring. (Unfortunately I'm not currently able to double-check
    manually.)

    Finally, if your /run file system is mounted noexec, I could see that.
    Are there any odd options in the output of 'mount'?

    -- Philip

    On Fri, Nov 9, 2012 at 2:29 PM, Aaron T. Myers <a...@cloudera.com<javascript:>
    wrote:
    +scm-users@
    bcc: cdh-user@

    Hi Nicolas,

    Sending this to scm-users@, which might be more helpful with this issue.

    I do have one comment/observation down below.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard <
    nicolas....@fifty-five.com <javascript:>> wrote:
    Hi everyone

    Follow up on my previous message.
    I played around a bit tried supressing and changing what machine had the
    jobtracker in cloudera manager 4.1.1.
    But on start everything looks fine the manager says started and good to
    go.
    However if I look in the jobtracker log and tasktracker logs I see
    errors: and a frightening nullpointer.

    Have I missed a step not installed a package, is unbuntu 12.04 not
    supported for mapreduce.
    If anyone has an idea what I am doing wrong.

    JobYTacker
    2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.JobTracker: Lost
    tracker 'tracker_ks25937.kimsufi.com:localhost/127.0.0.1:47149'
    2012-11-09 15:25:59,772 ERROR
    org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
    as:mapred (auth:SIMPLE) cause:java.io.IOException:
    java.lang.NullPointerException
    2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 13 on 8021, call
    heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@12fcf657, true, true,
    true, -1), rpc version=2, client version=32, methodsFingerPrint=-159967141
    from 37.59.44.188:45329: error: java.io.IOException:
    java.lang.NullPointerException
    java.io.IOException: java.lang.NullPointerException
    at
    org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2735)
    at
    org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2715)
    at
    org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3211)
    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
    2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.ScriptBasedMapping:
    Exception running
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py
    91.121.69.14
    java.io.IOException: Cannot run program
    "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py" (in
    directory "/run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied
    This looks to me to be the root cause. The question is why can't the JT
    run the topology script. Though it's a shame that it results in an NPE, I'm
    not entirely surprised that a properly configured topology script that is
    present but can't be executed for some reason would result in an error on
    the JT.

    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.run(Shell.java:188)
    at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
    at
    org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
    at
    org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
    at
    org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
    at
    org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2734)
    at
    org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2715)
    at
    org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3211)
    at
    org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2915)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
    Caused by: java.io.IOException: java.io.IOException: error=13,
    Permission denied
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 21 more


    TASKTracker:2012-11-06 10:29:36,641 INFO
    org.apache.hadoop.mapred.TaskTracker: Resending 'status' to '
    ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


    --


  • Nicolas Maillard at Nov 14, 2012 at 5:17 pm
    Hi everyone

    I have completely reinstalled my cluster from scrracth and followed excatly
    the automated installation
    https://ccp.cloudera.com/display/FREE4DOC/Automated+Installation+of+Cloudera+Manager+and+CDH

    But when I try to run a map reduce job. I get the same error as before.
    I try using the documentation "running an example with MRv1" found:
    https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode#InstallingCDH4onaSingleLinuxNodeinPseudo-distributedMode-exAppMRv1

    But no cigar I can't get map reduce working, but i haven't triend anything
    fancy its a 64bit ubunut 12.04.

    Anyone seen this before or just got the mapreduce working with cloudera
    CDH4.1.1.

    thanks


    On Tue, Nov 13, 2012 at 10:46 AM, nick wrote:

    Hi everyone and thanks for the help

    So I have gone ahead and checked my directories and files the rights and
    ownserhsip seem good. To be sure I have chmod 775 on all of them.
    *
    *
    *ls -ld / /run /run/cloudera-scm-agent /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *drwxr-xr-x 25 root root 4096 Nov 2 18:57 /*
    *drwxr-xr-x 23 root root 760 Nov 13 09:50 /run*
    *drwxrwxr-x 4 root root 100 Nov 5 17:56 /run/cloudera-scm-agent*
    *drwxrwxr-x 259 root root 5180 Nov 9 15:18 /run/cloudera-scm-agent/
    process*
    *drwxrwxr-x 3 mapred hadoop 360 Nov 9 15:17 /run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER*
    *-rwxrwxr-x 1 mapred hadoop 1382 Nov 9 15:17 /run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER/topology.py*
    *
    *
    I was not aware python was needed for cloudera manager, I stuck to the
    installation guide. However python is intalled an in the path.
    I made sure "/usr/bin/env" exists:
    *ls -ld /usr/bin/env*
    *-rwxr-xr-x 1 root root 27040 Apr 1 2012 /usr/bin/env*
    *
    *
    *
    *
    The mount seems ok to me:

    mount
    /dev/root on / type ext4 (rw,relatime,errors=remount-**
    ro,user_xattr,acl,barrier=1,**data=ordered)
    none on /proc type proc (rw,nosuid,nodev,noexec,**relatime)
    none on /sys type sysfs (rw,nosuid,nodev,noexec,**relatime)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,
    **relatime)
    none on /sys/fs/fuse/connections type fusectl (rw,relatime)
    /dev on /dev type devtmpfs (rw,relatime,size=12338840k,**
    nr_inodes=3084710,mode=755)
    none on /dev/pts type devpts (rw,nosuid,noexec,relatime,**gid=5,mode=620)
    none on /run type tmpfs (rw,nosuid,noexec,relatime,**
    size=2474428k,mode=755)
    none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,**
    relatime,size=5120k)
    none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
    /dev/sda2 on /home type ext4 (rw,relatime,user_xattr,acl,**
    barrier=1,data=ordered)
    *
    *
    *
    *
    I hope all of this info can help, it has been really complicated to get a
    hadoop/hbase cluster running.

    thanks for all the help

    regards
    On Saturday, November 10, 2012 4:29:22 AM UTC+1, Philip Zeyliger wrote:

    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/proce**
    ss/427-mapreduce-JOBTRACKER/**topology.py" (in directory
    "/run/cloudera-scm-agent/proce**ss/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied

    As Aaron points out below, this is unusual and is the cause of your
    errors in the log. Could you run "ls -ld / /run /run/cloudera-scm-agent
    /run/cloudera-scm-agent/**process /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py". Perhaps a permission
    is wrong along one of those directories.

    Alternately, does "/usr/bin/env" exist and does "python" exist in your
    path? If the interpreter for a shell script doesn't exist, I could see an
    error occurring. (Unfortunately I'm not currently able to double-check
    manually.)

    Finally, if your /run file system is mounted noexec, I could see that.
    Are there any odd options in the output of 'mount'?

    -- Philip
    On Fri, Nov 9, 2012 at 2:29 PM, Aaron T. Myers wrote:

    +scm-users@
    bcc: cdh-user@

    Hi Nicolas,

    Sending this to scm-users@, which might be more helpful with this issue.

    I do have one comment/observation down below.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard <
    nicolas....@fifty-five.**com> wrote:
    Hi everyone

    Follow up on my previous message.
    I played around a bit tried supressing and changing what machine had
    the jobtracker in cloudera manager 4.1.1.
    But on start everything looks fine the manager says started and good to
    go.
    However if I look in the jobtracker log and tasktracker logs I see
    errors: and a frightening nullpointer.

    Have I missed a step not installed a package, is unbuntu 12.04 not
    supported for mapreduce.
    If anyone has an idea what I am doing wrong.

    JobYTacker
    2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.**JobTracker:
    Lost tracker 'tracker_ks25937.kimsufi.com:l**ocalhost/127.0.0.1:47149'
    2012-11-09 15:25:59,772 ERROR org.apache.hadoop.security.**UserGroupInformation:
    PriviledgedActionException as:mapred (auth:SIMPLE)
    cause:java.io.IOException: java.lang.NullPointerException
    2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 13 on 8021, call heartbeat(org.apache.hadoop.**
    mapred.TaskTrackerStatus@**12fcf657, true, true, true, -1), rpc
    version=2, client version=32, methodsFingerPrint=-159967141 from
    37.59.44.188:45329: error: java.io.IOException:
    java.lang.NullPointerException
    java.io.IOException: java.lang.NullPointerException
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2735)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**Handler.run(Server.java:1687)
    2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.**ScriptBasedMapping:
    Exception running /run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER/topology.py
    91.121.69.14
    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py" (in directory
    "/run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER"):
    java.io.IOException: error=13, Permission denied
    This looks to me to be the root cause. The question is why can't the JT
    run the topology script. Though it's a shame that it results in an NPE, I'm
    not entirely surprised that a properly configured topology script that is
    present but can't be executed for some reason would result in an error on
    the JT.

    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:460)
    at org.apache.hadoop.util.Shell.**runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.**run(Shell.java:188)
    at org.apache.hadoop.util.Shell$**ShellCommandExecutor.execute(
    **Shell.java:381)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.**runResolveCommand(**
    ScriptBasedMapping.java:242)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.resolve(**ScriptBasedMapping.java:180)
    at org.apache.hadoop.net.**CachedDNSToSwitchMapping.**resolve(*
    *CachedDNSToSwitchMapping.java:**119)
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2734)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**Handler.run(Server.java:1687)
    Caused by: java.io.IOException: java.io.IOException: error=13,
    Permission denied
    at java.lang.UNIXProcess.<init>(**UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(**ProcessImpl.java:65)
    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:453)
    ... 21 more


    TASKTracker:2012-11-06 10:29:36,641 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


    --


  • Nicolas Maillard at Nov 15, 2012 at 4:43 pm
    Hi

    Ok a got down to it.
    For some reason the python file topology.py can be excecuted and returns a
    permission denied, I'm guessing this goes thriough a try catch mecanism
    which results in a null value, this is not caught or expected in
    resolveAndAddToTopology
    and ends up in a null pointer exceptions.
    This is what phillip was pointing to.

    What I can't maker sense of is python is installed if I do:
    /usr/bin/env python topology.py everything goes fine.

    If I move the topology.py to /home fro example and execute it:
    ./topology.py it's also good.

    For some reason it does not like beeing in /run somewhere.

    Does this ring a bell for anyone?

    regards
    On Wed, Nov 14, 2012 at 6:17 PM, Nicolas Maillard wrote:

    Hi everyone

    I have completely reinstalled my cluster from scrracth and followed
    excatly the automated installation

    https://ccp.cloudera.com/display/FREE4DOC/Automated+Installation+of+Cloudera+Manager+and+CDH

    But when I try to run a map reduce job. I get the same error as before.
    I try using the documentation "running an example with MRv1" found:

    https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode#InstallingCDH4onaSingleLinuxNodeinPseudo-distributedMode-exAppMRv1

    But no cigar I can't get map reduce working, but i haven't triend anything
    fancy its a 64bit ubunut 12.04.

    Anyone seen this before or just got the mapreduce working with cloudera
    CDH4.1.1.

    thanks


    On Tue, Nov 13, 2012 at 10:46 AM, nick wrote:

    Hi everyone and thanks for the help

    So I have gone ahead and checked my directories and files the rights and
    ownserhsip seem good. To be sure I have chmod 775 on all of them.
    *
    *
    *ls -ld / /run /run/cloudera-scm-agent /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *drwxr-xr-x 25 root root 4096 Nov 2 18:57 /*
    *drwxr-xr-x 23 root root 760 Nov 13 09:50 /run*
    *drwxrwxr-x 4 root root 100 Nov 5 17:56 /run/cloudera-scm-agent*
    *drwxrwxr-x 259 root root 5180 Nov 9 15:18 /run/cloudera-scm-agent/
    process*
    *drwxrwxr-x 3 mapred hadoop 360 Nov 9 15:17 /run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER*
    *-rwxrwxr-x 1 mapred hadoop 1382 Nov 9 15:17 /run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER/topology.py*
    *
    *
    I was not aware python was needed for cloudera manager, I stuck to the
    installation guide. However python is intalled an in the path.
    I made sure "/usr/bin/env" exists:
    *ls -ld /usr/bin/env*
    *-rwxr-xr-x 1 root root 27040 Apr 1 2012 /usr/bin/env*
    *
    *
    *
    *
    The mount seems ok to me:

    mount
    /dev/root on / type ext4 (rw,relatime,errors=remount-**
    ro,user_xattr,acl,barrier=1,**data=ordered)
    none on /proc type proc (rw,nosuid,nodev,noexec,**relatime)
    none on /sys type sysfs (rw,nosuid,nodev,noexec,**relatime)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,
    **relatime)
    none on /sys/fs/fuse/connections type fusectl (rw,relatime)
    /dev on /dev type devtmpfs (rw,relatime,size=12338840k,**
    nr_inodes=3084710,mode=755)
    none on /dev/pts type devpts (rw,nosuid,noexec,relatime,**gid=5,mode=620)
    none on /run type tmpfs (rw,nosuid,noexec,relatime,**
    size=2474428k,mode=755)
    none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,**
    relatime,size=5120k)
    none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
    /dev/sda2 on /home type ext4 (rw,relatime,user_xattr,acl,**
    barrier=1,data=ordered)
    *
    *
    *
    *
    I hope all of this info can help, it has been really complicated to get a
    hadoop/hbase cluster running.

    thanks for all the help

    regards
    On Saturday, November 10, 2012 4:29:22 AM UTC+1, Philip Zeyliger wrote:

    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/proce**
    ss/427-mapreduce-JOBTRACKER/**topology.py" (in directory
    "/run/cloudera-scm-agent/proce**ss/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied

    As Aaron points out below, this is unusual and is the cause of your
    errors in the log. Could you run "ls -ld / /run /run/cloudera-scm-agent
    /run/cloudera-scm-agent/**process /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py". Perhaps a permission
    is wrong along one of those directories.

    Alternately, does "/usr/bin/env" exist and does "python" exist in your
    path? If the interpreter for a shell script doesn't exist, I could see an
    error occurring. (Unfortunately I'm not currently able to double-check
    manually.)

    Finally, if your /run file system is mounted noexec, I could see that.
    Are there any odd options in the output of 'mount'?

    -- Philip
    On Fri, Nov 9, 2012 at 2:29 PM, Aaron T. Myers wrote:

    +scm-users@
    bcc: cdh-user@

    Hi Nicolas,

    Sending this to scm-users@, which might be more helpful with this
    issue.

    I do have one comment/observation down below.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard <
    nicolas....@fifty-five.**com> wrote:
    Hi everyone

    Follow up on my previous message.
    I played around a bit tried supressing and changing what machine had
    the jobtracker in cloudera manager 4.1.1.
    But on start everything looks fine the manager says started and good
    to go.
    However if I look in the jobtracker log and tasktracker logs I see
    errors: and a frightening nullpointer.

    Have I missed a step not installed a package, is unbuntu 12.04 not
    supported for mapreduce.
    If anyone has an idea what I am doing wrong.

    JobYTacker
    2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.**JobTracker:
    Lost tracker 'tracker_ks25937.kimsufi.com:l**ocalhost/127.0.0.1:47149'
    2012-11-09 15:25:59,772 ERROR org.apache.hadoop.security.**UserGroupInformation:
    PriviledgedActionException as:mapred (auth:SIMPLE)
    cause:java.io.IOException: java.lang.NullPointerException
    2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 13 on 8021, call heartbeat(org.apache.hadoop.**
    mapred.TaskTrackerStatus@**12fcf657, true, true, true, -1), rpc
    version=2, client version=32, methodsFingerPrint=-159967141 from
    37.59.44.188:45329: error: java.io.IOException:
    java.lang.NullPointerException
    java.io.IOException: java.lang.NullPointerException
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2735)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**
    Handler.run(Server.java:1687)
    2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.**ScriptBasedMapping:
    Exception running /run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER/topology.py
    91.121.69.14
    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py" (in directory
    "/run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER"):
    java.io.IOException: error=13, Permission denied
    This looks to me to be the root cause. The question is why can't the JT
    run the topology script. Though it's a shame that it results in an NPE, I'm
    not entirely surprised that a properly configured topology script that is
    present but can't be executed for some reason would result in an error on
    the JT.

    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:460)
    at org.apache.hadoop.util.Shell.**runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.**run(Shell.java:188)
    at org.apache.hadoop.util.Shell$**
    ShellCommandExecutor.execute(**Shell.java:381)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.**runResolveCommand(**
    ScriptBasedMapping.java:242)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.resolve(**ScriptBasedMapping.java:180)
    at org.apache.hadoop.net.**CachedDNSToSwitchMapping.**resolve(
    **CachedDNSToSwitchMapping.java:**119)
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2734)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:**
    1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**
    Handler.run(Server.java:1687)
    Caused by: java.io.IOException: java.io.IOException: error=13,
    Permission denied
    at java.lang.UNIXProcess.<init>(**UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(**ProcessImpl.java:65)
    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:453)
    ... 21 more


    TASKTracker:2012-11-06 10:29:36,641 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


    --


  • Philip Zeyliger at Nov 15, 2012 at 5:39 pm
    Nicolas--what OS are you using?

    I'd have thought that this (specifically the noexec):
    none on /run type tmpfs (rw,nosuid,noexec,relatime,**
    size=2474428k,mode=755)
    might be the issue, but I have the same setup on my dev box, and it seems
    to be working. You could try to change the mount options and see if the
    problem goes away.

    It's not a good solution, but, since you're blocked on getting your cluster
    to work, you can configure the following option manually, pointed to a
    script in /usr/local somewhere. (You can user our script; just copy it
    out.) You'll need to do this both for the JobTracker and Namenode
    configurations. That should get you unblocked.

    Topology Script File Name
    topology.script.file.name, net.topology.script.file.name

    Cheers,

    -- Philip
    On Thu, Nov 15, 2012 at 8:43 AM, Nicolas Maillard wrote:

    Hi

    Ok a got down to it.
    For some reason the python file topology.py can be excecuted and returns a
    permission denied, I'm guessing this goes thriough a try catch mecanism
    which results in a null value, this is not caught or expected in
    resolveAndAddToTopology
    and ends up in a null pointer exceptions.
    This is what phillip was pointing to.

    What I can't maker sense of is python is installed if I do:
    /usr/bin/env python topology.py everything goes fine.

    If I move the topology.py to /home fro example and execute it:
    ./topology.py it's also good.

    For some reason it does not like beeing in /run somewhere.

    Does this ring a bell for anyone?

    regards

    On Wed, Nov 14, 2012 at 6:17 PM, Nicolas Maillard <
    nicolas.maillard@fifty-five.com> wrote:
    Hi everyone

    I have completely reinstalled my cluster from scrracth and followed
    excatly the automated installation

    https://ccp.cloudera.com/display/FREE4DOC/Automated+Installation+of+Cloudera+Manager+and+CDH

    But when I try to run a map reduce job. I get the same error as before.
    I try using the documentation "running an example with MRv1" found:

    https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode#InstallingCDH4onaSingleLinuxNodeinPseudo-distributedMode-exAppMRv1

    But no cigar I can't get map reduce working, but i haven't triend
    anything fancy its a 64bit ubunut 12.04.

    Anyone seen this before or just got the mapreduce working with cloudera
    CDH4.1.1.

    thanks


    On Tue, Nov 13, 2012 at 10:46 AM, nick wrote:

    Hi everyone and thanks for the help

    So I have gone ahead and checked my directories and files the rights and
    ownserhsip seem good. To be sure I have chmod 775 on all of them.
    *
    *
    *ls -ld / /run /run/cloudera-scm-agent /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *drwxr-xr-x 25 root root 4096 Nov 2 18:57 /*
    *drwxr-xr-x 23 root root 760 Nov 13 09:50 /run*
    *drwxrwxr-x 4 root root 100 Nov 5 17:56 /run/cloudera-scm-agent*
    *drwxrwxr-x 259 root root 5180 Nov 9 15:18 /run/cloudera-scm-agent/
    process*
    *drwxrwxr-x 3 mapred hadoop 360 Nov 9 15:17 /run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER*
    *-rwxrwxr-x 1 mapred hadoop 1382 Nov 9 15:17 /run/cloudera-scm-agent/
    process/427-mapreduce-JOBTRACKER/topology.py*
    *
    *
    I was not aware python was needed for cloudera manager, I stuck to the
    installation guide. However python is intalled an in the path.
    I made sure "/usr/bin/env" exists:
    *ls -ld /usr/bin/env*
    *-rwxr-xr-x 1 root root 27040 Apr 1 2012 /usr/bin/env*
    *
    *
    *
    *
    The mount seems ok to me:

    mount
    /dev/root on / type ext4 (rw,relatime,errors=remount-**
    ro,user_xattr,acl,barrier=1,**data=ordered)
    none on /proc type proc (rw,nosuid,nodev,noexec,**relatime)
    none on /sys type sysfs (rw,nosuid,nodev,noexec,**relatime)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc
    (rw,nosuid,nodev,noexec,**relatime)
    none on /sys/fs/fuse/connections type fusectl (rw,relatime)
    /dev on /dev type devtmpfs (rw,relatime,size=12338840k,**
    nr_inodes=3084710,mode=755)
    none on /dev/pts type devpts (rw,nosuid,noexec,relatime,**
    gid=5,mode=620)
    none on /run type tmpfs (rw,nosuid,noexec,relatime,**
    size=2474428k,mode=755)
    none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,**
    relatime,size=5120k)
    none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
    /dev/sda2 on /home type ext4 (rw,relatime,user_xattr,acl,**
    barrier=1,data=ordered)
    *
    *
    *
    *
    I hope all of this info can help, it has been really complicated to get
    a hadoop/hbase cluster running.

    thanks for all the help

    regards
    On Saturday, November 10, 2012 4:29:22 AM UTC+1, Philip Zeyliger wrote:

    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/proce*
    *ss/427-mapreduce-JOBTRACKER/**topology.py" (in directory
    "/run/cloudera-scm-agent/proce**ss/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied

    As Aaron points out below, this is unusual and is the cause of your
    errors in the log. Could you run "ls -ld / /run /run/cloudera-scm-agent
    /run/cloudera-scm-agent/**process /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py". Perhaps a permission
    is wrong along one of those directories.

    Alternately, does "/usr/bin/env" exist and does "python" exist in your
    path? If the interpreter for a shell script doesn't exist, I could see an
    error occurring. (Unfortunately I'm not currently able to double-check
    manually.)

    Finally, if your /run file system is mounted noexec, I could see that.
    Are there any odd options in the output of 'mount'?

    -- Philip
    On Fri, Nov 9, 2012 at 2:29 PM, Aaron T. Myers wrote:

    +scm-users@
    bcc: cdh-user@

    Hi Nicolas,

    Sending this to scm-users@, which might be more helpful with this
    issue.

    I do have one comment/observation down below.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard <
    nicolas....@fifty-five.**com> wrote:
    Hi everyone

    Follow up on my previous message.
    I played around a bit tried supressing and changing what machine had
    the jobtracker in cloudera manager 4.1.1.
    But on start everything looks fine the manager says started and good
    to go.
    However if I look in the jobtracker log and tasktracker logs I see
    errors: and a frightening nullpointer.

    Have I missed a step not installed a package, is unbuntu 12.04 not
    supported for mapreduce.
    If anyone has an idea what I am doing wrong.

    JobYTacker
    2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.**JobTracker:
    Lost tracker 'tracker_ks25937.kimsufi.com:l**ocalhost/127.0.0.1:47149
    '
    2012-11-09 15:25:59,772 ERROR org.apache.hadoop.security.**UserGroupInformation:
    PriviledgedActionException as:mapred (auth:SIMPLE)
    cause:java.io.IOException: java.lang.NullPointerException
    2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC Server
    handler 13 on 8021, call heartbeat(org.apache.hadoop.**
    mapred.TaskTrackerStatus@**12fcf657, true, true, true, -1), rpc
    version=2, client version=32, methodsFingerPrint=-159967141 from
    37.59.44.188:45329: error: java.io.IOException:
    java.lang.NullPointerException
    java.io.IOException: java.lang.NullPointerException
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2735)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:*
    *1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:*
    *1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**
    Handler.run(Server.java:1687)
    2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.**ScriptBasedMapping:
    Exception running /run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER/topology.py
    91.121.69.14
    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py" (in directory
    "/run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER"):
    java.io.IOException: error=13, Permission denied
    This looks to me to be the root cause. The question is why can't the
    JT run the topology script. Though it's a shame that it results in an NPE,
    I'm not entirely surprised that a properly configured topology script that
    is present but can't be executed for some reason would result in an error
    on the JT.

    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:460)
    at org.apache.hadoop.util.Shell.**runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.**run(Shell.java:188)
    at org.apache.hadoop.util.Shell$**
    ShellCommandExecutor.execute(**Shell.java:381)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.**runResolveCommand(**
    ScriptBasedMapping.java:242)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.resolve(**ScriptBasedMapping.java:180)
    at org.apache.hadoop.net.**CachedDNSToSwitchMapping.**
    resolve(**CachedDNSToSwitchMapping.java:**119)
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2734)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:*
    *1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:*
    *1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**
    Handler.run(Server.java:1687)
    Caused by: java.io.IOException: java.io.IOException: error=13,
    Permission denied
    at java.lang.UNIXProcess.<init>(**UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(**ProcessImpl.java:65)
    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:453)
    ... 21 more


    TASKTracker:2012-11-06 10:29:36,641 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


    --


  • Philip Zeyliger at Nov 16, 2012 at 5:33 pm
    To close the loop, it turns out to have been the "noexec" setting on the
    mount point.
    On Thu, Nov 15, 2012 at 9:39 AM, Philip Zeyliger wrote:

    Nicolas--what OS are you using?

    I'd have thought that this (specifically the noexec):
    none on /run type tmpfs (rw,nosuid,noexec,relatime,**
    size=2474428k,mode=755)
    might be the issue, but I have the same setup on my dev box, and it seems
    to be working. You could try to change the mount options and see if the
    problem goes away.

    It's not a good solution, but, since you're blocked on getting your
    cluster to work, you can configure the following option manually, pointed
    to a script in /usr/local somewhere. (You can user our script; just copy
    it out.) You'll need to do this both for the JobTracker and Namenode
    configurations. That should get you unblocked.

    Topology Script File Name
    topology.script.file.name, net.topology.script.file.name

    Cheers,

    -- Philip

    On Thu, Nov 15, 2012 at 8:43 AM, Nicolas Maillard <
    nicolas.maillard@fifty-five.com> wrote:
    Hi

    Ok a got down to it.
    For some reason the python file topology.py can be excecuted and returns
    a permission denied, I'm guessing this goes thriough a try catch mecanism
    which results in a null value, this is not caught or expected in
    resolveAndAddToTopology
    and ends up in a null pointer exceptions.
    This is what phillip was pointing to.

    What I can't maker sense of is python is installed if I do:
    /usr/bin/env python topology.py everything goes fine.

    If I move the topology.py to /home fro example and execute it:
    ./topology.py it's also good.

    For some reason it does not like beeing in /run somewhere.

    Does this ring a bell for anyone?

    regards

    On Wed, Nov 14, 2012 at 6:17 PM, Nicolas Maillard <
    nicolas.maillard@fifty-five.com> wrote:
    Hi everyone

    I have completely reinstalled my cluster from scrracth and followed
    excatly the automated installation

    https://ccp.cloudera.com/display/FREE4DOC/Automated+Installation+of+Cloudera+Manager+and+CDH

    But when I try to run a map reduce job. I get the same error as before.
    I try using the documentation "running an example with MRv1" found:

    https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode#InstallingCDH4onaSingleLinuxNodeinPseudo-distributedMode-exAppMRv1

    But no cigar I can't get map reduce working, but i haven't triend
    anything fancy its a 64bit ubunut 12.04.

    Anyone seen this before or just got the mapreduce working with cloudera
    CDH4.1.1.

    thanks


    On Tue, Nov 13, 2012 at 10:46 AM, nick wrote:

    Hi everyone and thanks for the help

    So I have gone ahead and checked my directories and files the rights
    and ownserhsip seem good. To be sure I have chmod 775 on all of them.
    *
    *
    *ls -ld / /run /run/cloudera-scm-agent /run/cloudera-scm-agent/process
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *drwxr-xr-x 25 root root 4096 Nov 2 18:57 /*
    *drwxr-xr-x 23 root root 760 Nov 13 09:50 /run*
    *drwxrwxr-x 4 root root 100 Nov 5 17:56 /run/cloudera-scm-agent
    *
    *drwxrwxr-x 259 root root 5180 Nov 9 15:18
    /run/cloudera-scm-agent/process*
    *drwxrwxr-x 3 mapred hadoop 360 Nov 9 15:17
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER*
    *-rwxrwxr-x 1 mapred hadoop 1382 Nov 9 15:17
    /run/cloudera-scm-agent/process/427-mapreduce-JOBTRACKER/topology.py*
    *
    *
    I was not aware python was needed for cloudera manager, I stuck to the
    installation guide. However python is intalled an in the path.
    I made sure "/usr/bin/env" exists:
    *ls -ld /usr/bin/env*
    *-rwxr-xr-x 1 root root 27040 Apr 1 2012 /usr/bin/env*
    *
    *
    *
    *
    The mount seems ok to me:

    mount
    /dev/root on / type ext4 (rw,relatime,errors=remount-**
    ro,user_xattr,acl,barrier=1,**data=ordered)
    none on /proc type proc (rw,nosuid,nodev,noexec,**relatime)
    none on /sys type sysfs (rw,nosuid,nodev,noexec,**relatime)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc
    (rw,nosuid,nodev,noexec,**relatime)
    none on /sys/fs/fuse/connections type fusectl (rw,relatime)
    /dev on /dev type devtmpfs (rw,relatime,size=12338840k,**
    nr_inodes=3084710,mode=755)
    none on /dev/pts type devpts (rw,nosuid,noexec,relatime,**
    gid=5,mode=620)
    none on /run type tmpfs (rw,nosuid,noexec,relatime,**
    size=2474428k,mode=755)
    none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,**
    relatime,size=5120k)
    none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
    /dev/sda2 on /home type ext4 (rw,relatime,user_xattr,acl,**
    barrier=1,data=ordered)
    *
    *
    *
    *
    I hope all of this info can help, it has been really complicated to get
    a hadoop/hbase cluster running.

    thanks for all the help

    regards
    On Saturday, November 10, 2012 4:29:22 AM UTC+1, Philip Zeyliger wrote:

    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/proce
    **ss/427-mapreduce-JOBTRACKER/**topology.py" (in directory
    "/run/cloudera-scm-agent/proce**ss/427-mapreduce-JOBTRACKER"):
    java.io.IOException: error=13, Permission denied

    As Aaron points out below, this is unusual and is the cause of your
    errors in the log. Could you run "ls -ld / /run /run/cloudera-scm-agent
    /run/cloudera-scm-agent/**process /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER /run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py". Perhaps a
    permission is wrong along one of those directories.

    Alternately, does "/usr/bin/env" exist and does "python" exist in your
    path? If the interpreter for a shell script doesn't exist, I could see an
    error occurring. (Unfortunately I'm not currently able to double-check
    manually.)

    Finally, if your /run file system is mounted noexec, I could see that.
    Are there any odd options in the output of 'mount'?

    -- Philip
    On Fri, Nov 9, 2012 at 2:29 PM, Aaron T. Myers wrote:

    +scm-users@
    bcc: cdh-user@

    Hi Nicolas,

    Sending this to scm-users@, which might be more helpful with this
    issue.

    I do have one comment/observation down below.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Fri, Nov 9, 2012 at 6:28 AM, Nicolas Maillard <
    nicolas....@fifty-five.**com> wrote:
    Hi everyone

    Follow up on my previous message.
    I played around a bit tried supressing and changing what machine had
    the jobtracker in cloudera manager 4.1.1.
    But on start everything looks fine the manager says started and good
    to go.
    However if I look in the jobtracker log and tasktracker logs I see
    errors: and a frightening nullpointer.

    Have I missed a step not installed a package, is unbuntu 12.04 not
    supported for mapreduce.
    If anyone has an idea what I am doing wrong.

    JobYTacker
    2012-11-09 15:25:59,772 INFO org.apache.hadoop.mapred.**JobTracker:
    Lost tracker 'tracker_ks25937.kimsufi.com:l**ocalhost/
    127.0.0.1:47149'
    2012-11-09 15:25:59,772 ERROR org.apache.hadoop.security.**UserGroupInformation:
    PriviledgedActionException as:mapred (auth:SIMPLE)
    cause:java.io.IOException: java.lang.NullPointerException
    2012-11-09 15:25:59,773 INFO org.apache.hadoop.ipc.Server: IPC
    Server handler 13 on 8021, call heartbeat(org.apache.hadoop.**
    mapred.TaskTrackerStatus@**12fcf657, true, true, true, -1), rpc
    version=2, client version=32, methodsFingerPrint=-159967141 from
    37.59.44.188:45329: error: java.io.IOException:
    java.lang.NullPointerException
    java.io.IOException: java.lang.NullPointerException
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2735)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:
    **1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:
    **1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**
    Handler.run(Server.java:1687)
    2012-11-09 15:25:59,774 WARN org.apache.hadoop.net.**ScriptBasedMapping:
    Exception running /run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER/topology.py
    91.121.69.14
    java.io.IOException: Cannot run program "/run/cloudera-scm-agent/**
    process/427-mapreduce-**JOBTRACKER/topology.py" (in directory
    "/run/cloudera-scm-agent/**process/427-mapreduce-**JOBTRACKER"):
    java.io.IOException: error=13, Permission denied
    This looks to me to be the root cause. The question is why can't the
    JT run the topology script. Though it's a shame that it results in an NPE,
    I'm not entirely surprised that a properly configured topology script that
    is present but can't be executed for some reason would result in an error
    on the JT.

    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:460)
    at org.apache.hadoop.util.Shell.**runCommand(Shell.java:206)
    at org.apache.hadoop.util.Shell.**run(Shell.java:188)
    at org.apache.hadoop.util.Shell$**
    ShellCommandExecutor.execute(**Shell.java:381)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.**runResolveCommand(**
    ScriptBasedMapping.java:242)
    at org.apache.hadoop.net.**ScriptBasedMapping$**
    RawScriptBasedMapping.resolve(**ScriptBasedMapping.java:180)
    at org.apache.hadoop.net.**CachedDNSToSwitchMapping.**
    resolve(**CachedDNSToSwitchMapping.java:**119)
    at org.apache.hadoop.mapred.**JobTracker.**
    resolveAndAddToTopology(**JobTracker.java:2734)
    at org.apache.hadoop.mapred.**JobTracker.addNewTracker(**
    JobTracker.java:2715)
    at org.apache.hadoop.mapred.**JobTracker.processHeartbeat(**
    JobTracker.java:3211)
    at org.apache.hadoop.mapred.**JobTracker.heartbeat(**
    JobTracker.java:2915)
    at sun.reflect.**GeneratedMethodAccessor5.**invoke(Unknown
    Source)
    at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
    DelegatingMethodAccessorImpl.**java:25)
    at java.lang.reflect.Method.**invoke(Method.java:597)
    at org.apache.hadoop.ipc.**WritableRpcEngine$Server$**
    WritableRpcInvoker.call(**WritableRpcEngine.java:474)
    at org.apache.hadoop.ipc.RPC$**Server.call(RPC.java:898)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:
    **1693)
    at org.apache.hadoop.ipc.Server$**Handler$1.run(Server.java:
    **1689)
    at java.security.**AccessController.doPrivileged(**Native
    Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1332)
    at org.apache.hadoop.ipc.Server$**
    Handler.run(Server.java:1687)
    Caused by: java.io.IOException: java.io.IOException: error=13,
    Permission denied
    at java.lang.UNIXProcess.<init>(**UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(**ProcessImpl.java:65)
    at java.lang.ProcessBuilder.**start(ProcessBuilder.java:453)
    ... 21 more


    TASKTracker:2012-11-06 10:29:36,641 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,644 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,647 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,650 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,652 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,655 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,657 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,660 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,662 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,665 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,667 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,670 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1
    2012-11-06 10:29:36,672 INFO org.apache.hadoop.mapred.**TaskTracker:
    Resending 'status' to 'ks306783.kimsufi.com' with reponseId '-1


    --


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedNov 9, '12 at 10:29p
activeNov 16, '12 at 5:33p
posts7
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase