FAQ
Hi,
I am trying to setup a test system to host a distributed hbase installation.
No matter what I do, I get the below errors.

2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery fo
r block null bad datanode[0] nodes == null
2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
blo
ck locations. Source file "/tmp/mapred/system/jobtracker.info" - Aborting...
2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: Writing to
fil
e hdfs://
ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker.
info failed!
2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: FileSystem
is
not ready yet!
2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker: Failed to
init
ialize recovery manager.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/tmp/mapred/sys
tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
k(FSNamesystem.java:1417)
....

this is how i setup my config -
core-site.xml -
<configuration>

<property>
<name>fs.default.name</name>
<value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
</property>

</configuration>

hdfs-site.xml -
<configuration>

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.name.dir</name>
<value>/tmp/hbase</value>
</property>

<property>
<name>dfs.data.dir</name>
<value>/tmp/hbase</value>
</property>

</configuration>


mapred-site.xml -
<configuration>

<property>
<name>mapred.job.tracker</name>
<value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
</property>

<property>
<name>mapred.local.dir</name>
<value>/tmp/mapred_tmp</value>
</property>

<property>
<name>mapred.map.tasks</name>
<value>10</value>
</property>

<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
</property>

<property>
<name>mapred.system.dir</name>
<value>/tmp/mapred/system/</value>
</property>


</configuration>

i know that i am missing something really basic but not sure what it is. the
documentation says mapred.system.dir should be globally accessible. how do i
achieve that?

thanks
vinod

Search Discussions

  • Dejan Menges at Sep 29, 2011 at 11:49 am
    In core-site.xml, on first, you miss port in the end for HDFS:

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    Regards,
    Dejo

    On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
    wrote:
    Hi,
    I am trying to setup a test system to host a distributed hbase
    installation.
    No matter what I do, I get the below errors.

    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery fo
    r block null bad datanode[0] nodes == null
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could not
    get
    blo
    ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: Writing
    to
    fil
    e hdfs://
    ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker.
    info failed!
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
    FileSystem
    is
    not ready yet!
    2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker: Failed to
    init
    ialize recovery manager.
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /tmp/mapred/sys
    tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
    k(FSNamesystem.java:1417)
    ....

    this is how i setup my config -
    core-site.xml -
    <configuration>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    </configuration>

    hdfs-site.xml -
    <configuration>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>

    <property>
    <name>dfs.name.dir</name>
    <value>/tmp/hbase</value>
    </property>

    <property>
    <name>dfs.data.dir</name>
    <value>/tmp/hbase</value>
    </property>

    </configuration>


    mapred-site.xml -
    <configuration>

    <property>
    <name>mapred.job.tracker</name>
    <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
    </property>

    <property>
    <name>mapred.local.dir</name>
    <value>/tmp/mapred_tmp</value>
    </property>

    <property>
    <name>mapred.map.tasks</name>
    <value>10</value>
    </property>

    <property>
    <name>mapred.reduce.tasks</name>
    <value>2</value>
    </property>

    <property>
    <name>mapred.system.dir</name>
    <value>/tmp/mapred/system/</value>
    </property>


    </configuration>

    i know that i am missing something really basic but not sure what it is.
    the
    documentation says mapred.system.dir should be globally accessible. how do
    i
    achieve that?

    thanks
    vinod
  • Vinod Gupta Tankala at Sep 29, 2011 at 6:06 pm
    Thanks Dejo for pointing that out. I realized that earlier and fixed it. But
    I still hit the same problem.

    In my case, I only have a single host for now. But I am still trying to do a
    distributed setup by listing the machine itself as a slave in config and not
    using localhost anywhere. Does this even work? if not, I can try spending
    more time on pseudo-distributed setup for now.

    thanks

    On Thu, Sep 29, 2011 at 4:48 AM, Dejan Menges wrote:

    In core-site.xml, on first, you miss port in the end for HDFS:

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    Regards,
    Dejo

    On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
    wrote:
    Hi,
    I am trying to setup a test system to host a distributed hbase
    installation.
    No matter what I do, I get the below errors.

    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery fo
    r block null bad datanode[0] nodes == null
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could not
    get
    blo
    ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker: Writing
    to
    fil
    e hdfs://
    ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker.
    info failed!
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
    FileSystem
    is
    not ready yet!
    2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker: Failed to
    init
    ialize recovery manager.
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /tmp/mapred/sys
    tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
    k(FSNamesystem.java:1417)
    ....

    this is how i setup my config -
    core-site.xml -
    <configuration>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    </configuration>

    hdfs-site.xml -
    <configuration>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>

    <property>
    <name>dfs.name.dir</name>
    <value>/tmp/hbase</value>
    </property>

    <property>
    <name>dfs.data.dir</name>
    <value>/tmp/hbase</value>
    </property>

    </configuration>


    mapred-site.xml -
    <configuration>

    <property>
    <name>mapred.job.tracker</name>
    <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
    </property>

    <property>
    <name>mapred.local.dir</name>
    <value>/tmp/mapred_tmp</value>
    </property>

    <property>
    <name>mapred.map.tasks</name>
    <value>10</value>
    </property>

    <property>
    <name>mapred.reduce.tasks</name>
    <value>2</value>
    </property>

    <property>
    <name>mapred.system.dir</name>
    <value>/tmp/mapred/system/</value>
    </property>


    </configuration>

    i know that i am missing something really basic but not sure what it is.
    the
    documentation says mapred.system.dir should be globally accessible. how do
    i
    achieve that?

    thanks
    vinod
  • Dejan Menges at Sep 29, 2011 at 9:56 pm
    Hi Vinod,

    This sounds to me like a DNS issue. I have the same thing on development
    environments, and couple of things that made me big headache were:

    - Inconsistency of DNS and machine hostname, if not used localhost
    - On some systems in /etc/hosts you have localhost defined for both IPv4 and
    IPv6 (MacOS is one example) where you need to comment IPv6.
    - When I don't want to use localhost at all, I comment it out totally in
    /etc/hosts

    So, the approach I have in this case is to, prior starting doing anything
    with HBase, stop everything, clean the logs, format namenode, and start
    Hadoop only. Monitor logs, what won't be problem in this case as that's
    stand alone installation, to see if HDFS get up OK. You can also try coping
    file from and to your HDFS to be sure HDFS is working fine. Then you can
    start MapReduce (JobTracker and TaskTrackers) and later on HBase.

    I hope this will help :)

    Regards,
    Dejo

    On Thu, Sep 29, 2011 at 8:06 PM, Vinod Gupta Tankala
    wrote:
    Thanks Dejo for pointing that out. I realized that earlier and fixed it.
    But
    I still hit the same problem.

    In my case, I only have a single host for now. But I am still trying to do
    a
    distributed setup by listing the machine itself as a slave in config and
    not
    using localhost anywhere. Does this even work? if not, I can try spending
    more time on pseudo-distributed setup for now.

    thanks


    On Thu, Sep 29, 2011 at 4:48 AM, Dejan Menges <dejan.menges@gmail.com
    wrote:
    In core-site.xml, on first, you miss port in the end for HDFS:

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    Regards,
    Dejo

    On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
    wrote:
    Hi,
    I am trying to setup a test system to host a distributed hbase
    installation.
    No matter what I do, I get the below errors.

    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery fo
    r block null bad datanode[0] nodes == null
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could
    not
    get
    blo
    ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
    Writing
    to
    fil
    e hdfs://
    ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker
    .
    info failed!
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
    FileSystem
    is
    not ready yet!
    2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker:
    Failed
    to
    init
    ialize recovery manager.
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /tmp/mapred/sys
    tem/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
    k(FSNamesystem.java:1417)
    ....

    this is how i setup my config -
    core-site.xml -
    <configuration>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    </configuration>

    hdfs-site.xml -
    <configuration>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>

    <property>
    <name>dfs.name.dir</name>
    <value>/tmp/hbase</value>
    </property>

    <property>
    <name>dfs.data.dir</name>
    <value>/tmp/hbase</value>
    </property>

    </configuration>


    mapred-site.xml -
    <configuration>

    <property>
    <name>mapred.job.tracker</name>
    <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
    </property>

    <property>
    <name>mapred.local.dir</name>
    <value>/tmp/mapred_tmp</value>
    </property>

    <property>
    <name>mapred.map.tasks</name>
    <value>10</value>
    </property>

    <property>
    <name>mapred.reduce.tasks</name>
    <value>2</value>
    </property>

    <property>
    <name>mapred.system.dir</name>
    <value>/tmp/mapred/system/</value>
    </property>


    </configuration>

    i know that i am missing something really basic but not sure what it
    is.
    the
    documentation says mapred.system.dir should be globally accessible. how do
    i
    achieve that?

    thanks
    vinod
  • Vinod Gupta Tankala at Sep 29, 2011 at 10:26 pm
    Thanks Dejo.
    I suspected that running distributed mode with single host is going to be
    problematic. so i reverted the settings to pseudo-distributed and deleted
    everything and did a reformat of the namenode. now hdfs starts fine and i
    can see a live node in web console on port 50030.
    but when i start a mapred example task (grep-search one) after creating
    input dir in dfs, it fails with these kind of errors -

    2011-09-29 22:12:12,477 INFO org.apache.hadoop.mapred.JvmManager: In
    JvmRunner c
    onstructed JVM ID: jvm_201109292200_0004_m_-1121734218
    2011-09-29 22:12:12,477 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
    jvm
    _201109292200_0004_m_-1121734218 spawned.
    2011-09-29 22:12:12,480 INFO org.apache.hadoop.mapred.TaskController:
    Writing co
    mmands to /media/ephemeral0/hadoop/mapred/local

    /ttprivate/taskTracker/ec2-user/jobcache/job_201109292200_0004/attempt_20110
    9292200_0004_m_000015_2/taskjvm.sh
    2011-09-29 22:12:12,528 WARN org.apache.hadoop.mapred.DefaultTaskController:
    Exi
    t code from task is : 127
    2011-09-29 22:12:12,529 INFO org.apache.hadoop.mapred.DefaultTaskController:
    Out
    put from DefaultTaskController's launchTask follows:
    2011-09-29 22:12:12,529 INFO org.apache.hadoop.mapred.TaskController:
    2011-09-29 22:12:12,529 INFO org.apache.hadoop.mapred.JvmManager: JVM Not
    killed
    jvm_201109292200_0004_m_-1121734218 but just removed
    2011-09-29 22:12:12,530 WARN org.apache.hadoop.mapred.TaskRunner:
    attempt_201109
    292200_0004_m_000015_2 : Child Error
    java.io.IOException: Task process exit with nonzero status of 127.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


    now i wonder what could be the reason for this.

    thanks
    vinod
    On Thu, Sep 29, 2011 at 2:55 PM, Dejan Menges wrote:

    Hi Vinod,

    This sounds to me like a DNS issue. I have the same thing on development
    environments, and couple of things that made me big headache were:

    - Inconsistency of DNS and machine hostname, if not used localhost
    - On some systems in /etc/hosts you have localhost defined for both IPv4
    and
    IPv6 (MacOS is one example) where you need to comment IPv6.
    - When I don't want to use localhost at all, I comment it out totally in
    /etc/hosts

    So, the approach I have in this case is to, prior starting doing anything
    with HBase, stop everything, clean the logs, format namenode, and start
    Hadoop only. Monitor logs, what won't be problem in this case as that's
    stand alone installation, to see if HDFS get up OK. You can also try coping
    file from and to your HDFS to be sure HDFS is working fine. Then you can
    start MapReduce (JobTracker and TaskTrackers) and later on HBase.

    I hope this will help :)

    Regards,
    Dejo

    On Thu, Sep 29, 2011 at 8:06 PM, Vinod Gupta Tankala
    wrote:
    Thanks Dejo for pointing that out. I realized that earlier and fixed it.
    But
    I still hit the same problem.

    In my case, I only have a single host for now. But I am still trying to do
    a
    distributed setup by listing the machine itself as a slave in config and
    not
    using localhost anywhere. Does this even work? if not, I can try spending
    more time on pseudo-distributed setup for now.

    thanks


    On Thu, Sep 29, 2011 at 4:48 AM, Dejan Menges <dejan.menges@gmail.com
    wrote:
    In core-site.xml, on first, you miss port in the end for HDFS:

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    Regards,
    Dejo

    On Wed, Sep 28, 2011 at 6:21 PM, Vinod Gupta Tankala
    wrote:
    Hi,
    I am trying to setup a test system to host a distributed hbase
    installation.
    No matter what I do, I get the below errors.

    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery fo
    r block null bad datanode[0] nodes == null
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.hdfs.DFSClient: Could
    not
    get
    blo
    ck locations. Source file "/tmp/mapred/system/jobtracker.info" -
    Aborting...
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
    Writing
    to
    fil
    e hdfs://
    ec2-184-73-22-146.compute-1.amazonaws.com/tmp/mapred/system/jobtracker
    .
    info failed!
    2011-09-28 22:17:26,288 WARN org.apache.hadoop.mapred.JobTracker:
    FileSystem
    is
    not ready yet!
    2011-09-28 22:17:26,292 WARN org.apache.hadoop.mapred.JobTracker:
    Failed
    to
    init
    ialize recovery manager.
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /tmp/mapred/sys
    tem/jobtracker.info could only be replicated to 0 nodes, instead of
    1
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc
    k(FSNamesystem.java:1417)
    ....

    this is how i setup my config -
    core-site.xml -
    <configuration>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://ec2-184-73-22-146.compute-1.amazonaws.com/</value>
    </property>

    </configuration>

    hdfs-site.xml -
    <configuration>

    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>

    <property>
    <name>dfs.name.dir</name>
    <value>/tmp/hbase</value>
    </property>

    <property>
    <name>dfs.data.dir</name>
    <value>/tmp/hbase</value>
    </property>

    </configuration>


    mapred-site.xml -
    <configuration>

    <property>
    <name>mapred.job.tracker</name>
    <value>ec2-184-73-22-146.compute-1.amazonaws.com:9001</value>
    </property>

    <property>
    <name>mapred.local.dir</name>
    <value>/tmp/mapred_tmp</value>
    </property>

    <property>
    <name>mapred.map.tasks</name>
    <value>10</value>
    </property>

    <property>
    <name>mapred.reduce.tasks</name>
    <value>2</value>
    </property>

    <property>
    <name>mapred.system.dir</name>
    <value>/tmp/mapred/system/</value>
    </property>


    </configuration>

    i know that i am missing something really basic but not sure what it
    is.
    the
    documentation says mapred.system.dir should be globally accessible.
    how
    do
    i
    achieve that?

    thanks
    vinod

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedSep 28, '11 at 10:22p
activeSep 29, '11 at 10:26p
posts5
users2
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase