FAQ
I am trying to import data from a flat file into Hbase using a Map Reduce
job. There are close to 2 million rows. Mid way into the job, it starts
giving me connection problems and eventually kills the job. When the error
comes, the hbase shell also stops working.

This is what I get:

2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
Retrying connect to server: /171.69.102.52:60020. Already tried 0
time(s).

What could be going wrong?

Amandeep


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz

Search Discussions

  • Amandeep Khurana at Feb 21, 2009 at 5:47 am
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana wrote:

    I am trying to import data from a flat file into Hbase using a Map Reduce
    job. There are close to 2 million rows. Mid way into the job, it starts
    giving me connection problems and eventually kills the job. When the error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /171.69.102.52:60020. Already tried 0 time(s).

    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 5:56 am
    I dont know if this is related or not, but it seems to be. After this map
    reduce job, I tried to count the number of entries in the table in hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana wrote:

    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana wrote:

    I am trying to import data from a flat file into Hbase using a Map Reduce
    job. There are close to 2 million rows. Mid way into the job, it starts
    giving me connection problems and eventually kills the job. When the error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /171.69.102.52:60020. Already tried 0 time(s).

    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Stack at Feb 21, 2009 at 8:10 am
    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat to
    insert into your table?

    Which version of hbase?

    St.Ack
    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana wrote:

    I dont know if this is related or not, but it seems to be. After this map
    reduce job, I tried to count the number of entries in the table in hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana wrote:

    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it starts
    giving me connection problems and eventually kills the job. When the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass: Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 8:13 am
    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat to
    insert into your table?

    Which version of hbase?

    St.Ack
    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana wrote:

    I dont know if this is related or not, but it seems to be. After this map
    reduce job, I tried to count the number of entries in the table in hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 9:02 am
    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);

    Interestingly, the hbase shell isnt working now either. Its giving errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat to
    insert into your table?

    Which version of hbase?

    St.Ack
    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana wrote:

    I dont know if this is related or not, but it seems to be. After this map
    reduce job, I tried to count the number of entries in the table in hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 9:13 am
    I just did a "count 'in_table'", it did work. So, I truncated the table and
    started the map reduce job again to load the entire file. For some reason, I
    had to kill the job. After that, the count command gives the following
    error:


    count 'in_table'

    NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
    Trying to contact region server 171.69.102.51:60020 for region
    in_table,,1235194191883, row '', but failed after 5 attempts.
    Exceptions:
    org.apache.hadoop.hbase.NotServingRegionException:
    org.apache.hadoop.hbase.NotServingRegionException: in_table,,1235194191883
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)

    org.apache.hadoop.hbase.NotServingRegionException:
    org.apache.hadoop.hbase.NotServingRegionException: in_table,,1235194191883
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)


    It seems like the daemon is getting corrupt with the Map Red job somehow...
    I cant really understand whats happening.

    Amandeep

    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana wrote:

    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(
    conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);


    Interestingly, the hbase shell isnt working now either. Its giving errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat to
    insert into your table?

    Which version of hbase?

    St.Ack

    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    I dont know if this is related or not, but it seems to be. After this map
    reduce job, I tried to count the number of entries in the table in hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from

    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Stack at Feb 21, 2009 at 9:14 am
    It looks like regionserver hosting root crashed:

    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region

    How many servers you running?

    You made similar config. to that reported by Larry Compton in a mail from
    earlier today? (See FAQ and Troubleshooting page for more on his listed
    configs.)

    St.Ack

    On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana wrote:

    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);

    Interestingly, the hbase shell isnt working now either. Its giving errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat to
    insert into your table?

    Which version of hbase?

    St.Ack

    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    I dont know if this is related or not, but it seems to be. After this
    map
    reduce job, I tried to count the number of entries in the table in
    hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in
    `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in
    `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 9:24 am
    I have 1 master + 2 slaves.
    Am using 0.19.0 for both Hadoop and Hbase.
    I didnt change any config from the default except the hbase.rootdir and the
    hbase.master.

    I have gone through the FAQs but couldnt find anything. What exactly are you
    pointing to?


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 1:14 AM, stack wrote:

    It looks like regionserver hosting root crashed:

    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region

    How many servers you running?

    You made similar config. to that reported by Larry Compton in a mail from
    earlier today? (See FAQ and Troubleshooting page for more on his listed
    configs.)

    St.Ack

    On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana wrote:

    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);

    Interestingly, the hbase shell isnt working now either. Its giving errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat
    to
    insert into your table?

    Which version of hbase?

    St.Ack

    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    I dont know if this is related or not, but it seems to be. After this
    map
    reduce job, I tried to count the number of entries in the table in
    hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in
    `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in
    `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at
    org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
    out
    trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <
    amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When
    the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 9:45 am
    Here's another thing thats happening. I was trying to truncate the table.

    hbase(main):001:0> truncate 'in_table'
    Truncating in_table; it may take a while
    Disabling table...
    NativeException: org.apache.hadoop.hbase.RegionException: Retries exhausted,
    it took too long to wait for the table in_table to be disabled.
    from org/apache/hadoop/hbase/client/HBaseAdmin.java:387:in
    `disableTable'
    from org/apache/hadoop/hbase/client/HBaseAdmin.java:348:in
    `disableTable'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    from org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
    `call'
    from org/jruby/internal/runtime/methods/DynamicMethod.java:78:in `call'
    from org/jruby/runtime/CallSite.java:155:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:332:in `call'
    from org/jruby/evaluator/ASTInterpreter.java:649:in `callNode'
    from org/jruby/evaluator/ASTInterpreter.java:324:in `evalInternal'

    I left it for a few minutes and tried again. It worked. There was no load on
    the cluster at all. changed the config (both) and added
    dfs.datanode.socket.write.timeout property with value 0. I also defined the
    property in the job config.

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 1:23 AM, Amandeep Khurana wrote:

    I have 1 master + 2 slaves.
    Am using 0.19.0 for both Hadoop and Hbase.
    I didnt change any config from the default except the hbase.rootdir and the
    hbase.master.

    I have gone through the FAQs but couldnt find anything. What exactly are
    you pointing to?


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 1:14 AM, stack wrote:

    It looks like regionserver hosting root crashed:

    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
    trying
    to locate root region

    How many servers you running?

    You made similar config. to that reported by Larry Compton in a mail from
    earlier today? (See FAQ and Troubleshooting page for more on his listed
    configs.)

    St.Ack


    On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);

    Interestingly, the hbase shell isnt working now either. Its giving errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat
    to
    insert into your table?

    Which version of hbase?

    St.Ack

    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    I dont know if this is related or not, but it seems to be. After
    this
    map
    reduce job, I tried to count the number of entries in the table in
    hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from
    org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in
    `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in
    `metaScan'
    from
    org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at
    org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
    out
    trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <
    amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a
    Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When
    the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 10:15 am
    Yes, I noticed it this time. The regionserver gets slow or stops responding
    and then this error comes. How do I get this to work? Is there a way of
    limiting the resources that the map red job should take?

    I did make the changes in the config site similar to Larry Comton's config.
    It only made the job go from dying at 7% to 12% this time.

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 1:14 AM, stack wrote:

    It looks like regionserver hosting root crashed:

    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
    to locate root region

    How many servers you running?

    You made similar config. to that reported by Larry Compton in a mail from
    earlier today? (See FAQ and Troubleshooting page for more on his listed
    configs.)

    St.Ack

    On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana wrote:

    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code from the docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);

    Interestingly, the hbase shell isnt working now either. Its giving errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 12:10 AM, stack wrote:

    The table exists before you start the MR job?

    When you say 'midway through the job', are you using tableoutputformat
    to
    insert into your table?

    Which version of hbase?

    St.Ack

    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    I dont know if this is related or not, but it seems to be. After this
    map
    reduce job, I tried to count the number of entries in the table in
    hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in
    `metaScan'
    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in
    `metaScan'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in `execute'
    ... 145 levels...
    from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
    from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at
    org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child.main(Child.java:155)

    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
    out
    trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <
    amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase using a Map
    Reduce
    job. There are close to 2 million rows. Mid way into the job, it
    starts
    giving me connection problems and eventually kills the job. When
    the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0 time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz
  • Amandeep Khurana at Feb 21, 2009 at 6:56 pm
    Here's whats happening in the logs...

    I get these messages pretty often:
    2009-02-21 10:47:27,252 INFO org.apache.hadoop.hdfs.DFSClient: Could not
    complete file
    /hbase/in_table/compaction.dir/29712919/b2b/mapfiles/6353513045069085254/data
    retrying...


    Sometimes I get these too:
    2009-02-21 10:48:46,273 INFO org.apache.hadoop.hbase.regionserver.HRegion:
    Blocking updates for 'IPC Server handler 5 on 60020' on region
    in_table,,1235241411727: Memcache size 128.0m is >= than blocking 128.0m
    size

    Here's what it logs when the job starts to fail:
    2009-02-21 10:50:52,510 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
    Exception: java.net.SocketTimeoutException: 5000 millis timeout while
    waiting for channel to be ready for write. ch :
    java.nio.channels.SocketChannel[connected local=/171.69.102.51:8270 remote=/
    171.69.102.51:50010]
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
    at java.io.BufferedOutputStream.write(Unknown Source)
    at java.io.DataOutputStream.write(Unknown Source)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)

    2009-02-21 10:50:52,511 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
    Finished memcache flush of ~64.0m for region in_table,,1235241411727 in
    5144ms, sequence id=30842181, compaction requested=true
    2009-02-21 10:50:52,511 DEBUG
    org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
    requested for region in_table,,1235241411727/29712919 because:
    regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher
    2009-02-21 10:50:52,512 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block blk_-2896903198415069285_18306 bad datanode[0]
    171.69.102.51:50010
    2009-02-21 10:50:52,513 FATAL
    org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed with ioe:
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,513 FATAL org.apache.hadoop.hbase.regionserver.HLog:
    Could not append. Requesting close of log
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,515 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All
    datanodes 171.69.102.51:50010 are bad. Aborting...
    2009-02-21 10:50:52,515 FATAL org.apache.hadoop.hbase.regionserver.HLog:
    Could not append. Requesting close of log
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,515 INFO
    org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
    request=11, regions=2, stores=33, storefiles=167, storefileIndexSize=0,
    memcacheSize=1, usedHeap=156, maxHeap=963
    2009-02-21 10:50:52,515 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
    LogRoller exiting.
    2009-02-21 10:50:52,516 FATAL org.apache.hadoop.hbase.regionserver.HLog:
    Could not append. Requesting close of log
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,516 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All
    datanodes 171.69.102.51:50010 are bad. Aborting...
    2009-02-21 10:50:52,516 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All
    datanodes 171.69.102.51:50010 are bad. Aborting...
    2009-02-21 10:50:52,516 FATAL org.apache.hadoop.hbase.regionserver.HLog:
    Could not append. Requesting close of log
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,516 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All
    datanodes 171.69.102.51:50010 are bad. Aborting...
    2009-02-21 10:50:52,517 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 9 on 60020, call batchUpdates([B@cfdbc2,
    [Lorg.apache.hadoop.hbase.io.BatchUpdate;@64c0d9) from 171.69.102.51:8468:
    error: java.io.IOException: All datanodes 171.69.102.51:50010 are bad.
    Aborting...
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,517 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 1 on 60020, call batchUpdates([B@b0fc4d,
    [Lorg.apache.hadoop.hbase.io.BatchUpdate;@184425c) from 171.69.102.51:8469:
    error: java.io.IOException: All datanodes 171.69.102.51:50010 are bad.
    Aborting...
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,518 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 0 on 60020, call batchUpdates([B@20a9de,
    [Lorg.apache.hadoop.hbase.io.BatchUpdate;@706d7c) from 171.69.102.52:9279:
    error: java.io.IOException: All datanodes 171.69.102.51:50010 are bad.
    Aborting...
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:52,518 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 5 on 60020, call batchUpdates([B@1240afe,
    [Lorg.apache.hadoop.hbase.io.BatchUpdate;@14dc2e6) from 171.69.102.52:9280:
    error: java.io.IOException: All datanodes 171.69.102.51:50010 are bad.
    Aborting...
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:53,049 DEBUG org.apache.hadoop.hbase.RegionHistorian:
    Offlined
    2009-02-21 10:50:53,050 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
    server on 60020
    2009-02-21 10:50:53,050 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 0 on 60020: exiting
    2009-02-21 10:50:53,050 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
    Server listener on 60020
    2009-02-21 10:50:53,051 INFO
    org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
    2009-02-21 10:50:53,051 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
    Server Responder
    2009-02-21 10:50:53,052 INFO org.mortbay.util.ThreadedServer: Stopping
    Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
    2009-02-21 10:50:53,052 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 1 on 60020: exiting
    2009-02-21 10:50:53,052 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 3 on 60020: exiting
    2009-02-21 10:50:53,052 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 5 on 60020: exiting
    2009-02-21 10:50:53,052 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 7 on 60020: exiting
    2009-02-21 10:50:53,052 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 9 on 60020: exiting
    2009-02-21 10:50:53,052 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 2 on 60020: exiting
    2009-02-21 10:50:53,053 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 4 on 60020: exiting
    2009-02-21 10:50:53,053 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 6 on 60020: exiting
    2009-02-21 10:50:53,053 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
    handler 8 on 60020: exiting
    2009-02-21 10:50:54,503 INFO org.mortbay.http.SocketListener: Stopped
    SocketListener on 0.0.0.0:60030
    2009-02-21 10:50:54,670 INFO org.mortbay.util.Container: Stopped
    HttpContext[/logs,/logs]
    2009-02-21 10:50:54,671 INFO org.mortbay.util.Container: Stopped
    org.mortbay.jetty.servlet.WebApplicationHandler@c3c315
    2009-02-21 10:50:54,771 INFO org.mortbay.util.Container: Stopped
    WebApplicationContext[/static,/static]
    2009-02-21 10:50:54,772 INFO org.mortbay.util.Container: Stopped
    org.mortbay.jetty.servlet.WebApplicationHandler@aae86e
    2009-02-21 10:50:54,893 INFO org.mortbay.util.Container: Stopped
    WebApplicationContext[/,/]
    2009-02-21 10:50:54,893 INFO org.mortbay.util.Container: Stopped
    org.mortbay.jetty.Server@1f3ce5c
    2009-02-21 10:50:54,893 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
    closing log writer in
    hdfs://rndpc0:9000/hbase/log_171.69.102.51_1235215460389_60020
    2009-02-21 10:50:54,893 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
    abort
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
    2009-02-21 10:50:54,893 DEBUG
    org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
    in_table,,1235241411727
    2009-02-21 10:50:54,893 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
    Closing in_table,,1235241411727: compactions & flushes disabled
    2009-02-21 10:50:54,893 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
    waiting for compaction to complete for region in_table,,1235241411727
    2009-02-21 10:50:54,893 INFO
    org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
    regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
    2009-02-21 10:50:54,894 INFO
    org.apache.hadoop.hbase.regionserver.LogFlusher:
    regionserver/0:0:0:0:0:0:0:0:60020.logFlusher exiting
    2009-02-21 10:50:54,894 INFO
    org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
    regionserver/0:0:0:0:0:0:0:0:60020.majorCompactionChecker exiting
    2009-02-21 10:50:57,251 INFO org.apache.hadoop.hbase.Leases:
    regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closing leases2009-02-21
    10:50:57,251 INFO org.apache.hadoop.hbase.Leases:
    regionserver/0:0:0:0:0:0:0:0:60020.leaseChecker closed leases
    2009-02-21 10:51:01,199 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
    moving
    /hbase/in_table/compaction.dir/29712919/tac_product_hw_key/mapfiles/6810647399799866363
    to /hbase/in_table/29712919/tac_product_hw_key/mapfiles/4524637696729699312
    2009-02-21 10:51:01,943 INFO
    org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
    2009-02-21 10:51:02,587 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
    Completed compaction of 29712919/tac_product_hw_key store size is 17.0m
    2009-02-21 10:51:02,828 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
    Compaction size of 29712919/summary: 23.9m; Skipped 1 file(s), size:
    10737371
    2009-02-21 10:51:03,896 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
    Started compaction of 7 file(s) into
    /hbase/in_table/compaction.dir/29712919/summary/mapfiles/3985892544092790173

    and then followed by:

    2009-02-21 10:50:54,503 INFO org.mortbay.http.SocketListener: Stopped
    SocketListener on 0.0.0.0:60030
    2009-02-21 10:50:54,670 INFO org.mortbay.util.Container: Stopped
    HttpContext[/logs,/logs]
    2009-02-21 10:50:54,671 INFO org.mortbay.util.Container: Stopped
    org.mortbay.jetty.servlet.WebApplicationHandler@c3c315
    2009-02-21 10:50:54,771 INFO org.mortbay.util.Container: Stopped
    WebApplicationContext[/static,/static]
    2009-02-21 10:50:54,772 INFO org.mortbay.util.Container: Stopped
    org.mortbay.jetty.servlet.WebApplicationHandler@aae86e
    2009-02-21 10:50:54,893 INFO org.mortbay.util.Container: Stopped
    WebApplicationContext[/,/]
    2009-02-21 10:50:54,893 INFO org.mortbay.util.Container: Stopped
    org.mortbay.jetty.Server@1f3ce5c
    2009-02-21 10:50:54,893 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
    closing log writer in
    hdfs://rndpc0:9000/hbase/log_171.69.102.51_1235215460389_60020
    2009-02-21 10:50:54,893 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
    abort
    java.io.IOException: All datanodes 171.69.102.51:50010 are bad. Aborting...
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997)
    at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)






    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 3:17 AM, Amandeep Khurana wrote:

    I changed the config and restarted the cluster. This time the job went upto
    16% and the same problem started.

    I'll do the stuff with the logs now and see what comes out.


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 3:08 AM, Ryan Rawson wrote:

    you have to change hadoop-site.xml and restart HDFS.

    you should also change the logging to be more verbose in hbase - check out
    the hbase FAQ (link missing -ed).

    if you get the problem again, peruse the hbase logs and post what is going
    on there. the client errors dont really include the root cause on the
    regionserver side.

    good luck,
    -ryan


    On Sat, Feb 21, 2009 at 2:21 AM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    I have 1 master + 2 slaves. I did set the timout to zero. I'll set the
    xceivers to 2047 and try again. Can this be done in the job config or does
    the site.xml need to be changed and the cluster restarted?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Sat, Feb 21, 2009 at 2:16 AM, Ryan Rawson <ryanobjc@gmail.com>
    wrote:
    So the usual suspects are:

    - xcievers (i hvae mine set to 2047)
    - timeout (i have mine set to 0)

    I can import a few hundred million records with these settings.

    how many nodes do you have again?

    On Sat, Feb 21, 2009 at 2:14 AM, Amandeep Khurana <amansk@gmail.com>
    wrote:
    Yes, I noticed it this time. The regionserver gets slow or stops
    responding
    and then this error comes. How do I get this to work? Is there a way
    of
    limiting the resources that the map red job should take?

    I did make the changes in the config site similar to Larry Comton's config.
    It only made the job go from dying at 7% to 12% this time.

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

    On Sat, Feb 21, 2009 at 1:14 AM, stack wrote:

    It looks like regionserver hosting root crashed:

    org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
    out
    trying
    to locate root region

    How many servers you running?

    You made similar config. to that reported by Larry Compton in a
    mail
    from
    earlier today? (See FAQ and Troubleshooting page for more on his
    listed
    configs.)

    St.Ack


    On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana <
    amansk@gmail.com>
    wrote:
    Yes, the table exists before I start the job.

    I am not using TableOutputFormat. I picked up the sample code
    from
    the
    docs
    and am using it.

    Here's the job conf:

    JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
    FileInputFormat.setInputPaths(conf, new
    Path("import_data"));
    conf.setMapperClass(MapClass.class);
    conf.setNumReduceTasks(0);
    conf.setOutputFormat(NullOutputFormat.class);
    JobClient.runJob(conf);

    Interestingly, the hbase shell isnt working now either. Its
    giving
    errors
    even when I give the command "list"...



    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Sat, Feb 21, 2009 at 12:10 AM, stack <stack@duboce.net>
    wrote:
    The table exists before you start the MR job?

    When you say 'midway through the job', are you using
    tableoutputformat
    to
    insert into your table?

    Which version of hbase?

    St.Ack

    On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <
    amansk@gmail.com>
    wrote:
    I dont know if this is related or not, but it seems to be.
    After
    this
    map
    reduce job, I tried to count the number of entries in the
    table
    in
    hbase
    through the shell. It failed with the following error:

    hbase(main):002:0> count 'in_table'
    NativeException: java.lang.NullPointerException: null
    from java.lang.String:-1:in `<init>'
    from org/apache/hadoop/hbase/util/Bytes.java:92:in
    `toString'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
    `getMessage'
    from
    org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
    `<init>'
    from
    org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
    `getRegionServerWithRetries'
    from
    org/apache/hadoop/hbase/client/MetaScanner.java:56:in
    `metaScan'
    from
    org/apache/hadoop/hbase/client/MetaScanner.java:30:in
    `metaScan'
    from
    org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
    `getHTableDescriptor'
    from org/apache/hadoop/hbase/client/HTable.java:219:in
    `getTableDescriptor'
    from sun.reflect.NativeMethodAccessorImpl:-2:in `invoke0'
    from sun.reflect.NativeMethodAccessorImpl:-1:in `invoke'
    from sun.reflect.DelegatingMethodAccessorImpl:-1:in
    `invoke'
    from java.lang.reflect.Method:-1:in `invoke'
    from org/jruby/javasupport/JavaMethod.java:250:in
    `invokeWithExceptionHandling'
    from org/jruby/javasupport/JavaMethod.java:219:in
    `invoke'
    from org/jruby/javasupport/JavaClass.java:416:in
    `execute'
    ... 145 levels...
    from
    org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
    `call'
    from
    org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
    `call'
    from org/jruby/runtime/CallSite.java:123:in
    `cacheAndCall'
    from org/jruby/runtime/CallSite.java:298:in `call'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `__file__'
    from
    ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
    `load'
    from org/jruby/Ruby.java:512:in `runScript'
    from org/jruby/Ruby.java:432:in `runNormally'
    from org/jruby/Ruby.java:312:in `runFromMain'
    from org/jruby/Main.java:144:in `run'
    from org/jruby/Main.java:89:in `run'
    from org/jruby/Main.java:80:in `main'
    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in
    `count'
    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in
    `count'
    from (hbase):3:in `binding'


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana <
    amansk@gmail.com>
    wrote:
    Here's what it throws on the console:

    09/02/20 21:45:29 INFO mapred.JobClient: Task Id :
    attempt_200902201300_0019_m_000006_0, Status : FAILED
    java.io.IOException: table is null
    at
    IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
    at
    IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
    at
    org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at
    org.apache.hadoop.mapred.Child.main(Child.java:155)
    attempt_200902201300_0019_m_000006_0:
    org.apache.hadoop.hbase.client.NoServerForRegionException:
    Timed
    out
    trying
    to locate root region
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
    attempt_200902201300_0019_m_000006_0: at
    IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
    attempt_200902201300_0019_m_000006_0: at
    org.apache.hadoop.mapred.Child.main(Child.java:155)





    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz


    On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana <
    amansk@gmail.com
    wrote:
    I am trying to import data from a flat file into Hbase
    using
    a
    Map
    Reduce
    job. There are close to 2 million rows. Mid way into the
    job,
    it
    starts
    giving me connection problems and eventually kills the
    job.
    When
    the
    error
    comes, the hbase shell also stops working.

    This is what I get:

    2009-02-20 21:37:14,407 INFO
    org.apache.hadoop.ipc.HBaseClass:
    Retrying
    connect to server: /171.69.102.52:60020. Already tried 0
    time(s).
    What could be going wrong?

    Amandeep


    Amandeep Khurana
    Computer Science Graduate Student
    University of California, Santa Cruz

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 21, '09 at 5:44a
activeFeb 21, '09 at 6:56p
posts12
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Amandeep Khurana: 10 posts Stack: 2 posts

People

Translate

site design / logo © 2022 Grokbase