FAQ
Hey All,

Running into a problem storing data from a pig script storing results into
HBase.

We are getting the following error:

java.lang.NullPointerException
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.mapred.Child.main(Child.java:211)


We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
followed the instructions to get pig 0.8.0 to work with CDH3 from
thedatachef(Thanks!)
http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html.

The relevant line from the pig script is below. We've applied the
patch to get the "-noWAL" working:

STORE links INTO 'p' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
a:g','-noWAL');



Anyone know what could be causing this problem?


Thanks in advance,


Matt

Search Discussions

  • Dmitriy Ryaboy at Feb 14, 2011 at 10:02 pm
    I have a fix for that, just discovered it last night myself. The patch for
    0.89 doesn't work on storage (it only works for loading). Will update ticket
    later tonight once i get some ivy uglyness out of the way.

    D
    On Mon, Feb 14, 2011 at 1:57 PM, Matt Davies wrote:

    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)

    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .

    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Matt Davies at Feb 14, 2011 at 10:04 pm
    Thank you! We are looking forward to it.
    On Mon, Feb 14, 2011 at 3:01 PM, Dmitriy Ryaboy wrote:

    I have a fix for that, just discovered it last night myself. The patch for
    0.89 doesn't work on storage (it only works for loading). Will update
    ticket
    later tonight once i get some ivy uglyness out of the way.

    D
    On Mon, Feb 14, 2011 at 1:57 PM, Matt Davies wrote:

    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)

    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .

    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Jacob at Feb 14, 2011 at 10:19 pm
    Also,

    Since I encountered all of these issues when loading and storing and
    needed to 'get-shit-done', here's the modified version of hbase storage:

    https://github.com/infochimps/HbaseBulkloader

    that might help for now (it's not a long term solution). In it you'll
    find it can read full column families into pig bags
    (eg:
    "USING com.infochimps.hbase.HBaseStorage('my_fol_fam:')"
    )
    as well as the normal read and write functionality of HBaseStorage from
    the pig trunk but with the nulls taken care of. Writing to the WAL is
    turned off and hardcoded.

    Lastly, there's an additional class (for writing to hbase only) called
    'DynamicFamilyStorage' that gets the column family and column name from
    the records themselves that you might find useful.

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 15:03 -0700, Matt Davies wrote:
    Thank you! We are looking forward to it.
    On Mon, Feb 14, 2011 at 3:01 PM, Dmitriy Ryaboy wrote:

    I have a fix for that, just discovered it last night myself. The patch for
    0.89 doesn't work on storage (it only works for loading). Will update
    ticket
    later tonight once i get some ivy uglyness out of the way.

    D
    On Mon, Feb 14, 2011 at 1:57 PM, Matt Davies wrote:

    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)

    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .

    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Jacob at Feb 14, 2011 at 10:07 pm
    Thanks for the mention. Off the top of my head I got these sorts of
    errors when trying to store either null records or records with null
    fields. What happens if you FILTER out any null values you might have.
    Does the problem persist?

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote:
    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)
    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html.

    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Matt Davies at Feb 14, 2011 at 11:15 pm
    You are welcome! We did a filter before the store, and I confirmed that
    there weren't any nulls. Just something undocumented feature ;)
    On Mon, Feb 14, 2011 at 3:03 PM, jacob wrote:

    Thanks for the mention. Off the top of my head I got these sorts of
    errors when trying to store either null records or records with null
    fields. What happens if you FILTER out any null values you might have.
    Does the problem persist?

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote:
    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)
    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .
    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Matt Davies at Feb 15, 2011 at 10:46 pm
    Dmitriy,

    I saw the patch posted to PIG-1680 from last night. Thanks! We have been
    trying to apply it, and find that the patch can't match some of the hunks.
    Are you patching against trunk or a certain version?


    Thanks!
    -Matt
    On Mon, Feb 14, 2011 at 4:14 PM, Matt Davies wrote:

    You are welcome! We did a filter before the store, and I confirmed that
    there weren't any nulls. Just something undocumented feature ;)

    On Mon, Feb 14, 2011 at 3:03 PM, jacob wrote:

    Thanks for the mention. Off the top of my head I got these sorts of
    errors when trying to store either null records or records with null
    fields. What happens if you FILTER out any null values you might have.
    Does the problem persist?

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote:
    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)
    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .
    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Dmitriy Ryaboy at Feb 15, 2011 at 10:57 pm
    Thought it was against the trunk.. I did make it ignore whitespace changes,
    maybe there's a problem there. I'll email you another patch off-list, let's
    see if we can get it to work.

    D
    On Tue, Feb 15, 2011 at 2:45 PM, Matt Davies wrote:

    Dmitriy,

    I saw the patch posted to PIG-1680 from last night. Thanks! We have been
    trying to apply it, and find that the patch can't match some of the hunks.
    Are you patching against trunk or a certain version?


    Thanks!
    -Matt
    On Mon, Feb 14, 2011 at 4:14 PM, Matt Davies wrote:

    You are welcome! We did a filter before the store, and I confirmed that
    there weren't any nulls. Just something undocumented feature ;)

    On Mon, Feb 14, 2011 at 3:03 PM, jacob wrote:

    Thanks for the mention. Off the top of my head I got these sorts of
    errors when trying to store either null records or records with null
    fields. What happens if you FILTER out any null values you might have.
    Does the problem persist?

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote:
    Hey All,

    Running into a problem storing data from a pig script storing results into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at
    org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)
    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .
    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Matt Davies at Feb 15, 2011 at 11:01 pm
    Dmitriy,

    Appreciate the help. We have tried against trunk and here's the output.



    patching file src/org/apache/pig/backend/hadoop/hbase/HBaseTableInputFormat.java
    patching file src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java
    Hunk #6 FAILED at 80.
    Hunk #7 succeeded at 115 (offset 1 line).
    Hunk #9 succeeded at 167 (offset 1 line).
    Hunk #11 succeeded at 253 (offset 1 line).
    Hunk #13 succeeded at 288 (offset 1 line).
    Hunk #15 succeeded at 324 (offset 1 line).
    Hunk #17 succeeded at 377 (offset 1 line).
    Hunk #19 succeeded at 417 (offset 1 line).
    Hunk #21 succeeded at 451 (offset 1 line).
    1 out of 21 hunks FAILED -- saving rejects to file
    src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java.rej
    patching file test/hbase-site.xml
    patching file test/org/apache/pig/test/TestHBaseStorage.java
    Hunk #3 FAILED at 42.
    1 out of 3 hunks FAILED -- saving rejects to file
    test/org/apache/pig/test/TestHBaseStorage.java.rej
    patching file build.xml
    Hunk #1 succeeded at 51 (offset 1 line).
    Hunk #2 FAILED at 199.
    Hunk #3 succeeded at 587 (offset -83 lines).
    1 out of 3 hunks FAILED -- saving rejects to file build.xml.rej
    patching file ivy.xml
    Hunk #1 succeeded at 84 (offset -5 lines).
    patching file ivy/libraries.properties
    Hunk #1 succeeded at 28 with fuzz 2 (offset -1 lines).
    patching file src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
    Hunk #1 succeeded at 111 (offset 1 line).
    Hunk #3 succeeded at 139 (offset 1 line).
    Hunk #4 succeeded at 175 with fuzz 1 (offset -3 lines).
    Hunk #5 succeeded at 201 (offset 1 line).
    Hunk #6 succeeded at 229 (offset -3 lines).
    Hunk #7 succeeded at 270 (offset 1 line).
    Hunk #8 succeeded at 276 (offset -3 lines).
    Hunk #9 succeeded at 298 (offset 1 line).
    Hunk #10 succeeded at 320 (offset -3 lines).
    Hunk #11 FAILED at 385.
    Hunk #12 succeeded at 394 with fuzz 1 (offset -2 lines).
    Hunk #13 succeeded at 402 (offset -4 lines).
    Hunk #14 succeeded at 418 (offset -2 lines).
    Hunk #15 succeeded at 430 (offset -4 lines).
    Hunk #16 succeeded at 457 (offset -2 lines).
    Hunk #17 FAILED at 484.
    Hunk #18 succeeded at 484 (offset -13 lines).
    Hunk #19 succeeded at 514 (offset -2 lines).
    Hunk #20 succeeded at 539 (offset -19 lines).
    Hunk #21 succeeded at 564 (offset -2 lines).
    Hunk #22 succeeded at 566 with fuzz 2 (offset -22 lines).
    Hunk #23 succeeded at 605 (offset -2 lines).
    Hunk #24 succeeded at 595 (offset -22 lines).
    Hunk #25 succeeded at 635 (offset -2 lines).
    Hunk #26 FAILED at 651.
    Hunk #27 FAILED at 680.
    Hunk #28 succeeded at 673 (offset -24 lines).
    Hunk #29 succeeded at 705 (offset -2 lines).
    Hunk #30 succeeded at 694 with fuzz 1 (offset -24 lines).
    Hunk #31 succeeded at 746 (offset -2 lines).
    Hunk #32 succeeded at 740 (offset -24 lines).
    Hunk #33 succeeded at 775 (offset -2 lines).
    Hunk #34 succeeded at 771 (offset -24 lines).
    Hunk #35 succeeded at 806 (offset -2 lines).
    Hunk #36 succeeded at 806 (offset -24 lines).
    Hunk #37 succeeded at 892 (offset -2 lines).
    Hunk #38 succeeded at 914 (offset -24 lines).
    Hunk #39 succeeded at 948 (offset -2 lines).
    Hunk #40 succeeded at 936 (offset -24 lines).
    Hunk #41 succeeded at 974 (offset -2 lines).
    Hunk #42 FAILED at 1072.
    Hunk #43 succeeded at 1093 (offset -31 lines).
    Hunk #44 succeeded at 1158 (offset -2 lines).
    Hunk #45 succeeded at 1141 (offset -31 lines).
    Hunk #46 succeeded at 1221 (offset -2 lines).
    Hunk #47 succeeded at 1212 (offset -31 lines).
    Hunk #48 succeeded at 1264 (offset -2 lines).
    Hunk #49 FAILED at 1290.
    Hunk #50 FAILED at 1302.
    Hunk #51 FAILED at 1322.




    8 out of 51 hunks FAILED -- saving rejects to file
    src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java.rej


    On Tue, Feb 15, 2011 at 3:56 PM, Dmitriy Ryaboy wrote:

    Thought it was against the trunk.. I did make it ignore whitespace changes,
    maybe there's a problem there. I'll email you another patch off-list, let's
    see if we can get it to work.

    D
    On Tue, Feb 15, 2011 at 2:45 PM, Matt Davies wrote:

    Dmitriy,

    I saw the patch posted to PIG-1680 from last night. Thanks! We have been
    trying to apply it, and find that the patch can't match some of the hunks.
    Are you patching against trunk or a certain version?


    Thanks!
    -Matt
    On Mon, Feb 14, 2011 at 4:14 PM, Matt Davies wrote:

    You are welcome! We did a filter before the store, and I confirmed that
    there weren't any nulls. Just something undocumented feature ;)


    On Mon, Feb 14, 2011 at 3:03 PM, jacob <jacob.a.perkins@gmail.com>
    wrote:
    Thanks for the mention. Off the top of my head I got these sorts of
    errors when trying to store either null records or records with null
    fields. What happens if you FILTER out any null values you might have.
    Does the problem persist?

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote:
    Hey All,

    Running into a problem storing data from a pig script storing
    results
    into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at
    org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)
    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .
    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt
  • Dmitriy Ryaboy at Feb 16, 2011 at 12:02 am
    Matt and I got the patch to apply and work but another bug showed up.
    I'll see about fixing it later tonight.

    D
    On Tue, Feb 15, 2011 at 3:00 PM, Matt Davies wrote:

    Dmitriy,

    Appreciate the help. We have tried against trunk and here's the output.



    patching file
    src/org/apache/pig/backend/hadoop/hbase/HBaseTableInputFormat.java
    patching file src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java
    Hunk #6 FAILED at 80.
    Hunk #7 succeeded at 115 (offset 1 line).
    Hunk #9 succeeded at 167 (offset 1 line).
    Hunk #11 succeeded at 253 (offset 1 line).
    Hunk #13 succeeded at 288 (offset 1 line).
    Hunk #15 succeeded at 324 (offset 1 line).
    Hunk #17 succeeded at 377 (offset 1 line).
    Hunk #19 succeeded at 417 (offset 1 line).
    Hunk #21 succeeded at 451 (offset 1 line).
    1 out of 21 hunks FAILED -- saving rejects to file
    src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java.rej
    patching file test/hbase-site.xml
    patching file test/org/apache/pig/test/TestHBaseStorage.java
    Hunk #3 FAILED at 42.
    1 out of 3 hunks FAILED -- saving rejects to file
    test/org/apache/pig/test/TestHBaseStorage.java.rej
    patching file build.xml
    Hunk #1 succeeded at 51 (offset 1 line).
    Hunk #2 FAILED at 199.
    Hunk #3 succeeded at 587 (offset -83 lines).
    1 out of 3 hunks FAILED -- saving rejects to file build.xml.rej
    patching file ivy.xml
    Hunk #1 succeeded at 84 (offset -5 lines).
    patching file ivy/libraries.properties
    Hunk #1 succeeded at 28 with fuzz 2 (offset -1 lines).
    patching file
    src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
    Hunk #1 succeeded at 111 (offset 1 line).
    Hunk #3 succeeded at 139 (offset 1 line).
    Hunk #4 succeeded at 175 with fuzz 1 (offset -3 lines).
    Hunk #5 succeeded at 201 (offset 1 line).
    Hunk #6 succeeded at 229 (offset -3 lines).
    Hunk #7 succeeded at 270 (offset 1 line).
    Hunk #8 succeeded at 276 (offset -3 lines).
    Hunk #9 succeeded at 298 (offset 1 line).
    Hunk #10 succeeded at 320 (offset -3 lines).
    Hunk #11 FAILED at 385.
    Hunk #12 succeeded at 394 with fuzz 1 (offset -2 lines).
    Hunk #13 succeeded at 402 (offset -4 lines).
    Hunk #14 succeeded at 418 (offset -2 lines).
    Hunk #15 succeeded at 430 (offset -4 lines).
    Hunk #16 succeeded at 457 (offset -2 lines).
    Hunk #17 FAILED at 484.
    Hunk #18 succeeded at 484 (offset -13 lines).
    Hunk #19 succeeded at 514 (offset -2 lines).
    Hunk #20 succeeded at 539 (offset -19 lines).
    Hunk #21 succeeded at 564 (offset -2 lines).
    Hunk #22 succeeded at 566 with fuzz 2 (offset -22 lines).
    Hunk #23 succeeded at 605 (offset -2 lines).
    Hunk #24 succeeded at 595 (offset -22 lines).
    Hunk #25 succeeded at 635 (offset -2 lines).
    Hunk #26 FAILED at 651.
    Hunk #27 FAILED at 680.
    Hunk #28 succeeded at 673 (offset -24 lines).
    Hunk #29 succeeded at 705 (offset -2 lines).
    Hunk #30 succeeded at 694 with fuzz 1 (offset -24 lines).
    Hunk #31 succeeded at 746 (offset -2 lines).
    Hunk #32 succeeded at 740 (offset -24 lines).
    Hunk #33 succeeded at 775 (offset -2 lines).
    Hunk #34 succeeded at 771 (offset -24 lines).
    Hunk #35 succeeded at 806 (offset -2 lines).
    Hunk #36 succeeded at 806 (offset -24 lines).
    Hunk #37 succeeded at 892 (offset -2 lines).
    Hunk #38 succeeded at 914 (offset -24 lines).
    Hunk #39 succeeded at 948 (offset -2 lines).
    Hunk #40 succeeded at 936 (offset -24 lines).
    Hunk #41 succeeded at 974 (offset -2 lines).
    Hunk #42 FAILED at 1072.
    Hunk #43 succeeded at 1093 (offset -31 lines).
    Hunk #44 succeeded at 1158 (offset -2 lines).
    Hunk #45 succeeded at 1141 (offset -31 lines).
    Hunk #46 succeeded at 1221 (offset -2 lines).
    Hunk #47 succeeded at 1212 (offset -31 lines).
    Hunk #48 succeeded at 1264 (offset -2 lines).
    Hunk #49 FAILED at 1290.
    Hunk #50 FAILED at 1302.
    Hunk #51 FAILED at 1322.




    8 out of 51 hunks FAILED -- saving rejects to file

    src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java.rej


    On Tue, Feb 15, 2011 at 3:56 PM, Dmitriy Ryaboy wrote:

    Thought it was against the trunk.. I did make it ignore whitespace changes,
    maybe there's a problem there. I'll email you another patch off-list, let's
    see if we can get it to work.

    D
    On Tue, Feb 15, 2011 at 2:45 PM, Matt Davies wrote:

    Dmitriy,

    I saw the patch posted to PIG-1680 from last night. Thanks! We have
    been
    trying to apply it, and find that the patch can't match some of the hunks.
    Are you patching against trunk or a certain version?


    Thanks!
    -Matt

    On Mon, Feb 14, 2011 at 4:14 PM, Matt Davies <matt@mattdavies.net>
    wrote:
    You are welcome! We did a filter before the store, and I confirmed
    that
    there weren't any nulls. Just something undocumented feature ;)


    On Mon, Feb 14, 2011 at 3:03 PM, jacob <jacob.a.perkins@gmail.com>
    wrote:
    Thanks for the mention. Off the top of my head I got these sorts of
    errors when trying to store either null records or records with null
    fields. What happens if you FILTER out any null values you might
    have.
    Does the problem persist?

    --jacob
    @thedatachef
    On Mon, 2011-02-14 at 14:57 -0700, Matt Davies wrote:
    Hey All,

    Running into a problem storing data from a pig script storing
    results
    into
    HBase.

    We are getting the following error:

    java.lang.NullPointerException
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:126)
    at
    org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:81)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:364)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
    at
    org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:523)
    at
    org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at
    org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native
    Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)


    We are using CDH3b3, and HBase 0.90.0 (from Apache direct). We've
    followed the instructions to get pig 0.8.0 to work with CDH3 from
    thedatachef(Thanks!)
    http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html
    .
    The relevant line from the pig script is below. We've applied the
    patch to get the "-noWAL" working:

    STORE links INTO 'p' USING
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('a:t a:t2
    a:g','-noWAL');



    Anyone know what could be causing this problem?


    Thanks in advance,


    Matt

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedFeb 14, '11 at 9:57p
activeFeb 16, '11 at 12:02a
posts10
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase