FAQ
Hi, all.

A few of our engineers have been trying to debug this issue to no
avail:

Our dataset includes dates stored as JODA LocalDate. Even the
simplest queries fail, e.g.

(??<- [?a] ([[(LocalDate.)]] ?a))

The stack trace reports a NullPointerException in
org.joda.time.LocalDate.getValue() as being the culprit, but
inspection of the joda-time-1.6.2 source showed no obvious flaws.

We're running cascalog version 1.8.6 and joda-time 1.6.2.

Any insight you might be able to offer would be most appreciated.

Thanks in advance!
-John Bates

Search Discussions

  • Thomas Jack at Mar 20, 2012 at 2:47 am
    Can you share more of the stacktrace?

    My wild guess is that the default kryo serialization doesn't work for
    LocalDate. You may need to register a different serializer or create a new
    one.
    On Mar 19, 2012 8:52 PM, "John Bates" wrote:

    Hi, all.

    A few of our engineers have been trying to debug this issue to no
    avail:

    Our dataset includes dates stored as JODA LocalDate. Even the
    simplest queries fail, e.g.

    (??<- [?a] ([[(LocalDate.)]] ?a))

    The stack trace reports a NullPointerException in
    org.joda.time.LocalDate.getValue() as being the culprit, but
    inspection of the joda-time-1.6.2 source showed no obvious flaws.

    We're running cascalog version 1.8.6 and joda-time 1.6.2.

    Any insight you might be able to offer would be most appreciated.

    Thanks in advance!
    -John Bates
  • Sam Ritchie at Mar 20, 2012 at 5:04 pm
    Jack, based on my test this is right. John, you're going to have to write a custom Kryo serializer for LocalDate. I've got a bunch of examples here:

    https://github.com/Cascading/meat-locker/tree/master/src/jvm/com/twitter/meatlocker/kryo

    Once you've got that done you can register the serializer with the "cascading.kryo.registrations" JobConf option, as detailed in Cascading.Kryo:

    https://github.com/Cascading/cascading.kryo

    The simplest way to do this is to create a "job-conf.clj" file under the "resources" directory in your project. The file should contain the following:

    {"cascading.kryo.registrations" "org.joda.time.LocalDate,your.kryo.LocalDateSerializer"}

    This should get everything working again.

    Cheers,
    --
    Sam Ritchie
    Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

    On Monday, March 19, 2012 at 7:46 PM, Thomas Jack wrote:

    Can you share more of the stacktrace?
    My wild guess is that the default kryo serialization doesn't work for LocalDate. You may need to register a different serializer or create a new one.
    On Mar 19, 2012 8:52 PM, "John Bates" (mailto:johntbates@gmail.com)> wrote:
    Hi, all.

    A few of our engineers have been trying to debug this issue to no
    avail:

    Our dataset includes dates stored as JODA LocalDate. Even the
    simplest queries fail, e.g.

    (??<- [?a] ([[(LocalDate.)]] ?a))

    The stack trace reports a NullPointerException in
    org.joda.time.LocalDate.getValue() as being the culprit, but
    inspection of the joda-time-1.6.2 source showed no obvious flaws.

    We're running cascalog version 1.8.6 and joda-time 1.6.2.

    Any insight you might be able to offer would be most appreciated.

    Thanks in advance!
    -John Bates
  • John Bates at Mar 20, 2012 at 6:10 pm
    Excellent - we'll give that a try. Thanks very much!

    -bates
    On Mar 20, 10:04 am, Sam Ritchie wrote:
    Jack, based on my test this is right. John, you're going to have to write a custom Kryo serializer for LocalDate. I've got a bunch of examples here:

    https://github.com/Cascading/meat-locker/tree/master/src/jvm/com/twit...

    Once you've got that done you can register the serializer with the "cascading.kryo.registrations" JobConf option, as detailed in Cascading.Kryo:

    https://github.com/Cascading/cascading.kryo

    The simplest way to do this is to create a "job-conf.clj" file under the "resources" directory in your project. The file should contain the following:

    {"cascading.kryo.registrations" "org.joda.time.LocalDate,your.kryo.LocalDateSerializer"}

    This should get everything working again.

    Cheers,
    --
    Sam Ritchie
    Sent with Sparrow (http://www.sparrowmailapp.com/?sig)






    On Monday, March 19, 2012 at 7:46 PM, Thomas Jack wrote:
    Can you share more of the stacktrace?
    My wild guess is that the default kryo serialization doesn't work for LocalDate. You may need to register a different serializer or create a new one.
    On Mar 19, 2012 8:52 PM, "John Bates" (mailto:johntba...@gmail.com)> wrote:
    Hi, all.
    A few of our engineers have been trying to debug this issue to no
    avail:
    Our dataset includes dates stored as JODA LocalDate.  Even the
    simplest queries fail, e.g.
    (??<- [?a] ([[(LocalDate.)]] ?a))
    The stack trace reports a NullPointerException in
    org.joda.time.LocalDate.getValue() as being the culprit, but
    inspection of the joda-time-1.6.2 source showed no obvious flaws.
    We're running cascalog version 1.8.6 and joda-time 1.6.2.
    Any insight you might be able to offer would be most appreciated.
    Thanks in advance!
    -John Bates
  • Marc Limotte at Mar 20, 2012 at 9:57 pm
    Hi Sam.

    I'm trying this solution out, but for a slightly different scenario. I'm
    using Joda LocalTime. Also in John's example, the data to be serialized is
    in tuple data. I have a case where it's part of the custom operation
    parameters. E.g.:

    (defmapcatop [map-normalize [the-local-time-instance-is-passed-here]]
    ...)

    I wrote a simple JodaLocalTimeSerializer along the lines of your examples,
    and adjusted the jon-conf as suggested. From a kryo TRACE, it looks like
    it is finding it. But not using it at the right point.

    As I understand it, the mapcatop parameters are serialized into the
    jobconf. So I wonder if maybe the "cascading.kryo.registrations" have not
    taken effect at that point?

    I can provide the full kryo trace output if that helps.

    Marc
    On Tue, Mar 20, 2012 at 1:04 PM, Sam Ritchie wrote:

    Jack, based on my test this is right. John, you're going to have to write
    a custom Kryo serializer for LocalDate. I've got a bunch of examples here:


    https://github.com/Cascading/meat-locker/tree/master/src/jvm/com/twitter/meatlocker/kryo

    Once you've got that done you can register the serializer with the
    "cascading.kryo.registrations" JobConf option, as detailed in
    Cascading.Kryo:

    https://github.com/Cascading/cascading.kryo

    The simplest way to do this is to create a "job-conf.clj" file under the
    "resources" directory in your project. The file should contain the
    following:

    {"cascading.kryo.registrations"
    "org.joda.time.LocalDate,your.kryo.LocalDateSerializer"}

    This should get everything working again.

    Cheers,
    --
    Sam Ritchie
    Sent with Sparrow <http://www.sparrowmailapp.com/?sig>

    On Monday, March 19, 2012 at 7:46 PM, Thomas Jack wrote:

    Can you share more of the stacktrace?

    My wild guess is that the default kryo serialization doesn't work for
    LocalDate. You may need to register a different serializer or create a new
    one.
    On Mar 19, 2012 8:52 PM, "John Bates" wrote:

    Hi, all.

    A few of our engineers have been trying to debug this issue to no
    avail:

    Our dataset includes dates stored as JODA LocalDate. Even the
    simplest queries fail, e.g.

    (??<- [?a] ([[(LocalDate.)]] ?a))

    The stack trace reports a NullPointerException in
    org.joda.time.LocalDate.getValue() as being the culprit, but
    inspection of the joda-time-1.6.2 source showed no obvious flaws.

    We're running cascalog version 1.8.6 and joda-time 1.6.2.

    Any insight you might be able to offer would be most appreciated.

    Thanks in advance!
    -John Bates

  • Sam Ritchie at Mar 23, 2012 at 2:40 am
    Hey Marc,

    You're correct about this; same issue as I mentioned in the other email.
    I'll try to get this fixed up shortly.

    Cheers,
    Sam
    On Tue, Mar 20, 2012 at 2:56 PM, Marc Limotte wrote:

    Hi Sam.

    I'm trying this solution out, but for a slightly different scenario. I'm
    using Joda LocalTime. Also in John's example, the data to be serialized is
    in tuple data. I have a case where it's part of the custom operation
    parameters. E.g.:

    (defmapcatop [map-normalize [the-local-time-instance-is-passed-here]]
    ...)

    I wrote a simple JodaLocalTimeSerializer along the lines of your examples,
    and adjusted the jon-conf as suggested. From a kryo TRACE, it looks like
    it is finding it. But not using it at the right point.

    As I understand it, the mapcatop parameters are serialized into the
    jobconf. So I wonder if maybe the "cascading.kryo.registrations" have not
    taken effect at that point?

    I can provide the full kryo trace output if that helps.

    Marc
    On Tue, Mar 20, 2012 at 1:04 PM, Sam Ritchie wrote:

    Jack, based on my test this is right. John, you're going to have to
    write a custom Kryo serializer for LocalDate. I've got a bunch of examples
    here:


    https://github.com/Cascading/meat-locker/tree/master/src/jvm/com/twitter/meatlocker/kryo

    Once you've got that done you can register the serializer with the
    "cascading.kryo.registrations" JobConf option, as detailed in
    Cascading.Kryo:

    https://github.com/Cascading/cascading.kryo

    The simplest way to do this is to create a "job-conf.clj" file under the
    "resources" directory in your project. The file should contain the
    following:

    {"cascading.kryo.registrations"
    "org.joda.time.LocalDate,your.kryo.LocalDateSerializer"}

    This should get everything working again.

    Cheers,
    --
    Sam Ritchie
    Sent with Sparrow <http://www.sparrowmailapp.com/?sig>

    On Monday, March 19, 2012 at 7:46 PM, Thomas Jack wrote:

    Can you share more of the stacktrace?

    My wild guess is that the default kryo serialization doesn't work for
    LocalDate. You may need to register a different serializer or create a new
    one.
    On Mar 19, 2012 8:52 PM, "John Bates" wrote:

    Hi, all.

    A few of our engineers have been trying to debug this issue to no
    avail:

    Our dataset includes dates stored as JODA LocalDate. Even the
    simplest queries fail, e.g.

    (??<- [?a] ([[(LocalDate.)]] ?a))

    The stack trace reports a NullPointerException in
    org.joda.time.LocalDate.getValue() as being the culprit, but
    inspection of the joda-time-1.6.2 source showed no obvious flaws.

    We're running cascalog version 1.8.6 and joda-time 1.6.2.

    Any insight you might be able to offer would be most appreciated.

    Thanks in advance!
    -John Bates


    --
    Sam Ritchie, Twitter Inc
    703.662.1337
    @sritchie09

    (Too brief? Here's why! http://emailcharter.org)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedMar 20, '12 at 1:52a
activeMar 23, '12 at 2:40a
posts6
users4
websiteclojure.org
irc#clojure

People

Translate

site design / logo © 2021 Grokbase