Hey, I'm going to split this thread up to answer each of the issues. First
off, I've fixed the problem with Kryo objects in the first position of a
tuple. Go ahead and pull the latest 1.8.5-SNAPSHOT and you should be good
to go.

I also discovered a bug in my configuration merging code -- that's all
fixed, so your custom serializations should be back in action. Let me know
if this gets you back on track!
On Sat, Dec 10, 2011 at 7:47 PM, tomoj wrote:

Yeah, I patched your fork. Here is the problem with Clojure 1.2.1:
https://gist.github.com/7505aa3e77cd7f28d782

Another issue: after upgrading to the snapshot, code that worked
before is now breaking with a somewhat similar error:
https://gist.github.com/9fecf1d84ebf5143c3af . This despite the fact
that the query is wrapped in (with-job-conf {"io.serializations"

"org.apache.hadoop.io.serializer.WritableSerialization,my,custom,serializations"}).
Should this disable Kryo serialization entirely?

I get the same error here when sinking to an hfs-seqfile as with
stdout, and there are no memory source taps involved.

I haven't been able to reproduce yet with an analogous simple query on
primitive types, which makes me think there is something wrong with my
custom serialization, but the code works fine in 1.8.4. Any ideas?
I'll keep trying to extract a simple reproduction or maybe try to
bisect the Cascalog history...
On Dec 10, 6:40 pm, Sam Ritchie wrote:
Cascalog should definitely remain compatible with 1.2. Interesting about
BigInt, I hadn't gotten any exceptions running my tests in Cascalog with
1.2. Did you end up patching my fork of Carbonite? Can you send me the
error you were getting when you tried to run cascalog 1.8.5-SNAPSHOT with
1.2?

I think the EOFException is a problem with Cascalog's StdoutTap and
MemorySourceTap; I'll open up an issue and get this fixed right up. (Your
example works fine with a textline or sequencefile tap, but fails with
those two.)

Thanks for this, I knew we'd have a few kinks to iron out before this
worked :)

Sam


--
Sam Ritchie, Twitter Inc
703.662.1337
@sritchie09

(Too brief? Here's why! http://emailcharter.org)

Search Discussions

  • Tomoj at Dec 12, 2011 at 7:14 am
    Yeah, both problems solved, thanks! This is great.
    On Dec 11, 11:59 pm, Sam Ritchie wrote:
    Hey, I'm going to split this thread up to answer each of the issues. First
    off, I've fixed the problem with Kryo objects in the first position of a
    tuple. Go ahead and pull the latest 1.8.5-SNAPSHOT and you should be good
    to go.

    I also discovered a bug in my configuration merging code -- that's all
    fixed, so your custom serializations should be back in action. Let me know
    if this gets you back on track!
  • François Le Lay at Dec 12, 2011 at 12:14 pm
    Excellent, same for me, no more EOF Exceptions!
    I'm opening another topic for the BigInt issue.

    François
    On 12 déc, 08:14, tomoj wrote:
    Yeah, both problems solved, thanks! This is great.

    On Dec 11, 11:59 pm, Sam Ritchie wrote:






    Hey, I'm going to split this thread up to answer each of the issues. First
    off, I've fixed the problem with Kryo objects in the first position of a
    tuple. Go ahead and pull the latest 1.8.5-SNAPSHOT and you should be good
    to go.
    I also discovered a bug in my configuration merging code -- that's all
    fixed, so your custom serializations should be back in action. Let me know
    if this gets you back on track!

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedDec 12, '11 at 5:59a
activeDec 12, '11 at 12:14p
posts3
users3
websiteclojure.org
irc#clojure

People

Translate

site design / logo © 2022 Grokbase