FAQ
So I've spent quite a bit of time on trying to get lzo working with
cascalog.

Right now I'm trying to use cascalog-lzo, and I run into this error.

ERROR lzo.GPLNativeCodeLoader: Could not load native gpl library

java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path


I first tried compiling the hadoop-gpl-compression, and placing the
binaries and jar in HADOOP_HOME/lib/native/arch/ and HADOOP_HOME/lib/
respectively. Then I tried the same thing with twitter's hadoop-lzo
thinking maybe that would work instead? Neither worked.

I believe I need to somehow load the gplcompression jar inside the cascalog
project, but I thought cascalog-lzo was suppose to handle this for me?

I'm on arch-linux-64bit

My project just copies a txt file to an lzo compressed text file


(ns casc-test.core
(:use [cascalog.api]
[cascalog.more-taps :only (hfs-delimited)]
[cascalog.lzo])
(:gen-class))

(defn -main [in out & args]
(?<- (hfs-lzo-textline out)
[?doc ?line]
((hfs-delimited in :skip-header? true) ?doc ?line)))


Is there something special I need to do with hadoop, to allow lzo
compression, or is there something special I need to do with my project? or
both?

Any help is appreciated! Thank you


PS. Unfortunately the latest version of cascalog-lzo fails in lein deps
because the dependency "com.google:protobuf-java:jar:2.3.0" should be
"com.google.protobuf:protobuf-java:jar:2.3.0"
http://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.3.0/

Search Discussions

  • Marco Munizaga at Jan 11, 2013 at 2:54 am
    Doing this seems to work!

    export
    JAVA_LIBRARY_PATH=/home/marco/cascalog/lzo/hadoop-lzo/build/native/Linux-amd64-64/lib:/usr/lib/hadoop/lib/native/Linux-amd64-64
    export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native/Linux-amd64-64

    On Thursday, January 10, 2013 4:46:04 PM UTC-5, Marco Munizaga wrote:

    So I've spent quite a bit of time on trying to get lzo working with
    cascalog.

    Right now I'm trying to use cascalog-lzo, and I run into this error.

    ERROR lzo.GPLNativeCodeLoader: Could not load native gpl library

    java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path


    I first tried compiling the hadoop-gpl-compression, and placing the
    binaries and jar in HADOOP_HOME/lib/native/arch/ and HADOOP_HOME/lib/
    respectively. Then I tried the same thing with twitter's hadoop-lzo
    thinking maybe that would work instead? Neither worked.

    I believe I need to somehow load the gplcompression jar inside the
    cascalog project, but I thought cascalog-lzo was suppose to handle this for
    me?

    I'm on arch-linux-64bit

    My project just copies a txt file to an lzo compressed text file


    (ns casc-test.core
    (:use [cascalog.api]
    [cascalog.more-taps :only (hfs-delimited)]
    [cascalog.lzo])
    (:gen-class))

    (defn -main [in out & args]
    (?<- (hfs-lzo-textline out)
    [?doc ?line]
    ((hfs-delimited in :skip-header? true) ?doc ?line)))


    Is there something special I need to do with hadoop, to allow lzo
    compression, or is there something special I need to do with my project? or
    both?

    Any help is appreciated! Thank you


    PS. Unfortunately the latest version of cascalog-lzo fails in lein deps
    because the dependency "com.google:protobuf-java:jar:2.3.0" should be
    "com.google.protobuf:protobuf-java:jar:2.3.0"
    http://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.3.0/


Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedJan 10, '13 at 11:07p
activeJan 11, '13 at 2:54a
posts2
users1
websiteclojure.org
irc#clojure

1 user in discussion

Marco Munizaga: 2 posts

People

Translate

site design / logo © 2021 Grokbase