Jeremy, where you able to make it run on AMAZON elastic map reduce
machines?
i'v tried to copy the jars (both pig's and cassandra) to the new machine
set the PIG_HOME environment variable
even added the hadoop config files to the class path
and I'm getting this error
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage
java.lang.RuntimeException: Failed to create DataStorage
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.(HExecutionEngine.java:213)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
at org.apache.pig.PigServer.(PigServer.java:214)
at org.apache.pig.tools.grunt.Grunt.(Main.java:462)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Call to
ip-10-56-51-167.eu-west-1.compute.internal/10.56.51.167:9000 failed on local
exception: java.io.EOFExc
eption
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1139)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:111)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:180)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1514)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1548)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1530)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:720)
================================================================================
Amazon claims to run hadoop v 0.20, what am i doing wrong?
On Mon, Aug 1, 2011 at 5:55 PM, Jeremy Hanna wrote:Ah - just saw this, glad you got it working - cheers.
On Aug 1, 2011, at 5:43 AM, Shai Harel wrote:hey all, i'v successfully fixed this problem,
i was missing the cassandra jars,
so you actually need to build cassandra (ant) and then you need to jar it
(ant jar)
and only then it'll work
BTW if you have hue installed, remove it first!
On Mon, Aug 1, 2011 at 12:41 PM, Shai Harel wrote:
thanks for the help, i'v tried to be conservative and i'm using pig 0.8
&
cassandra 0.8
and still getting this error
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. Could not initialize class
org.apache.cassandra.thrift.SliceRange
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.cassandra.thrift.SliceRange
at
org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(Unknown
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:369)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:256)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.storeEx(PigServer.java:874)
at org.apache.pig.PigServer.store(PigServer.java:816)
at org.apache.pig.PigServer.openIterator(PigServer.java:728)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
does anyone else have this problem?
On Sun, Jul 31, 2011 at 2:04 PM, Jeremy Hanna <
jeremy.hanna1234@gmail.com>wrote:
with
Cassandra's schema validators. CassandraStorage gets the schema from
Cassandra and tries to unmarshal the data into Pig data types with the
schema information.
See if that helps.
On Jul 31, 2011, at 9:48 AM, Shai Harel wrote:
hey all, i'v been trying to query cassandra using my pig script,
so i used the contrib jar from cassandra. and i'm getting the
following
error...
some thrift failure err.... :|
ERROR 2998: Unhandled internal error.
org.apache.thrift.meta_data.FieldValueMetaData.<init>(BZ)V
java.lang.NoSuchMethodError:
org.apache.thrift.meta_data.FieldValueMetaData.<init>(BZ)V
at
org.apache.cassandra.thrift.SliceRange.<clinit>(SliceRange.java:149)
at
org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(Unknown
Source)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:369)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:256)
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.storeEx(PigServer.java:874)
at org.apache.pig.PigServer.store(PigServer.java:816)
at org.apache.pig.PigServer.openIterator(PigServer.java:728)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
does anyone managed to get this up and running?
i'm considering to rewrite the CassandraStorage.jar using Hector,
Any thoughts about that?