[Hive-user] [Hive] Error in semantic analysis: partition not found - CDH3 Beta

Carl Steinbach answered this - thanks

From: Ken.Barclay@wellsfargo.com
Sent: Thursday, August 12, 2010 7:19 PM
To: hive-user@hadoop.apache.org
Subject: [Hive] Error in semantic analysis: partition not found - CDH3 Beta

Hello,

[I posted the question below to Cloudera's getsatisfaction site but am cross-posting here in case hive-users folks have debugging suggestions. I'm really stuck on this one.]

I recently upgraded to CDH3 Beta. I had some Hive code working well in an earlier version of Hadoop 20 that created a table, then loaded data into it using LOAD DATA LOCAL INPATH. In CDH3, I now get a semantic error when I run the same LOAD command.

The table is created by

CREATE TABLE TOMCAT(identifier STRING, datestamp STRING, time_stamp STRING, seq STRING, server STRING, logline STRING) PARTITIONED BY(filedate STRING, app STRING, filename STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\011' STORED AS TEXTFILE;

and the load command used is:

LOAD DATA LOCAL INPATH '/var/www/petrify/mw.log.trustejb1' INTO TABLE TOMCAT PARTITION (filedate='2010-06-25', app='trustdomain', filename='mw.log.trustejb1');

The file is simple tab-delimited log data.
If I exclude the partition when I create the table, the data loads fine. But when I set up the partitions I get the stack trace below during the load.

I tried copying the data into HDFS and using LOAD DATA INPATH instead, but got the same error:

FAILED: Error in semantic analysis: line 1:110 Partition not found 'mw.log.trustejb1'

where 110 is the character position just after the word PARTITION in the query.
It seems like it doesn't think the table is partitioned, though I can see the partition keys listed when I do DESCRIBE EXTENDED on my table. (Output from that is below the error.) There were no errors in the logs or at the Thrift server console when I created the table.

Strangely, when I run SHOW PARTITIONS TOMCAT, it doesn't list anything.

Any help with this would be most welcome.

Thanks
Ken

10/08/12 15:11:40 INFO service.HiveServer: Running the query: LOAD DATA LOCAL INPATH '/var/www/petrify/trustdomain-rewritten/mw.log.trustejb1' INTO TABLE TOMCAT PARTITION (filedate='2010-06-25', app='trustdomain', filename='mw.log.trustejb1')
10/08/12 15:11:40 INFO parse.ParseDriver: Parsing command: LOAD DATA LOCAL INPATH '/var/www/petrify/trustdomain-rewritten/mw.log.trustejb1' INTO TABLE TOMCAT PARTITION (filedate='2010-06-25', app='trustdomain', filename='mw.log.trustejb1')
10/08/12 15:11:40 INFO parse.ParseDriver: Parse Completed
10/08/12 15:11:40 INFO hive.log: DDL: struct tomcat { string identifier, string datestamp, string time_stamp, string seq, string server, string logline}
10/08/12 15:11:40 ERROR metadata.Hive: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:831)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:799)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:418)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:620)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.(BaseSemanticAnalyzer.java:397)
at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:178)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:120)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

FAILED: Error in semantic analysis: line 1:110 Partition not found 'mw.log.trustejb1'
10/08/12 15:11:40 ERROR ql.Driver: FAILED: Error in semantic analysis: line 1:110 Partition not found 'mw.log.trustejb1'
org.apache.hadoop.hive.ql.parse.SemanticException: line 1:110 Partition not found 'mw.log.trustejb1'
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.(BaseSemanticAnalyzer.java:403)
at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:178)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275)
at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:120)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

DESCRIBE TABLE EXTENDED TOMCAT;
identifier string
datestamp string
time_stamp string
seq string
server string
logline string
filedate string
app string
filename string

Detailed Table Information Table(tableName:tomcat, dbName:default, owner:root, createTime:1281661047, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:identifier, type:string, comment:null), FieldSchema(name:datestamp, type:string, comment:null), FieldSchema(name:time_stamp, type:string, comment:null), FieldSchema(name:seq, type:string, comment:null), FieldSchema(name:server, type:string, comment:null), FieldSchema(name:logline, type:string, comment:null)], location:hdfs://hadoop-vm1/user/hive/warehouse/tomcat, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim= , serialization.format=9}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[FieldSchema(name:filedate, type:string, comment:null), FieldSchema(name:app, type:string, comment:null), FieldSchema(name:filename, type:string, comment:null)], parameters:{transient_lastDdlTime=1281661047})
Time taken: 0.086 seconds
10/08/12 18:53:08 INFO CliDriver: Time taken: 0.086 seconds



Ken Barclay

Integration Engineer

Wells Fargo Bank - ISD | 45 Fremont Street, 10th Floor | San Francisco, CA 94105
MAC A0194-100
Tel 415-222-6491

ken.barclay@wellsfargo.com

This message may contain confidential and/or privileged information, and is intended for the use of the addressee only. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.

Hi Ken,

I'm going to crosspost what I wrote on GetSatisfaction:

Looks like you ran into this bug:
https://issues.apache.org/jira/browse/HIVE-1428

Pradeep wrote a fix for this issue in 0.7, and we're in the process of
backporting it to 0.6, and ideally we're going to release 0.6.0 in time for
it to be included in the next version of CDH3. In the meantime I think your
best bet is to execute the LOAD DATA command via a CLI session that is not
using Thrift, i.e. one that is not connected to a standalone HiveServer or
HiveMetaStoreServer instance.

Sorry for the inconvenience!

Carl