Grokbase Groups Hive user June 2010
FAQ
Hi,

I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.



Thanks,

Pradeep



== session output ===



[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"

10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively

Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask

[pradeepk@chargesize:~/dev/howl]



== session output ===



/tmp/pradeepk/hive.log has:

2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result

org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result

at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)

at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result

at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)

at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)

at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)

at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

... 15 more



The thrift server messages are:

10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable

10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called

10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable

Search Discussions

  • Yq he at Jun 17, 2010 at 12:01 pm
    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing data
    in sequencefile format. However I have written a custom Serde to interpret
    the data. I am able to create the table fine but get the exception shown in
    the session output below when I try to add partition – any help would be
    greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action)
    location '/user/pradeepk/mytable/20091101/10’;"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
    core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
    core-default.xml, mapred-default.xml and hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    *Caused by: org.apache.thrift.TApplicationException: get_partition failed:
    unknown result*

    * at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    *

    * at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    *

    * at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    *

    * at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*

    * ... 15 more*



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default
    tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with
    implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable



  • Pradeep Kamath at Jun 17, 2010 at 4:10 pm
    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Ashish Thusoo at Jun 17, 2010 at 6:15 pm
    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 17, 2010 at 8:26 pm
    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Pradeep Kamath at Jun 18, 2010 at 5:20 pm
    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Ning Zhang at Jun 18, 2010 at 5:49 pm
    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde’
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable’;

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Edward Capriolo at Jun 18, 2010 at 6:23 pm

    On Fri, Jun 18, 2010 at 1:49 PM, Ning Zhang wrote:

    Pradeep,

    I ran the commands you provided and it succeeded with the expected
    behavior.

    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH
    so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here – I want to open a jira for
    the first one since I am now able to reproduce it even with text format with
    builtin Serdes. Essentially this is a bug in the thrift code (not sure if it
    is in the client or server) since the same alter table statement works fine
    when the hive client does not use thrift. Here are the details:



    *cat create_dummy.sql*

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    *hive -f create_dummy.sql*

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
    core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
    core-default.xml, mapred-default.xml and hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds

    * *

    *hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"*

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
    core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
    core-default.xml, mapred-default.xml and hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*



    *hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid = '100',action='view',testid='10')
    location '/user/pradeepk/dummy/20100602/100/view/10';"*

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
    core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
    core-default.xml, mapred-default.xml and hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    *OK*

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? – If it is the
    latter, I can open a jira with the above details.



    Thanks,

    Pradeep




    ------------------------------

    *From:* Pradeep Kamath
    *Sent:* Thursday, June 17, 2010 1:25 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde’

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable’;



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*

    If I don’t use thrift and use a hive-site.xml to directly talk to the db,
    the alter table seems to succeed:

    hive --auxpath ult-serde.jar *--config hive_conf_without_thrift* -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]*hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult
    where datestamp='20091101';"*

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
    core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
    core-default.xml, mapred-default.xml and hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.<init>(Path.java:90)

    at org.apache.hadoop.fs.Path.<init>(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep


    ------------------------------

    *From:* Ashish Thusoo
    *Sent:* Thursday, June 17, 2010 11:15 AM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command for
    this table.



    Ashish


    ------------------------------

    *From:* Pradeep Kamath
    *Sent:* Thursday, June 17, 2010 9:09 AM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error

    Sorry – that was a cut-paste error – I don’t have the action part – so I am
    specifying key-value pairs. Since what I am trying to do seems like a basic
    operation, I am wondering if it’s something to do with my Serde –
    unfortunately the error I see gives me no clue of what could be wrong – any
    help would be greatly appreciated!



    Thanks,

    Pradeep


    ------------------------------

    *From:* yq he
    *Sent:* Wednesday, June 16, 2010 5:54 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing data
    in sequencefile format. However I have written a custom Serde to interpret
    the data. I am able to create the table fine but get the exception shown in
    the session output below when I try to add partition – any help would be
    greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action)
    location '/user/pradeepk/mytable/20091101/10’;"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
    core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
    core-default.xml, mapred-default.xml and hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    *Caused by: org.apache.thrift.TApplicationException: get_partition failed:
    unknown result*

    * at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    *

    * at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    *

    * at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    *

    * at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*

    * ... 15 more*



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default
    tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with
    implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable







    Also I find that serde's and input formats should be put in hive_home/auxlib
    as well as hive_home/lib. auxlib directory is not created by default but if
    it exists the hive shell bootstrap picks it up.
  • Pradeep Kamath at Jun 18, 2010 at 6:27 pm
    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
    se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
    e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
    _client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
    on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
    bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
    b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
    jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
    hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
    /hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
    .jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
    c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
    lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
    bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
    bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
    /hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
    .jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
    xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
    /lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
    r:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
    .3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
    exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
    /lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
    ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
    me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
    ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
    erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
    ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
    client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
    ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
    ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
    jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
    lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
    ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
    e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
    locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:





    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Karthik at Jun 18, 2010 at 6:48 pm
    I have my custom SerDe classes (as jar files) under /home/hadoop/hive/lib folder and I have set "hive.aux.jars.path" property in my "hive-site.xml" file to this location (value: "file:///home/hadoop/hive/lib/")

    When I create a able (or query an existing table), I get a SerDe does not exist error, althought, it works fine if I set the value to full absolute path of the file name like this: "file:///home/hadoop/hive/lib/myserde.jar", but again, my HWI complains with same error even if I set full path name of the JAR.

    It also works well if I run "hive --auxpath /home/hadoop/hive/lib".

    I need to provide multiple JAR files under auxpath. I use Hive 0.5+20. Should I not relay on "hive.aux.jars.path" set in hive-site.xml and only use --auxpath param?

    Please help.

    Regards,
    Karthik.
  • Ning Zhang at Jun 18, 2010 at 6:51 pm
    We also use thrift for hive.metastore.uris. Then at the thrift server side, we use JDBC to mysql (javax.jdo.option.ConnectionURL). Can you check your thrift server side to see the JDO connectionURL is set correctly?


    On Jun 18, 2010, at 11:25 AM, Pradeep Kamath wrote:

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an “env” command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:


    I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde’
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable’;

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Paul Yang at Jun 18, 2010 at 7:00 pm
    If the JDO connectionURL wasn't set correctly on the Thrift server, then the create table should fail as well. The problem is that the create table succeeds, but the alter table fails, right?

    From: Ning Zhang
    Sent: Friday, June 18, 2010 11:52 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    We also use thrift for hive.metastore.uris. Then at the thrift server side, we use JDBC to mysql (javax.jdo.option.ConnectionURL). Can you check your thrift server side to see the JDO connectionURL is set correctly?


    On Jun 18, 2010, at 11:25 AM, Pradeep Kamath wrote:


    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde'
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable';

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 18, 2010 at 7:19 pm
    Yes - exactly - the create table works fine - I have been able to even
    create external tables and run select queries on them with my current
    thrift setup. I am using Oracle as a backend DB.



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:00 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    If the JDO connectionURL wasn't set correctly on the Thrift server, then
    the create table should fail as well. The problem is that the create
    table succeeds, but the alter table fails, right?



    From: Ning Zhang
    Sent: Friday, June 18, 2010 11:52 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    We also use thrift for hive.metastore.uris. Then at the thrift server
    side, we use JDBC to mysql (javax.jdo.option.ConnectionURL). Can you
    check your thrift server side to see the JDO connectionURL is set
    correctly?





    On Jun 18, 2010, at 11:25 AM, Pradeep Kamath wrote:



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
    se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
    e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
    _client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
    on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
    bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
    b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
    jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
    hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
    /hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
    .jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
    c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
    lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
    bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
    bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
    /hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
    .jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
    xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
    /lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
    r:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
    .3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
    exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
    /lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
    ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
    me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
    ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
    erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
    ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
    client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
    ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
    ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
    jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
    lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
    ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
    e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
    locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:





    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Paul Yang at Jun 18, 2010 at 7:10 pm
    Hi Pradeep,

    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running

    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Cheers,
    Paul

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde'
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable';

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 18, 2010 at 8:40 pm
    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
    se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
    e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
    _client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
    on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
    bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
    b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
    jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
    hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
    /hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
    .jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
    c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
    lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
    bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
    bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
    /hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
    .jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
    xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
    /lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
    r:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
    .3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
    exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
    /lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
    ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
    me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
    ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
    erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
    ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
    client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
    ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
    ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
    jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
    lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
    ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
    e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
    locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Paul Yang at Jun 18, 2010 at 9:19 pm
    Looks like the thrift python libraries aren't in your path. run:

    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/

    before trying the remote command

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Traceback (most recent call last):
    File "./ThriftHiveMetastore-remote", line 11, in ?
    from thrift.transport import TTransport
    ImportError: No module named thrift.transport

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Pradeep,

    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running

    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Cheers,
    Paul

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde'
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable';

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 18, 2010 at 9:52 pm
    Looks like the standalone script works fine against the existing
    partition:

    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
    reKeyTextOutputFormat', sortCols=[],
    inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
    .LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2
    100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    However when I tried to add another partition with the hive cli using
    thrift:

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    tail -30 /tmp/pradeepk/hive.log

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
    se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
    e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
    _client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
    on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
    bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
    b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
    jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
    hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
    /hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
    .jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
    c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
    lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
    bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
    bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
    /hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
    .jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
    xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
    /lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
    r:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
    .3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
    exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
    /lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
    ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
    me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
    ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
    erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
    ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
    client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
    ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
    ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
    jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
    lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
    ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
    e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
    locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Pradeep Kamath at Jun 21, 2010 at 9:12 pm
    I tried debugging in code a little more. Here is what I found:

    The code in ThriftHiveMetaStore eventually makes a call -
    get_partition() passing the partition key values for the partition I am
    trying to add using alter table. I assume this is to check that the
    partition doesn't already exist.

    I added a debug line in the following code:

    public Partition recv_get_partition() throws MetaException, TException

    {

    TMessage msg = iprot_.readMessageBegin();

    if (msg.type == TMessageType.EXCEPTION) {

    TApplicationException x = TApplicationException.read(iprot_);

    iprot_.readMessageEnd();

    throw x;

    }

    get_partition_result result = new get_partition_result();



    result.read(iprot_);

    System.err.println("XXX: result:" + result);

    iprot_.readMessageEnd();

    if (result.isSetSuccess()) {

    return result.success;

    }

    if (result.o1 != null) {

    throw result.o1;

    }

    throw new
    TApplicationException(TApplicationException.MISSING_RESULT,
    "get_partition failed: unknown result");

    }



    I also put a debug statements in the read() method:

    public void read(TProtocol iprot) throws TException {

    TField field;

    iprot.readStructBegin();

    System.err.println("XXX: Reading TProtocol object:");

    while (true)

    {

    field = iprot.readFieldBegin();

    System.err.println("XXX: field just read:" + field);

    if (field.type == TType.STOP) {

    break;



    I got

    XXX: Reading TProtocol object:

    XXX: field just read:<TField name:'' type:0 field-id:0>

    XXX: result:get_partition_result(success:null, o1:null)



    The field read in the thrift response message is of type "STOP" and with
    id of type SUCCESS. This seems right since there are no existing
    partitions. But the way the rest of the code handles this, results in
    the exception.



    Any pointers?



    TIA,

    Pradeep

    ________________________________

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the standalone script works fine against the existing
    partition:

    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
    reKeyTextOutputFormat', sortCols=[],
    inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
    .LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2
    100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    However when I tried to add another partition with the hive cli using
    thrift:

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    tail -30 /tmp/pradeepk/hive.log

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
    se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
    e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
    _client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
    on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
    bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
    b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
    jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
    hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
    /hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
    .jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
    c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
    lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
    bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
    bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
    /hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
    .jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
    xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
    /lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
    r:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
    .3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
    exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
    /lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
    ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
    me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
    ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
    erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
    ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
    client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
    ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
    ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
    jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
    lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
    ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
    e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
    locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Paul Yang at Jun 21, 2010 at 9:35 pm
    Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used.

    One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.

    Pradeep, are you interested in working on a patch for this problem?

    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    I tried debugging in code a little more. Here is what I found:
    The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist.
    I added a debug line in the following code:
    public Partition recv_get_partition() throws MetaException, TException
    {
    TMessage msg = iprot_.readMessageBegin();
    if (msg.type == TMessageType.EXCEPTION) {
    TApplicationException x = TApplicationException.read(iprot_);
    iprot_.readMessageEnd();
    throw x;
    }
    get_partition_result result = new get_partition_result();

    result.read(iprot_);
    System.err.println("XXX: result:" + result);
    iprot_.readMessageEnd();
    if (result.isSetSuccess()) {
    return result.success;
    }
    if (result.o1 != null) {
    throw result.o1;
    }
    throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
    }

    I also put a debug statements in the read() method:
    public void read(TProtocol iprot) throws TException {
    TField field;
    iprot.readStructBegin();
    System.err.println("XXX: Reading TProtocol object:");
    while (true)
    {
    field = iprot.readFieldBegin();
    System.err.println("XXX: field just read:" + field);
    if (field.type == TType.STOP) {
    break;

    I got
    XXX: Reading TProtocol object:
    XXX: field just read:<TField name:'' type:0 field-id:0>
    XXX: result:get_partition_result(success:null, o1:null)

    The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.

    Any pointers?

    TIA,
    Pradeep
    ________________________________
    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the standalone script works fine against the existing partition:
    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]

    However when I tried to add another partition with the hive cli using thrift:
    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    tail -30 /tmp/pradeepk/hive.log
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the thrift python libraries aren't in your path. run:

    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/

    before trying the remote command

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Traceback (most recent call last):
    File "./ThriftHiveMetastore-remote", line 11, in ?
    from thrift.transport import TTransport
    ImportError: No module named thrift.transport

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Pradeep,

    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running

    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Cheers,
    Paul

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde'
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable';

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 22, 2010 at 12:48 am
    Hi Paul,

    Yes I don't mind working on a patch though I have no knowledge of how
    thrift code works. I was also confused by your suggestion for a fix. I
    thought the fix would be in
    ThriftHiveMetaStore.Client.recv_get_partition() - to first check
    if(result.o1 != null) throw result.o1, then to just return
    result.success and not throw the "unknown result" exception. I tried
    something along these lines and the alter table worked. Am I missing
    something?



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Ah okay. So what's going on is that getPartition() is supposed to return
    null if the specified partition doesn't exist. But because Thrift can't
    handle null return values, we are seeing this exception. This isn't a
    problem with a local metastore because Thrift isn't used.



    One solution to this problem would be modify ObjectStore.getPartition()
    to throw a NoSuchObjectException instead of returning null for
    non-existent partitions. Then Hive.getPartition() can catch the
    exception and return null to retain the original behavior.



    Pradeep, are you interested in working on a patch for this problem?



    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    I tried debugging in code a little more. Here is what I found:

    The code in ThriftHiveMetaStore eventually makes a call -
    get_partition() passing the partition key values for the partition I am
    trying to add using alter table. I assume this is to check that the
    partition doesn't already exist.

    I added a debug line in the following code:

    public Partition recv_get_partition() throws MetaException, TException

    {

    TMessage msg = iprot_.readMessageBegin();

    if (msg.type == TMessageType.EXCEPTION) {

    TApplicationException x = TApplicationException.read(iprot_);

    iprot_.readMessageEnd();

    throw x;

    }

    get_partition_result result = new get_partition_result();



    result.read(iprot_);

    System.err.println("XXX: result:" + result);

    iprot_.readMessageEnd();

    if (result.isSetSuccess()) {

    return result.success;

    }

    if (result.o1 != null) {

    throw result.o1;

    }

    throw new
    TApplicationException(TApplicationException.MISSING_RESULT,
    "get_partition failed: unknown result");

    }



    I also put a debug statements in the read() method:

    public void read(TProtocol iprot) throws TException {

    TField field;

    iprot.readStructBegin();

    System.err.println("XXX: Reading TProtocol object:");

    while (true)

    {

    field = iprot.readFieldBegin();

    System.err.println("XXX: field just read:" + field);

    if (field.type == TType.STOP) {

    break;



    I got

    XXX: Reading TProtocol object:

    XXX: field just read:<TField name:'' type:0 field-id:0>

    XXX: result:get_partition_result(success:null, o1:null)



    The field read in the thrift response message is of type "STOP" and with
    id of type SUCCESS. This seems right since there are no existing
    partitions. But the way the rest of the code handles this, results in
    the exception.



    Any pointers?



    TIA,

    Pradeep

    ________________________________

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the standalone script works fine against the existing
    partition:

    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
    reKeyTextOutputFormat', sortCols=[],
    inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
    .LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2
    100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    However when I tried to add another partition with the hive cli using
    thrift:

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    tail -30 /tmp/pradeepk/hive.log

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
    se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
    e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
    _client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
    on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
    bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
    b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
    jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
    hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
    /hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
    .jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
    c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
    lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
    bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
    bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
    /hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
    .jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
    xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
    /lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
    r:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
    .3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
    exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
    /lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
    ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
    me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
    ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
    erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
    ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
    client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
    ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
    ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
    jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
    lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
    ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
    e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
    locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Paul Yang at Jun 22, 2010 at 1:26 am
    Well, the section of code you are referring to is automatically generated by the thrift compiler so we can't modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead.

    Check out this thread:
    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html


    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 5:46 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Paul,
    Yes I don't mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() - to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the "unknown result" exception. I tried something along these lines and the alter table worked. Am I missing something?

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used.

    One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.

    Pradeep, are you interested in working on a patch for this problem?

    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    I tried debugging in code a little more. Here is what I found:
    The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist.
    I added a debug line in the following code:
    public Partition recv_get_partition() throws MetaException, TException
    {
    TMessage msg = iprot_.readMessageBegin();
    if (msg.type == TMessageType.EXCEPTION) {
    TApplicationException x = TApplicationException.read(iprot_);
    iprot_.readMessageEnd();
    throw x;
    }
    get_partition_result result = new get_partition_result();

    result.read(iprot_);
    System.err.println("XXX: result:" + result);
    iprot_.readMessageEnd();
    if (result.isSetSuccess()) {
    return result.success;
    }
    if (result.o1 != null) {
    throw result.o1;
    }
    throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
    }

    I also put a debug statements in the read() method:
    public void read(TProtocol iprot) throws TException {
    TField field;
    iprot.readStructBegin();
    System.err.println("XXX: Reading TProtocol object:");
    while (true)
    {
    field = iprot.readFieldBegin();
    System.err.println("XXX: field just read:" + field);
    if (field.type == TType.STOP) {
    break;

    I got
    XXX: Reading TProtocol object:
    XXX: field just read:<TField name:'' type:0 field-id:0>
    XXX: result:get_partition_result(success:null, o1:null)

    The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.

    Any pointers?

    TIA,
    Pradeep
    ________________________________
    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the standalone script works fine against the existing partition:
    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]

    However when I tried to add another partition with the hive cli using thrift:
    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    tail -30 /tmp/pradeepk/hive.log
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the thrift python libraries aren't in your path. run:

    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/

    before trying the remote command

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Traceback (most recent call last):
    File "./ThriftHiveMetastore-remote", line 11, in ?
    from thrift.transport import TTransport
    ImportError: No module named thrift.transport

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Pradeep,

    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running

    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Cheers,
    Paul

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde'
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable';

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 22, 2010 at 5:22 pm
    Hi Paul,
    I have attached a draft of a patch (incomplete). The problem I am
    facing is that to make this change I seem to need to change the thrift
    idl (hive_metastore.thrift). Looking at the code structure it seems like
    the generate java files are checked in rather than generated through the
    build (is this so that the build does not depend on the thrift
    compiler?). I tried downloading and building thrift but hit issues in
    the make install - wanted to check with you if the patch looks correct
    so far or if I am on the wrong path. Also if the direction is right, I
    need some guidance on generating the java files from the modified idl.

    Thanks,
    Pradeep

    Paul Yang wrote:
    Well, the section of code you are referring to is automatically
    generated by the thrift compiler so we can't modify it. As the
    inability to return null is a limitation of Thrift, we can work around
    that issue by throwing an exception instead.



    Check out this thread:

    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html





    *From:* Pradeep Kamath
    *Sent:* Monday, June 21, 2010 5:46 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Hi Paul,

    Yes I don't mind working on a patch though I have no knowledge of
    how thrift code works. I was also confused by your suggestion for a
    fix. I thought the fix would be in
    ThriftHiveMetaStore.Client.recv_get_partition() -- to first check
    if(result.o1 != null) throw result.o1, then to just return
    result.success and not throw the "unknown result" exception. I tried
    something along these lines and the alter table worked. Am I missing
    something?



    Thanks,

    Pradeep



    ------------------------------------------------------------------------

    *From:* Paul Yang
    *Sent:* Monday, June 21, 2010 2:35 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Ah okay. So what's going on is that getPartition() is supposed to
    return null if the specified partition doesn't exist. But because
    Thrift can't handle null return values, we are seeing this exception.
    This isn't a problem with a local metastore because Thrift isn't used.



    One solution to this problem would be modify
    ObjectStore.getPartition() to throw a NoSuchObjectException instead of
    returning null for non-existent partitions. Then Hive.getPartition()
    can catch the exception and return null to retain the original behavior.



    Pradeep, are you interested in working on a patch for this problem?



    *From:* Pradeep Kamath
    *Sent:* Monday, June 21, 2010 2:11 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    I tried debugging in code a little more. Here is what I found:

    The code in ThriftHiveMetaStore eventually makes a call --
    get_partition() passing the partition key values for the partition I
    am trying to add using alter table. I assume this is to check that the
    partition doesn't already exist.

    I added a debug line in the following code:

    public Partition recv_get_partition() throws MetaException, TException

    {

    TMessage msg = iprot_.readMessageBegin();

    if (msg.type == TMessageType.EXCEPTION) {

    TApplicationException x = TApplicationException.read(iprot_);

    iprot_.readMessageEnd();

    throw x;

    }

    get_partition_result result = new get_partition_result();



    result.read(iprot_);

    *System.err.println("XXX: result:" + result);*

    iprot_.readMessageEnd();

    if (result.isSetSuccess()) {

    return result.success;

    }

    if (result.o1 != null) {

    throw result.o1;

    }

    * throw new
    TApplicationException(TApplicationException.MISSING_RESULT,
    "get_partition failed: unknown result");*

    }



    I also put a debug statements in the read() method:

    public void read(TProtocol iprot) throws TException {

    TField field;

    iprot.readStructBegin();

    * System.err.println("XXX: Reading TProtocol object:");*

    while (true)

    {

    field = iprot.readFieldBegin();

    * System.err.println("XXX: field just read:" + field);*

    if (field.type == TType.STOP) {

    break;



    *I got *

    *XXX: Reading TProtocol object:*

    *XXX: field just read:<TField name:'' type:0 field-id:0>*

    *XXX: result:get_partition_result(success:null, o1:null)*



    The field read in the thrift response message is of type "STOP" and
    with id of type SUCCESS. This seems right since there are no existing
    partitions. But the way the rest of the code handles this, results in
    the exception.



    Any pointers?



    TIA,

    Pradeep

    ------------------------------------------------------------------------

    *From:* Pradeep Kamath
    *Sent:* Friday, June 18, 2010 2:51 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    *Looks like the standalone script works fine against the existing
    partition:*

    *./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10*

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat',
    sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe',
    name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    *However when I tried to add another partition with the hive cli using
    thrift:*

    *hive -e "ALTER TABLE dummy add partition(datestamp = '20100602',
    srcid = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"*

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated.
    Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
    override properties of core-default.xml, mapred-default.xml and
    hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*



    *tail -30 /tmp/pradeepk/hive.log*

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ------------------------------------------------------------------------

    *From:* Paul Yang
    *Sent:* Friday, June 18, 2010 2:19 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    *From:* Pradeep Kamath
    *Sent:* Friday, June 18, 2010 1:38 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Sorry, don't know python -- so couldn't make sense out of the
    following error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote
    -h localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ------------------------------------------------------------------------

    *From:* Paul Yang
    *Sent:* Friday, June 18, 2010 12:10 PM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script
    called ThriftHiveMetastore-remote that can be used to test out the
    thrift server independent of the CLI. As a quick test to narrow down
    the problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=1



    Cheers,

    Paul



    *From:* Pradeep Kamath
    *Sent:* Friday, June 18, 2010 11:26 AM
    *To:* hive-user@hadoop.apache.org
    *Subject:* RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly.
    By default I have hive set up to use a hive-site.xml [2] which
    connects to a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    *HADOOP_CLASSPATH=/*home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar*:/home/y/libexec/hive_client/lib/libthrift.jar*:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    *CLASSPATH=/*home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar*:/home/y/libexec/hive_client/lib/libthrift.jar*:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    *cat /home/y/libexec/hive_client/conf/hive-site.xml *

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup
    variables. -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!--
    resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    * <name>hive.metastore.uris</name>*

    * <value>thrift://localhost:9080</value>*

    <description>host and port for the thrift metastore server</description>

    </property>



    <property>

    * <name>hive.metastore.local</name>*

    * <value>false</value>*

    <description>controls whether to connect to remove metastore server
    or open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ------------------------------------------------------------------------

    *From:* Ning Zhang
    *Sent:* Friday, June 18, 2010 10:50 AM
    *To:* hive-user@hadoop.apache.org
    *Subject:* Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar
    in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here -- I want to open a jira
    for the first one since I am now able to reproduce it even with text
    format with builtin Serdes. Essentially this is a bug in the thrift
    code (not sure if it is in the client or server) since the same alter
    table statement works fine when the hive client does not use thrift.
    Here are the details:



    *cat create_dummy.sql*

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    *hive -f create_dummy.sql*

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated.
    Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
    override properties of core-default.xml, mapred-default.xml and
    hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds

    * *

    *hive -e "ALTER TABLE dummy add partition(datestamp = '20100602',
    srcid = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"*

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated.
    Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
    override properties of core-default.xml, mapred-default.xml and
    hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*



    *hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"*

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated.
    Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
    override properties of core-default.xml, mapred-default.xml and
    hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    *OK*

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? -- If it
    is the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ------------------------------------------------------------------------

    *From:* Pradeep Kamath
    *Sent:* Thursday, June 17, 2010 1:25 PM
    *To:* hive-user@hadoop.apache.org *Subject:* RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat
    'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat
    'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar *--config hive_conf_without_thrift* -e
    "ALTER TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]*hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"*

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated.
    Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
    override properties of core-default.xml, mapred-default.xml and
    hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an
    empty string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.<init>(Path.java:90)

    at org.apache.hadoop.fs.Path.<init>(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ------------------------------------------------------------------------

    *From:* Ashish Thusoo
    *Sent:* Thursday, June 17, 2010 11:15 AM
    *To:* hive-user@hadoop.apache.org *Subject:* RE: alter table add partition error



    hmm... Can you send the exact command and also the create table
    command for this table.



    Ashish



    ------------------------------------------------------------------------

    *From:* Pradeep Kamath
    *Sent:* Thursday, June 17, 2010 9:09 AM
    *To:* hive-user@hadoop.apache.org *Subject:* RE: alter table add partition error

    Sorry -- that was a cut-paste error -- I don't have the action part --
    so I am specifying key-value pairs. Since what I am trying to do seems
    like a basic operation, I am wondering if it's something to do with my
    Serde -- unfortunately the error I see gives me no clue of what could
    be wrong -- any help would be greatly appreciated!



    Thanks,

    Pradeep



    ------------------------------------------------------------------------

    *From:* yq he
    *Sent:* Wednesday, June 16, 2010 5:54 PM
    *To:* hive-user@hadoop.apache.org *Subject:* Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key
    `action` seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath
    wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add
    partition -- any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '10',action) location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated.
    Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
    override properties of core-default.xml, mapred-default.xml and
    hdfs-default.xml respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result*

    *FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask*

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    *Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result*

    * at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)*

    * at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)*

    * at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)*

    * at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*

    * ... 15 more*



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable







  • Paul Yang at Jun 23, 2010 at 1:06 am
    Hey Pradeep,

    Yep, that's on the right path :) Since you made a change to hive_metastore.thrift, you should run the thrift compiler by calling 'ant thriftif' within trunk/metastore. But before you can do that, the thrift compiler has to be in your path. JVS has a nice write up on how to do this:

    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code

    Also, I've set up a JIRA for this issue at:

    https://issues.apache.org/jira/browse/HIVE-1428

    Can you post your patch there? It makes it a lot easier to track issues. Thanks!

    -Paul

    From: Pradeep Kamath
    Sent: Tuesday, June 22, 2010 10:22 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Paul,
    I have attached a draft of a patch (incomplete). The problem I am facing is that to make this change I seem to need to change the thrift idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - wanted to check with you if the patch looks correct so far or if I am on the wrong path. Also if the direction is right, I need some guidance on generating the java files from the modified idl.

    Thanks,
    Pradeep

    Paul Yang wrote:
    Well, the section of code you are referring to is automatically generated by the thrift compiler so we can't modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead.

    Check out this thread:
    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html


    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 5:46 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Paul,
    Yes I don't mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() - to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the "unknown result" exception. I tried something along these lines and the alter table worked. Am I missing something?

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used.

    One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.

    Pradeep, are you interested in working on a patch for this problem?

    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    I tried debugging in code a little more. Here is what I found:
    The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist.
    I added a debug line in the following code:
    public Partition recv_get_partition() throws MetaException, TException
    {
    TMessage msg = iprot_.readMessageBegin();
    if (msg.type == TMessageType.EXCEPTION) {
    TApplicationException x = TApplicationException.read(iprot_);
    iprot_.readMessageEnd();
    throw x;
    }
    get_partition_result result = new get_partition_result();

    result.read(iprot_);
    System.err.println("XXX: result:" + result);
    iprot_.readMessageEnd();
    if (result.isSetSuccess()) {
    return result.success;
    }
    if (result.o1 != null) {
    throw result.o1;
    }
    throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
    }

    I also put a debug statements in the read() method:
    public void read(TProtocol iprot) throws TException {
    TField field;
    iprot.readStructBegin();
    System.err.println("XXX: Reading TProtocol object:");
    while (true)
    {
    field = iprot.readFieldBegin();
    System.err.println("XXX: field just read:" + field);
    if (field.type == TType.STOP) {
    break;

    I got
    XXX: Reading TProtocol object:
    XXX: field just read:<TField name:'' type:0 field-id:0>
    XXX: result:get_partition_result(success:null, o1:null)

    The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.

    Any pointers?

    TIA,
    Pradeep
    ________________________________
    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the standalone script works fine against the existing partition:
    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]

    However when I tried to add another partition with the hive cli using thrift:
    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    tail -30 /tmp/pradeepk/hive.log
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the thrift python libraries aren't in your path. run:

    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/

    before trying the remote command

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Traceback (most recent call last):
    File "./ThriftHiveMetastore-remote", line 11, in ?
    from thrift.transport import TTransport
    ImportError: No module named thrift.transport

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Pradeep,

    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running

    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Cheers,
    Paul

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde'
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable';

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 23, 2010 at 4:05 pm
    In the second step of

    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code



    "Use the approved version of thrift. This is currently
    thrift-instant-r790732, which you can obtain from
    http://instant.thrift-rpc.org"



    Looks like that site is down since it is not responding to requests - is
    there an alternate url I can try? Also, is the approved version still
    thrift-instant-r790732?



    ________________________________

    From: Paul Yang
    Sent: Tuesday, June 22, 2010 6:06 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hey Pradeep,



    Yep, that's on the right path :) Since you made a change to
    hive_metastore.thrift, you should run the thrift compiler by calling
    'ant thriftif' within trunk/metastore. But before you can do that, the
    thrift compiler has to be in your path. JVS has a nice write up on how
    to do this:



    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code



    Also, I've set up a JIRA for this issue at:



    https://issues.apache.org/jira/browse/HIVE-1428



    Can you post your patch there? It makes it a lot easier to track issues.
    Thanks!



    -Paul



    From: Pradeep Kamath
    Sent: Tuesday, June 22, 2010 10:22 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Paul,
    I have attached a draft of a patch (incomplete). The problem I am
    facing is that to make this change I seem to need to change the thrift
    idl (hive_metastore.thrift). Looking at the code structure it seems like
    the generate java files are checked in rather than generated through the
    build (is this so that the build does not depend on the thrift
    compiler?). I tried downloading and building thrift but hit issues in
    the make install - wanted to check with you if the patch looks correct
    so far or if I am on the wrong path. Also if the direction is right, I
    need some guidance on generating the java files from the modified idl.

    Thanks,
    Pradeep

    Paul Yang wrote:

    Well, the section of code you are referring to is automatically
    generated by the thrift compiler so we can't modify it. As the inability
    to return null is a limitation of Thrift, we can work around that issue
    by throwing an exception instead.



    Check out this thread:

    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html





    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 5:46 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Paul,

    Yes I don't mind working on a patch though I have no knowledge of how
    thrift code works. I was also confused by your suggestion for a fix. I
    thought the fix would be in
    ThriftHiveMetaStore.Client.recv_get_partition() - to first check
    if(result.o1 != null) throw result.o1, then to just return
    result.success and not throw the "unknown result" exception. I tried
    something along these lines and the alter table worked. Am I missing
    something?



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Ah okay. So what's going on is that getPartition() is supposed to return
    null if the specified partition doesn't exist. But because Thrift can't
    handle null return values, we are seeing this exception. This isn't a
    problem with a local metastore because Thrift isn't used.



    One solution to this problem would be modify ObjectStore.getPartition()
    to throw a NoSuchObjectException instead of returning null for
    non-existent partitions. Then Hive.getPartition() can catch the
    exception and return null to retain the original behavior.



    Pradeep, are you interested in working on a patch for this problem?



    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    I tried debugging in code a little more. Here is what I found:

    The code in ThriftHiveMetaStore eventually makes a call -
    get_partition() passing the partition key values for the partition I am
    trying to add using alter table. I assume this is to check that the
    partition doesn't already exist.

    I added a debug line in the following code:

    public Partition recv_get_partition() throws MetaException, TException

    {

    TMessage msg = iprot_.readMessageBegin();

    if (msg.type == TMessageType.EXCEPTION) {

    TApplicationException x = TApplicationException.read(iprot_);

    iprot_.readMessageEnd();

    throw x;

    }

    get_partition_result result = new get_partition_result();



    result.read(iprot_);

    System.err.println("XXX: result:" + result);

    iprot_.readMessageEnd();

    if (result.isSetSuccess()) {

    return result.success;

    }

    if (result.o1 != null) {

    throw result.o1;

    }

    throw new
    TApplicationException(TApplicationException.MISSING_RESULT,
    "get_partition failed: unknown result");

    }



    I also put a debug statements in the read() method:

    public void read(TProtocol iprot) throws TException {

    TField field;

    iprot.readStructBegin();

    System.err.println("XXX: Reading TProtocol object:");

    while (true)

    {

    field = iprot.readFieldBegin();

    System.err.println("XXX: field just read:" + field);

    if (field.type == TType.STOP) {

    break;



    I got

    XXX: Reading TProtocol object:

    XXX: field just read:<TField name:'' type:0 field-id:0>

    XXX: result:get_partition_result(success:null, o1:null)



    The field read in the thrift response message is of type "STOP" and with
    id of type SUCCESS. This seems right since there are no existing
    partitions. But the way the rest of the code handles this, results in
    the exception.



    Any pointers?



    TIA,

    Pradeep

    ________________________________

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the standalone script works fine against the existing
    partition:

    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
    reKeyTextOutputFormat', sortCols=[],
    inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
    .LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2
    100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    However when I tried to add another partition with the hive cli using
    thrift:

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    tail -30 /tmp/pradeepk/hive.log

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libex
    ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
    e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
    r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
    e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
    libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
    nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
    ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
    lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
    6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
    y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
    ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
    xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
    ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
    .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
    ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
    plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
    /libexec/hive_client/lib/zookeeper-3.2.2.jar:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libex
    ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
    e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
    r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
    e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
    libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
    nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
    ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
    lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
    6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
    y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
    ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
    xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
    ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
    .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
    ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
    plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
    /libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • John Sichi at Jun 23, 2010 at 4:49 pm
    I don't know who owns that server, but I happened to have an old copy of the tarball lying around, so I've put it somewhere you can download it:

    http://ansible.eigenbase.org/download/thrift-instant-r790732.tar.gz

    For the long run we'll figure out what's up with the latest thrift and how to upgrade.

    JVS
    ________________________________________
    From: Pradeep Kamath [pradeepk@yahoo-inc.com]
    Sent: Wednesday, June 23, 2010 9:03 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    In the second step of
    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code

    “Use the approved version of thrift. This is currently thrift-instant-r790732, which you can obtain from http://instant.thrift-rpc.org”

    Looks like that site is down since it is not responding to requests – is there an alternate url I can try? Also, is the approved version still thrift-instant-r790732?

    ________________________________
    From: Paul Yang
    Sent: Tuesday, June 22, 2010 6:06 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hey Pradeep,

    Yep, that’s on the right path :) Since you made a change to hive_metastore.thrift, you should run the thrift compiler by calling ‘ant thriftif’ within trunk/metastore. But before you can do that, the thrift compiler has to be in your path. JVS has a nice write up on how to do this:

    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code

    Also, I’ve set up a JIRA for this issue at:

    https://issues.apache.org/jira/browse/HIVE-1428

    Can you post your patch there? It makes it a lot easier to track issues. Thanks!

    -Paul

    From: Pradeep Kamath
    Sent: Tuesday, June 22, 2010 10:22 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Paul,
    I have attached a draft of a patch (incomplete). The problem I am facing is that to make this change I seem to need to change the thrift idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - wanted to check with you if the patch looks correct so far or if I am on the wrong path. Also if the direction is right, I need some guidance on generating the java files from the modified idl.

    Thanks,
    Pradeep

    Paul Yang wrote:
    Well, the section of code you are referring to is automatically generated by the thrift compiler so we can’t modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead.

    Check out this thread:
    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html


    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 5:46 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Paul,
    Yes I don’t mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() – to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the “unknown result” exception. I tried something along these lines and the alter table worked. Am I missing something?

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Ah okay. So what’s going on is that getPartition() is supposed to return null if the specified partition doesn’t exist. But because Thrift can’t handle null return values, we are seeing this exception. This isn’t a problem with a local metastore because Thrift isn’t used.

    One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.

    Pradeep, are you interested in working on a patch for this problem?

    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    I tried debugging in code a little more. Here is what I found:
    The code in ThriftHiveMetaStore eventually makes a call – get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn’t already exist.
    I added a debug line in the following code:
    public Partition recv_get_partition() throws MetaException, TException
    {
    TMessage msg = iprot_.readMessageBegin();
    if (msg.type == TMessageType.EXCEPTION) {
    TApplicationException x = TApplicationException.read(iprot_);
    iprot_.readMessageEnd();
    throw x;
    }
    get_partition_result result = new get_partition_result();

    result.read(iprot_);
    System.err.println("XXX: result:" + result);
    iprot_.readMessageEnd();
    if (result.isSetSuccess()) {
    return result.success;
    }
    if (result.o1 != null) {
    throw result.o1;
    }
    throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
    }

    I also put a debug statements in the read() method:
    public void read(TProtocol iprot) throws TException {
    TField field;
    iprot.readStructBegin();
    System.err.println("XXX: Reading TProtocol object:");
    while (true)
    {
    field = iprot.readFieldBegin();
    System.err.println("XXX: field just read:" + field);
    if (field.type == TType.STOP) {
    break;

    I got
    XXX: Reading TProtocol object:
    XXX: field just read:<TField name:'' type:0 field-id:0>
    XXX: result:get_partition_result(success:null, o1:null)

    The field read in the thrift response message is of type “STOP” and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.

    Any pointers?

    TIA,
    Pradeep
    ________________________________
    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the standalone script works fine against the existing partition:
    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]

    However when I tried to add another partition with the hive cli using thrift:
    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    tail -30 /tmp/pradeepk/hive.log
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Looks like the thrift python libraries aren’t in your path. run:

    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/

    before trying the remote command

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry, don’t know python – so couldn’t make sense out of the following error when I run the command you suggested:
    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
    Traceback (most recent call last):
    File "./ThriftHiveMetastore-remote", line 11, in ?
    from thrift.transport import TTransport
    ImportError: No module named thrift.transport

    Thanks,
    Pradeep

    ________________________________
    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Pradeep,

    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running

    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Cheers,
    Paul

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Hi Ning,
    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an “env” command printed just before the following line in execHiveCmd.sh
    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"

    There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.

    The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.

    Any pointers appreciated!

    Thanks,
    Pradeep

    [1]
    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:

    [2]
    cat /home/y/libexec/hive_client/conf/hive-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>

    <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
    <!-- that are implied by Hadoop setup variables. -->
    <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
    <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
    <!-- resource). -->

    <!-- Hive Execution Parameters -->
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9080</value>
    <description>host and port for the thrift metastore server</description>
    </property>

    <property>
    <name>hive.metastore.local</name>
    <value>false</value>
    <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>

    <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/pradeepk/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
    </property>


    </configuration>
    ________________________________
    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Pradeep,

    I ran the commands you provided and it succeeded with the expected behavior.

    One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?

    Thanks,
    Ning

    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:

    I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:

    cat create_dummy.sql
    CREATE external TABLE if not exists dummy (

    partition_name string
    ,partition_id int
    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format delimited
    stored as textfile
    location '/user/pradeepk/dummy';

    hive -f create_dummy.sql
    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
    OK
    Time taken: 0.627 seconds

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
    OK
    Time taken: 5.849 seconds

    Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details.

    Thanks,
    Pradeep


    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Here are the create table and alter table statements:
    CREATE external TABLE if not exists mytable (

    bc string
    ,src_spaceid string
    ,srcpvid string
    ,dstpvid string
    ,dst_spaceid string
    ,page_params map<string, string>
    ,clickinfo map<string, string>
    ,viewinfo array<map<string, string>>

    )
    PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
    row format serde 'com.yahoo.mySerde’
    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
    location '/user/pradeepk/mytable’;

    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    I get the following error:
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"

    However I get errors when I try to run a query:
    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
    at org.apache.hadoop.fs.Path.(Path.java:50)
    at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
    at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    Any help is much appreciated.

    Pradeep

    ________________________________
    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    hmm... Can you send the exact command and also the create table command for this table.

    Ashish

    ________________________________
    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error
    Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated!

    Thanks,
    Pradeep

    ________________________________
    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error

    Hi Pradeep,

    partition definition need to be key-value pairs. partition key `action` seems missed the value.

    Thanks
    Yongqiang
    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:
    Hi,
    I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated.

    Thanks,
    Pradeep

    == session output ===

    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;"
    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
    Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
    FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    [pradeepk@chargesize:~/dev/howl]

    == session output ===

    /tmp/pradeepk/hive.log has:
    2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
    at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
    ... 15 more

    The thrift server messages are:
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
  • Pradeep Kamath at Jun 25, 2010 at 8:40 pm
    I have posted a patch for this - not sure how to write unit tests for it
    - so patch doesn't have tests.



    ________________________________

    From: Pradeep Kamath
    Sent: Wednesday, June 23, 2010 9:04 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    In the second step of

    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code



    "Use the approved version of thrift. This is currently
    thrift-instant-r790732, which you can obtain from
    http://instant.thrift-rpc.org"



    Looks like that site is down since it is not responding to requests - is
    there an alternate url I can try? Also, is the approved version still
    thrift-instant-r790732?



    ________________________________

    From: Paul Yang
    Sent: Tuesday, June 22, 2010 6:06 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hey Pradeep,



    Yep, that's on the right path :) Since you made a change to
    hive_metastore.thrift, you should run the thrift compiler by calling
    'ant thriftif' within trunk/metastore. But before you can do that, the
    thrift compiler has to be in your path. JVS has a nice write up on how
    to do this:



    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code



    Also, I've set up a JIRA for this issue at:



    https://issues.apache.org/jira/browse/HIVE-1428



    Can you post your patch there? It makes it a lot easier to track issues.
    Thanks!



    -Paul



    From: Pradeep Kamath
    Sent: Tuesday, June 22, 2010 10:22 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Paul,
    I have attached a draft of a patch (incomplete). The problem I am
    facing is that to make this change I seem to need to change the thrift
    idl (hive_metastore.thrift). Looking at the code structure it seems like
    the generate java files are checked in rather than generated through the
    build (is this so that the build does not depend on the thrift
    compiler?). I tried downloading and building thrift but hit issues in
    the make install - wanted to check with you if the patch looks correct
    so far or if I am on the wrong path. Also if the direction is right, I
    need some guidance on generating the java files from the modified idl.

    Thanks,
    Pradeep

    Paul Yang wrote:

    Well, the section of code you are referring to is automatically
    generated by the thrift compiler so we can't modify it. As the inability
    to return null is a limitation of Thrift, we can work around that issue
    by throwing an exception instead.



    Check out this thread:

    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html





    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 5:46 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Paul,

    Yes I don't mind working on a patch though I have no knowledge of how
    thrift code works. I was also confused by your suggestion for a fix. I
    thought the fix would be in
    ThriftHiveMetaStore.Client.recv_get_partition() - to first check
    if(result.o1 != null) throw result.o1, then to just return
    result.success and not throw the "unknown result" exception. I tried
    something along these lines and the alter table worked. Am I missing
    something?



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Ah okay. So what's going on is that getPartition() is supposed to return
    null if the specified partition doesn't exist. But because Thrift can't
    handle null return values, we are seeing this exception. This isn't a
    problem with a local metastore because Thrift isn't used.



    One solution to this problem would be modify ObjectStore.getPartition()
    to throw a NoSuchObjectException instead of returning null for
    non-existent partitions. Then Hive.getPartition() can catch the
    exception and return null to retain the original behavior.



    Pradeep, are you interested in working on a patch for this problem?



    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    I tried debugging in code a little more. Here is what I found:

    The code in ThriftHiveMetaStore eventually makes a call -
    get_partition() passing the partition key values for the partition I am
    trying to add using alter table. I assume this is to check that the
    partition doesn't already exist.

    I added a debug line in the following code:

    public Partition recv_get_partition() throws MetaException, TException

    {

    TMessage msg = iprot_.readMessageBegin();

    if (msg.type == TMessageType.EXCEPTION) {

    TApplicationException x = TApplicationException.read(iprot_);

    iprot_.readMessageEnd();

    throw x;

    }

    get_partition_result result = new get_partition_result();



    result.read(iprot_);

    System.err.println("XXX: result:" + result);

    iprot_.readMessageEnd();

    if (result.isSetSuccess()) {

    return result.success;

    }

    if (result.o1 != null) {

    throw result.o1;

    }

    throw new
    TApplicationException(TApplicationException.MISSING_RESULT,
    "get_partition failed: unknown result");

    }



    I also put a debug statements in the read() method:

    public void read(TProtocol iprot) throws TException {

    TField field;

    iprot.readStructBegin();

    System.err.println("XXX: Reading TProtocol object:");

    while (true)

    {

    field = iprot.readFieldBegin();

    System.err.println("XXX: field just read:" + field);

    if (field.type == TType.STOP) {

    break;



    I got

    XXX: Reading TProtocol object:

    XXX: field just read:<TField name:'' type:0 field-id:0>

    XXX: result:get_partition_result(success:null, o1:null)



    The field read in the thrift response message is of type "STOP" and with
    id of type SUCCESS. This seems right since there are no existing
    partitions. But the way the rest of the code handles this, results in
    the exception.



    Any pointers?



    TIA,

    Pradeep

    ________________________________

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the standalone script works fine against the existing
    partition:

    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
    reKeyTextOutputFormat', sortCols=[],
    inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
    .LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2
    100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    However when I tried to add another partition with the hive cli using
    thrift:

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    tail -30 /tmp/pradeepk/hive.log

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libex
    ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
    e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
    r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
    e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
    libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
    nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
    ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
    lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
    6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
    y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
    ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
    xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
    ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
    .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
    ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
    plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
    /libexec/hive_client/lib/zookeeper-3.2.2.jar:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libex
    ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
    e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
    r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
    e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
    libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
    nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
    ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
    lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
    6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
    y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
    ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
    xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
    ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
    .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
    ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
    plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
    /libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable
  • Pradeep Kamath at Jul 1, 2010 at 4:18 pm
    I am having difficulty starting a thrift service instance in the junit
    test for this patch -
    https://issues.apache.org/jira/browse/HIVE-1428?focusedCommentId=1288361
    4&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpane
    l#action_12883614 - any pointers would be appreciated.



    Thanks,

    Pradeep



    ________________________________

    From: Pradeep Kamath
    Sent: Friday, June 25, 2010 1:37 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    I have posted a patch for this - not sure how to write unit tests for it
    - so patch doesn't have tests.



    ________________________________

    From: Pradeep Kamath
    Sent: Wednesday, June 23, 2010 9:04 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    In the second step of

    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code



    "Use the approved version of thrift. This is currently
    thrift-instant-r790732, which you can obtain from
    http://instant.thrift-rpc.org"



    Looks like that site is down since it is not responding to requests - is
    there an alternate url I can try? Also, is the approved version still
    thrift-instant-r790732?



    ________________________________

    From: Paul Yang
    Sent: Tuesday, June 22, 2010 6:06 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hey Pradeep,



    Yep, that's on the right path :) Since you made a change to
    hive_metastore.thrift, you should run the thrift compiler by calling
    'ant thriftif' within trunk/metastore. But before you can do that, the
    thrift compiler has to be in your path. JVS has a nice write up on how
    to do this:



    http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code



    Also, I've set up a JIRA for this issue at:



    https://issues.apache.org/jira/browse/HIVE-1428



    Can you post your patch there? It makes it a lot easier to track issues.
    Thanks!



    -Paul



    From: Pradeep Kamath
    Sent: Tuesday, June 22, 2010 10:22 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Paul,
    I have attached a draft of a patch (incomplete). The problem I am
    facing is that to make this change I seem to need to change the thrift
    idl (hive_metastore.thrift). Looking at the code structure it seems like
    the generate java files are checked in rather than generated through the
    build (is this so that the build does not depend on the thrift
    compiler?). I tried downloading and building thrift but hit issues in
    the make install - wanted to check with you if the patch looks correct
    so far or if I am on the wrong path. Also if the direction is right, I
    need some guidance on generating the java files from the modified idl.

    Thanks,
    Pradeep

    Paul Yang wrote:

    Well, the section of code you are referring to is automatically
    generated by the thrift compiler so we can't modify it. As the inability
    to return null is a limitation of Thrift, we can work around that issue
    by throwing an exception instead.



    Check out this thread:

    http://publists.facebook.com/pipermail/thrift/2007-September/000085.html





    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 5:46 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Paul,

    Yes I don't mind working on a patch though I have no knowledge of how
    thrift code works. I was also confused by your suggestion for a fix. I
    thought the fix would be in
    ThriftHiveMetaStore.Client.recv_get_partition() - to first check
    if(result.o1 != null) throw result.o1, then to just return
    result.success and not throw the "unknown result" exception. I tried
    something along these lines and the alter table worked. Am I missing
    something?



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Monday, June 21, 2010 2:35 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Ah okay. So what's going on is that getPartition() is supposed to return
    null if the specified partition doesn't exist. But because Thrift can't
    handle null return values, we are seeing this exception. This isn't a
    problem with a local metastore because Thrift isn't used.



    One solution to this problem would be modify ObjectStore.getPartition()
    to throw a NoSuchObjectException instead of returning null for
    non-existent partitions. Then Hive.getPartition() can catch the
    exception and return null to retain the original behavior.



    Pradeep, are you interested in working on a patch for this problem?



    From: Pradeep Kamath
    Sent: Monday, June 21, 2010 2:11 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    I tried debugging in code a little more. Here is what I found:

    The code in ThriftHiveMetaStore eventually makes a call -
    get_partition() passing the partition key values for the partition I am
    trying to add using alter table. I assume this is to check that the
    partition doesn't already exist.

    I added a debug line in the following code:

    public Partition recv_get_partition() throws MetaException, TException

    {

    TMessage msg = iprot_.readMessageBegin();

    if (msg.type == TMessageType.EXCEPTION) {

    TApplicationException x = TApplicationException.read(iprot_);

    iprot_.readMessageEnd();

    throw x;

    }

    get_partition_result result = new get_partition_result();



    result.read(iprot_);

    System.err.println("XXX: result:" + result);

    iprot_.readMessageEnd();

    if (result.isSetSuccess()) {

    return result.success;

    }

    if (result.o1 != null) {

    throw result.o1;

    }

    throw new
    TApplicationException(TApplicationException.MISSING_RESULT,
    "get_partition failed: unknown result");

    }



    I also put a debug statements in the read() method:

    public void read(TProtocol iprot) throws TException {

    TField field;

    iprot.readStructBegin();

    System.err.println("XXX: Reading TProtocol object:");

    while (true)

    {

    field = iprot.readFieldBegin();

    System.err.println("XXX: field just read:" + field);

    if (field.type == TType.STOP) {

    break;



    I got

    XXX: Reading TProtocol object:

    XXX: field just read:<TField name:'' type:0 field-id:0>

    XXX: result:get_partition_result(success:null, o1:null)



    The field read in the thrift response message is of type "STOP" and with
    id of type SUCCESS. This seems right since there are no existing
    partitions. But the way the rest of the code handles this, results in
    the exception.



    Any pointers?



    TIA,

    Pradeep

    ________________________________

    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 2:51 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the standalone script works fine against the existing
    partition:

    ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10

    Partition(parameters={'transient_lastDdlTime': '1276881277'},
    tableName='dummy', createTime=1276881277, lastAccessTime=0,
    values=['20100602', '100', 'view', '10'], dbName='default',
    sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
    reKeyTextOutputFormat', sortCols=[],
    inputFormat='org.apache.hadoop.mapred.TextInputFormat',
    cols=[FieldSchema(comment=None, type='string', name='partition_name'),
    FieldSchema(comment=None, type='int', name='partition_id')],
    compressed=False, bucketCols=[], numBuckets=-1, parameters={},
    serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
    .LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
    location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2
    100602/100/view/10'))

    [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]



    However when I tried to add another partition with the hive cli using
    thrift:

    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='click',testid='10') location
    '/user/pradeepk/dummy/20100602/100/click/10';"

    10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    tail -30 /tmp/pradeepk/hive.log

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



    2010-06-18 14:49:14,124 ERROR exec.DDLTask
    (SessionState.java:printError(277)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    2010-06-18 14:49:14,124 ERROR ql.Driver
    (SessionState.java:printError(277)) - FAILED: Execution Error, return
    code 1 from org.apache.hadoop.hive.ql.exec.DDLTask



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 2:19 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Looks like the thrift python libraries aren't in your path. run:



    export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/



    before trying the remote command



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 1:38 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Sorry, don't know python - so couldn't make sense out of the following
    error when I run the command you suggested:

    [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
    localhost:9080 get_partition_by_name default dummy
    datestamp=20100602/srcid=100/action=view/testid=10

    Traceback (most recent call last):

    File "./ThriftHiveMetastore-remote", line 11, in ?

    from thrift.transport import TTransport

    ImportError: No module named thrift.transport



    Thanks,

    Pradeep



    ________________________________

    From: Paul Yang
    Sent: Friday, June 18, 2010 12:10 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Pradeep,



    In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
    ThriftHiveMetastore-remote that can be used to test out the thrift
    server independent of the CLI. As a quick test to narrow down the
    problem, after the partition is created, can you try running



    ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
    default dummy datestamp=20100602/srcid=100/action=view/testid=10



    Cheers,

    Paul



    From: Pradeep Kamath
    Sent: Friday, June 18, 2010 11:26 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Hi Ning,

    I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
    from an "env" command printed just before the following line in
    execHiveCmd.sh

    exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"



    There is only one occurrence of libthrift.jar whose cksum matches the
    one from lib/libthrift.jar from a hive checkout build.



    The hive_config_without_thrift is a conf directory containing a
    hive-site.xml with jdo settings to connect to the database directly. By
    default I have hive set up to use a hive-site.xml [2] which connects to
    a thrift server running on localhost.



    Any pointers appreciated!



    Thanks,

    Pradeep



    [1]

    HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
    lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
    1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
    /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
    ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
    ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
    ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
    exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
    client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
    eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
    -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
    .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
    /y/libexec/hive_client/lib/derby.jar:/home/y/libex
    ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
    e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
    r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
    e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
    libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
    nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
    ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
    lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
    6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
    y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
    ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
    xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
    ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
    .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
    ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
    plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
    /libexec/hive_client/lib/zookeeper-3.2.2.jar:



    CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
    ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
    home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
    xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
    b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
    dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
    e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
    ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
    lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
    nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
    .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
    /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
    xec/hive_client/lib/derby.jar:/home/y/libex
    ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
    e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
    r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
    e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
    contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
    libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
    nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
    0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
    ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
    lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
    6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
    y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
    ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
    xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
    ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
    .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
    ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
    plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
    /libexec/hive_client/lib/zookeeper-3.2.2.jar:



    [2]

    cat /home/y/libexec/hive_client/conf/hive-site.xml

    <?xml version="1.0"?>

    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>



    <configuration>



    <!-- Hive Configuration can either be stored in this file or in the
    hadoop configuration files -->

    <!-- that are implied by Hadoop setup variables.
    -->

    <!-- Aside from Hadoop setup variables - this file is provided as a
    convenience so that Hive -->

    <!-- users do not have to edit hadoop configuration files (that may be
    managed as a centralized -->

    <!-- resource).
    -->



    <!-- Hive Execution Parameters -->

    <property>

    <name>hive.metastore.uris</name>

    <value>thrift://localhost:9080</value>

    <description>host and port for the thrift metastore
    server</description>

    </property>



    <property>

    <name>hive.metastore.local</name>

    <value>false</value>

    <description>controls whether to connect to remove metastore server or
    open a new metastore server in Hive Client JVM</description>

    </property>



    <property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/pradeepk/hive/warehouse</value>

    <description>location of default database for the
    warehouse</description>

    </property>





    </configuration>

    ________________________________

    From: Ning Zhang
    Sent: Friday, June 18, 2010 10:50 AM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Pradeep,



    I ran the commands you provided and it succeeded with the expected
    behavior.



    One possibility is that there are multiple versions of libthrift.jar in
    your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
    CLASSPATH so that no other libthrift.jar is there? What is in
    hive_config_without_thrift?



    Thanks,

    Ning



    On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:



    I think there are two separate issues here - I want to open a jira for
    the first one since I am now able to reproduce it even with text format
    with builtin Serdes. Essentially this is a bug in the thrift code (not
    sure if it is in the client or server) since the same alter table
    statement works fine when the hive client does not use thrift. Here are
    the details:



    cat create_dummy.sql

    CREATE external TABLE if not exists dummy (



    partition_name string

    ,partition_id int

    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format delimited

    stored as textfile

    location '/user/pradeepk/dummy';



    hive -f create_dummy.sql

    10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt

    OK

    Time taken: 0.627 seconds



    hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
    = '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask



    hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
    partition(datestamp = '20100602', srcid =
    '100',action='view',testid='10') location
    '/user/pradeepk/dummy/20100602/100/view/10';"

    10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt

    OK

    Time taken: 5.849 seconds



    Is there some thrift setting I am missing or is this a bug? - If it is
    the latter, I can open a jira with the above details.



    Thanks,

    Pradeep





    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 1:25 PM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    Here are the create table and alter table statements:

    CREATE external TABLE if not exists mytable (



    bc string

    ,src_spaceid string

    ,srcpvid string

    ,dstpvid string

    ,dst_spaceid string

    ,page_params map<string, string>

    ,clickinfo map<string, string>

    ,viewinfo array<map<string, string>>



    )

    PARTITIONED BY ( datestamp string, srcid string, action string, testid
    string )

    row format serde 'com.yahoo.mySerde'

    stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
    outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'

    location '/user/pradeepk/mytable';



    hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
    partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    I get the following error:

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    If I don't use thrift and use a hive-site.xml to directly talk to the
    db, the alter table seems to succeed:

    hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid =
    '19174',action='click',testid='NOTESTID') location
    '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"



    However I get errors when I try to run a query:

    [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
    hive_conf_without_thrift -e "select src_spaceid from
    ult_search_austria_ult where datestamp='20091101';"

    10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks is set to 0 since there's no reduce operator

    java.lang.IllegalArgumentException: Can not create a Path from an empty
    string

    at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

    at org.apache.hadoop.fs.Path.(Path.java:50)

    at
    org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)

    at
    org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
    .java:603)

    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)

    at
    org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)

    at
    org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)

    at
    org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)



    Any help is much appreciated.



    Pradeep



    ________________________________

    From: Ashish Thusoo
    Sent: Thursday, June 17, 2010 11:15 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error



    hmm... Can you send the exact command and also the create table command
    for this table.



    Ashish



    ________________________________

    From: Pradeep Kamath
    Sent: Thursday, June 17, 2010 9:09 AM
    To: hive-user@hadoop.apache.org
    Subject: RE: alter table add partition error

    Sorry - that was a cut-paste error - I don't have the action part - so I
    am specifying key-value pairs. Since what I am trying to do seems like a
    basic operation, I am wondering if it's something to do with my Serde -
    unfortunately the error I see gives me no clue of what could be wrong -
    any help would be greatly appreciated!



    Thanks,

    Pradeep



    ________________________________

    From: yq he
    Sent: Wednesday, June 16, 2010 5:54 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: alter table add partition error



    Hi Pradeep,



    partition definition need to be key-value pairs. partition key `action`
    seems missed the value.



    Thanks

    Yongqiang

    On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath wrote:

    Hi,

    I am trying to create an external table against already existing
    data in sequencefile format. However I have written a custom Serde to
    interpret the data. I am able to create the table fine but get the
    exception shown in the session output below when I try to add partition
    - any help would be greatly appreciated.



    Thanks,

    Pradeep



    == session output ===



    [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
    TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
    location '/user/pradeepk/mytable/20091101/10';"

    10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
    found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
    use core-site.xml, mapred-site.xml and hdfs-site.xml to override
    properties of core-default.xml, mapred-default.xml and hdfs-default.xml
    respectively

    Hive history
    file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt

    FAILED: Error in metadata: org.apache.thrift.TApplicationException:
    get_partition failed: unknown result

    FAILED: Execution Error, return code 1 from
    org.apache.hadoop.hive.ql.exec.DDLTask

    [pradeepk@chargesize:~/dev/howl]



    == session output ===



    /tmp/pradeepk/hive.log has:

    2010-06-16 17:09:00,841 ERROR exec.DDLTask
    (SessionState.java:printError(269)) - FAILED: Error in metadata:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    org.apache.hadoop.hive.ql.metadata.HiveException:
    org.apache.thrift.TApplicationException: get_partition failed: unknown
    result

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)

    at
    org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)

    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)

    at
    org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
    55)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)

    at
    org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)

    at
    org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
    a:39)

    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
    Impl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

    Caused by: org.apache.thrift.TApplicationException: get_partition
    failed: unknown result

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
    _partition(ThriftHiveMetastore.java:931)

    at
    org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
    ition(ThriftHiveMetastore.java:899)

    at
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
    taStoreClient.java:500)

    at
    org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)

    ... 15 more



    The thrift server messages are:

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
    db=default tbl=mytable

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
    with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

    10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
    called

    10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore

    10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
    db=default tbl=mytable

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedJun 17, '10 at 12:01p
activeJul 1, '10 at 4:18p
posts27
users8
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase