[Hive-user] can hive currently support appending data to the table?

Hive does not support appending to exist partition ot table yet. For non bucketed table a workaround is to insert the new data into a temp table and move the files to the destination by hand.

Of course you are welcome to add this to the project.
Thanks,
Ning
------
Sent from my blackberry

________________________________
From: luocanrao <[email protected]>
To: [email protected] <[email protected]>
Sent: Fri May 07 07:22:38 2010
Subject: can hive currently support appending data to the table?

I read the hive wiki, it says:
The output of each of the select statements is written to the chosen table (or partition).
Currently the OVERWRITE keyword is mandatory and implies that the contents of the chosen table or partition are replaced with the output of corresponding select statement

can hive currently support appending data to the table?

We can also add more partitions to a table (or a sub partition to a
partition) to simulate appending to table/partition.

There is no real efficiency loss except that we will create a new
directory/file per append.

Zheng

Thanks for your reply, can you give me some example about adding a sub partition to a Partition?
How can I achieve that creating a new directory/file per append
<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p>&nbsp;</o:p>
-----邮件原件-----
发件人: Zheng Shao
发送时间: 2010年5月8日 2:29
收件人: [email protected]
主题: Re: can hive currently support appending data to the table?
<o:p>&nbsp;</o:p>
We can also add more partitions to a table (or a sub partition to a
partition) to simulate appending to table/partition.
<o:p>&nbsp;</o:p>
There is no real efficiency loss except that we will create a new directory/file per append.
<o:p>&nbsp;</o:p>
Zheng
<o:p>&nbsp;</o:p>
On Fri, May 7, 2010 at 7:51 AM, Ning Zhang &lt;[email protected]&gt; wrote:
&gt; Hive does not support appending to exist partition ot table yet. For
&gt; non bucketed table a workaround is to insert the new data into a temp
&gt; table and move the files to the destination by hand.
&gt;<o:p>&nbsp;</o:p>
&gt; Of course you are welcome to add this to the project.
&gt; Thanks,
&gt; Ning
&gt; ------
&gt; Sent from my blackberry
&gt;<o:p>&nbsp;</o:p>
&gt; ________________________________
&gt; From: luocanrao &lt;[email protected]&gt;
&gt; To: hive-us[email protected] &lt;[email protected]&gt;
&gt; Sent: Fri May 07 07:22:38 2010
&gt; Subject: can hive currently support appending data to the table?
&gt;<o:p>&nbsp;</o:p>
&gt; I read the hive wiki, it says:
&gt;<o:p>&nbsp;</o:p>
&gt; The output of each of the select statements is written to the chosen
&gt; table (or partition).
&gt;<o:p>&nbsp;</o:p>
&gt; Currently the OVERWRITE keyword is mandatory and implies that the
&gt; contents of the chosen table or partition are replaced with the output
&gt; of corresponding select statement
&gt;<o:p>&nbsp;</o:p>
&gt;<o:p>&nbsp;</o:p>
&gt;<o:p>&nbsp;</o:p>
&gt; can hive currently support appending data to the table?
<o:p>&nbsp;</o:p>
<o:p>&nbsp;</o:p>
<o:p>&nbsp;</o:p>

Please refer to the Hive Language Manual:
http://wiki.apache.org/hadoop/Hive/LanguageManual/DML#Inserting_data_into_Hive_Tables_from_queries

Loading data into a sub-partition of a table is the same as loading data
into regular table with the partition predicate specified. For example:

LOAD DATA LOCAL INPATH '/path/to/data/file' OVERWRITE INTO TABLE table_name
partition *( part_column = part_column_value)*

More information on manipulating partitions as a whole is available here:
http://wiki.apache.org/hadoop/Hive/LanguageManual/DDL#Add_Partitions

Hope this helps.
-Arvind