[Hive-user] Can Hive 0.7 Rebuild partitions ?

On May 19, 2011, at 2:25 AM, Jasper Knulst wrote:

Hi,

I have a partitioned external table on Hive 0.7. New subfolders are regularly added to the base table HDFS folder.
I now have to perform this scan myself and let an external tool create new partitions by generating and firing ALTER TABLE ADD PARTITION commands.

Is there an easier way to have hive scan the base table folder to see if there are any new partitions around? Something like REBUILD PARTITIONS perhaps??




Couldn't find anything similar on the Hive/LanguageManual/DDL

--
Kind Regards



Jasper

On Thu, May 19, 2011 at 2:25 AM, Jasper Knulst wrote:

Hi,

I have a partitioned external table on Hive 0.7. New subfolders are
regularly added to the base table HDFS folder.
I now have to perform this scan myself and let an external tool create new
partitions by generating and firing ALTER TABLE ADD PARTITION commands.

Is there an easier way to have hive scan the base table folder to see if
there are any new partitions around? Something like REBUILD PARTITIONS
perhaps??




Couldn't find anything similar on the Hive/LanguageManual/DDL

--
Kind Regards



Jasper

On Thu, May 19, 2011 at 2:01 PM, Tim Spence wrote:

Is this functionality handled by ALTER TABLE [name] RECOVER PARTITIONS?
Take a look at this presentation for context:
http://www.slideshare.net/AmazonWebServices/aws-office-hours-amazon-elastic-mapreduce

Best of luck,
Tim

On Thu, May 19, 2011 at 3:23 PM, Igor Tatarinov wrote:

That's Amazon's extension to Hive and it's really handy.

On Thu, May 19, 2011 at 15:52, Roberto Congiu wrote:
I agree it's useful, especially for external tables, that may be loaded by
an external process that may 'forget' to issue a ADD PARTITION.
A 'sync partitions' feature to sync metadata with directories would be
really handy.


--
Roberto Congiu -Data Engineer - OpenX
20 E Del Mar blvd, Pasadena, CA