[Hive-user] Deleted input files after load

On Fri, Jan 22, 2010 at 10:44 AM, Shiva wrote:

Hi,
For the first time I used Hive to load couple of word count data input
files into tables with and without OVERWRITE. Both the times the input file
in HDFS got deleted. Is that a expected behavior? And couldn't find any
definitive answer on the Hive wiki.

hive> LOAD DATA INPATH '/user/vmplanet/output/part-00000' OVERWRITE
INTO TABLE t_word_count;

Env.: Using Hadoop 0.20.1 and latest Hive on Ubuntu 9.10 running in VMware.

Thanks,
Shiva

On Fri, Jan 22, 2010 at 10:51 AM, Bill Graham wrote:
Hive doesn't delete the files upon load, it moves them to a location under
the Hive warehouse directory. Try looking under
/user/hive/warehouse/t_word_count.

On Fri, Jan 22, 2010 at 11:05 AM, Zheng Shao wrote:

If you want the files to stay there, you can try "CREATE EXTERNAL
TABLE" with a location (instead of create table + load)

Zheng
input
file


--
Yours,
Zheng

On 1/22/10 3:09 PM, Shiva wrote:
I can try that. Here is what I am trying to do.

Load some fact data from a file (say weblogs moved to HDFS after some
cleanup and transformation) and then do summarization at daily or weekly
level. In that case, I would like to create a one fact table which get
loaded with daily data and bring dimensional data from MySQL to perform
summarization.

I appreciate any input on this technique, performance and how I can get
dimensional data to Hive (from MySQL -> to file -> HDFS -> Hive).
Thanks,
Shiva

On Fri, Jan 22, 2010 at 1:53 PM, Eric Sammer wrote:
Shiva:

This is very common. I use Hive to do something very similar.

Cloudera has a tool called sqoop that will "export" MySQL tables to
files on HDFS that Hive can understand. Once there, you can easily join
the data in your Hive queries.

http://www.cloudera.com/hadoop-sqoop

Sqoop is smarter than just doing an export to a local file system and
then copying to HDFS and should save you a fair amount of time and
effort. Check out the link.

Hope this helps.
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

On 1/25/10 11:47 AM, Shiva wrote:
Eric,
Thanks for the details. I took a quick look at the link and it seems
like a tool that I would help me here. Do I need to download whole
Cloudera's Distribution for Hadoop <http://www.cloudera.com/hadoop> just
to get sqoop? I already have Hadoop, Hive and Pig setup completed.
I appreciate your input,
Shiva

On Mon, Jan 25, 2010 at 9:04 AM, Eric Sammer wrote:
Shiva:

As far as I know, sqoop is only available bundled with Cloudera's
distribution (CDH). That said, you may be able to download the tarball
version of CDH and just pull sqoop out of it. I've never tried this,
though, and I don't know if Sqoop depends on anything specific in CDH.
Someone from Cloudera might be able to fill in the blanks here.

For what it's worth, CDH2 is a great distribution with some nice
packaging and configuration layout on top of the ASF distro along with
some helpful patches. It's worth checking out.

The tarball version of CDH is available at:
http://archive.cloudera.com/cdh/testing/hadoop-0.20.1+152.tar.gz

Best of luck!
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

On Fri, Jan 22, 2010 at 10:51 AM, Bill Graham wrote:

Hive doesn't delete the files upon load, it moves them to a location under
the Hive warehouse directory. Try looking under
/user/hive/warehouse/t_word_count.

On Fri, Jan 22, 2010 at 2:58 PM, Shiva wrote:
Yeap. I found them there.  Why would Hive move it?
Cheers,
Shiva