Grokbase Groups Pig user October 2008
FAQ
My latest stuff looks at apache logs, aggregates to txt files, then I have a simple perl script that +='s into mysql tables. A few thoughts

* Would sure be nice if I could just STORE my aggregations into any jdbc-friendly database, like mysql, instead of text files. Anyone work on such a thing? I could do the simple case(s), but would need some help with more complicated ones.

* How about a MOVE function? Would be nice to move files once done processing them.

* I have yet to get into hadoop, but it would be nice to have an incoming directory, then a processed directory. Really, I would like to have a daemon that watches a directory that churns through logs exactly once. That's kind of how hadoop works, right?
* How about a LOAD function that can read from S3, or maybe the MOVE could move from S3 to local storage, or vice versa?Thoughts?

Thanks,
Earl


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 6 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedOct 18, '08 at 5:18a
activeOct 21, '08 at 7:24a
posts6
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase