Grokbase Groups Pig user October 2008
My latest stuff looks at apache logs, aggregates to txt files, then I have a simple perl script that +='s into mysql tables. A few thoughts

* Would sure be nice if I could just STORE my aggregations into any jdbc-friendly database, like mysql, instead of text files. Anyone work on such a thing? I could do the simple case(s), but would need some help with more complicated ones.

* How about a MOVE function? Would be nice to move files once done processing them.

* I have yet to get into hadoop, but it would be nice to have an incoming directory, then a processed directory. Really, I would like to have a daemon that watches a directory that churns through logs exactly once. That's kind of how hadoop works, right?
* How about a LOAD function that can read from S3, or maybe the MOVE could move from S3 to local storage, or vice versa?Thoughts?


Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 6 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedOct 18, '08 at 5:18a
activeOct 21, '08 at 7:24a



site design / logo © 2021 Grokbase