I have a setup where logs are periodically bundled up and dumped into
hadoop dfs as large sequence file.
It works fine for all my map reduce jobs.
Now i need to handle adhoc queries for pulling out logs based on user
and time range.
I really dont need a full indexer (like lucene) for this purpose.
My first thought is to run a periodic mapreduce to generate a large
text file sorted by user id.
The text file will have (sequence file name, offset) to retrieve the logs ....
I am guessing many of you ran into similar requirements... Any
suggestions on doing this better?