FAQ
Hello Everyone,
New to Lucene..

We currently roughly 100Gig of log files. We are needing to build a search
application that can return rows of data from the files and combine the
results?

Does Lucene index the content in the files?

Will it be able to find matching criteria say a date and then return the
next X number of lines?

The indexing piece, how does it work if I am looking for something that has
been added to the file since the indexing has taken place?

Thanks,
Chris

Search Discussions

  • Paul Smith at Oct 14, 2008 at 9:35 pm

    On 15/10/2008, at 7:37 AM, Chris Gilliam wrote:

    Hello Everyone,
    New to Lucene..

    We currently roughly 100Gig of log files. We are needing to build
    a search
    application that can return rows of data from the files and combine
    the
    results?

    Does Lucene index the content in the files?

    Will it be able to find matching criteria say a date and then
    return the
    next X number of lines?

    The indexing piece, how does it work if I am looking for something
    that has
    been added to the file since the indexing has taken place?
    I'm working on just such a thing over on my Apache Lab project called
    "Pinpoint":

    http://svn.apache.org/repos/asf/labs/pinpoint/trunk/

    It is designed to process log4j environments, soaking from network-
    based appenders, of by importing log files based on a pattern. It's
    relatively early days but the pinpoint-service layer recently ran
    through a production load test environment soaking 500-1000 events
    per/second no problems at all.

    The pinpoint-search layer provides an API to query a
    'context' (usually a days worth of logs) to find matching events, and
    then to be able to view the temporal context of specific matching
    events so you can see them inline with their related logs (either by
    the Thread that created the original matching event, +/-10 minutes,
    or by showing all logging events around that time +/- 10 minutes).

    Really good for production triage. I'm trying to work on a Web-based
    app that allows a dynamic sort-of-OLAP analysis of log files (charts,
    trends, etc).

    cheers,

    Paul


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedOct 14, '08 at 8:38p
activeOct 14, '08 at 9:35p
posts2
users2
websitelucene.apache.org

2 users in discussion

Paul Smith: 1 post Chris Gilliam: 1 post

People

Translate

site design / logo © 2022 Grokbase