FAQ
Hello list,

has anyone built a log-analyzer based on Lucene?
Our logs are so big that grep takes more hours to do what I want it to do.
I'm sure Lucene would solve it.

Thanks in advance
paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Steven A Rowe at Jan 13, 2011 at 1:40 pm
    Hi Paul,

    I saw this yesterday, but haven't tried it myself:

    http://karussell.wordpress.com/2010/10/27/feeding-solr-with-its-own-logs/

    The author has a project called "Sogger" - Solr + Logger? - that can read various forms of logs.

    Steve
    -----Original Message-----
    From: Paul Libbrecht
    Sent: Thursday, January 13, 2011 7:54 AM
    To: java-user@lucene.apache.org
    Subject: lucene-based log searcher?


    Hello list,

    has anyone built a log-analyzer based on Lucene?
    Our logs are so big that grep takes more hours to do what I want it to do.
    I'm sure Lucene would solve it.

    Thanks in advance
    paul

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Umesh Prasad at Jan 13, 2011 at 2:00 pm
    Checkout http://code.google.com/p/lucene-log4j/

    On Thu, Jan 13, 2011 at 7:09 PM, Steven A Rowe wrote:

    Hi Paul,

    I saw this yesterday, but haven't tried it myself:

    http://karussell.wordpress.com/2010/10/27/feeding-solr-with-its-own-logs/

    The author has a project called "Sogger" - Solr + Logger? - that can read
    various forms of logs.

    Steve
    -----Original Message-----
    From: Paul Libbrecht
    Sent: Thursday, January 13, 2011 7:54 AM
    To: java-user@lucene.apache.org
    Subject: lucene-based log searcher?


    Hello list,

    has anyone built a log-analyzer based on Lucene?
    Our logs are so big that grep takes more hours to do what I want it to do.
    I'm sure Lucene would solve it.

    Thanks in advance
    paul

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    ---
    Thanks & Regards
    Umesh Prasad
  • Benzion G at Jan 13, 2011 at 5:15 pm
    Hi,

    I almost finished these days to write such a tool.

    It reads via FTPClient the log files and adds to Lucene index.

    Via small Web application you can see/filter/sort the log messages.
    --
    View this message in context: http://lucene.472066.n3.nabble.com/lucene-based-log-searcher-tp2247969p2249215.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Chensheng at Jan 14, 2011 at 6:06 am
    Hi, I have a similar case.
    If I only index some fixed fields with single-word/number, instead of full-text search,
    is Lucene any better than a relational database?


    ------------------ Original ------------------
    From: "Benzion G"<benzionk@yahoo.com>;
    Date: Fri, Jan 14, 2011 01:15 AM
    To: "java-user"<java-user@lucene.apache.org>;

    Subject: Re: lucene-based log searcher?



    Hi,

    I almost finished these days to write such a tool.

    It reads via FTPClient the log files and adds to Lucene index.

    Via small Web application you can see/filter/sort the log messages.
    --
    View this message in context: http://lucene.472066.n3.nabble.com/lucene-based-log-searcher-tp2247969p2249215.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Benzion G at Jan 14, 2011 at 6:37 am
    If you can define one searchable word for the whole log message and index
    this word only maybe you can use DB instead.
    I apply Analyzer to every and whole log message text - and then I can
    quickly find all messages, that have e.g. "NullPointerException" or "user
    not found" text or anything else.

    In DB using select with "LIKE '%something%'" will do full table scan.
    --
    View this message in context: http://lucene.472066.n3.nabble.com/lucene-based-log-searcher-tp2247969p2253738.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Lukáš Vlček at Jan 14, 2011 at 8:18 am
    Hi,

    I haven't tried that myself so far but may be you can check logstash as well
    (http://www.loggly.com/2010/11/do-you-have-a-logstash/).

    Regards,
    Lukas
    On Fri, Jan 14, 2011 at 7:36 AM, Benzion G wrote:


    If you can define one searchable word for the whole log message and index
    this word only maybe you can use DB instead.
    I apply Analyzer to every and whole log message text - and then I can
    quickly find all messages, that have e.g. "NullPointerException" or "user
    not found" text or anything else.

    In DB using select with "LIKE '%something%'" will do full table scan.
    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/lucene-based-log-searcher-tp2247969p2253738.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Matthieu Huin at Jan 14, 2011 at 9:39 am
    Hello,

    I've been working on logs and metadata and I've been working on this
    specific problem for quite a while.

    if you have a constant, predefined number of fields associated to each
    log, a relational database might perform just fine and offer you more
    possibilities regarding log analysis (statistics, aggregation, min/max
    ... ). As a side note, you might not be aware that PostgreSQL ( > 8.3 )
    comes with built-in full text search capabilities, so you don't have to
    sacrifice the functionality. A column-based database would probably be
    also worth trying.

    On the other hand, if fields can vary in type and number for each log,
    the relational database model will perform very badly : if you normalize
    a lot you'll end up with a compact database schema that takes a lot of
    time to query ( and queries can become hellishly hard to write); if you
    don't you're losing most of what a database would bring you in the first
    place. This is a very close problem to the one discussed here :
    http://www.pui.ch/phred/archives/2005/04/tags-database-schemas.html this
    link is worth a read if you want to go on with databases.

    Do not forget also that unless you use a column-based database, you
    might probably want to use indices on your tables in order to speed up
    some lookups ... But it will come at a hefty price at insertion time
    after you reach a certain amount of indexed logs.

    Concerning my project (where we fall in the second case), current tests
    with lucene are very encouraging. Sure, we lose some analytics
    capabilities and the hard disk footprint is rather huge, but the query
    response time, be it from full-text searches or complex queries on
    specific fields, is outstanding.


    Of course, in the end, it all depends on what you need to do. Just my
    personal experience here ...

    Regards,

    Matthieu Huin

    Le 14/01/2011 06:47, chensheng a écrit :
    Hi, I have a similar case.
    If I only index some fixed fields with single-word/number, instead of full-text search,
    is Lucene any better than a relational database?


    ------------------ Original ------------------
    From: "Benzion G"<benzionk@yahoo.com>;
    Date: Fri, Jan 14, 2011 01:15 AM
    To: "java-user"<java-user@lucene.apache.org>;

    Subject: Re: lucene-based log searcher?



    Hi,

    I almost finished these days to write such a tool.

    It reads via FTPClient the log files and adds to Lucene index.

    Via small Web application you can see/filter/sort the log messages.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJan 13, '11 at 12:54p
activeJan 14, '11 at 9:39a
posts8
users7
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase