FAQ
I am getting a "Too Many Open Files" Exception. I've read the FAQ about
lowering the merge factor (currently set to 25), issuing a ulimit -n
<number>, etc... but I am still getting the "Too Many Open Files"
Exception (yes... I'm making sure I close all writer/searchers/reader
and I only have one open at a time).



Here is my situation:



I have one application dedicated to building the index on one JVM
(server 1). I have several other applications doing the searching in
another JVM (server 2). I have the merge merge docs set to 999999, max
buffered docs = 3000, and max merge factor = 25. Server 2 is all set to
default. I am getting the "Too Many Open Files" Exception on server
2... The index in question has 576,239 documents, 48 fields, and
6,521,618 terms (actually... this just turned into a two part question).




The first part would be why am I getting the "Too Many Open Files"
Exception.



The second part is: I have a total of 24 different indices. Each index
is dedicated to one specific part of the application (ie: an index for
customers, index for products, index for media location, etc). Each
index has its own searcher/reader (and only one searcher/reader per
index). After server 1 finishes it's scheduled job to update the index,
it notifies server 2 to renew the appropriate searcher/reader. Question
is should (could) I merge all 24 indices into one big index? What is
the optimal thing to do (the indices range from 5MB - 900MB... most of
the are 30MB-50MB).



I have two indices that are "large", one being 700MB and the other being
900MB. Optimizing these two indices takes a long time, so I only
optimize it once (last scheduled job of the night). The 700MB index is
scheduled to update every 4 hours, while the 900MB index is schedule to
update every 30 minutes. Should I optimize more often throughout the
day (changes varies)?



I know this email is all over the place... so any feedback is
appreciated.



Thanks,



Van

Search Discussions

  • Chris Hostetter at Jul 4, 2007 at 4:41 am
    : I am getting a "Too Many Open Files" Exception. I've read the FAQ about
    : lowering the merge factor (currently set to 25), issuing a ulimit -n
    : <number>, etc... but I am still getting the "Too Many Open Files"
    : Exception (yes... I'm making sure I close all writer/searchers/reader
    : and I only have one open at a time).

    so ... what is your ulimit set to? how many files are in your index
    directory? have you tried running lsof on your processes to see all the
    files you have open?

    the anoying thing about "Too Many Open Files" type errors, is that the
    method/class/library that encounters the problem isn't neccessarily the
    guy that caused hte problem ... it's just the straw that breaks the camels
    back. it could be that you aren't closing some network socket (sockets
    are files too) and it may not have anything to do with lucene.

    or it could just be that your index has a ton of files because of hte
    number of segments and the number of fields, and between it, and all the
    jars your app loads it really does just use 1 too many files for your
    ulimit.



    -Hoss


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Van Nguyen at Jul 5, 2007 at 6:24 pm
    : so ... what is your ulimit set to?
    Issuing a "limit descriptors", I see that I have it set to 1024

    : how many files are in your index directory?
    In the directory that I'm getting this particular error: 3
    I have 24 different index directories... I think the most I saw at that
    particular time in any one index was 20

    : have you tried running lsof on your processes to see all the
    : files you have open?
    I'm not too familiar with this command. Do I need to issue this command
    for the processes I'm using (should it be the command: apache, jboss,
    etc or process id?) or for the whole system? If I do issue the command,
    does each line it comes back with considered an "open file"?

    : or it could just be that your index has a ton of files because of the
    : number of segments and the number of fields, and between it, and all
    the
    : jars your app loads it really does just use 1 too many files for your
    : ulimit.
    How do you calculate this? I found this formula: (1 + mergeFacotr) *
    FilesPerSegment... but am unsure how to apply it to my indices. Keep in
    mind I have 24 different indices. Do I apply this formula per index?

    Thanks

    Van

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Chris Hostetter at Jul 9, 2007 at 2:41 am
    : Issuing a "limit descriptors", I see that I have it set to 1024

    : In the directory that I'm getting this particular error: 3
    : I have 24 different index directories... I think the most I saw at that
    : particular time in any one index was 20

    as i said ... it doesn't matter where in the code your encounter the
    error, or what directory the line of code that throws the errors happens
    to be operating on, what matters is the totla number of file handles in
    use by the process.

    since you didn't tel us what the total number if files for all of your 24
    index is, let's assume it 15 ... 15*24 is 360 ... assuming you open new
    readers before closing the olde ones you might have as many as 720 files
    open at once just considering the searching aspects of the index files ...
    thta doesn't even count filehanldes open involved in writting to the index
    directories.

    and we're still just talking about index files, there's all the jars/class
    files your process has to open, plus any network connections. Solr for
    example, on startup while sitting idle and not counting the files in the
    index directory has 64 files open ... thta's just jars and basic shared
    libraries the JVM needs.

    it wouldn't supprise me if 1024 was way to low for an application
    maintaining 24 different indexes and dealing with concurrent network
    connections.

    : : have you tried running lsof on your processes to see all the
    : : files you have open?
    : I'm not too familiar with this command. Do I need to issue this command

    lsof -p [pid] | wc -l

    The number that comes back is the one that will cuase you problems once it
    approaches your open files limit.




    -Hoss


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Van Nguyen at Jul 5, 2007 at 9:24 pm
    Ok... after spending time looking at the code... I see that a method is
    not closing a TokenStream in one of the classes (a class that is
    instantiated quite often) - I would imagine this could quite possibly be
    the culprit?

    Van

    -----Original Message-----
    From: Chris Hostetter
    Sent: Tuesday, July 03, 2007 9:41 PM
    To: java-user@lucene.apache.org
    Subject: Re: Too Many Open files Exception

    : I am getting a "Too Many Open Files" Exception. I've read the FAQ
    about
    : lowering the merge factor (currently set to 25), issuing a ulimit -n
    : <number>, etc... but I am still getting the "Too Many Open Files"
    : Exception (yes... I'm making sure I close all writer/searchers/reader
    : and I only have one open at a time).

    so ... what is your ulimit set to? how many files are in your index
    directory? have you tried running lsof on your processes to see all the
    files you have open?

    the anoying thing about "Too Many Open Files" type errors, is that the
    method/class/library that encounters the problem isn't neccessarily the
    guy that caused hte problem ... it's just the straw that breaks the
    camels
    back. it could be that you aren't closing some network socket (sockets
    are files too) and it may not have anything to do with lucene.

    or it could just be that your index has a ton of files because of hte
    number of segments and the number of fields, and between it, and all the
    jars your app loads it really does just use 1 too many files for your
    ulimit.



    -Hoss


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Chris Hostetter at Jul 9, 2007 at 2:48 am
    : Ok... after spending time looking at the code... I see that a method is
    : not closing a TokenStream in one of the classes (a class that is
    : instantiated quite often) - I would imagine this could quite possibly be
    : the culprit?

    can you be more specific about the code in question?

    I'm not sure what class is responsible for closing TokenStreams when
    documents get added ... but the only way i can imagine this might possible
    be causing you problems is if you are using Field instances built from a
    Reader (because failure to close the TokenStream *might* mean failure to
    close the Reader -- but i'm just guessing there)

    if you aren't constructing your Document's using Fields built with
    Readers, then even if there is a bug with closing TokenStreams it isn't
    causing your problem.




    -Hoss


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJul 3, '07 at 9:56p
activeJul 9, '07 at 2:48a
posts6
users2
websitelucene.apache.org

2 users in discussion

Van Nguyen: 3 posts Chris Hostetter: 3 posts

People

Translate

site design / logo © 2022 Grokbase