FAQ
Hello,

We keep hitting the too many open files exception. Looking at lsof we have
a lot (several thousand) of entries like this:

java 19339 root 1619u sock 0,7 0t0
682291383 can't identify protocol


However, netstat -a doesn't show any of these.

Can anyone suggest a way to diagnose what these socket entries are? Happy
to post any more information as needed.


Cheers,
Colin


--
Colin Howe
@colinhowe

VP of Engineering
Conversocial Ltd
conversocial.com

Search Discussions

  • Markus Jelsma at Mar 14, 2012 at 10:18 am
    Are you running trunk and have auto-commit enabled? Then disable
    auto-commit. Even if you increase ulimits it will continue to swallow
    all available file descriptors.
    On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe wrote:
    Hello,

    We keep hitting the too many open files exception. Looking at lsof we
    have
    a lot (several thousand) of entries like this:

    java 19339 root 1619u sock 0,7
    0t0
    682291383 can't identify protocol


    However, netstat -a doesn't show any of these.

    Can anyone suggest a way to diagnose what these socket entries are?
    Happy
    to post any more information as needed.


    Cheers,
    Colin
    --
    Markus Jelsma - CTO - Openindex
    http://www.linkedin.com/in/markus17
    050-8536600 / 06-50258350
  • Colin Howe at Mar 14, 2012 at 10:23 am
    Currently using 3.4.0. We have autocommit enabled but we manually do
    commits every 100 documents anyway... I can turn it off if you think that
    might help.


    Cheers,
    Colin


    On Wed, Mar 14, 2012 at 10:24 AM, Markus Jelsma
    wrote:
    Are you running trunk and have auto-commit enabled? Then disable
    auto-commit. Even if you increase ulimits it will continue to swallow all
    available file descriptors.

    On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe wrote:

    Hello,

    We keep hitting the too many open files exception. Looking at lsof we have
    a lot (several thousand) of entries like this:

    java 19339 root 1619u sock 0,7 0t0
    682291383 can't identify protocol


    However, netstat -a doesn't show any of these.

    Can anyone suggest a way to diagnose what these socket entries are? Happy
    to post any more information as needed.


    Cheers,
    Colin
    --
    Markus Jelsma - CTO - Openindex
    http://www.linkedin.com/in/**markus17<http://www.linkedin.com/in/markus17>
    050-8536600 / 06-50258350


    --
    Colin Howe
    @colinhowe

    VP of Engineering
    Conversocial Ltd
    conversocial.com
  • Michael Kuhlmann at Mar 14, 2012 at 10:27 am
    I had the same problem, without auto-commit.

    I never really found out what exactly the reason was, but I think it was
    because commits were triggered before a previous commit had the chance
    to finish.

    We now commit after every minute or 1000 (quite large) documents,
    whatever comes first. And we never optimize. We haven't had this
    exceptions for months now.

    Good luck!
    -Kuli

    Am 14.03.2012 11:22, schrieb Colin Howe:
    Currently using 3.4.0. We have autocommit enabled but we manually do
    commits every 100 documents anyway... I can turn it off if you think that
    might help.


    Cheers,
    Colin


    On Wed, Mar 14, 2012 at 10:24 AM, Markus Jelsma
    wrote:
    Are you running trunk and have auto-commit enabled? Then disable
    auto-commit. Even if you increase ulimits it will continue to swallow all
    available file descriptors.


    On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe<colin@conversocial.com>
    wrote:
    Hello,

    We keep hitting the too many open files exception. Looking at lsof we have
    a lot (several thousand) of entries like this:

    java 19339 root 1619u sock 0,7 0t0
    682291383 can't identify protocol


    However, netstat -a doesn't show any of these.

    Can anyone suggest a way to diagnose what these socket entries are? Happy
    to post any more information as needed.


    Cheers,
    Colin
    --
    Markus Jelsma - CTO - Openindex
    http://www.linkedin.com/in/**markus17<http://www.linkedin.com/in/markus17>
    050-8536600 / 06-50258350
  • Colin Howe at Mar 14, 2012 at 11:08 am
    After some more digging around I discovered that there was a bug reported
    in jetty 6: https://jira.codehaus.org/browse/JETTY-1458

    This prompted me to upgrade to Jetty 7 and things look a bit more stable
    now :)


    On Wed, Mar 14, 2012 at 10:26 AM, Michael Kuhlmann wrote:

    I had the same problem, without auto-commit.

    I never really found out what exactly the reason was, but I think it was
    because commits were triggered before a previous commit had the chance to
    finish.

    We now commit after every minute or 1000 (quite large) documents, whatever
    comes first. And we never optimize. We haven't had this exceptions for
    months now.

    Good luck!
    -Kuli

    Am 14.03.2012 11:22, schrieb Colin Howe:
    Currently using 3.4.0. We have autocommit enabled but we manually do
    commits every 100 documents anyway... I can turn it off if you think that
    might help.


    Cheers,
    Colin


    On Wed, Mar 14, 2012 at 10:24 AM, Markus Jelsma
    wrote:

    Are you running trunk and have auto-commit enabled? Then disable
    auto-commit. Even if you increase ulimits it will continue to swallow all
    available file descriptors.


    On Wed, 14 Mar 2012 10:13:55 +0000, Colin Howe<colin@conversocial.com>
    wrote:

    Hello,
    We keep hitting the too many open files exception. Looking at lsof we
    have
    a lot (several thousand) of entries like this:

    java 19339 root 1619u sock 0,7
    0t0
    682291383 can't identify protocol


    However, netstat -a doesn't show any of these.

    Can anyone suggest a way to diagnose what these socket entries are?
    Happy
    to post any more information as needed.


    Cheers,
    Colin
    --
    Markus Jelsma - CTO - Openindex
    http://www.linkedin.com/in/****markus17<http://www.linkedin.com/in/**markus17>
    <http://www.linkedin.**com/in/markus17<http://www.linkedin.com/in/markus17>
    050-8536600 / 06-50258350

    --
    Colin Howe
    @colinhowe

    VP of Engineering
    Conversocial Ltd
    conversocial.com
  • Michael Kuhlmann at Mar 14, 2012 at 11:40 am
    Ah, good to know! Thank you!

    I already had Jetty under suspicion, but we had this failure quite often
    in October and November, when the bug was not yet reported.

    -Kuli

    Am 14.03.2012 12:08, schrieb Colin Howe:
    After some more digging around I discovered that there was a bug reported
    in jetty 6: https://jira.codehaus.org/browse/JETTY-1458

    This prompted me to upgrade to Jetty 7 and things look a bit more stable
    now :)
  • Erick Erickson at Mar 14, 2012 at 2:18 pm
    Colin:

    FYI, you might consider just setting up the autocommit (or commitWithin if
    you're using SolrJ) for some reasonable interval (I often use 10 minutes or so).

    Even though you've figured it is a Tomcat issue, each
    commit causes searcher re-opens, perhaps replication in a master/slave
    setup, increased merges etc. It works, but it's also resource intensive...

    FWIW
    Erick
    On Wed, Mar 14, 2012 at 6:40 AM, Michael Kuhlmann wrote:
    Ah, good to know! Thank you!

    I already had Jetty under suspicion, but we had this failure quite often in
    October and November, when the bug was not yet reported.

    -Kuli

    Am 14.03.2012 12:08, schrieb Colin Howe:
    After some more digging around I discovered that there was a bug reported
    in jetty 6:  https://jira.codehaus.org/browse/JETTY-1458

    This prompted me to upgrade to Jetty 7 and things look a bit more stable
    now :)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupsolr-user @
categorieslucene
postedMar 14, '12 at 10:14a
activeMar 14, '12 at 2:18p
posts7
users4
websitelucene.apache.org...

People

Translate

site design / logo © 2017 Grokbase