Grokbase › Groups › Lucene › java-user › August 2010
FAQ
Hi all,


We're observing search threads slowing down during directory copies performed during updates to the index. The thread dump shows search threads blocked on a FSDirectory$FSIndexInput$Descriptor instance:



"Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for monitor entry [0x988ed000..0x988edf30]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
- waiting to lock <0xc79c9900> (a org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
at org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
at org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
at org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.Searcher.search(Searcher.java:118)

...


The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior to applying changes (addDocument calls) on the current "available" segment. This takes >7 mins to copy a directory of size 1.4G. During this time window, the searches are slow and the above thread stacks are observed.


Could there be any system level limits we're hitting?


Our test environment is:
lucene-2.3.2
4x2.6 GHz, 16G memory
Red Hat 3.4.6-9


Thanks and regards,


- Gagan

Search Discussions

  • Anshum at Aug 23, 2010 at 7:48 am ⇧
    Seems like a case of I/O issues. You may be reading content off the index
    while performing searches while the I/O for copy is also happening.

    --
    Anshum Gupta
    http://ai-cafe.blogspot.com

    On Mon, Aug 23, 2010 at 1:12 PM, wrote:


    Hi all,


    We're observing search threads slowing down during directory copies
    performed during updates to the index. The thread dump shows search threads
    blocked on a FSDirectory$FSIndexInput$Descriptor instance:



    "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
    monitor entry [0x988ed000..0x988edf30]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
    - waiting to lock <0xc79c9900> (a
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
    at
    org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
    at
    org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
    at
    org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
    at
    org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
    at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
    at
    org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
    at
    org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
    at
    org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
    at org.apache.lucene.search.Searcher.search(Searcher.java:118)

    ...


    The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
    to applying changes (addDocument calls) on the current "available" segment.
    This takes >7 mins to copy a directory of size 1.4G. During this time
    window, the searches are slow and the above thread stacks are observed.


    Could there be any system level limits we're hitting?


    Our test environment is:
    lucene-2.3.2
    4x2.6 GHz, 16G memory
    Red Hat 3.4.6-9


    Thanks and regards,


    - Gagan
  • Gaganb at Aug 23, 2010 at 8:21 am ⇧
    We suspected the same at first, and so modified the flow to do the copy of the new version "before" searching is activated on it. So, there are no searches on the new version dir at the time of copy - yet we still see slowness during the disk copy.


    Thanks,
    - Gagan





    -----Original Message-----
    From: Anshum <anshumg@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Mon, Aug 23, 2010 1:17 pm
    Subject: Re: slow search threads during a disk copy


    Seems like a case of I/O issues. You may be reading content off the index
    while performing searches while the I/O for copy is also happening.

    --
    Anshum Gupta
    http://ai-cafe.blogspot.com

    On Mon, Aug 23, 2010 at 1:12 PM, wrote:


    Hi all,


    We're observing search threads slowing down during directory copies
    performed during updates to the index. The thread dump shows search threads
    blocked on a FSDirectory$FSIndexInput$Descriptor instance:



    "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
    monitor entry [0x988ed000..0x988edf30]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
    - waiting to lock <0xc79c9900> (a
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
    at
    org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
    at
    org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
    at
    org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
    at
    org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
    at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
    at
    org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
    at
    org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
    at
    org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
    at org.apache.lucene.search.Searcher.search(Searcher.java:118)

    ...


    The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
    to applying changes (addDocument calls) on the current "available" segment.
    This takes >7 mins to copy a directory of size 1.4G. During this time
    window, the searches are slow and the above thread stacks are observed.


    Could there be any system level limits we're hitting?


    Our test environment is:
    lucene-2.3.2
    4x2.6 GHz, 16G memory
    Red Hat 3.4.6-9


    Thanks and regards,


    - Gagan
  • Ian Lea at Aug 23, 2010 at 8:42 am ⇧
    Well, there still sounds like plenty of scope for IO contention. Are
    source and dest directories on the same disk? What does iostat show?
    Could you use rsync rather than cp -rp, or something else that only
    copies changes?


    --
    Ian.

    On Mon, Aug 23, 2010 at 9:20 AM, wrote:

    We suspected the same at first, and so modified the flow to do the copy of the new version "before" searching is activated on it. So, there are no searches on the new version dir at the time of copy - yet we still see slowness during the disk copy.


    Thanks,
    - Gagan





    -----Original Message-----
    From: Anshum <anshumg@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Mon, Aug 23, 2010 1:17 pm
    Subject: Re: slow search threads during a disk copy


    Seems like a case of I/O issues. You may be reading content off the index
    while performing searches while the I/O for copy is also happening.

    --
    Anshum Gupta
    http://ai-cafe.blogspot.com

    On Mon, Aug 23, 2010 at 1:12 PM, wrote:


    Hi all,


    We're observing search threads slowing down during directory copies
    performed during updates to the index. The thread dump shows search threads
    blocked on a FSDirectory$FSIndexInput$Descriptor instance:



    "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
    monitor entry [0x988ed000..0x988edf30]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
    - waiting to lock <0xc79c9900> (a
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
    at
    org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
    at
    org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
    at
    org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
    at
    org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
    at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
    at
    org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
    at
    org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
    at
    org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
    at org.apache.lucene.search.Searcher.search(Searcher.java:118)

    ...


    The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
    to applying changes (addDocument calls) on the current "available" segment.
    This takes >7 mins to copy a directory of size 1.4G. During this time
    window, the searches are slow and the above thread stacks are observed.


    Could there be any system level limits we're hitting?


    Our test environment is:
    lucene-2.3.2
    4x2.6 GHz, 16G memory
    Red Hat 3.4.6-9


    Thanks and regards,


    - Gagan

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Gaganb at Aug 23, 2010 at 9:44 am ⇧
    Yes, all version directories are on the same disk. iostat output should be useful. Using rsync is complicated because of legacy reasons - there's a change impact to manage.


    Intererstingly, the copy is quite fast (around 30s) when there are no searches in progress.



    Thanks,
    - Gagan


    -----Original Message-----
    From: Ian Lea <ian.lea@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Mon, Aug 23, 2010 2:10 pm
    Subject: Re: slow search threads during a disk copy


    Well, there still sounds like plenty of scope for IO contention. Are
    source and dest directories on the same disk? What does iostat show?
    Could you use rsync rather than cp -rp, or something else that only
    copies changes?


    --
    Ian.

    On Mon, Aug 23, 2010 at 9:20 AM, wrote:

    We suspected the same at first, and so modified the flow to do the copy of the
    new version "before" searching is activated on it. So, there are no searches on
    the new version dir at the time of copy - yet we still see slowness during the
    disk copy.

    Thanks,
    - Gagan





    -----Original Message-----
    From: Anshum <anshumg@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Mon, Aug 23, 2010 1:17 pm
    Subject: Re: slow search threads during a disk copy


    Seems like a case of I/O issues. You may be reading content off the index
    while performing searches while the I/O for copy is also happening.

    --
    Anshum Gupta
    http://ai-cafe.blogspot.com

    On Mon, Aug 23, 2010 at 1:12 PM, wrote:


    Hi all,


    We're observing search threads slowing down during directory copies
    performed during updates to the index. The thread dump shows search threads
    blocked on a FSDirectory$FSIndexInput$Descriptor instance:



    "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting for
    monitor entry [0x988ed000..0x988edf30]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
    - waiting to lock <0xc79c9900> (a
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
    at
    org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
    at
    org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
    at
    org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
    at
    org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
    at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
    at
    org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
    at
    org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
    at
    org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
    at org.apache.lucene.search.Searcher.search(Searcher.java:118)

    ...


    The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command prior
    to applying changes (addDocument calls) on the current "available" segment.
    This takes >7 mins to copy a directory of size 1.4G. During this time
    window, the searches are slow and the above thread stacks are observed.


    Could there be any system level limits we're hitting?


    Our test environment is:
    lucene-2.3.2
    4x2.6 GHz, 16G memory
    Red Hat 3.4.6-9


    Thanks and regards,


    - Gagan

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Anshum at Aug 23, 2010 at 10:40 am ⇧
    There is bound to be IO contention. I'm sure iostat will give you a much
    better picture on it.

    --
    Anshum Gupta
    http://ai-cafe.blogspot.com

    On Mon, Aug 23, 2010 at 3:13 PM, wrote:

    Yes, all version directories are on the same disk. iostat output should be
    useful. Using rsync is complicated because of legacy reasons - there's a
    change impact to manage.


    Intererstingly, the copy is quite fast (around 30s) when there are no
    searches in progress.



    Thanks,
    - Gagan


    -----Original Message-----
    From: Ian Lea <ian.lea@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Mon, Aug 23, 2010 2:10 pm
    Subject: Re: slow search threads during a disk copy


    Well, there still sounds like plenty of scope for IO contention. Are
    source and dest directories on the same disk? What does iostat show?
    Could you use rsync rather than cp -rp, or something else that only
    copies changes?


    --
    Ian.

    On Mon, Aug 23, 2010 at 9:20 AM, wrote:

    We suspected the same at first, and so modified the flow to do the copy
    of the
    new version "before" searching is activated on it. So, there are no
    searches on
    the new version dir at the time of copy - yet we still see slowness during
    the
    disk copy.

    Thanks,
    - Gagan





    -----Original Message-----
    From: Anshum <anshumg@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Mon, Aug 23, 2010 1:17 pm
    Subject: Re: slow search threads during a disk copy


    Seems like a case of I/O issues. You may be reading content off the index
    while performing searches while the I/O for copy is also happening.

    --
    Anshum Gupta
    http://ai-cafe.blogspot.com

    On Mon, Aug 23, 2010 at 1:12 PM, wrote:


    Hi all,


    We're observing search threads slowing down during directory copies
    performed during updates to the index. The thread dump shows search
    threads
    blocked on a FSDirectory$FSIndexInput$Descriptor instance:



    "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting
    for
    monitor entry [0x988ed000..0x988edf30]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
    - waiting to lock <0xc79c9900> (a
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
    at
    org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
    at
    org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
    at
    org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
    at
    org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
    at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
    at
    org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
    at
    org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
    at
    org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
    at
    org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
    at org.apache.lucene.search.Searcher.search(Searcher.java:118)

    ...


    The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command
    prior
    to applying changes (addDocument calls) on the current "available"
    segment.
    This takes >7 mins to copy a directory of size 1.4G. During this time
    window, the searches are slow and the above thread stacks are observed.


    Could there be any system level limits we're hitting?


    Our test environment is:
    lucene-2.3.2
    4x2.6 GHz, 16G memory
    Red Hat 3.4.6-9


    Thanks and regards,


    - Gagan

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

  • Toke Eskildsen at Aug 24, 2010 at 1:15 pm ⇧

    On Mon, 2010-08-23 at 11:43 +0200, gaganb@graffiti.net wrote:
    Intererstingly, the copy is quite fast (around 30s) when there are no searches in progress.
    I agree with Anshum: This looks very much like IO contention.

    However, it might not just be a case of seek-time trouble: We've had
    similar problems on our Linux server with when copying large files from
    harddisk A to A while performing searches on SSD B.

    Our low-level guys tells me that this was due to the write-buffer being
    filled and that reads were blocked while it was flushed (AFAIR). We've
    experimented by mounting with sync instead of async which gave us much
    better response times during copying at the cost of a substantially
    slower copy. dirsync should also be worth looking into.

    Regards,
    Toke Eskildsen


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedAug 23, '10 at 7:43a
activeAug 24, '10 at 1:15p
posts7
users4
websitelucene.apache.org

4 users in discussion

Gaganb: 3 posts Anshum: 2 posts Toke Eskildsen: 1 post Ian Lea: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase