FAQ
Dear Experts,

I have a strange problem where select q=*:* is returning different number of
documents. Sometime its returning numFound = 5866712 and sometimes it
returns numFound = 5852274. *numFound is always one of these 2 values.*

Here is the query:

*http://localhost:5011/solr/mycollection/select?q=*:*&rows=0*


I am running Solr in cloud mode and this problem is occurring with both
solr-4.5.1 and solr-4.10.0. I have exactly same data indexed in both
versions. 4.5.1 is running on a 8 nodes cluster (4x2 shards) and solr-4.10.0
is running on a 4 node (2x2 shards)cluster.

Size of whole collection is ~4GB. No simultaneous indexing is going on. The
last indexing happened 20 days ago. *Collection is optimized.*

What might be the problem here?

I tried reloading the collection, restarting the whole cluster and changing
the *rows* param in query. But still I am having this problem.




--
View this message in context: http://lucene.472066.n3.nabble.com/Mismatch-in-numFound-in-q-query-tp4163911.html
Sent from the Solr - User mailing list archive at Nabble.com.

Search Discussions

  • Shawn Heisey at Oct 12, 2014 at 7:59 pm

    On 10/12/2014 12:26 PM, vidit.asthana wrote:
    I have a strange problem where select q=*:* is returning different number of
    documents. Sometime its returning numFound = 5866712 and sometimes it
    returns numFound = 5852274. *numFound is always one of these 2 values.*

    Here is the query:

    *http://localhost:5011/solr/mycollection/select?q=*:*&rows=0*


    I am running Solr in cloud mode and this problem is occurring with both
    solr-4.5.1 and solr-4.10.0. I have exactly same data indexed in both
    versions. 4.5.1 is running on a 8 nodes cluster (4x2 shards) and solr-4.10.0
    is running on a 4 node (2x2 shards)cluster.
    I really need to make a wiki page for this. It would save so much
    typing! I also need to boil it down to a small-scale real-world example
    and show how the numbers get calculated and what goes wrong, which means
    I need to have a complete understanding of the problem, and at this
    moment, I don't have that.

    This is a problem that's unique to distributed indexes. What causes it
    is having documents with the same value in the uniqueKey field indexed
    in more than one shard.

    It is not a bug, it's a result of the way that results from multiple
    shards are combined into one result. The only way to "fix" this problem
    would involve so much additional processing that it would make all
    queries extremely slow.

    If you're using automatic document routing, then your routing algorithm
    may have changed at some point, and you didn't re-index. If you're
    using manual document routing, then some documents were indexed on the
    wrong shard, and later indexed on another shard as well.

    Preventing the problem is easy -- always index documents onto the
    correct shard. Fixing the problem at this point might involve clearing
    your index and re-indexing from scratch, unless you can figure out which
    documents have been indexed on more than one shard and you can delete
    them from the incorrect shard(s).

    Thanks,
    Shawn
  • Jan Høydahl at Oct 12, 2014 at 9:46 pm
    If you don't want "downtime", you could add a <field name="indextime" type="tdate" default="NOW" /> field to your schema, reload, do a full re-index on top of your existing index, and then delete all documents that were not updated, via a delelteByQuery, e.g.: indextime:[* TO NOW-1DAY]

    --
    Jan Høydahl, search solution architect
    Cominvent AS - www.cominvent.com

    12. okt. 2014 kl. 21:59 skrev Shawn Heisey <apache@elyograg.org>:
    On 10/12/2014 12:26 PM, vidit.asthana wrote:
    I have a strange problem where select q=*:* is returning different number of
    documents. Sometime its returning numFound = 5866712 and sometimes it
    returns numFound = 5852274. *numFound is always one of these 2 values.*

    Here is the query:

    *http://localhost:5011/solr/mycollection/select?q=*:*&rows=0*


    I am running Solr in cloud mode and this problem is occurring with both
    solr-4.5.1 and solr-4.10.0. I have exactly same data indexed in both
    versions. 4.5.1 is running on a 8 nodes cluster (4x2 shards) and solr-4.10.0
    is running on a 4 node (2x2 shards)cluster.
    I really need to make a wiki page for this. It would save so much
    typing! I also need to boil it down to a small-scale real-world example
    and show how the numbers get calculated and what goes wrong, which means
    I need to have a complete understanding of the problem, and at this
    moment, I don't have that.

    This is a problem that's unique to distributed indexes. What causes it
    is having documents with the same value in the uniqueKey field indexed
    in more than one shard.

    It is not a bug, it's a result of the way that results from multiple
    shards are combined into one result. The only way to "fix" this problem
    would involve so much additional processing that it would make all
    queries extremely slow.

    If you're using automatic document routing, then your routing algorithm
    may have changed at some point, and you didn't re-index. If you're
    using manual document routing, then some documents were indexed on the
    wrong shard, and later indexed on another shard as well.

    Preventing the problem is easy -- always index documents onto the
    correct shard. Fixing the problem at this point might involve clearing
    your index and re-indexing from scratch, unless you can figure out which
    documents have been indexed on more than one shard and you can delete
    them from the incorrect shard(s).

    Thanks,
    Shawn

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupsolr-user @
categorieslucene
postedOct 12, '14 at 6:26p
activeOct 12, '14 at 9:46p
posts3
users3
websitelucene.apache.org...

People

Translate

site design / logo © 2020 Grokbase