On Dec 18, 2007 3:49 AM, Olly Betts wrote:
On Fri, Dec 14, 2007 at 11:18:12AM -0800, Andrey wrote:
from my own experience, breaking up into dbs will not cause a big
preformance lost, like from 1sec to 2 secs, it just works like querying 1 db
after cached up
We are all missing the points here. There are two types of Xapian users.
1. Search engine using less than 1 million documents or data can be
fit in memory.
2. Search engine using 1-100 million documents and data is much larger
People who are testing performance on data that can easily fit into
server memory, their data is cashed in memory and their performance
measurements is high and distorted. We must measure the performance
when searches are not cashed to memory but sitting on hard disk. Only
then we can see the real performance of searches as the hard disk
spins and find the correct data. Than OS (Linux) place the result into
cache if available. The second same search will use cache instead of
hard disk and the performance is too high and invalid.
Users of all search engines platforms are surprise that some searches
takes very long, specially those that are not in cache. Because they
run their performance on cache not on hard disk. Quickly they find
their scalability problem and broken promises. In my case having
100-500GB data on hard disk, the data cannot fit into memory and using
two databases is two times slower than using single database. That is
why I keep saying that indexing performance of single database is the
most important, because the search performance follows.
I would be suprised if there was a large overhead - there's a bit of
extra work from opening the databases, and a small amount from having
a "MultiPostList". The combined size of the split databases is usually
a little larger than the combined one, which may increase VM pressure a
If you do profile and find there's a significant difference, it would
be interesting to see comparable profiles for the two cases to see where
the extra time is spent.
maybe you can try to duplicate another copy of your db and serach over them
together, its very easy with just 1 extra line
You'd also need to generate the equivalent combined database (e.g. by
using xapian-compact with the same input twice).
But just duplicating the data isn't an accurate recreation of searching
a real database split in two though. I don't know if it actually would
make a difference, but it might.
Xapian-discuss mailing list