FAQ
We've a centralized lucene index running on a nfs share. This index gets
an update per 30 min. The LuceneServer nodes will notice the update and
copy the index (about 2,5gig) to a local tmpfs directory. Searching is
way faster in our case compared to a local disk.
However eks' concerns are valid and in the beginning we had some issues
with the OS starting to swap out things without a noticeable reason.
However this swapping occured rarely and did not affect performance
severly. Now the boxes got some more main memory and the problem is
gone. Two LuceneServer instances are sufficient to handle up to 60 heavy
search requests per second, which involves also some object generation,
etc.

I've tried RAMDirectory first, but we went the tmpfs way quite fast.
Sadly I can't remember what the main reason of our decision has been.

eks dev wrote:
I do not know much about RAM FS, but I know for sure if you have enough memory for RAMDirectory, you should go for it. That gives you the fastest and the most stable performance, no OS swaps, no sudden performance drops... Uwe's tip is very good, if you/OS occasionally need RAM for other things, so that OS can borrow some from your index. This swapping comes with price, which can or cannot be ok for you.


----- Original Message ----

From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
To: java-user@lucene.apache.org
Sent: Thursday, 23 July, 2009 18:55:57
Subject: Re: Loading an index into memory

I haven't verified this myself, but I remember talking to somebody who tried
MMapDirectory and compared it to simply using tmpfs (RAM FS). The result was
that MMapDirectory had some memory overhead, so putting the index on tmpfs was
more memory-efficient. I guess this person had read-only indices, so tmpfs was
an option.

Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR


----- Original Message ----

From: Uwe Schindler
To: java-user@lucene.apache.org
Sent: Thursday, July 23, 2009 9:47:24 AM
Subject: RE: Loading an index into memory

The size is in bytes and the RAMDirectory stores the bytes in bytes, so size
is equal. I would suggest to not copy the dir into a RAMdirectory. It is
better to use MMapDirectory in this case, as it "swaps" the files into
address space like a normal OS swap file. The OS kernel will automatically
swap needed parts into physical RAM. In this case the Java Heap is not
wasted and only needed parts are swapped into RAM.

-----
UWE SCHINDLER
Webserver/Middleware Development
PANGAEA - Publishing Network for Geoscientific and Environmental Data
MARUM - University of Bremen
Room 2500, Leobener Str., D-28359 Bremen
Tel.: +49 421 218 65595
Fax: +49 421 218 65505
http://www.pangaea.de/
E-mail: uschindler@pangaea.de

-----Original Message-----
From: Dragon Fly
Sent: Thursday, July 23, 2009 3:38 PM
To: java-user@lucene.apache.org
Subject: Loading an index into memory


Hi,

I have a question regarding RAMDirectory. I have a 5 GB index on disk and
it is opened like the following:

searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

Approximately how much memory is needed to load the index? 5GB of memory
or 10GB because of Unicode? Does the entire index get loaded into memory
or only parts of it? Thank you.


_________________________________________________________________
Windows LiveT HotmailR: Celebrate the moment with your favorite sports
pics. Check it out.
http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
_WL_QA_HM_sports_photos_072009&cat=sports
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

--
Thomas Becker
Senior JEE Developer

net mobile AG
Zollhof 17
40221 Düsseldorf
GERMANY

Phone: +49 211 97020-195
Fax: +49 211 97020-949
Mobile: +49 173 5146567 (private)
E-Mail: mailto:thomas.becker@net-m.de
Internet: http://www.net-m.de

Registergericht: Amtsgericht Düsseldorf, HRB 48022
Vorstand: Theodor Niehues (Vorsitzender), Frank Hartmann,
Kai Markus Kulas, Dieter Plassmann
Vorsitzender des
Aufsichtsrates: Dr. Michael Briem


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 7 of 7 | next ›
Discussion Overview
groupjava-user @
categorieslucene
postedJul 23, '09 at 1:37p
activeJul 24, '09 at 8:21a
posts7
users6
websitelucene.apache.org

People

Translate

site design / logo © 2021 Grokbase