FAQ
Hi,

I have a question regarding RAMDirectory. I have a 5 GB index on disk and it is opened like the following:

searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

Approximately how much memory is needed to load the index? 5GB of memory or 10GB because of Unicode? Does the entire index get loaded into memory or only parts of it? Thank you.


_________________________________________________________________
Windows Live™ Hotmail®: Celebrate the moment with your favorite sports pics. Check it out.
http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM_WL_QA_HM_sports_photos_072009&cat=sports

Search Discussions

  • Uwe Schindler at Jul 23, 2009 at 3:27 pm
    The size is in bytes and the RAMDirectory stores the bytes in bytes, so size
    is equal. I would suggest to not copy the dir into a RAMdirectory. It is
    better to use MMapDirectory in this case, as it "swaps" the files into
    address space like a normal OS swap file. The OS kernel will automatically
    swap needed parts into physical RAM. In this case the Java Heap is not
    wasted and only needed parts are swapped into RAM.

    -----
    UWE SCHINDLER
    Webserver/Middleware Development
    PANGAEA - Publishing Network for Geoscientific and Environmental Data
    MARUM - University of Bremen
    Room 2500, Leobener Str., D-28359 Bremen
    Tel.: +49 421 218 65595
    Fax: +49 421 218 65505
    http://www.pangaea.de/
    E-mail: uschindler@pangaea.de
    -----Original Message-----
    From: Dragon Fly
    Sent: Thursday, July 23, 2009 3:38 PM
    To: java-user@lucene.apache.org
    Subject: Loading an index into memory


    Hi,

    I have a question regarding RAMDirectory. I have a 5 GB index on disk and
    it is opened like the following:

    searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

    Approximately how much memory is needed to load the index? 5GB of memory
    or 10GB because of Unicode? Does the entire index get loaded into memory
    or only parts of it? Thank you.


    _________________________________________________________________
    Windows LiveT HotmailR: Celebrate the moment with your favorite sports
    pics. Check it out.
    http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
    _WL_QA_HM_sports_photos_072009&cat=sports

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Erick Erickson at Jul 23, 2009 at 3:55 pm
    What are you trying to accomplish? I'd insure that my performance wasa
    problem before doing anything. If you're thinking "it's in RAM so it
    has to be faster" you might be surprised.....

    So gather evidence that you have a problem before you jump to
    providing a solution.

    Erick
    On Thu, Jul 23, 2009 at 9:47 AM, Uwe Schindler wrote:

    The size is in bytes and the RAMDirectory stores the bytes in bytes, so
    size
    is equal. I would suggest to not copy the dir into a RAMdirectory. It is
    better to use MMapDirectory in this case, as it "swaps" the files into
    address space like a normal OS swap file. The OS kernel will automatically
    swap needed parts into physical RAM. In this case the Java Heap is not
    wasted and only needed parts are swapped into RAM.

    -----
    UWE SCHINDLER
    Webserver/Middleware Development
    PANGAEA - Publishing Network for Geoscientific and Environmental Data
    MARUM - University of Bremen
    Room 2500, Leobener Str., D-28359 Bremen
    Tel.: +49 421 218 65595
    Fax: +49 421 218 65505
    http://www.pangaea.de/
    E-mail: uschindler@pangaea.de
    -----Original Message-----
    From: Dragon Fly
    Sent: Thursday, July 23, 2009 3:38 PM
    To: java-user@lucene.apache.org
    Subject: Loading an index into memory


    Hi,

    I have a question regarding RAMDirectory. I have a 5 GB index on disk and
    it is opened like the following:

    searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

    Approximately how much memory is needed to load the index? 5GB of memory
    or 10GB because of Unicode? Does the entire index get loaded into memory
    or only parts of it? Thank you.


    _________________________________________________________________
    Windows LiveT HotmailR: Celebrate the moment with your favorite sports
    pics. Check it out.
    http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
    _WL_QA_HM_sports_photos_072009&cat=sports

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Dragon Fly at Jul 23, 2009 at 4:45 pm
    Thank you both.
    Date: Thu, 23 Jul 2009 11:55:58 -0400
    Subject: Re: Loading an index into memory
    From: erickerickson@gmail.com
    To: java-user@lucene.apache.org

    What are you trying to accomplish? I'd insure that my performance wasa
    problem before doing anything. If you're thinking "it's in RAM so it
    has to be faster" you might be surprised.....

    So gather evidence that you have a problem before you jump to
    providing a solution.

    Erick
    On Thu, Jul 23, 2009 at 9:47 AM, Uwe Schindler wrote:

    The size is in bytes and the RAMDirectory stores the bytes in bytes, so
    size
    is equal. I would suggest to not copy the dir into a RAMdirectory. It is
    better to use MMapDirectory in this case, as it "swaps" the files into
    address space like a normal OS swap file. The OS kernel will automatically
    swap needed parts into physical RAM. In this case the Java Heap is not
    wasted and only needed parts are swapped into RAM.

    -----
    UWE SCHINDLER
    Webserver/Middleware Development
    PANGAEA - Publishing Network for Geoscientific and Environmental Data
    MARUM - University of Bremen
    Room 2500, Leobener Str., D-28359 Bremen
    Tel.: +49 421 218 65595
    Fax: +49 421 218 65505
    http://www.pangaea.de/
    E-mail: uschindler@pangaea.de
    -----Original Message-----
    From: Dragon Fly
    Sent: Thursday, July 23, 2009 3:38 PM
    To: java-user@lucene.apache.org
    Subject: Loading an index into memory


    Hi,

    I have a question regarding RAMDirectory. I have a 5 GB index on disk and
    it is opened like the following:

    searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

    Approximately how much memory is needed to load the index? 5GB of memory
    or 10GB because of Unicode? Does the entire index get loaded into memory
    or only parts of it? Thank you.


    _________________________________________________________________
    Windows LiveT HotmailR: Celebrate the moment with your favorite sports
    pics. Check it out.
    http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
    _WL_QA_HM_sports_photos_072009&cat=sports

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    _________________________________________________________________
    Bing™ brings you maps, menus, and reviews organized in one place. Try it now.
    http://www.bing.com/search?q=restaurants&form=MLOGEN&publ=WLHMTAG&crea=TXT_MLOGEN_Local_Local_Restaurants_1x1
  • Otis Gospodnetic at Jul 23, 2009 at 4:55 pm
    I haven't verified this myself, but I remember talking to somebody who tried MMapDirectory and compared it to simply using tmpfs (RAM FS). The result was that MMapDirectory had some memory overhead, so putting the index on tmpfs was more memory-efficient. I guess this person had read-only indices, so tmpfs was an option.

    Otis
    --
    Sematext is hiring -- http://sematext.com/about/jobs.html?mls
    Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR


    ----- Original Message ----
    From: Uwe Schindler <uschindler@pangaea.de>
    To: java-user@lucene.apache.org
    Sent: Thursday, July 23, 2009 9:47:24 AM
    Subject: RE: Loading an index into memory

    The size is in bytes and the RAMDirectory stores the bytes in bytes, so size
    is equal. I would suggest to not copy the dir into a RAMdirectory. It is
    better to use MMapDirectory in this case, as it "swaps" the files into
    address space like a normal OS swap file. The OS kernel will automatically
    swap needed parts into physical RAM. In this case the Java Heap is not
    wasted and only needed parts are swapped into RAM.

    -----
    UWE SCHINDLER
    Webserver/Middleware Development
    PANGAEA - Publishing Network for Geoscientific and Environmental Data
    MARUM - University of Bremen
    Room 2500, Leobener Str., D-28359 Bremen
    Tel.: +49 421 218 65595
    Fax: +49 421 218 65505
    http://www.pangaea.de/
    E-mail: uschindler@pangaea.de
    -----Original Message-----
    From: Dragon Fly
    Sent: Thursday, July 23, 2009 3:38 PM
    To: java-user@lucene.apache.org
    Subject: Loading an index into memory


    Hi,

    I have a question regarding RAMDirectory. I have a 5 GB index on disk and
    it is opened like the following:

    searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

    Approximately how much memory is needed to load the index? 5GB of memory
    or 10GB because of Unicode? Does the entire index get loaded into memory
    or only parts of it? Thank you.


    _________________________________________________________________
    Windows LiveT HotmailR: Celebrate the moment with your favorite sports
    pics. Check it out.
    http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
    _WL_QA_HM_sports_photos_072009&cat=sports

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Eks dev at Jul 23, 2009 at 5:30 pm
    I do not know much about RAM FS, but I know for sure if you have enough memory for RAMDirectory, you should go for it. That gives you the fastest and the most stable performance, no OS swaps, no sudden performance drops... Uwe's tip is very good, if you/OS occasionally need RAM for other things, so that OS can borrow some from your index. This swapping comes with price, which can or cannot be ok for you.


    ----- Original Message ----
    From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
    To: java-user@lucene.apache.org
    Sent: Thursday, 23 July, 2009 18:55:57
    Subject: Re: Loading an index into memory

    I haven't verified this myself, but I remember talking to somebody who tried
    MMapDirectory and compared it to simply using tmpfs (RAM FS). The result was
    that MMapDirectory had some memory overhead, so putting the index on tmpfs was
    more memory-efficient. I guess this person had read-only indices, so tmpfs was
    an option.

    Otis
    --
    Sematext is hiring -- http://sematext.com/about/jobs.html?mls
    Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR


    ----- Original Message ----
    From: Uwe Schindler
    To: java-user@lucene.apache.org
    Sent: Thursday, July 23, 2009 9:47:24 AM
    Subject: RE: Loading an index into memory

    The size is in bytes and the RAMDirectory stores the bytes in bytes, so size
    is equal. I would suggest to not copy the dir into a RAMdirectory. It is
    better to use MMapDirectory in this case, as it "swaps" the files into
    address space like a normal OS swap file. The OS kernel will automatically
    swap needed parts into physical RAM. In this case the Java Heap is not
    wasted and only needed parts are swapped into RAM.

    -----
    UWE SCHINDLER
    Webserver/Middleware Development
    PANGAEA - Publishing Network for Geoscientific and Environmental Data
    MARUM - University of Bremen
    Room 2500, Leobener Str., D-28359 Bremen
    Tel.: +49 421 218 65595
    Fax: +49 421 218 65505
    http://www.pangaea.de/
    E-mail: uschindler@pangaea.de
    -----Original Message-----
    From: Dragon Fly
    Sent: Thursday, July 23, 2009 3:38 PM
    To: java-user@lucene.apache.org
    Subject: Loading an index into memory


    Hi,

    I have a question regarding RAMDirectory. I have a 5 GB index on disk and
    it is opened like the following:

    searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

    Approximately how much memory is needed to load the index? 5GB of memory
    or 10GB because of Unicode? Does the entire index get loaded into memory
    or only parts of it? Thank you.


    _________________________________________________________________
    Windows LiveT HotmailR: Celebrate the moment with your favorite sports
    pics. Check it out.
    http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
    _WL_QA_HM_sports_photos_072009&cat=sports

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Thomas Becker at Jul 24, 2009 at 8:21 am
    We've a centralized lucene index running on a nfs share. This index gets
    an update per 30 min. The LuceneServer nodes will notice the update and
    copy the index (about 2,5gig) to a local tmpfs directory. Searching is
    way faster in our case compared to a local disk.
    However eks' concerns are valid and in the beginning we had some issues
    with the OS starting to swap out things without a noticeable reason.
    However this swapping occured rarely and did not affect performance
    severly. Now the boxes got some more main memory and the problem is
    gone. Two LuceneServer instances are sufficient to handle up to 60 heavy
    search requests per second, which involves also some object generation,
    etc.

    I've tried RAMDirectory first, but we went the tmpfs way quite fast.
    Sadly I can't remember what the main reason of our decision has been.

    eks dev wrote:
    I do not know much about RAM FS, but I know for sure if you have enough memory for RAMDirectory, you should go for it. That gives you the fastest and the most stable performance, no OS swaps, no sudden performance drops... Uwe's tip is very good, if you/OS occasionally need RAM for other things, so that OS can borrow some from your index. This swapping comes with price, which can or cannot be ok for you.


    ----- Original Message ----

    From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
    To: java-user@lucene.apache.org
    Sent: Thursday, 23 July, 2009 18:55:57
    Subject: Re: Loading an index into memory

    I haven't verified this myself, but I remember talking to somebody who tried
    MMapDirectory and compared it to simply using tmpfs (RAM FS). The result was
    that MMapDirectory had some memory overhead, so putting the index on tmpfs was
    more memory-efficient. I guess this person had read-only indices, so tmpfs was
    an option.

    Otis
    --
    Sematext is hiring -- http://sematext.com/about/jobs.html?mls
    Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR


    ----- Original Message ----

    From: Uwe Schindler
    To: java-user@lucene.apache.org
    Sent: Thursday, July 23, 2009 9:47:24 AM
    Subject: RE: Loading an index into memory

    The size is in bytes and the RAMDirectory stores the bytes in bytes, so size
    is equal. I would suggest to not copy the dir into a RAMdirectory. It is
    better to use MMapDirectory in this case, as it "swaps" the files into
    address space like a normal OS swap file. The OS kernel will automatically
    swap needed parts into physical RAM. In this case the Java Heap is not
    wasted and only needed parts are swapped into RAM.

    -----
    UWE SCHINDLER
    Webserver/Middleware Development
    PANGAEA - Publishing Network for Geoscientific and Environmental Data
    MARUM - University of Bremen
    Room 2500, Leobener Str., D-28359 Bremen
    Tel.: +49 421 218 65595
    Fax: +49 421 218 65505
    http://www.pangaea.de/
    E-mail: uschindler@pangaea.de

    -----Original Message-----
    From: Dragon Fly
    Sent: Thursday, July 23, 2009 3:38 PM
    To: java-user@lucene.apache.org
    Subject: Loading an index into memory


    Hi,

    I have a question regarding RAMDirectory. I have a 5 GB index on disk and
    it is opened like the following:

    searcher = new IndexSearcher (new RAMDirectory (indexDirectory));

    Approximately how much memory is needed to load the index? 5GB of memory
    or 10GB because of Unicode? Does the entire index get loaded into memory
    or only parts of it? Thank you.


    _________________________________________________________________
    Windows LiveT HotmailR: Celebrate the moment with your favorite sports
    pics. Check it out.
    http://www.windowslive.com/Online/Hotmail/Campaign/QuickAdd?ocid=TXT_TAGLM
    _WL_QA_HM_sports_photos_072009&cat=sports
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    Thomas Becker
    Senior JEE Developer

    net mobile AG
    Zollhof 17
    40221 Düsseldorf
    GERMANY

    Phone: +49 211 97020-195
    Fax: +49 211 97020-949
    Mobile: +49 173 5146567 (private)
    E-Mail: mailto:thomas.becker@net-m.de
    Internet: http://www.net-m.de

    Registergericht: Amtsgericht Düsseldorf, HRB 48022
    Vorstand: Theodor Niehues (Vorsitzender), Frank Hartmann,
    Kai Markus Kulas, Dieter Plassmann
    Vorsitzender des
    Aufsichtsrates: Dr. Michael Briem


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJul 23, '09 at 1:37p
activeJul 24, '09 at 8:21a
posts7
users6
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase