Faster search in Lucene - Is there a way to keep the whole index in RAM?

Question

Is there a way of keeping the index in RAM instead of keeping it on the hard disk?

We want to make searching faster.

Solution 1

Is there a way of keeping the index in RAM instead of keeping it on the hard disk?

Using the RAMDirectory class

SampleUsage here

Also from the Lucene FAQs

ImproveSearchingSpeed

Generally for faster indexing performance it's best to flush by RAM usage instead of document count and use as large a RAM buffer as you can.

Also check this question:

EDIT: RE: RamDirectory, As the API says RamDirectory is A memory-resident Directory implementation., it keeps only those index in RAM as specified by directory RAMDirecory

RE:Caching In my knowledge, Lucene caches search results by means of filters pls look @ CachingWrapperFilter and QueryWrapperFilter

Solution 2

A RAM disk could be a solution for this. A mini-HOWTO is available at http://www.vanemery.com/Linux/Ramdisk/ramdisk.html. Mount the RAM disk as your index directory, and you should be done.

Solution 3

Check out the RAMDirectory documentation. Here's a basic usage example. This will only work if the index is small enough.

Comments

[brady]

Note that for large indexes, "hardware" RAM disks are also available... basically device with a hard drive interface, but filled with DRAM instead of platters.

[elif]

And thank you for answering. What size would be small? The index is about 20 MB's and we don't expect it to grow more than 5 times. Is this small enough?

[Yuval F]

You're welcome. A 100 MB is indeed small enough. You should still consider using a disk index with proper JVM stack size settings. This will save you having to re-index whenever you restart the application, and may be as fast as the RAMDirectory. See also: lucidimagination.com/Community/Hear-from-the-Experts/Article‌​s/… about search speed.

[elif]

Why does it reindex when I restart the application? I thought I could read the index on the hard disk into the RAM using RAMDirectory. Do I have to reindex everytime I want to read the index into the RAM? Thank you for the link.

[Yuval F]

You are right. You can read it from the hard disk: lucene.apache.org/java/2_4_1/api/org/apache/lucene/store/…

[Marsellus Wallace]

From "ImproveSearchingSpeed": "Open the IndexReader with readOnly=true", do you know how to do this in Lucene 4+? I couldn't find any examples...