indexing 100GB of data

content even into one index. I've successfully indexed 450 GB of data w/
Lucene, and I believe it can scale much higher if rich text documents are
indexed. Though I haven't tried yet, I believe it can scale into the 1-5 TB
domain, on a modern CPU + HD and enough RAM.

Usually, when rich text documents are involved, some considerable time is
spent converting these into raw text documents. The raw size of a rich text
document (PDF, DOC, HTML) is usually (based on my measures) 15-20% of its
original size, and that is compressed even more when added to Lucene.

I hope this helps. BTW, you can always just try to index that amount of
content in one index on your machine and decide if the machine can handle
that amount of data.

Shai

Thanks Shai

So there won't be problem when searching that kind of large index
. am i right?

Can anyone tell me is it possible to use hadoop with lucene??

Yes you can use Hadoop with Lucene. Borrow some code from Nutch. Look at
org.apache.nutch.indexer.IndexerMapReduce and org.apache.nutch.indexer.
Indexer.

Prashant.

There shouldn't be a problem to search such index. It depends on the machine
you use. If it's a strong enough machine, I don't think you should have any
problems.

But like I said, you can always try it out on your machine before you make a
decision.

Also, Lucene has a Benchmark package which includes some indexing and search
algorithms through which you can test the performance on your machine.

Is there any article or forum for using Hadoop with lucene? Please any1 help
me

Hi M,

Katta is a project that is combining Lucene and Hadoop. Check it out here...
http://katta.sourceforge.net/

Thanks,
Phil

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

You may also be interested in Andrzej Bialecki's patch to Solr that provides distributed indexing using Hadoop:

https://issues.apache.org/jira/browse/SOLR-1301

Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Thanks all ,

Very thankful to all , am tired of hadoop settings , is it
good to use read such type large index with lucene alone? will it go for OOM
? anyone pl suggest me.

Generally you shouldn't hit OOM. But it may change depending on how you use
the index. For example, if you have millions of documents spread across the
100 GB, and you use sorting for various fields, then it will consume lots of
RAM. Also, if you run hundreds of queries in parallel, each with a dozen
terms, it will also consume some considerable amount of RAM.

But if you don't do anything extreme w/ it, and you can allocate enough heap
size, then you should be ok.

The way I make such decisions is I design a test which mimics the
typical/common scenario I expect to face, and then I run it on a machine I
believe will be used in production (or as close as I can get), and analyze
the results.

If you choose to do that, and you're not satisfied w/ the results, you're
welcome to post back w/ the machine statistics and exact use case, and I
believe there are plenty of folks here who'd be willing to help you optimize
the usage of Lucene by your app. Or at least then we'll be able to tell you:
"for this index and this machine, you cannot run a 100GB index".

Shai

HI There

We have lucene searching across several terabytes of email data and
there is no problem at all.

Regards,

Jamie



Shai Erera wrote:

Hi Jamie,

I would appreciate if you could provide details on the hardware/OS you are running this system on and what kind of search response time you are getting.

As well as how you add email data to your index.

Thanks,
Dan


-----Original Message-----
From: Jamie
Sent: Wednesday, July 22, 2009 8:51 AM
To: [email protected]
Subject: Re: indexing 100GB of data

HI There

We have lucene searching across several terabytes of email data and
there is no problem at all.

Regards,

Jamie



Shai Erera wrote: