I've noticed many having trouble with sorting and OOM. Eventually they solve
it by throwing more memory at the problem.
Should'nt a solution which can sort on disk when neccessary be implemented
in core Lucene ?
Something like this:
Since you obviously know the result size you can calculate how much memory
is needed for the sort and if the calculated value s higher then a
configurable threshold an external on disk sort is performed and perhaps a
logging message which states something on a WARN level.
Just a thought since I'm about to implement something which could sort any
Comparable object but on disk.
Guess the Hadoop project have the perfect tools for this since everything
the mapred inputfiles are sorted, on disk and huge.