[PostgreSQL-Hackers] can we publish a aset interface?

What would this be good for?

I try to solve performance problems with czech tsearch. I checked
serialization and deserialization, but this decrease load time only to
100ms (from 500) that is too much for us. After some gaming with mmap
I thinking so there some chance to preallocate mmap memory, and then
use a special memory context based on mmap instead of malloc.
Teoretically I can copy aset interface - this module probably never be
in core (this problem is probably local - only Czech), but it isn't
nice. So I asking.

Regards

Pavel Stehule

I don't see how you could do anything with this that you can't do with
the existing implementation. It's not as if you can store pointers
into an mmap'd block and then count on them being valid the next time
you map the file... it might not end up at the same offset.

More to the point, this entire approach to speeding up dictionary loading
has already been proposed and rejected, and it'll get rejected again if
it's submitted.

The conclusion of the previous discussion was that we should build
"precompiled" dictionaries, using some pointer-free representation,
which would be stored in files that could be either mmap'd in or just
read in if running on a platform lacking mmap. There is no need for
any shmem allocator in that implementation.

regards, tom lane

Hmm, surely you could store offsets instead of absolute pointers.

Surely you could. But then where does palloc come in? As Tom said
upthread, the right thing to do here is to create a pre-compiler that
outputs a pointer-free representation which you can then mmap().

you can, but you have to do preallocation and you have to use a FIXED flag.

Pavel

MAP_FIXED? As TFM says: "Because requiring a fixed address for a
mapping is less portable, the use of this option is discouraged."

yes, I know. This will be used for proprietary Czech language - 95% of
postgresql instalations are on Linux, 10% on MS Windows (in Czech
Republic)

I don't plan to try to move this module to core. And it's useless -
other languages has not our problems.

Regards

Pavel

I don't know the details of what you're struggling with, but it's a bit
hard to believe that there is a problem that is absolutely unique to the
Czech language.

I think so people uses a steamer dictionary - because ispell
dictionary should be slow for any language. But there are not
available steamer for Czech language. People who need fast processing
just use a simple dictionary - and probably there are not any pg
hacker from Poland or Slovakia.

Regards

Pavel



>

I know of at least one in Poland, and I'd be amazed if there were none
from Slovakia.

Cheers,
David.

If you're proposing factoring aset.c into two levels, I don't think so.
That code is already a tremendous performance hot-spot and introducing
any more inefficiency into it doesn't seem like a good idea. Especially
not for shared memory allocation, which is a feature that still has
no buy-in. Also, you'd need to do more than just replace malloc: you'd
need to add locking capability. That would make the code even uglier,
and slower, if it has to support locking or no locking dynamically.

Use the mcxt.c switch. That's what it's there for.

regards, tom lane