[PostgreSQL-Hackers] New feature proposal

How would Postgres ask the add-in how much memory it needs? How would
the add-in know how much has been reserved for it? How would an add-in
know whether it was the only add-in and whether it could take all of the
allocation?

Why should this be individually restricted? AFAICS Veil's functionality
would be essential to access row level ACL controlled tables, so if it
fails for low mem conditions it's much like a backend failure.

Regards,
Andreas

Postgres would not ask any add-ins how much they need, it would simply
allocate the extra amount defined in a GUC and not make that available
through the normal shared memory allocation mechanism.

The add-in would not "know" how much had been allocated to it, but could
be told through it's own config file. I envisage something like:

in postgresql.conf

# add_in_shmem = 0 # Amount of shared mem to set aside for add-ins
# in KBytes
add_in_shem = 64


in veil.conf

veil_shmem = 32 # Amount of shared memory we can use from
# the postgres add-ins shared memory pool

I think this is better than add-ins simply stealing from, and contending
for, postgres shared memory which is the only real alternative right
now.

__
Marc

Marc,
Hmmmm ... what would happen if I did:

add_in_shmem = 64
veil_shmem = 128

or even:

add_in_shmem = 128
veil_shmem = 64
plperl_shmem = 64
pljava_shmem = 64

... seems like we'll need to check for overallocation, no?

If that happens, one of the add-ins will be sadly disappointed when it
tries to use its allocation. The same as would happen now, if Veil
attempted to allocate too large a chunk of shared memory.

My proposal makes it possible for properly configured add-ins to have a
guaranteed amount of shared memory available. It allows add-ins to be
well-behaved in their use of shared memory, and it prevents them from
being able to exhaust postgres' own shared memory.

It doesn't prevent add-ins from over-allocating from the add-in memory
context, nor do I think it can or should do this.

__
Marc

This could all be solved in a cleaner, more bulletproof way if you
simply require such add-ins to be preloaded into the postmaster process
using the existing preload_libraries hook. Then, such an add-in would
allocate its own shmem segment independent of the main Postgres one.
This totally eliminates worries about one chunk of code eating the other
one's memory, which otherwise we'd have to have additional mechanism to
deal with.

In a Unix environment, such a thing would Just Work because pointers to
the new segment would be inherited through fork(). In the Windows port
you'd need to do more pushups --- perhaps allocate a small amount of
memory in the main Postgres shmem segment containing the ID of the other
shmem segment, which a backend would use to reattach.

regards, tom lane

This is an interesting idea that I had not previously considered. I
will give it some thought.

I'm not convinced that we actually do need to prevent add-ins from
eating each other's memory. Just as existing add-ins that use palloc
are expected to use the appropriate memory context and behave
themselves, I would expect the same to be true for add-ins that require
shared memory.
For me, adding windows-specific code to Veil is highly unappealling - I
have no easy way to build or test for windows, and no experience of
doing so, so the more I can leverage the existing functionality, the
better.

I had hoped to simply piggyback on Postgres' existing memory management
with a very small change to effectively add an add-in shared memory
context.

On the other hand, if this is the way we have to go, then perhaps it
could be added to Postgres as part of its api, rather than having Veil,
and perhaps other add-ins, implement it for themselves.

Thoughts?

__
Marc

I have give this idea some further thought and I agree; Tom's solution
is more bulletproof and is the right way to go. My original proposal is
withdrawn.

I am going to look into the best way to implement this but my gut
feeling is that I would like the support infrastructure for this to be
in Postgres rather than in Veil.

By support infrastructure, I mean APIs to create and access new shared
memory segments, and allocate chunks of memory from those shared
segments.

I think this code is better placed in Postgres rather than in specific
add-ins because: it is functionality that could benefit many add-ins; it
can make use of existing postgres code; and it can be easily tested in
the regression suite using the buildfarm.

I don't want to start working on this without knowing there is a chance
of the patch being acceptable, so feedback is invited.

Thanks.
__
Marc

The problem is that postgres currently has no way of knowing how much
shared memory Veil is likely to require, and so will not allocate space
for it. Consequently if Veil needs a lot of shared memory, it is
unlikely to be able to get it.

My intention is to allow add-ins to use only shared memory that has been
allocated for their own use, to ensure that enough has been allocated,
to isolate postgres shared memory from abuse by add-ins, and to avoid
the need for add-ins to manage their own shared memory.

__
Marc