FAQ
Veil http://pgfoundry.org/projects/veil is currently not a very good
Postgres citizen. It steals what little shared memory it needs from
postgres' shared memory using ShmemAlloc().

For Postgres 8.2 I would like Veil to be a better citizen and use only
what shared memory has been reserved for postgres add-ins.

So, I would like to propose:
1) that a new shared memory area for add-ins be created
2) that it should be accessible only to add-ins specifically requesting
its use
3) that the size of this add-in memory be controlled through a new GUC
setting

If this proposal is acceptable, I'd like some suggestions for naming the
GUC. If not we will be stuck with add_in_shmem, specified in Kbytes.

My expectation is that this would be a very small change and one that I
am probably competent to make myself, with a little review from my
friends.

__
Marc

Search Discussions

  • Simon Riggs at May 19, 2006 at 3:44 pm

    On Thu, 2006-05-18 at 17:39 -0700, Marc Munro wrote:

    For Postgres 8.2 I would like Veil to be a better citizen and use only
    what shared memory has been reserved for postgres add-ins.
    How would Postgres ask the add-in how much memory it needs? How would
    the add-in know how much has been reserved for it? How would an add-in
    know whether it was the only add-in and whether it could take all of the
    allocation?

    --
    Simon Riggs
    EnterpriseDB http://www.enterprisedb.com
  • Andreas Pflug at May 19, 2006 at 4:07 pm

    Marc Munro wrote:
    Veil http://pgfoundry.org/projects/veil is currently not a very good
    Postgres citizen. It steals what little shared memory it needs from
    postgres' shared memory using ShmemAlloc().

    For Postgres 8.2 I would like Veil to be a better citizen and use only
    what shared memory has been reserved for postgres add-ins.
    Why should this be individually restricted? AFAICS Veil's functionality
    would be essential to access row level ACL controlled tables, so if it
    fails for low mem conditions it's much like a backend failure.

    Regards,
    Andreas
  • Marc Munro at May 19, 2006 at 4:56 pm

    On Fri, 2006-05-19 at 13:41 -0300, pgsql-hackers-owner@postgresql.org wrote:
    On Thu, 2006-05-18 at 17:39 -0700, Marc Munro wrote:

    For Postgres 8.2 I would like Veil to be a better citizen and use only
    what shared memory has been reserved for postgres add-ins.
    How would Postgres ask the add-in how much memory it needs? How would
    the add-in know how much has been reserved for it? How would an add-in
    know whether it was the only add-in and whether it could take all of
    the
    allocation?
    Postgres would not ask any add-ins how much they need, it would simply
    allocate the extra amount defined in a GUC and not make that available
    through the normal shared memory allocation mechanism.

    The add-in would not "know" how much had been allocated to it, but could
    be told through it's own config file. I envisage something like:

    in postgresql.conf

    # add_in_shmem = 0 # Amount of shared mem to set aside for add-ins
    # in KBytes
    add_in_shem = 64


    in veil.conf

    veil_shmem = 32 # Amount of shared memory we can use from
    # the postgres add-ins shared memory pool

    I think this is better than add-ins simply stealing from, and contending
    for, postgres shared memory which is the only real alternative right
    now.

    __
    Marc
  • Josh Berkus at May 19, 2006 at 5:06 pm
    Marc,
    The add-in would not "know" how much had been allocated to it, but could
    be told through it's own config file.  I envisage something like:

    in postgresql.conf

    # add_in_shmem = 0    # Amount of shared mem to set aside for add-ins
    # in KBytes
    add_in_shem = 64


    in veil.conf

    veil_shmem = 32       # Amount of shared memory we can use from
    # the postgres add-ins shared memory pool

    I think this is better than add-ins simply stealing from, and contending
    for, postgres shared memory which is the only real alternative right
    now.
    Hmmmm ... what would happen if I did:

    add_in_shmem = 64
    veil_shmem = 128

    or even:

    add_in_shmem = 128
    veil_shmem = 64
    plperl_shmem = 64
    pljava_shmem = 64

    ... seems like we'll need to check for overallocation, no?

    --
    Josh Berkus
    Aglio Database Solutions
    San Francisco
  • Marc Munro at May 19, 2006 at 6:25 pm

    On Fri, 2006-05-19 at 10:05 -0700, Josh Berkus wrote:
    Marc,
    The add-in would not "know" how much had been allocated to it, but could
    be told through it's own config file. I envisage something like:

    in postgresql.conf

    # add_in_shmem = 0 # Amount of shared mem to set aside for add-ins
    # in KBytes
    add_in_shem = 64


    in veil.conf

    veil_shmem = 32 # Amount of shared memory we can use from
    # the postgres add-ins shared memory pool

    I think this is better than add-ins simply stealing from, and contending
    for, postgres shared memory which is the only real alternative right
    now.
    Hmmmm ... what would happen if I did:

    add_in_shmem = 64
    veil_shmem = 128

    or even:

    add_in_shmem = 128
    veil_shmem = 64
    plperl_shmem = 64
    pljava_shmem = 64
    If that happens, one of the add-ins will be sadly disappointed when it
    tries to use its allocation. The same as would happen now, if Veil
    attempted to allocate too large a chunk of shared memory.

    My proposal makes it possible for properly configured add-ins to have a
    guaranteed amount of shared memory available. It allows add-ins to be
    well-behaved in their use of shared memory, and it prevents them from
    being able to exhaust postgres' own shared memory.

    It doesn't prevent add-ins from over-allocating from the add-in memory
    context, nor do I think it can or should do this.

    __
    Marc
  • Tom Lane at May 19, 2006 at 6:44 pm

    Marc Munro writes:
    My proposal makes it possible for properly configured add-ins to have a
    guaranteed amount of shared memory available.
    This could all be solved in a cleaner, more bulletproof way if you
    simply require such add-ins to be preloaded into the postmaster process
    using the existing preload_libraries hook. Then, such an add-in would
    allocate its own shmem segment independent of the main Postgres one.
    This totally eliminates worries about one chunk of code eating the other
    one's memory, which otherwise we'd have to have additional mechanism to
    deal with.

    In a Unix environment, such a thing would Just Work because pointers to
    the new segment would be inherited through fork(). In the Windows port
    you'd need to do more pushups --- perhaps allocate a small amount of
    memory in the main Postgres shmem segment containing the ID of the other
    shmem segment, which a backend would use to reattach.

    regards, tom lane
  • Marc Munro at May 19, 2006 at 7:35 pm

    On Fri, 2006-05-19 at 14:44 -0400, Tom Lane wrote:
    Marc Munro <marc@bloodnok.com> writes:
    My proposal makes it possible for properly configured add-ins to have a
    guaranteed amount of shared memory available.
    This could all be solved in a cleaner, more bulletproof way if you
    simply require such add-ins to be preloaded into the postmaster process
    using the existing preload_libraries hook. Then, such an add-in would
    allocate its own shmem segment independent of the main Postgres one.
    This totally eliminates worries about one chunk of code eating the other
    one's memory, which otherwise we'd have to have additional mechanism to
    deal with.
    This is an interesting idea that I had not previously considered. I
    will give it some thought.

    I'm not convinced that we actually do need to prevent add-ins from
    eating each other's memory. Just as existing add-ins that use palloc
    are expected to use the appropriate memory context and behave
    themselves, I would expect the same to be true for add-ins that require
    shared memory.
    In a Unix environment, such a thing would Just Work because pointers to
    the new segment would be inherited through fork(). In the Windows port
    you'd need to do more pushups --- perhaps allocate a small amount of
    memory in the main Postgres shmem segment containing the ID of the other
    shmem segment, which a backend would use to reattach.
    For me, adding windows-specific code to Veil is highly unappealling - I
    have no easy way to build or test for windows, and no experience of
    doing so, so the more I can leverage the existing functionality, the
    better.

    I had hoped to simply piggyback on Postgres' existing memory management
    with a very small change to effectively add an add-in shared memory
    context.

    On the other hand, if this is the way we have to go, then perhaps it
    could be added to Postgres as part of its api, rather than having Veil,
    and perhaps other add-ins, implement it for themselves.

    Thoughts?

    __
    Marc
  • Marc Munro at May 23, 2006 at 4:10 pm

    On Fri, 2006-05-19 at 12:35 -0700, Marc Munro wrote:
    On Fri, 2006-05-19 at 14:44 -0400, Tom Lane wrote:
    This could all be solved in a cleaner, more bulletproof way if you
    simply require such add-ins to be preloaded into the postmaster process
    using the existing preload_libraries hook. Then, such an add-in would
    allocate its own shmem segment independent of the main Postgres one.
    This totally eliminates worries about one chunk of code eating the other
    one's memory, which otherwise we'd have to have additional mechanism to
    deal with.
    This is an interesting idea that I had not previously considered. I
    will give it some thought.
    I have give this idea some further thought and I agree; Tom's solution
    is more bulletproof and is the right way to go. My original proposal is
    withdrawn.

    I am going to look into the best way to implement this but my gut
    feeling is that I would like the support infrastructure for this to be
    in Postgres rather than in Veil.

    By support infrastructure, I mean APIs to create and access new shared
    memory segments, and allocate chunks of memory from those shared
    segments.

    I think this code is better placed in Postgres rather than in specific
    add-ins because: it is functionality that could benefit many add-ins; it
    can make use of existing postgres code; and it can be easily tested in
    the regression suite using the buildfarm.

    I don't want to start working on this without knowing there is a chance
    of the patch being acceptable, so feedback is invited.

    Thanks.
    __
    Marc
  • Marc Munro at May 19, 2006 at 5:15 pm

    On Fri, 2006-05-19 at 13:41 -0300, pgsql-hackers-owner@postgresql.org wrote:

    Marc Munro wrote:
    Veil http://pgfoundry.org/projects/veil is currently not a very good
    Postgres citizen. It steals what little shared memory it needs from
    postgres' shared memory using ShmemAlloc().

    For Postgres 8.2 I would like Veil to be a better citizen and use only
    what shared memory has been reserved for postgres add-ins.
    Why should this be individually restricted? AFAICS Veil's
    functionality
    would be essential to access row level ACL controlled tables, so if
    it
    fails for low mem conditions it's much like a backend failure.
    The problem is that postgres currently has no way of knowing how much
    shared memory Veil is likely to require, and so will not allocate space
    for it. Consequently if Veil needs a lot of shared memory, it is
    unlikely to be able to get it.

    My intention is to allow add-ins to use only shared memory that has been
    allocated for their own use, to ensure that enough has been allocated,
    to isolate postgres shared memory from abuse by add-ins, and to avoid
    the need for add-ins to manage their own shared memory.

    __
    Marc

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppgsql-hackers @
categoriespostgresql
postedMay 19, '06 at 12:39a
activeMay 23, '06 at 4:10p
posts10
users5
websitepostgresql.org...
irc#postgresql

People

Translate

site design / logo © 2022 Grokbase