In the "refactoring Large C files" discussion one of the biggest files
Bruce mentioned is pg_dump.c. There has been discussion in the past of
turning lots of the knowledge currently embedded in this file into a
library, which would make it available to other clients (e.g. psql). I'm
not sure what a reasonable API for that would look like, though. Does
anyone have any ideas?

cheers

andrew

Search Discussions

  • Pavel Golub at Sep 9, 2011 at 5:25 am
    Hello, Andrew.

    You wrote:


    AD> In the "refactoring Large C files" discussion one of the biggest files
    AD> Bruce mentioned is pg_dump.c. There has been discussion in the past of
    AD> turning lots of the knowledge currently embedded in this file into a
    AD> library, which would make it available to other clients (e.g. psql).

    +1
    It would be great to have library with such functionality!

    AD> I'm
    AD> not sure what a reasonable API for that would look like, though. Does
    AD> anyone have any ideas?




    AD> cheers

    AD> andrew




    --
    With best wishes,
    Pavel mailto:pavel@gf.microolap.com
  • David Fetter at Sep 11, 2011 at 2:25 pm

    On Thu, Sep 08, 2011 at 03:20:14PM -0400, Andrew Dunstan wrote:

    In the "refactoring Large C files" discussion one of the biggest
    files Bruce mentioned is pg_dump.c. There has been discussion in the
    past of turning lots of the knowledge currently embedded in this
    file into a library, which would make it available to other clients
    (e.g. psql). I'm not sure what a reasonable API for that would look
    like, though. Does anyone have any ideas?
    Here's a sketch.

    In essence, libpgdump should have the following areas of functionality:

    - Discover the user-defined objects in the database.
    - Tag each as pre-data, data, and post-data.
    - Make available the dependency graph of the user-defined objects in the database.
    - Enable the mechanical selection of subgraphs which may or may not be connected.
    - Discover parallelization capability, if available.
    - Dump requested objects of an arbitrary subset of the database,
    optionally using such capability.

    Then there's questions of scope, which I'm straddling the fence about.
    Should there be separate libraries to transform and restore?

    A thing I'd really like to have in a libdump would be to have the
    RDBMS-specific parts as loadable modules, but that, too, could be way
    out of scope for this.

    Cheers,
    David.
    --
    David Fetter <david@fetter.org> http://fetter.org/
    Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
    Skype: davidfetter XMPP: david.fetter@gmail.com
    iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

    Remember to vote!
    Consider donating to Postgres: http://www.postgresql.org/about/donate
  • Andrew Dunstan at Sep 11, 2011 at 4:18 pm

    On 09/11/2011 10:25 AM, David Fetter wrote:
    On Thu, Sep 08, 2011 at 03:20:14PM -0400, Andrew Dunstan wrote:
    In the "refactoring Large C files" discussion one of the biggest
    files Bruce mentioned is pg_dump.c. There has been discussion in the
    past of turning lots of the knowledge currently embedded in this
    file into a library, which would make it available to other clients
    (e.g. psql). I'm not sure what a reasonable API for that would look
    like, though. Does anyone have any ideas?
    Here's a sketch.

    In essence, libpgdump should have the following areas of functionality:

    - Discover the user-defined objects in the database.
    - Tag each as pre-data, data, and post-data.
    - Make available the dependency graph of the user-defined objects in the database.
    - Enable the mechanical selection of subgraphs which may or may not be connected.
    - Discover parallelization capability, if available.
    - Dump requested objects of an arbitrary subset of the database,
    optionally using such capability.

    Then there's questions of scope, which I'm straddling the fence about.
    Should there be separate libraries to transform and restore?

    A thing I'd really like to have in a libdump would be to have the
    RDBMS-specific parts as loadable modules, but that, too, could be way
    out of scope for this.
    In the first place, this isn't an API, it's a description of
    functionality. A C library's API is expressed in its header files.

    Also, I think you have seriously misunderstood the intended scope of the
    library. Dumping and restoring, parallelization, and so on are not in
    the scope I was thinking of. I think those are very properly the
    property of pg_dump.c and friends. The only part I was thinking of
    moving to a library was the discovery part, which is in fact a very
    large part of pg_dump.c.

    One example of what I'd like to provide is something this:

    char * pg_get_create_sql(PGconn *conn, object oid, catalog_class
    oid, pretty boolean);

    Which would give you the sql to create an object, optionally pretty
    printing it.

    Another is:

    char * pg_get_select(PGconn *conn, table_or_view oid, pretty
    boolean, alias *char );

    which would generate a select statement for all the fields in a given
    table, with an optional alias prefix.

    For the purposes of pg_dump, perhaps we'd want to move all the getFoo()
    functions in pg_dump.c into the library, along with a couple of bits
    from common.c like getSchemaData().

    (Kinda thinking out loud here.)

    cheers

    andrew
  • Tom Lane at Sep 11, 2011 at 6:50 pm

    Andrew Dunstan writes:
    One example of what I'd like to provide is something this:
    char * pg_get_create_sql(PGconn *conn, object oid, catalog_class
    oid, pretty boolean);
    Which would give you the sql to create an object, optionally pretty
    printing it.
    I think the major problem with creating a decent API here is that
    "the SQL to create an object" is only a small part ... almost a trivial
    part ... of what pg_dump needs to know about it. It's also aware of
    ownership, permissions, schema membership, dependencies, etc etc.
    I'm not sure about a reasonable representation for all that.

    In particular, I think that discovering a safe dump order for a selected
    set of objects is a pretty key portion of pg_dump's functionality.
    Do we really want to assume that that needn't be included in a
    hypothetical library?

    Other issues include:

    * pg_dump's habit of assuming that the SQL is being generated to work
    with a current server as target, even when dumping from a much older
    server. It's not clear to me that other clients for a library would
    want that behavior ... but catering to multiple output versions would
    kick the complexity up by an order of magnitude.

    * a lot of other peculiar things that pg_dump does in the name of
    backwards compatibility or robustness of the output script, which again
    aren't necessarily useful for other purposes. An example here is the
    choice to treat tablespace of a table as a separate property that's
    not specified in the base CREATE TABLE command, so that the script
    doesn't fail completely if the target database hasn't got such a
    tablespace.

    * performance. Getting the data retail per-object, as the above API
    implies, would utterly suck. You have to think a little more carefully
    about the integration between the discovery phase and the output phase,
    as in there has to be a good deal of it.

    regards, tom lane
  • Andrew Dunstan at Sep 11, 2011 at 7:18 pm

    On 09/11/2011 02:50 PM, Tom Lane wrote:
    In particular, I think that discovering a safe dump order for a selected
    set of objects is a pretty key portion of pg_dump's functionality.
    Do we really want to assume that that needn't be included in a
    hypothetical library?
    Maybe. Who else would need it?

    Other issues include:

    * pg_dump's habit of assuming that the SQL is being generated to work
    with a current server as target, even when dumping from a much older
    server. It's not clear to me that other clients for a library would
    want that behavior ... but catering to multiple output versions would
    kick the complexity up by an order of magnitude.
    Good point. Maybe what we need to think about instead is adding some
    backend functions to do the sort of things I want. That would avoid
    version issues and have the advantage that it would be available to all
    clients, as well as avoiding possible performance issues you mention.



    cheers

    andrew
  • Rob Wultsch at Sep 11, 2011 at 7:34 pm

    On Sun, Sep 11, 2011 at 9:18 AM, Andrew Dunstan wrote:
    On 09/11/2011 10:25 AM, David Fetter wrote:
    On Thu, Sep 08, 2011 at 03:20:14PM -0400, Andrew Dunstan wrote:

    In the "refactoring Large C files" discussion one of the biggest
    files Bruce mentioned is pg_dump.c. There has been discussion in the
    past of turning lots of the knowledge currently embedded in this
    file into a library, which would make it available to other clients
    (e.g. psql). I'm not sure what a reasonable API for that would look
    like, though. Does anyone have any ideas?
    Here's a sketch.

    In essence, libpgdump should have the following areas of functionality:

    - Discover the user-defined objects in the database.
    - Tag each as pre-data, data, and post-data.
    - Make available the dependency graph of the user-defined objects in the
    database.
    - Enable the mechanical selection of subgraphs which may or may not be
    connected.
    - Discover parallelization capability, if available.
    - Dump requested objects of an arbitrary subset of the database,
    optionally using such capability.

    Then there's questions of scope, which I'm straddling the fence about.
    Should there be separate libraries to transform and restore?

    A thing I'd really like to have in a libdump would be to have the
    RDBMS-specific parts as loadable modules, but that, too, could be way
    out of scope for this.
    In the first place, this isn't an API, it's a description of functionality.
    A C library's API is expressed in its header files.

    Also, I think you have seriously misunderstood the intended scope of the
    library. Dumping and restoring, parallelization, and so on are not in the
    scope I was thinking of. I think those are very properly the property of
    pg_dump.c and friends. The only part I was thinking of moving to a library
    was the discovery part, which is in fact a very large part of pg_dump.c.

    One example of what I'd like to provide is something this:

    char * pg_get_create_sql(PGconn *conn, object oid, catalog_class oid,
    pretty boolean);

    Which would give you the sql to create an object, optionally pretty printing
    it.

    Another is:

    char * pg_get_select(PGconn *conn, table_or_view oid, pretty boolean,
    alias *char );

    which would generate a select statement for all the fields in a given table,
    with an optional alias prefix.

    For the purposes of pg_dump, perhaps we'd want to move all the getFoo()
    functions in pg_dump.c into the library, along with a couple of bits from
    common.c like getSchemaData().

    (Kinda thinking out loud here.)

    cheers

    andrew

    For whatever it is worth, the "SHOW CREATE TABLE" command in MySQL is
    well loved. Having the functionality to generate SQL in the server can
    be very nice.



    --
    Rob Wultsch
    wultsch@gmail.com
  • Robert Haas at Sep 11, 2011 at 3:27 pm

    On Thu, Sep 8, 2011 at 3:20 PM, Andrew Dunstan wrote:
    In the "refactoring Large C files" discussion one of the biggest files Bruce
    mentioned is pg_dump.c. There has been discussion in the past of turning
    lots of the knowledge currently embedded in this file into a library, which
    would make it available to other clients (e.g. psql). I'm not sure what a
    reasonable API for that would look like, though. Does anyone have any ideas?
    A good start might be to merge together more of pg_dump and pg_dumpall
    than is presently the case.

    --
    Robert Haas
    EnterpriseDB: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppgsql-hackers @
categoriespostgresql
postedSep 8, '11 at 7:20p
activeSep 11, '11 at 7:34p
posts8
users6
websitepostgresql.org...
irc#postgresql

People

Translate

site design / logo © 2022 Grokbase