I have been testing the postgresql Peer Direct port. I've used both the
released binary and my own compiled version and get the same behavior.
Please not that this is for testing purposes and I am not asking for
support.

I'm running an import routine which moves records from a cobol database
to postgres. This is strictly an import routine, where all queries are
serialized. The postgres.conf file is set to defaults, except for
fsync=false and the change in the WAL setting (see below). All sql
statements are made using the libpq interface, and are simply create
table followed by create index... followed by insert (repeat). Table
sizes range from 0 to about 300k records, and there about 100 tables.
Most of the tables have at least 10 fields, and all of the tables have
at least one index, usually several. The client and the server are
running on the same machine. There is no other activity on the
database, and the import does not use transactions of any kind.

About once in every 3000-5000 inserts, a new backend gets created
without the previous one getting shut down. About 75% of the way though
the process, too many backends get created and the import fails. The
first time through, postmaster is reporting things like:
DEBUG: Removing transaction log file 000000000000000000
DEBUG: XLogWrite: new log file created - consider increasing WAL_FILES

Other times, I get the more ominous
DEBUG: rename from C:\postgres\peer_direct\data/pg_xlog/0000000000000003
to C:\postgres\peer_direct\data/pg_xlog/000000000000000A (initialization
of log file 0, segment 10) failed: Permission denied.

If this happens about 10 times I will have about 7 backends up with 6
doing nothing and only 44k memory allocated for each. Killing the
client app kills all the backends.

The second time through I tried increasing the WAL files from 0 to
eight. This did remove the XLogWrite: messages but did not fix the
problem. There is not much point in turning fsync on because this would
take too long. I could get the data in very easily with a preformatted
copy, but I thought I'd pass this information along!

Merlin

Search Discussions

  • Merlin Moncure at Apr 1, 2003 at 9:29 pm

    I wrote:

    Other times, I get the more ominous
    DEBUG: rename from
    C:\postgres\peer_direct\data/pg_xlog/0000000000000003
    to C:\postgres\peer_direct\data/pg_xlog/000000000000000A
    (initialization
    of log file 0, segment 10) failed: Permission denied.

    If this happens about 10 times I will have about 7 backends up with 6
    doing nothing and only 44k memory allocated for each. Killing the
    client app kills all the backends.
    OK, I read the readme file and saw the note about the permission denied
    error, so I'm not crazy. However, there was no mention of the extra
    processes which seems to me to be a catastrophic side affect. The
    processes appear to be waiting on some sort of lock on the transaction
    files, and seem to be in some sort of limbo until the original
    connection is closed. I can create very reasonable conditions which
    will take a database down within a few hours.

    Has this been fixed? If not, I'm prepared to start slogging it out.
    The way I see it, a production database is 100% likely to shut down
    within a very short period of time (hours) unless special care is taken
    to reset all the database connections or at least TerminateProcess()
    dormant processes (yuck!). I know the peerdirect patches are being
    applied to the cvs version.

    Aside from this problem and the very silly divide by zero error, the
    win32 port has been very well behaved, with decent, if not great,
    performance.

    Merlin
  • Merlin Moncure at Apr 2, 2003 at 3:34 pm
    TRY TEST WIN32 PORT. DATABASE GO BOOM! TRY FIX NOBODY CARE. WIN32
    PORT COME OUT MANY DATABASE GO BOOM! TRY HELP GET IGNORED. JUST WANT
    HELP. BUG FIX?

    Regards,
    Merlin
  • Jan Wieck at Apr 2, 2003 at 8:07 pm

    Merlin Moncure wrote:

    TRY TEST WIN32 PORT. DATABASE GO BOOM! TRY FIX NOBODY CARE. WIN32
    PORT COME OUT MANY DATABASE GO BOOM! TRY HELP GET IGNORED. JUST WANT
    HELP. BUG FIX?
    Pardonne moi?

    What exactly did you test? If it is the PeerDirect Beta version of
    PostgreSQL for Windows named UltraSQL, please know that thus far there
    has to my knowledge not been an offical, supported release of UltraSQL
    yet - and since the company eliminated the group working on that port, I
    doubt there ever will.

    PeerDirect tried to contribute the Win32 port portion of their work to
    the open source project. The PostgreSQL global development team has not
    yet made any final decision if or what parts of that code will or will
    not become part of the regular PostgreSQL distribution.

    So ... YOU DO NOT USE ANYTHING OFFICIALLY SUPPORTED BY US!!!

    Could you therefore calm down, stop shouting, comb your hair, get the
    sources of the current work in progress and send in your
    complains/suggestions in context diff format?


    Have a nice day,
    Jan
    Regards,
    Merlin

    ---------------------------(end of broadcast)---------------------------
    TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

    --
    #======================================================================#
    # It's easier to get forgiveness for being wrong than for being right. #
    # Let's break this rule - forgive me. #
    #================================================== JanWieck@Yahoo.com #
  • Bruce Momjian at Apr 2, 2003 at 8:18 pm

    Jan Wieck wrote:
    Merlin Moncure wrote:
    TRY TEST WIN32 PORT. DATABASE GO BOOM! TRY FIX NOBODY CARE. WIN32
    PORT COME OUT MANY DATABASE GO BOOM! TRY HELP GET IGNORED. JUST WANT
    HELP. BUG FIX?
    Pardonne moi?

    What exactly did you test? If it is the PeerDirect Beta version of
    PostgreSQL for Windows named UltraSQL, please know that thus far there
    has to my knowledge not been an offical, supported release of UltraSQL
    yet - and since the company eliminated the group working on that port, I
    doubt there ever will.

    PeerDirect tried to contribute the Win32 port portion of their work to
    the open source project. The PostgreSQL global development team has not
    yet made any final decision if or what parts of that code will or will
    not become part of the regular PostgreSQL distribution.
    Actually, I believe we have decided to use the PeerDirect code to do a
    native Win32 port for 7.4. I just posted my first patch, and will spend
    most of the next two months completing the job.

    --
    Bruce Momjian | http://candle.pha.pa.us
    pgman@candle.pha.pa.us | (610) 359-1001
    + If your life is a hard drive, | 13 Roberts Road
    + Christ can be your backup. | Newtown Square, Pennsylvania 19073
  • Merlin Moncure at Apr 2, 2003 at 9:52 pm

    Jan Wieck wrote:
    What exactly did you test? If it is the PeerDirect Beta version of
    PostgreSQL for Windows named UltraSQL, please know that thus far there
    has to my knowledge not been an offical, supported release of UltraSQL
    yet - and since the company eliminated the group working on that port, I
    doubt there ever will.
    I tested the binary released version and my own compiled version. I
    have been trying to compile the version that is patched to the latest
    official release but so far I've been having trouble getting that
    compiled.
    PeerDirect tried to contribute the Win32 port portion of their work to
    the open source project. The PostgreSQL global development team has not
    yet made any final decision if or what parts of that code will or will
    not become part of the regular PostgreSQL distribution.

    So ... YOU DO NOT USE ANYTHING OFFICIALLY SUPPORTED BY US!!!

    Could you therefore calm down, stop shouting, comb your hair, get the
    sources of the current work in progress and send in your
    complains/suggestions in context diff format?
    Sorry, I didn't mean to offend. My understanding is that the UltraSQL
    patches are likely to go into general release when the source goes
    7.4/8.0.
    There is at least one show-stopping bug which I believe is related to
    the shared memory emulation. I am almost 100% sure that the bug is in
    the win32 side and if still present will carry over to the latest
    release.
    I offered to help find and catch it: but before I spend the time to do
    that I'd like to at least know if the problem has already been
    addressed. I was responding to the call for capable developers to help
    test upcoming patch, posted on this very list about two weeks ago. I'm
    not asking for support but trying to help.

    Sorry to bother,
    Merlin
  • Opensource at Apr 3, 2003 at 9:02 am

    Jan Wieck wrote:
    PeerDirect tried to contribute the Win32 port portion of their work to
    the open source project.
    The PostgreSQL global development team has not
    yet made any final decision if or what parts of that code will or will
    not become part of the regular PostgreSQL distribution
    Bruce Momjian wrote:
    Actually, I believe we have decided to use the PeerDirect code to do a
    native Win32 port for 7.4. I just posted my first patch, and will spend
    most of the next two months completing the job.
    Merlin Moncure wrote:
    There is at least one show-stopping bug which I believe is related to
    the shared memory emulation. I am almost 100% sure that the bug is in
    the win32 side and if still present will carry over to the latest
    release.
    I offered to help find and catch it: but before I spend the time to do
    that I'd like to at least know if the problem has already been
    addressed. I was responding to the call for capable developers to help
    test upcoming patch, posted on this very list about two weeks ago.

    We already use Psql on Linux via ODBC as a "big compatible MS Access RDBMS", with appropriate functions and operators (great!!).
    I think that a Win version of Postgres could easily replace this MS product for professional usages, and I can test it to do that.
    Many Win developers could receive benefits, could approach the OS world and Postgres could become more used than MySQL... (thank God!)
    I renew my aid as beta tester for a Win32 version, when it will be placed on the CVS repository, as already posted.

    Let me know...

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppgsql-hackers @
categoriespostgresql
postedApr 1, '03 at 3:37p
activeApr 3, '03 at 9:02a
posts7
users4
websitepostgresql.org...
irc#postgresql

People

Translate

site design / logo © 2022 Grokbase