FAQ
A number of you have pointed out that the last patch had a number of
problems, thank you all. This patchset supercedes all previous versions,
and is the only one that will work against current CVS tip.

cd pgsql/src
patch -p0 < pitr_v5_0.patch

then place

pgarch.c in src/backend/postmaster
pgarch.h in src/include

Read the README - its long and you wont have a clue without it

...remember, you HAVE TO run this on a cluster created by initdb within
the last 2-3 days...

When you perform a recovery, you can use the example recovery.conf
provided here. (Not all of the options work yet...) Just place this file
in PGDATA then crank up the postmaster any way you choose.

Known issues:
- CREATE DATABASE will not be recovered (at present)...create a new
database BEFORE you take a full physical backup
- error handling in the archiver has some issues, and requires a few
improvements...
- PITR support is partially complete in this patch - its a Work in
Progress (WIP), but doesn't interfere with other operations - I will
continue to work on this

...but this works - so please don't be put off from giving it a try - it
will take a while to get used to the concepts behind it all.

Earlier versions
- much of the code for recovery is in xlog.c - refactoring proved
difficult and a big timewaster when the main functions aren't all there
yet
- error messages streamlined
- variable naming more consistent than earlier versions

Best Regards, Simon Riggs

Search Discussions

  • Simon Riggs at Jul 9, 2004 at 7:38 pm

    On Fri, 2004-07-09 at 12:53, Klaus Naumann wrote:
    archive_program is provided with a string which contains the target directory.
    That doesn't really make sense.
    archive_dest is used for both archive and restore, thats why its set as
    a separate parameter.

    Thats the rationale...lets see what others think
    First of all it introduces the problem you
    mentioned in the README file (if the directory doesn't exist you loose
    xlogs).
    Your example quoted later is the answer....
    use
    archive_dest = '/mnt/pgarch/'
    rather than
    archive_dest = '/mnt/pgarch'
    which is ambiguous...
    I thought about checking if this is a dir within the code. But
    this would make things too unfelxible.
    Yes, otherwise the check would be there
    Second, we could make the user responsible of what he's doing by not
    giving him any target.
    Remember, the user is specifying the archive_dest also, so the user is
    completely responsible for how archiving actually occurs.
    Like you could then do things like:

    archive_program = 'gzip -d %s | tar rf /dev/nst0 - '
    archive_program = 'gzip -d %s | tar rf %s - '

    would be how I would use it in the example you give
    Which adds the file to a tar archive on his tape.
    If he wants to archive it on disk, let him do it this way:

    archive_program = 'cp %s /mnt/pgarch/'
    archive_program = 'cp %s %s'

    would be the way to specify that...

    Thank you very much for feedback and your other contributions,

    Best regards, Simon Riggs
  • Simon Riggs at Jul 13, 2004 at 11:09 am
    Following a suggestion and patch from Klaus Naumann, the recovery.conf
    file can now accept comments....

    No patch supplied at present (anoncvs is down), but here is the
    annotated recovery.conf.sample

    Best Regards, Simon Riggs
  • Simon Riggs at Jul 13, 2004 at 10:47 pm
    New release of patch, at v5_1 ... for serious testing
    what's in
    - Point in Time Recovery now works....please check carefully
    - additional options in recovery.conf
    (including code contributed to PITR from Klaus Naumann)

    what's not (yet)
    - Timelines...though I think they are useful, they may not be critical
    - handling of local/UTC times (the variable is there...)

    The number of permutations is increasing, and available time is
    decreasing....not a full retest, OK.

    On Thu, 2004-07-08 at 19:11, Simon Riggs wrote:

    cd pgsql/src
    patch -p0 < pitr_v5_0.patch

    then place

    pgarch.c in src/backend/postmaster
    pgarch.h in src/include

    Read the README - its long and you wont have a clue without it

    ...remember, you HAVE TO run this on a cluster created by initdb within
    the last 2-3 days...

    When you perform a recovery, you can use the example recovery.conf
    provided here. (Not all of the options work yet...) Just place this file
    in PGDATA then crank up the postmaster any way you choose.

    Known issues:
    - CREATE DATABASE will not be recovered (at present)...create a new
    database BEFORE you take a full physical backup
    - error handling in the archiver has some issues, and requires a few
    improvements...
    ...but this works - so please don't be put off from giving it a try - it
    will take a while to get used to the concepts behind it all.

    Earlier versions
    - much of the code for recovery is in xlog.c - refactoring proved
    difficult and a big timewaster when the main functions aren't all there
    yet
    - error messages streamlined
    - variable naming more consistent than earlier versions

    Best Regards, Simon Riggs
  • Bruce Momjian at Jul 13, 2004 at 10:58 pm

    Simon Riggs wrote:
    New release of patch, at v5_1 ... for serious testing
    what's in
    - Point in Time Recovery now works....please check carefully
    - additional options in recovery.conf
    (including code contributed to PITR from Klaus Naumann)

    what's not (yet)
    - Timelines...though I think they are useful, they may not be critical
    I am not fond of the timeline idea, especially for 7.5. Let's get usage
    cases submitted first. I can imagine timelines as causing significant
    confusion during restore, which is the last thing we want to do.

    --
    Bruce Momjian | http://candle.pha.pa.us
    pgman@candle.pha.pa.us | (610) 359-1001
    + If your life is a hard drive, | 13 Roberts Road
    + Christ can be your backup. | Newtown Square, Pennsylvania 19073
  • Simon Riggs at Jul 13, 2004 at 11:23 pm

    On Tue, 2004-07-13 at 23:58, Bruce Momjian wrote:
    Simon Riggs wrote:
    New release of patch, at v5_1 ... for serious testing
    what's in
    - Point in Time Recovery now works....please check carefully
    - additional options in recovery.conf
    (including code contributed to PITR from Klaus Naumann)

    what's not (yet)
    - Timelines...though I think they are useful, they may not be critical
    I am not fond of the timeline idea, especially for 7.5. Let's get usage
    cases submitted first. I can imagine timelines as causing significant
    confusion during restore, which is the last thing we want to do.
    Well, I really want to finish this, so I do agree.

    Exhaustion is setting in....I need other eyes to test and fix the bugs.

    Best Regards, Simon Riggs
  • Tom Lane at Jul 14, 2004 at 4:20 am

    Bruce Momjian writes:
    Simon Riggs wrote:
    what's not (yet)
    - Timelines...though I think they are useful, they may not be critical
    I am not fond of the timeline idea, especially for 7.5. Let's get usage
    cases submitted first. I can imagine timelines as causing significant
    confusion during restore, which is the last thing we want to do.
    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.

    We really need to get this right the first time.

    regards, tom lane
  • Bruce Momjian at Jul 14, 2004 at 4:36 am

    Tom Lane wrote:
    Bruce Momjian <pgman@candle.pha.pa.us> writes:
    Simon Riggs wrote:
    what's not (yet)
    - Timelines...though I think they are useful, they may not be critical
    I am not fond of the timeline idea, especially for 7.5. Let's get usage
    cases submitted first. I can imagine timelines as causing significant
    confusion during restore, which is the last thing we want to do.
    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.

    We really need to get this right the first time.
    I assume they could just restore from backup and try again.

    --
    Bruce Momjian | http://candle.pha.pa.us
    pgman@candle.pha.pa.us | (610) 359-1001
    + If your life is a hard drive, | 13 Roberts Road
    + Christ can be your backup. | Newtown Square, Pennsylvania 19073
  • Tom Lane at Jul 14, 2004 at 4:45 am

    Bruce Momjian writes:
    Tom Lane wrote:
    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.
    I assume they could just restore from backup and try again.
    Sure, if they don't mind losing whatever transactions they processed
    before realizing how broken their database was. That's not going to be
    an acceptable answer for the sort of installations that need PITR in the
    first place.

    I think it's really important to get this right the first time, both for
    reliability's sake and because we are expecting people to write their
    own archiving scripts. If we change the xlog segment naming convention
    later on, then we will break all those scripts.

    regards, tom lane
  • Simon Riggs at Jul 14, 2004 at 7:15 am

    On Wed, 2004-07-14 at 05:45, Tom Lane wrote:
    Bruce Momjian <pgman@candle.pha.pa.us> writes:
    Tom Lane wrote:
    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.
    I assume they could just restore from backup and try again.
    Sure, if they don't mind losing whatever transactions they processed
    before realizing how broken their database was. That's not going to be
    an acceptable answer for the sort of installations that need PITR in the
    first place.

    I think it's really important to get this right the first time, both for
    reliability's sake and because we are expecting people to write their
    own archiving scripts. If we change the xlog segment naming convention
    later on, then we will break all those scripts.
    I agree, but I'm going to have a rest day while people test what is
    already there in case there are further code changes....which nods
    towards both of your concerns.

    BTW, one test last night broke because of the lack of timelines...

    Best Regards, Simon Riggs
  • Bruce Momjian at Jul 14, 2004 at 2:50 pm

    Tom Lane wrote:
    Bruce Momjian <pgman@candle.pha.pa.us> writes:
    Tom Lane wrote:
    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.
    I assume they could just restore from backup and try again.
    Sure, if they don't mind losing whatever transactions they processed
    before realizing how broken their database was. That's not going to be
    an acceptable answer for the sort of installations that need PITR in the
    first place.

    I think it's really important to get this right the first time, both for
    reliability's sake and because we are expecting people to write their
    own archiving scripts. If we change the xlog segment naming convention
    later on, then we will break all those scripts.
    We don't have anything hardcoded based on those file names, at last in
    PostgreSQL.

    --
    Bruce Momjian | http://candle.pha.pa.us
    pgman@candle.pha.pa.us | (610) 359-1001
    + If your life is a hard drive, | 13 Roberts Road
    + Christ can be your backup. | Newtown Square, Pennsylvania 19073
  • Tom Lane at Jul 14, 2004 at 3:00 pm

    Bruce Momjian writes:
    Tom Lane wrote:
    I think it's really important to get this right the first time, both for
    reliability's sake and because we are expecting people to write their
    own archiving scripts. If we change the xlog segment naming convention
    later on, then we will break all those scripts.
    We don't have anything hardcoded based on those file names, at last in
    PostgreSQL.
    That's because we've punted the whole problem of archive-segment
    management off to the users.

    If we did implement this functionality ourselves then I'd be less
    worried, since we'd know that future changes would affect only our
    own code. But as things stand, we will have very unhappy PITR users
    if we change the naming convention later.

    regards, tom lane
  • Simon Riggs at Jul 14, 2004 at 11:00 pm

    On Wed, 2004-07-14 at 16:00, Tom Lane wrote:
    Bruce Momjian <pgman@candle.pha.pa.us> writes:
    Tom Lane wrote:
    I think it's really important to get this right the first time, both for
    reliability's sake and because we are expecting people to write their
    own archiving scripts. If we change the xlog segment naming convention
    later on, then we will break all those scripts.
    We don't have anything hardcoded based on those file names, at last in
    PostgreSQL.
    Well, I think we do. There's two places where the filename format and
    length matters and there are numerous calls to calculate filenames from
    log,seg pairs. ...and much of the current patch would need reworking
    thoroughly to make sure all differences were changed.

    Which is why I was striving for a solution that retained the filename
    make-up, by adding a very large number to the log value (we just aren't
    gonna run out...I did the math in another post).
    That's because we've punted the whole problem of archive-segment
    management off to the users.

    If we did implement this functionality ourselves then I'd be less
    worried, since we'd know that future changes would affect only our
    own code. But as things stand, we will have very unhappy PITR users
    if we change the naming convention later.
    Yes, if we are going to change the xlog filename format, we must do it
    now. The change must be in effect whether or not you use archive_mode.

    ...Is there agreement with my previous posts on this....marked "Point in
    Time Recovery" over the last few days?
    Is that what we should implement?

    Overall, the timeline concept is only worth implementing if:
    - we archive xlogs
    - we recover them
    - we recover them to a point in time/txnid

    We agreed that the last part wasn't the priority for beta freeze. I'm
    willing to spend more time on the timeline idea as long as I've got some
    idea that we will be committing what has been developed so far. It takes
    effort to keep the patch viable against changes because new commits
    update the catalog version, which invalidates all my test databases, as
    well as any changes I have to track down. ...and I've been doing that
    for a month now - getting much better though, thanks.

    If we can review what we have now, I would be most pleased. Until we
    commit at least some of it, I'm the only developer and I would like to
    open this up to allow others to contribute more easily.

    Best Regards, Simon Riggs
  • Christopher Kings-Lynne at Dec 3, 2004 at 4:39 am
    I am not fond of the timeline idea, especially for 7.5. Let's get usage
    cases submitted first. I can imagine timelines as causing significant
    confusion during restore, which is the last thing we want to do.
    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.

    We really need to get this right the first time.

    regards, tom lane

    ---------------------------(end of broadcast)---------------------------
    TIP 8: explain analyze is your friend
  • Christopher Kings-Lynne at Dec 3, 2004 at 4:47 am
    Please ignore- seems some old mail of mine got sent waaay late...

    Christopher Kings-Lynne wrote:
    I am not fond of the timeline idea, especially for 7.5. Let's get usage
    cases submitted first. I can imagine timelines as causing significant
    confusion during restore, which is the last thing we want to do.

    I think that judgment is exactly backward. *Not* having timelines is
    what will cause serious and possibly fatal mistakes during restore:
    people will hand the wrong xlog files to restore and the software will
    be unable to recognize the inconsistency.

    We really need to get this right the first time.

    regards, tom lane

    ---------------------------(end of broadcast)---------------------------
    TIP 8: explain analyze is your friend


    ---------------------------(end of broadcast)---------------------------
    TIP 2: you can get off all lists at once with the unregister command
    (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppgsql-patches @
categoriespostgresql
postedJul 8, '04 at 6:12p
activeDec 3, '04 at 4:47a
posts15
users4
websitepostgresql.org
irc#postgresql

People

Translate

site design / logo © 2021 Grokbase