FAQ
Hello,


I'm a "bit" in a loss here: I was running Mailman 2.1.2 on Debian 3.0
with no problems. Using apt-get... to upgrade Mailman to 2.1.4
produced an error. Well, besides the fact that all my archives got
deleted (why would a script (!) do that??), I finally got Mailman back
up and running version 2.1.4. I had to fix a zillion permissions, and
it took me forever. Probably not a Mailman-thing, I know, and the
package-maintainer isn't my friend, I can tell you that. But the
following should be only Mailman-related:

After having fixed the web_page issue I could even access my lists.
All of them. But then the disaster: I have lost some 3.500 subscribers
in 3 different lists. Many other lists have been untouched, at least
no loss there. I do not understand why only those 3 were affected any
why anything got lost there in the first place.

The status is, that I have a config.pck that is way smaller than my
old ones in backup (e.g. 163 KB versus 533 KB). I'm missing _a lot_ of
subscribers (plus of course the fact that now there are probably
people in the list again that had unsubscribed!) and I can't recover.
Why? Well, I thought that backing up /lists would be sufficient since
all relevant data is in those files. Apparently not!? When I copy my
old config.pck to the directory, overwriting the current (new) file
(permissions set to list:list, 0660), then that file gets stripped in
size when I save and/or start qrunner again. And of course the
memberlist is still the old, small one, and interestingly enough, the
settings (web_page_url e.g.) are lost even though that file should
have the settings in it as they were obviously correct before. I do
have config.pck.tmp.XXX-files which I tried to rename to config.pck.
Again, it gets stripped down and only has a totally outdated number of
subscribers. Why would that happen? What else could I do to recover
those thousands of lost subscribers? Where _exactly_ are those
addresses actually stored? Any why doesn't this seem to be
documented somewhere? I really searched pretty much everywhere. Barry
in
http://mail.python.org/pipermail/mailman-developers/2002-August/012795.html
didn't really help me.

I really would appreciate any help I can get. My lists are not for fun
and we need them. I still don't understand why the backup doesn't
help. But maybe I'm simply missing something.

Thank you very much, a very desperate

Alex

Search Discussions

  • Tokio Kikuchi at Jun 6, 2004 at 12:21 am
    Hi,

    Have you done 'bin/update' after moving the backup config.pck file
    into the newly installed list directory ?

    Alex Dupont wrote:
    Hello,


    I'm a "bit" in a loss here: I was running Mailman 2.1.2 on Debian 3.0
    with no problems. Using apt-get... to upgrade Mailman to 2.1.4
    produced an error. Well, besides the fact that all my archives got
    deleted (why would a script (!) do that??), I finally got Mailman back
    up and running version 2.1.4. I had to fix a zillion permissions, and
    it took me forever. Probably not a Mailman-thing, I know, and the
    package-maintainer isn't my friend, I can tell you that. But the
    following should be only Mailman-related:
    (snip)

    I think every packager should disclose their configure options so
    that upgrading can be done through our source distribution and the
    list dbs automatically updated !

    --
    Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp
    http://weather.is.kochi-u.ac.jp/
  • Alex Dupont at Jun 6, 2004 at 5:28 pm
    Hello Tokio,


    [sorry, this got only mailed to you personally; again to the list;
    strange list setting btw...]
    Have you done 'bin/update' after moving the backup config.pck file
    into the newly installed list directory ?
    "No updates are necessary." The more I think about it I get the
    feeling that my backup-files somehow got screwed up... Still, can no
    one tell me where I can at least find the subscribed addresses? That
    information has to be stored _somewhere_??
    I think every packager should disclose their configure options so
    that upgrading can be done through our source distribution and the
    list dbs automatically updated !
    Agreed. However, using an update script provided by the distribution -
    specifically made for that application - should do _all_ the necessary
    work without screwing anything up. That's how it works in about 95% of
    the time. Errors can always happen. Yet it is beyond my comprehension
    how any script performing an upgrade would delete any file without
    making an automatic backup of it. And this has absolutely nothing to
    do with the fact that one should always backup before any such
    operation. Because sometimes the upgrade is performed by accident.
    Believe me, that happens...

    Greetings
    Alex
  • Tokio Kikuchi at Jun 6, 2004 at 11:54 pm

    Have you done 'bin/update' after moving the backup config.pck file
    into the newly installed list directory ?

    "No updates are necessary." The more I think about it I get the
    feeling that my backup-files somehow got screwed up... Still, can no
    one tell me where I can at least find the subscribed addresses? That
    information has to be stored _somewhere_??
    Have you also tried "bin/update --force" ?

    Another useful script in the bin directory may be dumpdb.
    It will dump all the data in your backup file if the format is not
    corrupted.

    --
    Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp
    http://weather.is.kochi-u.ac.jp/
  • Alex Dupont at Jun 7, 2004 at 7:49 am
    Hello,

    Have you also tried "bin/update --force" ?
    I have now... It produced quite a bit of output, such as e.g.:
    "Updating mailing list: LISTNAME
    Updating the held requests database. // every list
    - updating old private mbox file
    - updating old public mbox file
    Fixing language templates: LISTNAME // every list
    [...]
    updating old qfiles"

    Especially the list where I have lost subscribers (most of them at
    least) gave me "Your installation seems up-to-date, great!" regarding
    the mbox-file. Not important (the archive), but interesting.

    However, it didn't help at all. Again, it stripped the backup-file
    (that I had renamed to config.pck before executing the above command)
    from 533 KB to 83 KB - and still the old (no good) subscriber list.

    Something that really disturbs me, is the fact that when I
    rename the old file, my list settings show a status that is totally
    outdated - I can tell right away by the fact that the URL is incorrect
    (it had been changed to a sub-domain since with no redirect). I'm at a
    point where I am almost certain that something went wrong with/during
    the backup.
    The only thing I don't understand is why the backup-file (even though
    it contains some old settings) has a significantly bigger size than
    the one created now when I start the qrunner (or doing /bin/update).
    But maybe one of my assumptions is incorrect: I thought till now that
    the fact that it has a bigger size is connected with the fact that the
    subscriber list is much bigger than in/with the smaller, new
    config.pck. But maybe that doesn't have anything to do with it? Yet I
    have lists with 15 subscribers and there my config.pck has like 4 KB.
    So it should be connected?
    Another useful script in the bin directory may be dumpdb.
    It will dump all the data in your backup file if the format is not
    corrupted.
    Doing "perl dumpdb /var/lib/mailman/lists/LISTNAME/config.pck" prints
    the following error:
    "Traceback (most recent[...]):
    File "dumpdb", line 134, in ?
    msg = main()
    File "dumpdb", line 126, in main
    m = pickle.load(open(filename))
    EOFError"

    Now this doesn't really help me. Except that something is definitely
    wrong. :-)

    Thank you very much for your efforts, I really appreciate it!

    Greetings
    Alex
  • Tokio Kikuchi at Jun 7, 2004 at 8:05 am
    Hi again.
    Have you also tried "bin/update --force" ?
    However, it didn't help at all. Again, it stripped the backup-file
    (that I had renamed to config.pck before executing the above command)
    from 533 KB to 83 KB - and still the old (no good) subscriber list.
    I guess there is config.pck.last and mailman fallbacks to it knowing
    the backup config.pck is corrupted.
    Doing "perl dumpdb /var/lib/mailman/lists/LISTNAME/config.pck" prints
    the following error:
    "Traceback (most recent[...]):
    File "dumpdb", line 134, in ?
    msg = main()
    File "dumpdb", line 126, in main
    m = pickle.load(open(filename))
    EOFError"
    Hey! perl won't do the job, python do. Hmm, but the traceback says your
    config.pck is broken. You should use 'cat' or 'od' to read what your
    config.pck really has.
    Now this doesn't really help me. Except that something is definitely
    wrong. :-)
    Sorry, but something is definitely wrong!

    --
    Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp
    http://weather.is.kochi-u.ac.jp/
  • Alex Dupont at Jun 7, 2004 at 9:11 am
    Hi,


    sorry for this rather long email, but I made some observations, see
    below.
    I guess there is config.pck.last and mailman fallbacks to it knowing
    the backup config.pck is corrupted.
    Anticipating that, I had deleted the .last-file, even replaced them
    with the backup-file. No difference.
    Doing "perl dumpdb /var/lib/mailman/lists/LISTNAME/config.pck" prints
    the following error:
    "Traceback (most recent[...]):
    File "dumpdb", line 134, in ?
    msg = main()
    File "dumpdb", line 126, in main
    m = pickle.load(open(filename))
    EOFError"
    Hey! perl won't do the job, python do.
    Ups. :-) It didn't really change anything, though... (interesting
    actually). Still gives me the same error message.
    Hmm, but the traceback says your config.pck is broken. You should
    use 'cat' or 'od' to read what your config.pck really has.
    "od" gives me an almost endless list of 9 columns with numbers. The
    first column has 7 digits, the rest 6. At the end it sort of runs out.
    Example:

    "...
    2021620 063060 063143 060543 061544 063065 031542 033065 052562
    2021640 000125 052400 065420 061565 062550 057562 062147 063500"
    2021660 074155 062056 000145
    2021665"

    "cat" on the other hand gives me an endless list (eventually crashing
    my Putty...) of email-addresses, separated by a combination of letters
    and numbers (I guess the encrypted passwords?), everything without any
    spaces.
    I searched for some of those addresses in the current (short) member
    list and, surprise, some addresses were not found! Meaning that this
    file in fact contains the list of addresses that should be in the
    config.pck (but are not)! I am sort of relieved to see that the
    addresses are not entirely lost... Yet I have to manage to extract
    them somehow from that file and input them again.

    Is there any way to extract the addresses with some sort of script
    maybe and output them to a file that can then be used to reimport them
    to the unbroken (new) config.pck? I see a problem because of the fact
    that the password-sequences are directly connected to the addresses -
    that way, it seems to me, it will be impossible to tell the difference
    when an address starts and when it ends (and the password begins/ends).
    Although, I just noticed that the password seems to always start with
    a minor "r" and ends with a capital "U". I don't know if that goes for
    every of those passwords.
    I don't really know anything about writing scripts etc. so there is no
    way that I can write something like that myself. Anybody here who
    could and would do that for me?

    Since "cat" ultimately crashed my Putty I can't see any obvious
    errors. "od" just printed that list, neatly, with no interruptions
    that I could see. So I have no idea what the problem is. Also, the
    error message that dumpdb gives me doesn't tell _me_ anything. Anyone
    any idea what it means? That would probably help narrow down the
    problem.

    You know what? I think I just noticed something! Looking at the
    passwords (assuming those are encrypted passwords) I noticed some
    characters that reminded me of an error when I try to backup stuff
    with RAR. Those characters are such as "?", "?", "%", "?" and so on
    (hoping that they will be displayed correctly with everybody). I
    have the feeling that they have nothing to do with a legitimate
    encoding of plain-text letter-passwords! If this is the problem, then
    the list might simply not be able to be read by any import-script, or
    whatever is accessing that file, because of those characters. If we
    could either replace those characters (if they are in fact
    illegitimate, there) or at least get rid of them, maybe dumping the
    entire password information (probably = extracting the addresses),
    then we probably solved the problem. Hey, guys, I think this is kind
    of exiting! :-)
    Sorry, but something is definitely wrong!
    No kidding! :-) But we're getting there! I'm positive that we can sort
    this out. If we manage to, I would really suggest that this procedure
    is documented somewhere so it can maybe save some severe headaches for
    other people!

    Greetings
    Alex
  • Tokio Kikuchi at Jun 7, 2004 at 12:35 pm
    Hi, it's time to go to bed here in Japan. ;-)
    Hmm, but the traceback says your config.pck is broken. You should
    use 'cat' or 'od' to read what your config.pck really has.

    "od" gives me an almost endless list of 9 columns with numbers. The
    (snip)
    then how about 'strings' ?

    Other useful tool may be bin/check_db. Use -h option for usage.

    If the .pck file is really a pickle file and not broken, following
    operations give you no error. I suspect the package maintainer
    had done something to the DB structure.

    mailman% cd <prefix>/lists/<listname>
    mailman% cp <prefix>/bin/paths.py .
    mailman% python
    Python 2.2.3 (#1, Jun 23 2003, 15:54:19)
    [GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
    Type "help", "copyright", "credits" or "license" for more information.
    import paths
    import pickle
    x = pickle.load(file('config.pck'))
    print x (.... gives you contents of database)
    --
    Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp
    http://weather.is.kochi-u.ac.jp/
  • Alex Dupont at Jun 8, 2004 at 6:47 am
    Hi,

    then how about 'strings' ?
    Email-Addresses. A lot of them, although I can't see how many. But I
    fear something is not right as it (partly) looks as follows:

    ---
    email0 at address0.comr
    kJVe0ZnAr
    email at address.comr
    John Doe
    email2 at address2.comr U
    Jane Doe
    email3 at address3.comr!U
    Janis Doe
    l Elikarar"U
    email4 at address4.comr6U
    Jim Doe
    llerr7U
    email5 at address5.comr8U
    Jack Doer9U
    U&email6 at address6.comr:U
    Joey Doer;U
    ---

    The structure is obvious. But I don't understand why sometimes
    there is only the address and sometimes it begins with e.g. "U&"
    and/or the name also sometimes has an "rXU"-combination attached to
    it. Some passwords are encrypted (at the beginning of the list), some
    addresses don't have any pw (which is fine AFAIK as that comes from
    old days) but some combinations seem rather mixed up, see above. I am
    not able to stop the output in time when I thought I also saw some kind
    of gap in the list.
    Other useful tool may be bin/check_db. Use -h option for usage.
    With both config.pck (the small, new one, and the old, big one) -v
    gives me "okay". But: it alters my backup-config.pck to the new,
    small size. So apparently it dumps the information in there from some
    part on.
    print x (.... gives you contents of database)
    "Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "/usr/lib/python2.3/pickle.py", line 1390, in load
    return Unpickler(file).load()
    File "/usr/lib/python2.3/pickle.py", line 872, in load
    dispatch[key](self)
    File "/usr/lib/python2.3/pickle.py", line 894, in load_eof
    raise EOFError
    EOFError"

    Again, an error message that is of no help to me, unfortunately...

    Greetings
    Alex

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedJun 5, '04 at 1:47p
activeJun 8, '04 at 6:47a
posts9
users2
websitelist.org

2 users in discussion

Alex Dupont: 5 posts Tokio Kikuchi: 4 posts

People

Translate

site design / logo © 2022 Grokbase