FAQ
Hi:

I am having a problem regenerating my archive.

Everything was going along fine, with daily updates being added to the
archive. I then wanted to refresh the entire archive and ran bin/arch
--wipe list1

it failed.

Archive articles go from 0000 to 66990

list1.mbox is: 526313754 Aug 21 20:19 list1.mbox

The server NetBSD 3.1, has 2GB memory

bin/arch first went through every article

Week-of-Mon-20070702
#65136 <20070703134009.74f75a8e at viola.tamara-b.org>
figuring article archives
[...]

then after many

Updating HTML for article 4584
Updating HTML for article 4585
Updating HTML for article 4586
Updating index files for archive [Week-of-Mon-20050321]
Date
Subject
Author
Thread
Computing threaded index
Updating HTML for article 4587
Updating HTML for article 4588
Updating HTML for article 4589
Updating HTML for article 4590
Updating HTML for article 4591
Updating HTML for article 4592
Updating HTML for article 4593

[...]

Updating HTML for article 9353
Updating index files for archive [Week-of-Mon-20041122]
Traceback (most recent call last):
File "bin/arch", line 200, in ?
main()
File "bin/arch", line 190, in main
archiver.close()
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 322,
in close self.update_dirty_archives()
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 538,
in update_dirty_archives self.update_archive(i)
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperArch.py", line 1121,
in update_archive self.__super_update_archive(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 439,
in update_archive self.__set_parameters(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 354,
in __set_parameters firstdate = self.database.firstdate(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
209, in firstdate self.__openIndices(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
240, in __openIndices self.__closeIndices()
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
265, in __closeIndices index.close()
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
187, in close fp.write(marshal.dumps(self.dict))
MemoryError

So I ran a subset of the archive:

bin/arch -s 10000 -e 66990 list1

And after a while:
Updating HTML for article 11689
Updating HTML for article 11690
Updating index files for archive [Week-of-Mon-20050815]
Traceback (most recent call last):
File "bin/arch", line 200, in ?
main()
File "bin/arch", line 190, in main
archiver.close()
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 322,
in close self.update_dirty_archives()
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 538,
in update_dirty_archives self.update_archive(i)
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperArch.py", line 1121,
in update_archive self.__super_update_archive(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 439,
in update_archive self.__set_parameters(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 354,
in __set_parameters firstdate = self.database.firstdate(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
209, in firstdate self.__openIndices(archive)
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
240, in __openIndices self.__closeIndices()
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
265, in __closeIndices index.close()
File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
187, in close fp.write(marshal.dumps(self.dict))
MemoryError

====================
_dirty_archives ??? something corrupt?

Is there a fix for this?


I can successfully run about the last 10,000

bin/arch -s 50000 -e 66990 list1

But that is about 1/7th of the archive
Can I produce a complete archive in chunks of 10,000? How?

Best regards
Bob

Search Discussions

  • Mark Sapiro at Aug 22, 2007 at 3:01 pm

    Bob wrote:
    Everything was going along fine, with daily updates being added to the
    archive. I then wanted to refresh the entire archive and ran bin/arch
    --wipe list1

    it failed.

    Archive articles go from 0000 to 66990

    list1.mbox is: 526313754 Aug 21 20:19 list1.mbox

    The server NetBSD 3.1, has 2GB memory

    bin/arch first went through every article

    Week-of-Mon-20070702
    #65136 <20070703134009.74f75a8e at viola.tamara-b.org>
    figuring article archives
    [...]

    then after many

    Updating HTML for article 4584
    Updating HTML for article 4585
    Updating HTML for article 4586
    Updating index files for archive [Week-of-Mon-20050321]
    Date
    Subject
    Author
    Thread
    Computing threaded index
    Updating HTML for article 4587
    Updating HTML for article 4588
    Updating HTML for article 4589
    Updating HTML for article 4590
    Updating HTML for article 4591
    Updating HTML for article 4592
    Updating HTML for article 4593

    [...]

    Updating HTML for article 9353
    Updating index files for archive [Week-of-Mon-20041122]
    Traceback (most recent call last):
    File "bin/arch", line 200, in ?
    main()
    File "bin/arch", line 190, in main
    archiver.close()
    File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 322,
    in close self.update_dirty_archives()
    File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 538,
    in update_dirty_archives self.update_archive(i)
    File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperArch.py", line 1121,
    in update_archive self.__super_update_archive(archive)
    File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 439,
    in update_archive self.__set_parameters(archive)
    File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 354,
    in __set_parameters firstdate = self.database.firstdate(archive)
    File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
    209, in firstdate self.__openIndices(archive)
    File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
    240, in __openIndices self.__closeIndices()
    File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
    265, in __closeIndices index.close()
    File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
    187, in close fp.write(marshal.dumps(self.dict))
    MemoryError

    This problem is mentioned in the description of the -e option of
    bin/arch --help.

    I can successfully run about the last 10,000

    bin/arch -s 50000 -e 66990 list1

    But that is about 1/7th of the archive
    Can I produce a complete archive in chunks of 10,000? How?

    bin/arch --wipe -e 99999 list1
    bin/arch -s 10000 -e 19999 list1
    bin/arch -s 20000 -e 29999 list1
    ...
    bin/arch -s 60000 list1

    Note that --wipe is specified only on the first command.

    Also, it is a good idea to stop Mailman during this process. The reason
    is that if a new message arrives and is archived say between the
    second and third commands, it will be assigned message number 20000 in
    the new archive, but it will be message number 66991 in the .mbox.
    This will guarantee that if you ever rebuild the archive again, all
    message numbers from 20000 on will change, thus invalidating any saved
    archive URLs.

    Finally, it is a good idea to first check the .mbox file with
    bin/cleanarch to make sure there are no unescaped From_ lines in
    message bodies.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Aug 22, 2007 at 3:12 pm

    Mark Sapiro wrote:
    bin/arch --wipe -e 99999 list1

    Ooops! That should be

    bin/arch --wipe -e 9999 list1

    bin/arch -s 10000 -e 19999 list1
    bin/arch -s 20000 -e 29999 list1
    ...
    bin/arch -s 60000 list1
    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedAug 22, '07 at 12:55p
activeAug 22, '07 at 3:12p
posts3
users2
websitelist.org

2 users in discussion

Mark Sapiro: 2 posts Bob: 1 post

People

Translate

site design / logo © 2022 Grokbase