[Not sure if my first attempt made it out.]


Hello,


We recently upgrade from Debian 5 to 6, and are now having issues with
Mailman. Messages are still flowing properly, but the web archives are not
being generated since the upgrade.


When I run 'bin/arch mylistname' I get the following:


[...]
figuring article archives
2005-October
/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:176:
UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal
  self.dict = marshal.load(fp)
/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:74:
UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal
  self.sorted.sort()
Updating index files for archive [2004-December]
[...]
Updating HTML for article 214
Pickling archive state into
/usr/local/mailman-2.1.20/archives/private/mylistname/pipermail.pck
Traceback (most recent call last):
  File "bin/arch", line 201, in <module>
    main()
  File "bin/arch", line 189, in main
    archiver.processUnixMailbox(fp, start, end)
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/pipermail.py", line
586, in processUnixMailbox
    self.add_article(a)
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/pipermail.py", line
638, in add_article
    article.parentID = parentID = self.get_parent_info(arch, article)
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/pipermail.py", line
658, in get_parent_info
    if self.database.hasArticle(archive, article.in_reply_to):
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
279, in hasArticle
    self.__openIndices(archive)
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
257, in __openIndices
    t = DumbBTree(os.path.join(arcdir, archive + '-' + i))
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
66, in __init__
    self.load()
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
185, in load
    self.__sort(dirty=1)
  File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
74, in __sort
    self.sorted.sort()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xea in position 3:
ordinal not in range(128)


We're running 2.1.13, but as you can see, the problem persists even with
the latest release. We're using the stock Debian Python (2.6.6), but
Mailman for tarballs.


Thanks for any info.


Regards,
David

Search Discussions

  • Stephen J. Turnbull at Sep 1, 2015 at 4:15 pm
    David Magda writes:

    When I run 'bin/arch mylistname' I get the following: >
    [...]
    figuring article archives
    2005-October
    /usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:176:
    UnicodeWarning: Unicode equal comparison failed to convert both arguments
    to Unicode - interpreting them as being unequal
    self.dict = marshal.load(fp)
    /usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:74:
    UnicodeWarning: Unicode equal comparison failed to convert both arguments
    to Unicode - interpreting them as being unequal
    self.sorted.sort()
    Updating index files for archive [2004-December]
    [...]
    Updating HTML for article 214
    Pickling archive state into
    /usr/local/mailman-2.1.20/archives/private/mylistname/pipermail.pck
    Traceback (most recent call last):

    It would appear that you have non-ASCII character in the header of the
    214th message of December 2004 (or maybe it's the 214th message
    overall). That message doesn't conform to the mail standards and
    should be repaired.


    Since pipermail is constructing an index, I would guess that you have
    a localized date header, a display name with an accented character in
    it, or a subject with an accented character in it. The character in
    question is e with a caret in the Latin-1 set, I don't know if that's
    the intended character set though.
  • David Magda at Sep 1, 2015 at 5:26 pm
    [Actually send the reply to the list as well.]

    On Tue, September 1, 2015 12:15, Stephen J. Turnbull wrote:
    David Magda writes:
    When I run 'bin/arch mylistname' I get the following:

    [...]
    figuring article archives
    2005-October
    /usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:176:
    UnicodeWarning: Unicode equal comparison failed to convert both
    arguments
    to Unicode - interpreting them as being unequal
    self.dict = marshal.load(fp)
    /usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:74:
    UnicodeWarning: Unicode equal comparison failed to convert both
    arguments
    to Unicode - interpreting them as being unequal
    self.sorted.sort()
    Updating index files for archive [2004-December]
    [...]
    Updating HTML for article 214
    Pickling archive state into
    /usr/local/mailman-2.1.20/archives/private/mylistname/pipermail.pck
    Traceback (most recent call last):
    It would appear that you have non-ASCII character in the header of the
    214th message of December 2004 (or maybe it's the 214th message
    overall). That message doesn't conform to the mail standards and
    should be repaired.

    Since pipermail is constructing an index, I would guess that you have
    a localized date header, a display name with an accented character in
    it, or a subject with an accented character in it. The character in
    question is e with a caret in the Latin-1 set, I don't know if that's
    the intended character set though.

    Looking at the mbox, there was only one place where \xea was in the
    header, in a Subject line, using `grep --color='auto' -P -n "\xea"`. I
    manually edited the mbox (making a copy first) and remove the accented-e
    character with an ASCII "e", and I'm still getting the error (I did this
    before e-mail the list). There are other places which have \xea, but not
    in any headers.


    The 214 is the message count from a state file. Every time I rerun the
    command the number is higher, but it seems to die in the same place. In
    the middle of the output we have a "UnicodeWarning":


    [...]
    #00104 <...@acm.org>
    figuring article archives
    2005-September
    #00105 <...@mikep>
    figuring article archives
    2005-September
    #00106 <...@acm.org>
    figuring article archives
    2005-September
    #00107 <...@mail.gmail.com>
    figuring article archives
    2005-October
    /usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:176:
    UnicodeWarning: Unicode equal comparison failed to convert both arguments
    to Unicode - interpreting them as being unequal
       self.dict = marshal.load(fp)
    /usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py:74:
    UnicodeWarning: Unicode equal comparison failed to convert both arguments
    to Unicode - interpreting them as being unequal
       self.sorted.sort()
    Updating index files for archive [2004-December]
       Date
       Subject
       Author
       Thread
    Computing threaded index
    Updating HTML for article 757
    Updating HTML for article 864
    Updating HTML for article 866
    Updating index files for archive [2005-April]
       Date
    [...]


    Then the error at the end:


    [...]
    Updating index files for archive [2005-August]
    [...]
    Updating HTML for article 947
    Updating HTML for article 840
    Updating index files for archive [2005-September]
       Date
       Subject
       Author
       Thread
    Computing threaded index
    Updating HTML for article 841
    Updating HTML for article 842
    Updating HTML for article 843
    Updating HTML for article 952
    Updating HTML for article 845
    Updating HTML for article 966
    Updating HTML for article 846
    Updating HTML for article 847
    Updating HTML for article 848
    Updating HTML for article 957
    Updating HTML for article 958
    Updating HTML for article 961
    Updating HTML for article 962
    Updating HTML for article 963
    Updating HTML for article 964
    Updating HTML for article 965
    Updating HTML for article 851
    Updating HTML for article 960
    Updating HTML for article 861
    Updating HTML for article 859
    Updating HTML for article 860
    Updating HTML for article 970
    Pickling archive state into
    /usr/local/mailman-2.1.20/archives/private/reactome-help/pipermail.pck
    Traceback (most recent call last):
       File "bin/arch", line 201, in <module>
         main()
       File "bin/arch", line 189, in main
         archiver.processUnixMailbox(fp, start, end)
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/pipermail.py", line
    586, in processUnixMailbox
         self.add_article(a)
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/pipermail.py", line
    638, in add_article
         article.parentID = parentID = self.get_parent_info(arch, article)
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/pipermail.py", line
    658, in get_parent_info
         if self.database.hasArticle(archive, article.in_reply_to):
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
    279, in hasArticle
         self.__openIndices(archive)
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
    257, in __openIndices
         t = DumbBTree(os.path.join(arcdir, archive + '-' + i))
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
    66, in __init__
         self.load()
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
    185, in load
         self.__sort(dirty=1)
       File "/usr/local/mailman-2.1.20/Mailman/Archiver/HyperDatabase.py", line
    74, in __sort
         self.sorted.sort()
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xea in position 3:
    ordinal not in range(128)


    It gets to 2005-September, creates the index files, enumerates 22
    articles, pickles the archive state, and then dies. The 23rd (and later)
    message/s don't appear to have non-ASCII.


    Can I patched pipermail.py or HyperDatabase.py (or ???) in some way to
    work around this? I have LANG=en_US.UTF-8 and LC_TIME=en_DK.UTF8 in my
    shell environment: does that make a difference?


    This used to work just fine, so I'm wonder what happened with the OS
    upgrade. I should have a copy of the VM pre-upgrade in case that's
    helpful.


    Thanks for the help.
  • Mark Sapiro at Sep 1, 2015 at 5:53 pm

    On 09/01/2015 10:26 AM, David Magda wrote:
    Looking at the mbox, there was only one place where \xea was in the
    header, in a Subject line, using `grep --color='auto' -P -n "\xea"`. I
    manually edited the mbox (making a copy first) and remove the accented-e
    character with an ASCII "e", and I'm still getting the error (I did this
    before e-mail the list). There are other places which have \xea, but not
    in any headers.



    There shouldn't be any non-ascii in a mbox. Well, maybe in a
    "Content-Transfer-Encoding: 8bit" (or binary?) body part, but certainly
    not in any headers.


    I don't know what you are grepping, but if it's the mbox, you shouldn't
    be looking for "\xea", you should be looking for "?".



    The 214 is the message count from a state file. Every time I rerun the
    command the number is higher, but it seems to die in the same place. In
    the middle of the output we have a "UnicodeWarning":



    Are you running bin/arch with the --wipe option? If not, you are
    repeatedly adding the same messages to the archive which is why the
    number increases.




    ...
    Can I patched pipermail.py or HyperDatabase.py (or ???) in some way to
    work around this? I have LANG=en_US.UTF-8 and LC_TIME=en_DK.UTF8 in my
    shell environment: does that make a difference?



    Probably not.



    This used to work just fine, so I'm wonder what happened with the OS
    upgrade. I should have a copy of the VM pre-upgrade in case that's
    helpful.



    Python's default encoding could make a difference if it was something
    other than ascii previously.


    --
    Mark Sapiro <mark@msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Stephen J. Turnbull at Sep 1, 2015 at 6:35 pm
    Mark Sapiro writes:

    I don't know what you are grepping, but if it's the mbox, you shouldn't
    be looking for "\xea", you should be looking for "?".

    At least on recent BSD-based systems "\xea" is a well-defined escape
    sequence, interpreted as the hexadecimal representation of a byte.
    Dunno about GNU or proprietary systems. (POSIX.2)

    Can I patched pipermail.py or HyperDatabase.py (or ???) in some way to
    work around this? I have LANG=en_US.UTF-8 and LC_TIME=en_DK.UTF8 in my
    shell environment: does that make a difference?
      >
    Probably not.

    Actually, yes, it may. If you previously had LANG=en_US.ISO8859-1 (or
    similar), then Python's default encoding may have allowed all bytes.
    On the other hand, 0xEA is not a legal byte in modern UTF-8 (it's out
    of the range of legal Unicode scalars as a leading byte and it can't
    be a trailing byte).

    This used to work just fine,

    s/just fine/incorrectly but conveniently for the sysadmin/. :-)


    I suppose it's possible that a Python upgrade wiped out a patch or
    configuration that told Python to use a Latin-N default encoding, so
    it reverted to ASCII.


    I suspect that Mailman's copy of the email libraries has also evolved
    quite a bit since 2.1.9 (I think that's what you upgraded from?), and
    if it was a Mailman provided by the OS vendor, all bets are off. Who
    knows what patches they may have applied.
  • David Magda at Sep 1, 2015 at 7:16 pm

    On Tue, September 1, 2015 14:35, Stephen J. Turnbull wrote:
    Mark Sapiro writes:
    I don't know what you are grepping, but if it's the mbox, you shouldn't
    be looking for "\xea", you should be looking for "??".
    At least on recent BSD-based systems "\xea" is a well-defined escape
    sequence, interpreted as the hexadecimal representation of a byte.
    Dunno about GNU or proprietary systems. (POSIX.2)

    This is GNU grep under Debian.

    s/just fine/incorrectly but conveniently for the sysadmin/. :-)

    I suppose it's possible that a Python upgrade wiped out a patch or
    configuration that told Python to use a Latin-N default encoding, so
    it reverted to ASCII.

    Debian is fairly conservative about these things, and AFAIK, the OS
    default was to use UTF-8, either en_US or en_CA. The locale did not change
    during the upgrade.

    I suspect that Mailman's copy of the email libraries has also evolved
    quite a bit since 2.1.9 (I think that's what you upgraded from?), and
    if it was a Mailman provided by the OS vendor, all bets are off. Who
    knows what patches they may have applied.

    We are running 2.1.13 from tarballs and so the Mailman code did not change
    when the archive web page generation stopped. The only thing that changed
    was the version of Python (2.5 -> 2.6?) under the OS.


    Doing a "arch --wipe mylist" seems to have solved the issue, though now
    I'm curious to know why \xea was a problem before but suddenly isn't after
    the wipe.
  • Mark Sapiro at Sep 2, 2015 at 1:02 am

    On 09/01/2015 12:16 PM, David Magda wrote:
    On Tue, September 1, 2015 14:35, Stephen J. Turnbull wrote:
    Mark Sapiro writes:
    I don't know what you are grepping, but if it's the mbox, you shouldn't
    be looking for "\xea", you should be looking for "??".
    At least on recent BSD-based systems "\xea" is a well-defined escape
    sequence, interpreted as the hexadecimal representation of a byte.
    Dunno about GNU or proprietary systems. (POSIX.2)
    This is GNU grep under Debian.



    In my testing with GNU grep on Ubuntu 15.04, 'grep "\xea"' interprets \x
    as a literal x and therefore looks for the string "xea", not for the
    character whose hex value is EA.



    We are running 2.1.13 from tarballs and so the Mailman code did not change
    when the archive web page generation stopped. The only thing that changed
    was the version of Python (2.5 -> 2.6?) under the OS.

    Doing a "arch --wipe mylist" seems to have solved the issue, though now
    I'm curious to know why \xea was a problem before but suddenly isn't after
    the wipe.



    Here's what I suspect was going on.


    Your first run of bin/arch encountered some non-ascii in a header and
    threw the exception, but not before writing bad data to the pipermail
    database for that month.


    You then "fixed" the non-ascii in the input mbox, but subsequent runs of
    bin/arch still encountered the bad data in the database when they got to
    that month.


    Finally, you added the --wipe option and that removed everythin and
    rebuilt from scratch and as there was no non-ascii in the mbox headers,
    it worked.


    As to why this didn't happen before, see my next reply.


    --
    Mark Sapiro <mark@msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • David Magda at Sep 2, 2015 at 1:38 am

    On Sep 1, 2015, at 21:02, Mark Sapiro wrote:

    In my testing with GNU grep on Ubuntu 15.04, 'grep "\xea"' interprets \x
    as a literal x and therefore looks for the string "xea", not for the
    character whose hex value is EA.

    For the record/archives: GNU grep also as the ?-P? option, which allows Perl regexes (PCRE), and \xhh searches for characters with hex code hh (per pcrepattern(3)):


      http://stackoverflow.com/questions/3001177/ how-do-i-grep-for-all-non-ascii-characters-in-unix

    Doing a "arch --wipe mylist" seems to have solved the issue, though now
    I'm curious to know why \xea was a problem before but suddenly isn't after
    the wipe.

    Here's what I suspect was going on.

    Your first run of bin/arch encountered some non-ascii in a header and
    threw the exception, but not before writing bad data to the pipermail
    database for that month.

    You then "fixed" the non-ascii in the input mbox, but subsequent runs of
    bin/arch still encountered the bad data in the database when they got to
    that month.

    Finally, you added the --wipe option and that removed everythin and
    rebuilt from scratch and as there was no non-ascii in the mbox headers,
    it worked.

    As to why this didn't happen before, see my next reply.

    Sounds plausible.
  • Stephen J. Turnbull at Sep 2, 2015 at 6:31 am
    David Magda writes:

    Debian is fairly conservative about these things,

    Debian may be conservative, but individual maintainers vary. Cf. that
    OpenSSL patch fiasco. There was no excuse for that patch, the
    maintainer just thought he was smarter than the OpenSSL people.
    Mailman itself has spent a fair amount of time on questions that ended
    up being related to Debian patches (specifically the execrable
    mailman-to-postfix script, although that did at least have a good
    reason behind it from the point of view of the distro, it was very
    fragile but Debian saw no need to fix it).

    and AFAIK, the OS default was to use UTF-8, either en_US or
    en_CA. The locale did not change during the upgrade.

    OK, unless your old Debian was *really* old, that's very likely true
    and a locale change was not the source of any of the issues.

    We are running 2.1.13 from tarballs

    Sorry, I confused your thread with somebody else's, then.


    At 2.1.13, don't you have any DMARC problems? Or do you just prohibit
    users from posting from p=reject addresses?

    and so the Mailman code did not change when the archive web page
    generation stopped. The only thing that changed was the version of
    Python (2.5 -> 2.6?) under the OS.

    Python 2.5 was released in September 2006. That's long enough ago
    that I'm sure handling of Unicode and charsets changed significantly
    between then and the release of 2.6 in October 2008, although I'm not
    familiar with the details. In Python 2, str is "promoted" to unicode
    when operated on (eg, concatenation and comparison), but that is
    inherently fragile, and it's quite likely the rules (eg, default
    codecs and error handlers) changed between Python 2.5 and 2.6.


    Stability of the system is of course a concern, but you may want to
    consider installing python2.7 as an alternative (but *not* the system
    default), and pointing Mailman at that in setup.py. Going forward,
    it's unlikely that python2.7 will see more changes (barring a major
    security issue, there will be no more releases AFAIK), and Mailman
    2.1.20 is very near the end of the line for Mailman 2, as well (but
    keep your fingers crossed against a Yahoo!-style email security
    debacle that DMARC doesn't handle -- who knows what the freemail
    services might come up with in that scenario :-( ).

    Doing a "arch --wipe mylist" seems to have solved the issue, though
    now I'm curious to know why \xea was a problem before but suddenly
    isn't after the wipe.

    I expect Mark's explanation is correct, that without --wipe you had an
    existing partial archive and processing that is where the error occurred.
  • Laura Creighton at Sep 2, 2015 at 7:30 am
    If some code has
    from __future__ import unicode_literals
    that might go a long way to explaining things.


    see:
    see: https://docs.python.org/2/whatsnew/2.6.html
  • David Magda at Sep 1, 2015 at 7:09 pm

    On Tue, September 1, 2015 13:53, Mark Sapiro wrote:


    There shouldn't be any non-ascii in a mbox. Well, maybe in a
    "Content-Transfer-Encoding: 8bit" (or binary?) body part, but certainly
    not in any headers.

    I don't know what you are grepping, but if it's the mbox, you shouldn't
    be looking for "\xea", you should be looking for "?".

    Yes, there are a number of messages with a C-T-E of 8bit.

    Are you running bin/arch with the --wipe option? If not, you are
    repeatedly adding the same messages to the archive which is why the
    number increases.

    Doing a run with "--wipe" seems to have fixed things. Doing a complete
    're-build' (from message 0) finished without errors, and sending a test
    message seems to have now generated a proper archive HTML page for
    September 2015. Previously the last archive web page was for a message
    sent August 25, even though a few messages had come between then and now.


    Could the new version of Python been chocking on a binary file whose
    format has changed from the old version? Is it prudent to do an "arch
    --wipe" when changing versions of Python?
  • Mark Sapiro at Sep 2, 2015 at 1:14 am

    On 09/01/2015 12:09 PM, David Magda wrote:
    Could the new version of Python been chocking on a binary file whose
    format has changed from the old version? Is it prudent to do an "arch
    --wipe" when changing versions of Python?



    The exception is thrown in Python's sort() method. There are no
    documented changes in this method's calling sequence between Python 2.5
    and 2.6, but it's quite possible there were changes in the details of
    the key comparison code that are responsible.


    No it is not prudent to do an "arch --wipe" when changing versions of
    Python?. It is not a good idea to ever do "arch --wipe" unless it's
    really necessary. See the section "NOTE ON MBOX ANOMALIES:" at
    <http://wiki.list.org/x/4030681>.


    --
    Mark Sapiro <mark@msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • David Magda at Sep 2, 2015 at 1:55 am

    On Sep 1, 2015, at 21:14, Mark Sapiro wrote:
    On 09/01/2015 12:09 PM, David Magda wrote:

    Could the new version of Python been chocking on a binary file whose
    format has changed from the old version? Is it prudent to do an "arch
    --wipe" when changing versions of Python?

    The exception is thrown in Python's sort() method. There are no
    documented changes in this method's calling sequence between Python 2.5
    and 2.6, but it's quite possible there were changes in the details of
    the key comparison code that are responsible.

    No it is not prudent to do an "arch --wipe" when changing versions of
    Python?. It is not a good idea to ever do "arch --wipe" unless it's
    really necessary. See the section "NOTE ON MBOX ANOMALIES:" at
    <http://wiki.list.org/x/4030681>.

    The message in question seems to be from October 2005, and it has the "Subject: Voc? [?]? header. Running under Debian 5 (Python 2.5), archive processing wasn?t an issue: messages were coming into the mbox and the HTML page was being generated until August 25. I then did the upgrade on that day.


    The mbox then no longer has messages until September 1, the day when I did ?arch ?wipe?, even though the mail.log shows messages coming in (and going out AFAICT).


    So would there be anything besides a ??wipe? that could otherwise be done to prevent this archive breakage? Attempt a ?cleanarch" first? Seems strange that a message from 2005 could stop archive processing for current years/months.
  • Mark Sapiro at Sep 2, 2015 at 2:43 am

    On 09/01/2015 06:55 PM, David Magda wrote:
    The message in question seems to be from October 2005, and it has the "Subject: Voc? [?]? header. Running under Debian 5 (Python 2.5), archive processing wasn?t an issue: messages were coming into the mbox and the HTML page was being generated until August 25. I then did the upgrade on that day.

    The mbox then no longer has messages until September 1, the day when I did ?arch ?wipe?, even though the mail.log shows messages coming in (and going out AFAICT).



    So what has probably happened is that archiving has been broken since
    the upgrade on Aug 25. There should be a an error message with traceback
    in Mailman's 'error' log for each message that wasn't archived. More
    importantly, the unarchived messages should all be in Mailman's 'shunt'
    queue and running Mailman's bin/unshunt should process them for the archive.



    So would there be anything besides a ??wipe? that could otherwise be done to prevent this archive breakage? Attempt a ?cleanarch" first? Seems strange that a message from 2005 could stop archive processing for current years/months.



    Presumably, you did some kind of bin/arch process on Aug 25. The answer
    is don't do that. Or if you didn't do that, then the answer may lie in
    whatever changed in Python.


    You should never be running bin/arch without --wipe unless you are
    importing archives from elsewhere to be added to an existing archive,
    and even in that case, it might be preferable to stop Mailman,
    concatenate the existing Mailman mbox and import mbox into a combined
    mbox and run 'bin/arch --wipe'. Then after success, start Mailman again.


    You should also consider running the contrib/mmdsr script from the
    Mailman source distro or if not that, somehow looking a Mailman's error
    log frequently for problems to discover them sooner.


    --
    Mark Sapiro <mark@msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • David Magda at Sep 2, 2015 at 2:31 pm

    On Tue, September 1, 2015 22:43, Mark Sapiro wrote:
    On 09/01/2015 06:55 PM, David Magda wrote:

    The message in question seems to be from October 2005, and it has the
    "Subject: Voc? [
    ]? header. Running under Debian 5 (Python 2.5), archive
    processing wasn?t an issue: messages were coming into the mbox and the
    HTML page was being generated until August 25. I then did the upgrade on
    that day.

    The mbox then no longer has messages until September 1, the day when I
    did ?arch ?wipe?, even though the mail.log shows messages coming in (and
    going out AFAICT).

    So what has probably happened is that archiving has been broken since
    the upgrade on Aug 25. There should be a an error message with traceback
    in Mailman's 'error' log for each message that wasn't archived. More
    importantly, the unarchived messages should all be in Mailman's 'shunt'
    queue and running Mailman's bin/unshunt should process them for the
    archive.

    The last error message that is in the 'error' log is from Aug 24:


    mailman at mail1:~$ ls -lrth logs/
    total 16M
    -rw-rw-r-- 1 mailman mailman 1.2K Mar 2 2014 fromusenet
    -rw-rw-r-- 1 mailman mailman 63K Mar 18 09:57 locks
    -rw-rw-r-- 1 www-data mailman 3.1K Mar 31 18:04 mischief
    -rw-rw-r-- 1 mailman mailman 808K Aug 24 08:00 smtp-failure
    -rw-rw-r-- 1 mailman mailman 354K Aug 24 08:14 bounce
    -rw-rw-r-- 1 www-data mailman 1.1M Aug 24 13:14 error
    -rw-rw-r-- 1 mailman mailman 584K Aug 31 10:45 qrunner
    -rw-rw-r-- 1 www-data mailman 1.6M Sep 1 12:09 vette
    -rw-rw-r-- 1 www-data mailman 278K Sep 2 09:03 subscribe
    -rw-rw-r-- 1 mailman mailman 3.8M Sep 2 09:48 smtp
    -rw-rw-r-- 1 mailman mailman 2.7M Sep 2 09:48 post

    So would there be anything besides a ??wipe? that could otherwise be
    done to prevent this archive breakage? Attempt a ?cleanarch" first?
    Seems strange that a message from 2005 could stop archive processing for
    current years/months.
    Presumably, you did some kind of bin/arch process on Aug 25. The answer
    is don't do that. Or if you didn't do that, then the answer may lie in
    whatever changed in Python.

    Not that I'm aware of, unless the Debian package runs it in some way
    pre-/post-upgrade. I didn't start looking at bin/arch until I noticed the
    breakage.

    You should never be running bin/arch without --wipe unless you are
    importing archives from elsewhere to be added to an existing archive,
    and even in that case, it might be preferable to stop Mailman,
    concatenate the existing Mailman mbox and import mbox into a combined
    mbox and run 'bin/arch --wipe'. Then after success, start Mailman again.

    You should also consider running the contrib/mmdsr script from the
    Mailman source distro or if not that, somehow looking a Mailman's error
    log frequently for problems to discover them sooner.

    We have a few more hosts to upgrade, so I'll have to look more closely at
    this. However some of these hosts have quite a few lists, so I'm not sure
    how practical it will be examine every one to see if they're working
    properly.
  • Mark Sapiro at Sep 2, 2015 at 8:33 pm

    On 09/02/2015 07:31 AM, David Magda wrote:
    On Tue, September 1, 2015 22:43, Mark Sapiro wrote:

    So what has probably happened is that archiving has been broken since
    the upgrade on Aug 25. There should be a an error message with traceback
    in Mailman's 'error' log for each message that wasn't archived. More
    importantly, the unarchived messages should all be in Mailman's 'shunt'
    queue and running Mailman's bin/unshunt should process them for the
    archive.
    The last error message that is in the 'error' log is from Aug 24:



    Hmmm...



    Presumably, you did some kind of bin/arch process on Aug 25. The answer
    is don't do that. Or if you didn't do that, then the answer may lie in
    whatever changed in Python.
    Not that I'm aware of, unless the Debian package runs it in some way
    pre-/post-upgrade. I didn't start looking at bin/arch until I noticed the
    breakage.



    No. The package wouldn't do that.


    It is difficult to understand why a list would stop archiving without
    throwing and logging exceptions. Even if ArchRunner were somehow not
    running, the messages would be queued in its queue and processed when it
    eventually started.



    You should also consider running the contrib/mmdsr script from the
    Mailman source distro or if not that, somehow looking a Mailman's error
    log frequently for problems to discover them sooner.
    We have a few more hosts to upgrade, so I'll have to look more closely at
    this. However some of these hosts have quite a few lists, so I'm not sure
    how practical it will be examine every one to see if they're working
    properly.



    mmdsr (Mailman daily summary report) is a daily summary for the whole
    server run at midnight by cron. It won't directly tell you if messages
    are not being archived, but it will give you a listing of what's in
    Mailman's queues and various error log entries.


    --
    Mark Sapiro <mark@msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedSep 1, '15 at 11:22a
activeSep 2, '15 at 8:33p
posts16
users4
websitelist.org

People

Translate

site design / logo © 2021 Grokbase