FAQ
I recently changed my Mailman 2.1.9 installation to use and support
Unicode. I cannot update to a newer Mailman version as long as my Debian
Etch hasn't been upgraded to the current Debian version, as I have been
told by my ISP.

I have found various error reports in /var/log/mailman/error and fixed
everything to have a usable Mailman again. But there are still many
shunted emails in /qfiles/retry/, /qfiles/shunt/ and /qfiles/virgin/.

When I try

$ /var/lib/mailman/bin/unshunt
$ -bash: unshunt: command not known

What am I doing wrong? When I try

$ /var/lib/mailman/bin/python unshunt

it simply returns to the prompt after a while, without having proceeded
the shunted files, as far as I can see. Is my "unshunt" kind of damaged?
Do I miss some files?

Thanks in advance for any help,
--UlfDunkel

Search Discussions

  • Mark Sapiro at May 25, 2009 at 2:58 pm

    Ulf Dunkel wrote:
    I have found various error reports in /var/log/mailman/error and fixed
    everything to have a usable Mailman again. But there are still many
    shunted emails in /qfiles/retry/, /qfiles/shunt/ and /qfiles/virgin/.

    The files in qfiles/retry/ and qfiles/virgin/ are not shunted. If they
    are *.pck files, they should be processed normally by RetryRunner and
    VirginRunner respectively if those runners are running. If they are
    *.bak files, they are 'backup' queue entries that were possibly left
    behind if the runner died while processing a message. If so, they
    should be reprocessed the next time the runner starts.

    When I try

    $ /var/lib/mailman/bin/unshunt
    $ -bash: unshunt: command not known

    Does /var/lib/mailman/bin/unshunt exist and is it executable (mode
    -rwxr-xr-x)?

    What am I doing wrong? When I try

    $ /var/lib/mailman/bin/python unshunt

    Do you really have a python in /var/lib/mailman/bin or did you perhaps
    mean

    $ python /var/lib/mailman/bin/unshunt

    it simply returns to the prompt after a while, without having proceeded
    the shunted files, as far as I can see. Is my "unshunt" kind of damaged?

    Unshunt will process *.bak and *.pck files in qfiles/shunt/ first
    renaming the .bak files to .pck and then attempting to restore the
    .pck files to their original queue. For any .pck files it can't
    requeue, it should print a "Cannot unshunt message xxxxx, skipping"
    message and leave that file as a .bak in the shunt/ queue.

    You can look at these files with bin/dumpdb -p to see the message and
    the metadata. Each file should contain two objects, the message and
    the metadata. The metadata should have a 'whichq' attribute indicating
    which queue to put it in.

    Is it possible that the messages are being unshunted and then throwing
    exceptions in processing and being shunted again?

    What do mailman's error and smtp-failure (for retry) logs say?

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Ulf Dunkel at May 25, 2009 at 4:02 pm
    Hi Mark.
    The files in qfiles/retry/ and qfiles/virgin/ are not shunted. If they
    are *.pck files, they should be processed normally by RetryRunner and
    VirginRunner respectively if those runners are running. If they are
    *.bak files, they are 'backup' queue entries that were possibly left
    behind if the runner died while processing a message. If so, they
    should be reprocessed the next time the runner starts.
    /qfiles/retry/ contains *.pck files.
    /qfiles/virgin/ contains one *.pck.tmp file only.

    Does /var/lib/mailman/bin/unshunt exist and is it executable (mode
    -rwxr-xr-x)?
    Yes, it exists there physically (no link or stuff) and is executable
    with mode -rwxr-xr-x.

    What am I doing wrong? When I try

    $ /var/lib/mailman/bin/python unshunt

    Do you really have a python in /var/lib/mailman/bin or did you perhaps
    mean

    $ python /var/lib/mailman/bin/unshunt
    Yepp, of course I meant the latter. Sorry.

    Unshunt will process *.bak and *.pck files in qfiles/shunt/ first
    renaming the .bak files to .pck and then attempting to restore the
    .pck files to their original queue. For any .pck files it can't
    requeue, it should print a "Cannot unshunt message xxxxx, skipping"
    message and leave that file as a .bak in the shunt/ queue.
    Sorry, but it doesn't print anything and simply quits. Afterwards, all
    existing .pck files in the shunt/ folder have been updated to the
    current date+time. That seems to be all. :-o
    You can look at these files with bin/dumpdb -p to see the message and
    the metadata. Each file should contain two objects, the message and
    the metadata. The metadata should have a 'whichq' attribute indicating
    which queue to put it in.
    I can see these data for all files, seem to be fine.

    Is it possible that the messages are being unshunted and then throwing
    exceptions in processing and being shunted again?

    What do mailman's error and smtp-failure (for retry) logs say?
    error says:
    ---snip---
    [...]
    May 25 17:57:06 2009 (5320) Uncaught runner exception: 'utf8' codec
    can't decode bytes in position 30-35: unsupported Unicode code range
    May 25 17:57:06 2009 (5320) Traceback (most recent call last):
    File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
    File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
    File "/usr/lib/mailman/Mailman/Queue/VirginRunner.py", line 38, in
    _dispose
    return IncomingRunner._dispose(self, mlist, msg, msgdata)
    File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in
    _dispose
    more = self._dopipeline(mlist, msg, msgdata, pipeline)
    File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in
    _dopipeline
    sys.modules[modname].process(mlist, msg, msgdata)
    File "/usr/lib/mailman/Mailman/Handlers/CookHeaders.py", line 188, in
    process
    i18ndesc = uheader(mlist, mlist.description, 'List-Id', maxlinelen�8)
    File "/usr/lib/mailman/Mailman/Handlers/CookHeaders.py", line 65, in
    uheader
    return Header(s, charset, maxlinelen, header_name, continuation_ws)
    File "/usr/lib/mailman/pythonlib/email/Header.py", line 188, in __init__
    self.append(s, charset, errors)
    File "/usr/lib/mailman/pythonlib/email/Header.py", line 272, in append
    ustr = unicode(s, incodec, errors)
    UnicodeDecodeError: 'utf8' codec can't decode bytes in position 30-35:
    unsupported Unicode code range

    May 25 17:57:06 2009 (5320) SHUNTING:
    1243264080.014122+b4e9af652fe7695880609b2edfb94ed8841bbfb3
    ---snap---


    smtp-failure says:
    ----- snip -----
    [...]
    May 25 17:56:38 2009 (5319) All recipients refused:
    {'udo_discussion-owner at lists.udo-open-source.org': (451, 'Temporary
    local problem - please try later')}, msg
    May 25 17:56:38 2009 (5319) delivery to
    udo_discussion-owner at lists.udo-open-source.org failed with code 451:
    Temporary local problem - please try later
    ----- snap -----

    What the heck should I do? I have no idea where else I should search for
    "wrong" utf8 stuff and what "Temporary local problem" means.

    As always, Mark: I really appreciate your help. :-)

    --UlfDunkel
  • Mark Sapiro at May 25, 2009 at 4:46 pm

    Ulf Dunkel wrote:
    /qfiles/retry/ contains *.pck files.

    Which are messages that have "retryable" delivery problems which should
    be retried at intervals (default 15 minutes) for some period (default
    5 days) or until they are delivered. There should be smtp-failure log
    entries for the failed deliveries.

    /qfiles/virgin/ contains one *.pck.tmp file only.

    This is a Mailman generated message that never got fully queued because
    the process queueing it died.

    Does /var/lib/mailman/bin/unshunt exist and is it executable (mode
    -rwxr-xr-x)?
    Yes, it exists there physically (no link or stuff) and is executable
    with mode -rwxr-xr-x.

    Then the #! first line points at a non-existent python, or if it says
    "#! @PYTHON@", this is a file that has not been configured by
    Mailman's configure - i.e it is a file unpacked from the tarball and
    not processed through configure and make.

    What am I doing wrong? When I try

    $ /var/lib/mailman/bin/python unshunt

    Do you really have a python in /var/lib/mailman/bin or did you perhaps
    mean

    $ python /var/lib/mailman/bin/unshunt
    Yepp, of course I meant the latter. Sorry.

    Unshunt will process *.bak and *.pck files in qfiles/shunt/ first
    renaming the .bak files to .pck and then attempting to restore the
    .pck files to their original queue. For any .pck files it can't
    requeue, it should print a "Cannot unshunt message xxxxx, skipping"
    message and leave that file as a .bak in the shunt/ queue.
    Sorry, but it doesn't print anything and simply quits. Afterwards, all
    existing .pck files in the shunt/ folder have been updated to the
    current date+time. That seems to be all. :-o

    Which indicates they were unshunted, processed and encountered the same
    exception as before and were shunted again.

    You can look at these files with bin/dumpdb -p to see the message and
    the metadata. Each file should contain two objects, the message and
    the metadata. The metadata should have a 'whichq' attribute indicating
    which queue to put it in.
    I can see these data for all files, seem to be fine.

    Is it possible that the messages are being unshunted and then throwing
    exceptions in processing and being shunted again?

    What do mailman's error and smtp-failure (for retry) logs say?
    error says:
    ---snip---
    [...]
    May 25 17:57:06 2009 (5320) Uncaught runner exception: 'utf8' codec
    can't decode bytes in position 30-35: unsupported Unicode code range
    May 25 17:57:06 2009 (5320) Traceback (most recent call last):
    File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
    File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
    File "/usr/lib/mailman/Mailman/Queue/VirginRunner.py", line 38, in
    _dispose
    return IncomingRunner._dispose(self, mlist, msg, msgdata)
    File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in
    _dispose
    more = self._dopipeline(mlist, msg, msgdata, pipeline)
    File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in
    _dopipeline
    sys.modules[modname].process(mlist, msg, msgdata)
    File "/usr/lib/mailman/Mailman/Handlers/CookHeaders.py", line 188, in
    process
    i18ndesc = uheader(mlist, mlist.description, 'List-Id', maxlinelen�8)
    File "/usr/lib/mailman/Mailman/Handlers/CookHeaders.py", line 65, in
    uheader
    return Header(s, charset, maxlinelen, header_name, continuation_ws)
    File "/usr/lib/mailman/pythonlib/email/Header.py", line 188, in __init__
    self.append(s, charset, errors)
    File "/usr/lib/mailman/pythonlib/email/Header.py", line 272, in append
    ustr = unicode(s, incodec, errors)
    UnicodeDecodeError: 'utf8' codec can't decode bytes in position 30-35:
    unsupported Unicode code range

    May 25 17:57:06 2009 (5320) SHUNTING:
    1243264080.014122+b4e9af652fe7695880609b2edfb94ed8841bbfb3
    ---snap---


    smtp-failure says:
    ----- snip -----
    [...]
    May 25 17:56:38 2009 (5319) All recipients refused:
    {'udo_discussion-owner at lists.udo-open-source.org': (451, 'Temporary
    local problem - please try later')}, msg
    May 25 17:56:38 2009 (5319) delivery to
    udo_discussion-owner at lists.udo-open-source.org failed with code 451:
    Temporary local problem - please try later
    ----- snap -----

    What the heck should I do? I have no idea where else I should search for
    "wrong" utf8 stuff and what "Temporary local problem" means.

    The "Temporary local problem" message is reported by your MTA to
    Mailman. Check the MTA's logs which may have more information.

    Regarding the utf8 issue, your OP says "I recently changed my Mailman
    2.1.9 installation to use and support Unicode." The error occurs when
    CookHeaders is trying to encode the list's 'description' for the
    List-ID header. I suspect that the charset for the list's preferred
    language is now utf-8, but the list's description is still iso-8859-1
    or whatever and thus contains non utf-8 codes.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Ulf Dunkel at May 25, 2009 at 5:53 pm
    Hi Mark.
    Does /var/lib/mailman/bin/unshunt exist and is it executable (mode
    -rwxr-xr-x)?
    Yes, it exists there physically (no link or stuff) and is executable
    with mode -rwxr-xr-x.

    Then the #! first line points at a non-existent python, or if it says
    "#! @PYTHON@", this is a file that has not been configured by
    Mailman's configure - i.e it is a file unpacked from the tarball and
    not processed through configure and make.
    The first line reads like this:
    ----- snip -----
    #! /usr/bin/python
    ----- snap -----

    ... and of course /usr/bin/python exists, but is a link to
    /usr/bin/python2.4 - is this okay?

    The "Temporary local problem" message is reported by your MTA to
    Mailman. Check the MTA's logs which may have more information.
    This looks rather strange to me (identical in exim4/mainlog and
    exim4/rejectlog):
    ----- snip -----
    2009-05-25 17:56:38 H=localhost (calamus.calamus.net) [127.0.0.1]
    F=<udo_discussion-bounces at lists.udo-open-source.org> temporarily
    rejected RCPT <udo-discussion-owner at lists.udo-open-source.org>: remote
    host address is the local host
    ----- snap -----

    I have really NO IDEA what this means and how I can fix it. ;-)
    Both domains (calamus.net and udo-open-source.org) are on my server.

    Regarding the utf8 issue, your OP says "I recently changed my Mailman
    2.1.9 installation to use and support Unicode." The error occurs when
    CookHeaders is trying to encode the list's 'description' for the
    List-ID header. I suspect that the charset for the list's preferred
    language is now utf-8, but the list's description is still iso-8859-1
    or whatever and thus contains non utf-8 codes.
    Thank you. I'll check these ...

    PS: By the way: Checking the .pck contents, I see much SPAM stuff. Would
    it be okay to simply delete those .pck files and concentrate on my own
    messages in both the /retry/ and /shunt/ folders?

    --UlfDunkel
  • Mark Sapiro at May 25, 2009 at 6:45 pm

    Ulf Dunkel wrote:
    Does /var/lib/mailman/bin/unshunt exist and is it executable (mode
    -rwxr-xr-x)?
    Yes, it exists there physically (no link or stuff) and is executable
    with mode -rwxr-xr-x.

    Then the #! first line points at a non-existent python, or if it says
    "#! @PYTHON@", this is a file that has not been configured by
    Mailman's configure - i.e it is a file unpacked from the tarball and
    not processed through configure and make.
    The first line reads like this:
    ----- snip -----
    #! /usr/bin/python
    ----- snap -----

    ... and of course /usr/bin/python exists, but is a link to
    /usr/bin/python2.4 - is this okay?

    It should be. I am now out of ideas as to why bash would report

    $ /var/lib/mailman/bin/unshunt
    $ -bash: unshunt: command not known

    Although that doesn't seem to be an exact copy/paste of the command and
    response.

    The "Temporary local problem" message is reported by your MTA to
    Mailman. Check the MTA's logs which may have more information.
    This looks rather strange to me (identical in exim4/mainlog and
    exim4/rejectlog):
    ----- snip -----
    2009-05-25 17:56:38 H=localhost (calamus.calamus.net) [127.0.0.1]
    F=<udo_discussion-bounces at lists.udo-open-source.org> temporarily
    rejected RCPT <udo-discussion-owner at lists.udo-open-source.org>: remote
    host address is the local host
    ----- snap -----

    I have really NO IDEA what this means and how I can fix it. ;-)
    Both domains (calamus.net and udo-open-source.org) are on my server.

    This is an exim configuration question.
    <http://www.exim.org/howto/mailman21.html> may help.

    Is lists.udo-open-source.org in the "domainlist local_domains" list in
    exim.conf?

    Regarding the utf8 issue, your OP says "I recently changed my Mailman
    2.1.9 installation to use and support Unicode." The error occurs when
    CookHeaders is trying to encode the list's 'description' for the
    List-ID header. I suspect that the charset for the list's preferred
    language is now utf-8, but the list's description is still iso-8859-1
    or whatever and thus contains non utf-8 codes.
    Thank you. I'll check these ...

    PS: By the way: Checking the .pck contents, I see much SPAM stuff. Would
    it be okay to simply delete those .pck files and concentrate on my own
    messages in both the /retry/ and /shunt/ folders?

    Yes. Just delete them.

    --
    Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Ulf Dunkel at May 26, 2009 at 7:18 am
    Hello Mark.
    It should be. I am now out of ideas as to why bash would report

    $ /var/lib/mailman/bin/unshunt
    $ -bash: unshunt: command not known

    Although that doesn't seem to be an exact copy/paste of the command and
    response.
    The litteral console output was:

    ----- snip -----
    calamus:~# cd /var/lib/mailman/bin/
    calamus:/var/lib/mailman/bin# unshunt
    -bash: unshunt: command not found
    calamus:/var/lib/mailman/bin#
    ----- snap -----

    ----- snip -----
    2009-05-25 17:56:38 H=localhost (calamus.calamus.net) [127.0.0.1]
    F=<udo_discussion-bounces at lists.udo-open-source.org> temporarily
    rejected RCPT <udo-discussion-owner at lists.udo-open-source.org>: remote
    host address is the local host
    ----- snap -----

    I have really NO IDEA what this means and how I can fix it. ;-)
    Both domains (calamus.net and udo-open-source.org) are on my server.

    This is an exim configuration question.
    <http://www.exim.org/howto/mailman21.html> may help.

    Is lists.udo-open-source.org in the "domainlist local_domains" list in
    exim.conf?
    Oh, it was only defined in Apache's virtual domains. Thank you for
    pointing me to this. I fixed it now.

    PS: By the way: Checking the .pck contents, I see much SPAM stuff. Would
    it be okay to simply delete those .pck files and concentrate on my own
    messages in both the /retry/ and /shunt/ folders?

    Yes. Just delete them.
    Everything seems to run fine again. Thanks a bunch.

    --UlfDunkel
  • Stephen J. Turnbull at May 26, 2009 at 7:40 am
    Ulf Dunkel writes:
    The litteral console output was: >
    ----- snip -----
    calamus:~# cd /var/lib/mailman/bin/
    calamus:/var/lib/mailman/bin# unshunt
    -bash: unshunt: command not found
    calamus:/var/lib/mailman/bin#
    ----- snap -----
    On Unix, for various reasons, the current directory is normally *not*
    in the search path for commands. You need to prefix the command name
    with the directory name (for current directory, you can abbreviate
    that to "./") like this:

    calamus:/var/lib/mailman/bin# ./unshunt <options and identifiers>
  • Ulf Dunkel at May 26, 2009 at 8:28 am
    Hi Stephen.
    On Unix, for various reasons, the current directory is normally *not*
    in the search path for commands. You need to prefix the command name
    with the directory name (for current directory, you can abbreviate
    that to "./") like this:

    calamus:/var/lib/mailman/bin# ./unshunt <options and identifiers>
    Cool - thank you for pointing me to this. Works fine. I just wonder why
    various other executables in the mailman/bin folder are executed without
    this trick, but this one doesn't.

    U se
    N erd's
    I ntelligence
    X ternal
  • Stefan Förster at May 26, 2009 at 8:00 pm

    * Ulf Dunkel wrote:
    On Unix, for various reasons, the current directory is normally *not*
    in the search path for commands. You need to prefix the command name
    with the directory name (for current directory, you can abbreviate
    that to "./") like this:

    calamus:/var/lib/mailman/bin# ./unshunt <options and identifiers>
    Cool - thank you for pointing me to this. Works fine. I just wonder why
    various other executables in the mailman/bin folder are executed without
    This is getting way offtopic, but a quick

    type command

    follwed if needed by

    ls -l $(which command)

    where "command" is one of the binaries you can execute (i.e. without a
    leading "./" in mailman/bin) will probably enlighten you :-)


    Ciao
    Stefan
    --
    Stefan F?rster http://www.incertum.net/ Public Key: 0xBBE2A9E9

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedMay 25, '09 at 10:03a
activeMay 26, '09 at 8:00p
posts10
users4
websitelist.org

People

Translate

site design / logo © 2021 Grokbase