FAQ
I have mailman 2.1.9 with the htdig patch installed. Set up to do local
search.

I rebuilt the archives from the list mbox file a couple of weeks ago,
and htdig would search on it, deliver pages, and they could be viewed.

This week (several days later), to view a post requires entering
a password before viewing. Trying to pull up a second page gets an
error message. This affects using the list admin password as well as
user passwords. To view any hit, a user account name or the admin
pasword (not both) has to be entered in a validation screen.

The mailman error log reports
htsearch for list: mercedes, cause: auth, detail: -10-

Browsers have cookies enabled (mine certainly does). Where do I look?
Permissions on the private directory were 2770; I reset them to 2774.
All of the files in private/listname are owner/group both mailman,
permissions on files are 666, on directories, 2775.

Where do I go from here?

Hank

--
Hank van Cleef (vancleef at lostwells.net, hvanclee at nyx.net)
1986 420SEL "A stranger in paradise" (Fremont Co. Wyoming)
1986 GMC 1500 6.2 diesel pickup "Seen one, seen them all"

Search Discussions

  • Mark Sapiro at Jan 19, 2007 at 5:31 pm

    Hank van Cleef wrote:
    I rebuilt the archives from the list mbox file a couple of weeks ago,
    and htdig would search on it, deliver pages, and they could be viewed.

    This week (several days later), to view a post requires entering
    a password before viewing.

    Did you change the archive from public to private? Do you have to enter
    the password each time (not normal) or only once (normal for a private
    archive)?

    Trying to pull up a second page gets an
    error message.

    What error message? If it's the 'we hit a bug' message, what's in
    Mailman's error log?

    This affects using the list admin password as well as
    user passwords. To view any hit, a user account name or the admin
    pasword (not both) has to be entered in a validation screen.

    Normal for a private archive, but should only be required the first
    time.

    The mailman error log reports
    htsearch for list: mercedes, cause: auth, detail: -10-

    Looking briefly at the htdig patch, I think this comes from mmsearch.py
    when private archive authorization fails.

    Browsers have cookies enabled (mine certainly does). Where do I look?
    Permissions on the private directory were 2770; I reset them to 2774.

    They shouldn't be 2774. They might need to be 2771 for public archive
    access to work. Otherwise 2770 should be fine. At this point the
    process should be running as the 'mailman' group because of the SETGID
    on the wrapper.

    All of the files in private/listname are owner/group both mailman,
    permissions on files are 666, on directories, 2775.

    Where do I go from here?

    I'm not sure anything is wrong, but if you are being required to
    authorize more than once per session per list, then there is a
    problem. What about normal archive access without searching? does that
    work?

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Mark Sapiro at Jan 19, 2007 at 7:28 pm

    Hank van Cleef wrote:
    The esteemed Mark Sapiro has said:
    Did you change the archive from public to private? Do you have to enter
    the password each time (not normal) or only once (normal for a private
    archive)?
    I used bin/arch to build the archives from the mbox file, all in the
    private directory. Then manually ran nightly_htdig to get the data
    bases built.

    But was the archive 'private' or 'public' (admin Archiving options)
    before the problem started?

    The scenario plays like this:

    On cold entry (no cookies in the browser) go to the archives page from
    the main page. User validation screen pops up. Enter username and
    password, and you get to the pipermail archives. You can now read those
    all day without needing to revalidate.

    Good.

    Now, set up an htdig search, which returns a few pages of hits. Click
    on one of those hits, and the validation screen pops up again.
    Validate, and the message is displayed. Go back and click on another
    hit. The validation screen pops up again, and displays the page after
    another validation.

    Trying to display the second page makes the

    I assume you mean the second page of hits.

    htdig Archives Access Failure

    template page pop up.

    This happens with either a user or the admin password active.

    The error log message that accompanies this is:

    Jan 19 11:12:55 2007 (23983) htsearch for list: mercedes, cause: auth,
    detail: -10-

    It seems clear to me that the htdig process is not able to find the validation
    cookie.

    I don't think that is the issue. It seems to me that the htdig process
    finds the cookie the first time, but then removes it. This explains
    why you need to revalidate when you try to visit the message, and it
    explains why you get mmsearch.py's authorization error when you try to
    get the second page of hits.

    If this were compiled code and not Python, I'd dbx the thing,
    but don't know enough about Python to know how to use the debugging
    resources. If there's some sort of crash and traceback message, I don't
    know where to look for that either.

    There wouldn't be a traceback in this case.

    For reference, the htdig (3.1.6) build was a default ./configure build.
    It wouldn't compile with either the Solaris Studio 11 CC or g++ 4.1.1.
    I loaded g++ 2.95.3 on another Solaris 9 machine, did a build there, NFS
    mounted the source tree, and did a make install to get it on this
    machine. The only additional step needed to get htdig to run its own
    rundig was to copy libstdc++ into /usr/lib (a bug in the Solaris
    Companion Disk g++ build).

    The default overrides in mm_cfg.py relative to archiving (you'll notice
    that some are commented out----I copied this file from the old host
    installation, and left the last guy's commented-out stuff in place)

    # HTDIG_MAILMAN_LINK = 'htdig-mailman'
    HTDIG_RUNDIG_PATH = '/opt/www/htdig/bin/rundig'
    HTDIG_HTSEARCH_PATH = '/opt/www/cgi-bin/htsearch'
    USE_HTDIG = 1

    # ARCHIVE_TO_MBOX = 1

    GZIP_ARCHIVE_TXT_FILES = 0

    It seems that htdig per se is working fine. It returns the hits. I
    think the problem is in mmsearch.py. It checks to be sure it is
    authorized for the private archive, and it is. It does the search and
    displays the results and somehow in this latter step the cookie is
    removed.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan
  • Hank van Cleef at Jan 19, 2007 at 10:20 pm
    The esteemed Mark Sapiro has said:

    But was the archive 'private' or 'public' (admin Archiving options)
    before the problem started?
    We're going back to Jan. 11 (when this started), and I'm trying to
    remember what I changed then. I suspect that I ran check_perms around
    then an fixed what it griped about, and the permissions on the private
    directory was one such. Let's assume this was the case (I can easily
    check, but I don't think setting the directory permission open really
    proves anything. Nonetheless, I should do it to see that it does clear
    the fault).
    The scenario plays like this:

    On cold entry (no cookies in the browser) go to the archives page from
    the main page. User validation screen pops up. Enter username and
    password, and you get to the pipermail archives. You can now read those
    all day without needing to revalidate.

    Good.

    Now, set up an htdig search, which returns a few pages of hits. Click
    on one of those hits, and the validation screen pops up again.
    Validate, and the message is displayed. Go back and click on another
    hit. The validation screen pops up again, and displays the page after
    another validation.

    Trying to display the second page makes the

    I assume you mean the second page of hits.
    Yes. Htdig will do the search and display a page of returns. Anything
    beyond that requires revalidation.
    It seems clear to me that the htdig process is not able to find the validation
    cookie.

    I don't think that is the issue. It seems to me that the htdig process
    finds the cookie the first time, but then removes it. This explains
    why you need to revalidate when you try to visit the message, and it
    explains why you get mmsearch.py's authorization error when you try to
    get the second page of hits.
    Yes, it's the second time in. Doing a second search on top of the first
    (i.e. changing the htdig search return display and telling it to search
    again) gets the error.
    >
    I have been reading the code (and Python in a Nutshell).

    mmsearch.py has
    if mlist.archive_private:
    if not mlist.WebAuthenticate((mm_cfg.AuthUser,
    mm_cfg.AuthListModerator,
    mm_cfg.AuthListAdmin,
    mm_cfg.AuthSiteAdmin),
    '', ''):
    raise _search_exception(listname, 'auth', '-10-')

    What I'm seeing is consistent with the "if not" condition. And if this
    is the second pass through this, rather than the first, then it implies
    that something in the code following is involved. I've included a cut
    and paste of exactly what is in the mmsearch.py, and will be doing some
    creative code-reading to see if this call differs from other calls to
    the same function.

    I've located the WebAuthenticate() function in SecurityManager.py and
    note that it is used in several places, all of which behave as
    expected.
    >
    One question comes to mind: why are we revalidating here? I'll assume
    without having researched it that the entry to the pipermail archives
    has its own validation, and in my configuration, you can't get to the
    archive search except from that post-validated page.

    Theoretically, I suppose I could work around for the moment by simply
    commenting out that check, but since something's getting trashed
    somwhere, think it better to pursue debugging the problem.
    If this were compiled code and not Python, I'd dbx the thing,
    but don't know enough about Python to know how to use the debugging
    resources. If there's some sort of crash and traceback message, I don't
    know where to look for that either.
    I'm busy reading the O'Reilly python book on debugging to see where I
    might get some hooks into this.
    There wouldn't be a traceback in this case.
    No, I can see that the code has an exit.
    It seems that htdig per se is working fine. It returns the hits. I
    think the problem is in mmsearch.py. It checks to be sure it is
    authorized for the private archive, and it is. It does the search and
    displays the results and somehow in this latter step the cookie is
    removed.
    I'll guess that you're right about the htdig installation. However some
    more testing is showing is not that the cookie is being removed, but
    that we've got something that's nulling something.

    Running with the admin password set, I go from the admin area to the
    main list page, then to the archives, where I can read the pipermail
    archives repeatedly.

    Do an htdig search, get the first page of hits back, and try to look at
    one. Anything entered in either the username or password field clears
    the block. One character in either field suffices, and it does not have to be
    alphanumeric.

    After having gone throught this exercise several times, I can go back to
    the admin pages without having to revalidate---the cookie is still set.

    Hank

    --
    Hank van Cleef (vancleef at lostwells.net, hvanclee at nyx.net)
    1986 420SEL "A stranger in paradise" (Fremont Co. Wyoming)
    1986 GMC 1500 6.2 diesel pickup "Seen one, seen them all"
  • Mark Sapiro at Jan 19, 2007 at 11:36 pm

    Hank van Cleef wrote:
    The esteemed Mark Sapiro has said:

    But was the archive 'private' or 'public' (admin Archiving options)
    before the problem started?
    We're going back to Jan. 11 (when this started), and I'm trying to
    remember what I changed then. I suspect that I ran check_perms around
    then an fixed what it griped about, and the permissions on the private
    directory was one such. Let's assume this was the case (I can easily
    check, but I don't think setting the directory permission open really
    proves anything. Nonetheless, I should do it to see that it does clear
    the fault).

    What I'm trying to determine here is if the authorization in
    mmsearch.py has 'always' been broken and it was only noticed because
    the archive was changed from 'public' (not requiring authorization) to
    'private', or if it worked before and stopped working.


    Yes, it's the second time in. Doing a second search on top of the first
    (i.e. changing the htdig search return display and telling it to search
    again) gets the error.
    I have been reading the code (and Python in a Nutshell).

    mmsearch.py has
    if mlist.archive_private:
    if not mlist.WebAuthenticate((mm_cfg.AuthUser,
    mm_cfg.AuthListModerator,
    mm_cfg.AuthListAdmin,
    mm_cfg.AuthSiteAdmin),
    '', ''):
    raise _search_exception(listname, 'auth', '-10-')

    Yes. This code says "you can't be here if you don't have the cookie".
    The first time through you have the cookie and all is good. If you
    then go to a message, you don't have the cookie, but that code allows
    you to reauthorize, so you get it again, but you had to reauthorize.
    If you come through mmsearch.py a second time, you have no cookie and
    you get the exception.

    What I'm seeing is consistent with the "if not" condition. And if this
    is the second pass through this, rather than the first, then it implies
    that something in the code following is involved. I've included a cut
    and paste of exactly what is in the mmsearch.py, and will be doing some
    creative code-reading to see if this call differs from other calls to
    the same function.

    I don't see the cut and paste, but I have the patch code, so I don't
    need it.

    I've located the WebAuthenticate() function in SecurityManager.py and
    note that it is used in several places, all of which behave as
    expected.
    One question comes to mind: why are we revalidating here?

    To prevent you from just going directly to an mmsearch URL and
    searching without authorization. We don't attempt to reauthorize here,
    because if you are playing by the rules, you are already authorized.
    We just check.

    I'll assume
    without having researched it that the entry to the pipermail archives
    has its own validation, and in my configuration, you can't get to the
    archive search except from that post-validated page.

    If you are playing by the rules.

    Theoretically, I suppose I could work around for the moment by simply
    commenting out that check, but since something's getting trashed
    somwhere, think it better to pursue debugging the problem.

    You don't want to comment out the check because it will allow
    unauthorized users to search the archive by invoking mmsearch
    directly. They won't be able to view the messages, but they can get
    information from the search results.

    I'll guess that you're right about the htdig installation. However some
    more testing is showing is not that the cookie is being removed, but
    that we've got something that's nulling something.

    Running with the admin password set, I go from the admin area to the
    main list page, then to the archives, where I can read the pipermail
    archives repeatedly.

    Good, but I'm not sure that's relevant to the problem.

    Do an htdig search, get the first page of hits back, and try to look at
    one. Anything entered in either the username or password field clears
    the block. One character in either field suffices, and it does not have to be
    alphanumeric.

    Puzzling, but what then is the exact URL that the search results page
    links to for the message.

    After having gone throught this exercise several times, I can go back to
    the admin pages without having to revalidate---the cookie is still set.

    If the cookie isn't being removed, then perhaps there is something in
    your web server specific to the URLs that don't work that is not
    passing the cookie to the environment. This now seems very likely, and
    would explain why you can reauthorize with 'invalid' data.

    Look at
    <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.045.htp>
    and
    <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.065.htp>
    and see if there are any clues there.

    For example are the non-working and only the non-working URLs http
    scheme that's being redirected to an https scheme URL in your web
    server.

    --
    Mark Sapiro <msapiro at value.net> The highway is for gamblers,
    San Francisco Bay Area, California better use your sense - B. Dylan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmailman-users @
categoriespython
postedJan 19, '07 at 3:50p
activeJan 19, '07 at 11:36p
posts5
users2
websitelist.org

2 users in discussion

Mark Sapiro: 3 posts Hank van Cleef: 2 posts

People

Translate

site design / logo © 2021 Grokbase