FAQ
I am using Windows 98, python 2.3, ClientCookie 0.4.3a.

When I do:
import ClientCookie
import os
c = ClientCookie.MSIECookieJar(delayload=1)
c.load_from_registry()

I get the response:
Traceback (most recent call last):
File "C:\My Documents\mycookies.py", line 33, in ?
c.load_from_registry()
File "C:\PYTHON23\Lib\site-packages\ClientCookie\_MSIECookieJar.py",
line 230, in load_from_registry
self.load(filename, ignore_discard, ignore_expires)
File "C:\PYTHON23\Lib\site-packages\ClientCookie\_MSIECookieJar.py",
line 245, in load
self._really_load(index, filename, ignore_discard, ignore_expires)
File "C:\PYTHON23\Lib\site-packages\ClientCookie\_MSIECookieJar.py",
line 258, in _really_load
user_name = string.lower(os.environ['USERNAME'])
File "C:\PYTHON23\Lib\os.py", line 417, in __getitem__
return self.data[key.upper()]
KeyError: 'USERNAME'

Basically, it doesn't like USERNAME as an environment variable.

Search Discussions

  • Syver Enstad at Aug 14, 2003 at 10:29 am

    cartermark46 at ukmail.com (Mark Carter) writes:

    I am using Windows 98, python 2.3, ClientCookie 0.4.3a.

    When I do:
    import ClientCookie
    import os
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry()

    I get the response:
    Traceback (most recent call last):
    File "C:\My Documents\mycookies.py", line 33, in ?
    c.load_from_registry()
    File "C:\PYTHON23\Lib\site-packages\ClientCookie\_MSIECookieJar.py",
    line 230, in load_from_registry
    self.load(filename, ignore_discard, ignore_expires)
    File "C:\PYTHON23\Lib\site-packages\ClientCookie\_MSIECookieJar.py",
    line 245, in load
    self._really_load(index, filename, ignore_discard, ignore_expires)
    File "C:\PYTHON23\Lib\site-packages\ClientCookie\_MSIECookieJar.py",
    line 258, in _really_load
    user_name = string.lower(os.environ['USERNAME'])
    File "C:\PYTHON23\Lib\os.py", line 417, in __getitem__
    return self.data[key.upper()]
    KeyError: 'USERNAME'

    Basically, it doesn't like USERNAME as an environment variable.
    I suspect this only works on "real" win32 (the NT family) operating
    systems, which define USERNAME, and not hybrid dos/win32 systems like
    the win9x family.
  • John J. Lee at Aug 14, 2003 at 12:05 pm

    Syver Enstad <syver at inout.no> writes:

    cartermark46 at ukmail.com (Mark Carter) writes:
    I am using Windows 98, python 2.3, ClientCookie 0.4.3a.

    When I do:
    import ClientCookie
    import os
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry()

    I get the response:
    [...]
    KeyError: 'USERNAME'

    Basically, it doesn't like USERNAME as an environment variable.
    I suspect this only works on "real" win32 (the NT family) operating
    systems, which define USERNAME, and not hybrid dos/win32 systems like
    the win9x family.
    Anybody know the best way to get the username without win32all
    installed?

    Mark: I don't have a win9x box, but try replacing that
    os.environ['USERNAME'] with getpass.getuser(). You'll need to stick
    an import getpass in there too, of course. Let me know if that works.

    The rest of the code should work OK on win9x.


    John
  • Mark Carter at Aug 14, 2003 at 7:19 pm
    jjl at pobox.com (John J. Lee) wrote in message news:<87fzk4sav2.fsf at pobox.com>...
    Syver Enstad <syver at inout.no> writes:
    cartermark46 at ukmail.com (Mark Carter) writes:
    I am using Windows 98, python 2.3, ClientCookie 0.4.3a.

    When I do:
    import ClientCookie
    import os
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry()

    I get the response:
    [...]
    KeyError: 'USERNAME'

    Basically, it doesn't like USERNAME as an environment variable.
    I suspect this only works on "real" win32 (the NT family) operating
    systems, which define USERNAME, and not hybrid dos/win32 systems like
    the win9x family.
    Anybody know the best way to get the username without win32all
    installed?

    Mark: I don't have a win9x box, but try replacing that
    os.environ['USERNAME'] with getpass.getuser(). You'll need to stick
    an import getpass in there too, of course. Let me know if that works.

    The rest of the code should work OK on win9x.


    John
    My apologies - the bug exists in my own brain, not in the code. I had in mind
    that a fix would be required in ClientCookie - but now I know better.

    To get it to work, I simply called:
    os.environ['USERNAME'] = 'mark carter' # or whatever your name is
    before calling
    c.load_from_registry() # c is a MSIECookieJar

    It's an embarassingly simple solution to the problem - but it took me a while
    to figure it out. Although there is nothing wrong with code, perhaps it would
    help new users of the library to mention it. Maybe the solution is somewhat
    obvious in retrospect - but it took me a while before the penny dropped.

    Another approach - which I actually prefer - is to copy the cookie to the local
    directory and load it using load_cookie_data(). At first, I was having no end
    of problems with it - until I discovered that you should use the (binary)
    cookie, not an ASCII cookie that you obtain from performing a cookie export
    from MSIE. ... and don't forget to call ClientCookie.MSIECookieJar() WITHOUT
    the delayload argument.

    So, sorry for the false alarm - but hope my investigations will prove useful to
    others.
  • Mark Carter at Aug 15, 2003 at 9:34 am

    trying the getpass fix I suggested? I don't know if getpass.getuser
    works on w9x, but it's worth a shot.
    On a win95 machine, running a fresh installation of python3.2, typing
    import getpass
    print getpass.getuser()
    returns
    ImportError: No module named pwd.

    I also tried getuid() - but that didn't work, either (it's in UNIX
    only). *Sigh* It appears that we are stuck with the workaround that I
    suggested.

    I'll try it on a win 98 machine, and report back the results asap.
  • Mark Carter at Aug 16, 2003 at 9:31 am
    I'll try it on a win 98 machine, and report back the results asap.
    ---
    USERNAME

    I promised that I would try getuser() on a win 98 machine. Alas,
    typing:

    import getpass
    print getpass.getuser()

    produces:

    ImportError: No module named pwd

    So, it's the same problem as occurs on win95 (unsuprisingly)

    ---
    DELAYLOAD

    John: Why without the delayload argument?

    My response:

    The following code:

    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_cookie_data("hemscott-cookie.bin")
    import urllib2
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    request2 = urllib2.Request(url)
    c.add_cookie_header(request2)
    response2 = urllib2.urlopen(request2)
    print response2.geturl()
    print response2.info() # headers
    for line in response2.readlines(): # body
    print line


    produces the error:

    Traceback (most recent call last):
    File "C:\My Documents\markc\ClientCookie\clp04.py", line 9, in ?
    c.add_cookie_header(request2)
    File "C:\PYTHON23\Lib\site-packages\ClientCookie\_ClientCookie.py",
    line 1170, in add_cookie_header
    cookies.extend(self._get_cookies_for_domain(
    File "C:\PYTHON23\Lib\site-packages\ClientCookie\_ClientCookie.py",
    line 1050, in _get_cookies_for_domain
    return self._cookies_for_domain(domain, request, unverifiable)
    File "C:\PYTHON23\lib\site-packages\ClientCookie\_MSIECookieJar.py",
    line 112, in _cookies_for_domain
    if self.delayload and cookies["//+delayload"] is not None:
    KeyError: '//+delayload'

    whereas running it without delayload=1 causes it to run successfully.
  • John J. Lee at Aug 16, 2003 at 10:33 am
    cartermark46 at ukmail.com (Mark Carter) writes:
    [...]
    John: Why without the delayload argument?

    My response:

    The following code:

    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_cookie_data("hemscott-cookie.bin")
    import urllib2
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    request2 = urllib2.Request(url)
    c.add_cookie_header(request2)
    response2 = urllib2.urlopen(request2)
    print response2.geturl()
    print response2.info() # headers
    for line in response2.readlines(): # body
    print line
    Take note of this comment from the web page:
    # Don't copy this blindly! You probably want to follow the examples
    # above, not this one.
    No matter how many times and how many places I say this, everybody
    seems to be driven to make the same mistakes. I guess people don't
    believe it's as simple as calling urlopen, though I explictly say it
    *is* that simple in most cases in the second sentence of the
    documentation.

    DON'T USE *BOTH* add_cookie_header/extract_cookies *AND* urlopen.
    urlopen is all you need, unless you're not using urllib2. Probably
    won't do any actual harm, but it's completely pointless and
    obfuscatory. You're also failing to use ClientCookie.urlopen (which
    unlike urllib2.urlopen, knows about cookies), because you blindly
    copied the third example from the web page. You're also using Request
    objects for no apparent reason, and you're attempting to fetch the
    same url twice, again because you're blindly copying. Sigh -- if
    people have to blindly copy, why not copy from the code I actually
    *tell* people to copy from, instead of the code I explictly tell
    people *not* to copy from? Ngghhh! :-)

    You want something like this:

    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_cookie_data("hemscott-cookie.bin")
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = ClientCookie.urlopen(url)

    print response.read()
    response.close()

    produces the error:

    Traceback (most recent call last): [...]
    whereas running it without delayload=1 causes it to run successfully.
    Anyway, you have found another bug, so all is forgiven.

    (the MSIE delayload feature is still a bit of a mess -- luckily, I can
    pin the blame on the original author of that code, since it's pretty
    directly ported from Perl ;)


    John
  • John J. Lee at Aug 16, 2003 at 11:05 am
    jjl at pobox.com (John J. Lee) writes:
    [...]
    Traceback (most recent call last): [...]
    whereas running it without delayload=1 causes it to run successfully.
    [...]

    I just uploaded a new version with this fixed and a username argument
    to load / load_from_registry (only needed for win9x).

    Let me know if it works or not: I don't have a networked Windows
    machine.


    John
  • Mark Carter at Aug 17, 2003 at 11:35 am

    *tell* people to copy from, instead of the code I explictly tell
    people *not* to copy from? Ngghhh! :-)
    Anyway, you have found another bug, so all is forgiven.
    (the MSIE delayload feature is still a bit of a mess -- luckily, I can
    pin the blame on the original author of that code, since it's pretty
    directly ported from Perl ;)

    If we shadows have offended,
    Think but this, and all is mended-
    That you have but slumb'red here
    While these visions did appear.
    And this weak and idle theme,
    No more yielding but a dream,
    Gentles, do not reprehend.
    If you pardon, we will mend.
    And, as I am an honest Puck,
    If we have unearned luck
    Now to scape the serpent's tongue,
    We will make amends ere long;
    Else the Puck a liar call.
    So, good night unto you all.
    Give me your hands, if we be friends,
    And Robin shall restore amends.


    But seriously ... please standby and I will inform you of the outcome
    of your changes.
  • Gary Feldman at Aug 17, 2003 at 4:45 pm

    On 16 Aug 2003 11:33:13 +0100, jjl at pobox.com (John J. Lee) wrote:
    Take note of this comment from the web page:
    # Don't copy this blindly! You probably want to follow the examples
    # above, not this one.
    Since the purpose of that example seems to be to show how things work under
    the hood, may I suggest putting it on a separate page, and replacing it
    here with something like "Here is a <a href...>lower level example</a> that
    shows how this works, though you would rarely want to implement things at
    this low level."

    Gary
  • John J. Lee at Aug 18, 2003 at 1:43 pm

    Gary Feldman <gafStopSpamData at ziplink.stopallspam.net> writes:
    On 16 Aug 2003 11:33:13 +0100, jjl at pobox.com (John J. Lee) wrote:


    Take note of this comment from the web page:
    # Don't copy this blindly! You probably want to follow the examples
    # above, not this one.
    Since the purpose of that example seems to be to show how things work under
    the hood, may I suggest putting it on a separate page, and replacing it
    here with something like "Here is a <a href...>lower level example</a> that
    shows how this works, though you would rarely want to implement things at
    this low level."
    Hmm, good idea, but I really don't want to split the documentation up
    -- one page is simpler -- and the example is instructive for people
    who actually want to understand what the module does. And if people
    manage to miss that comment (highlighted in orange, for heavens sake),
    well...


    John
  • Gary Feldman at Aug 18, 2003 at 7:15 pm

    On 18 Aug 2003 14:43:06 +0100, jjl at pobox.com (John J. Lee) wrote:
    Hmm, good idea, but I really don't want to split the documentation up
    -- one page is simpler -- and the example is instructive for people
    Then definitely blockquote it (or indent it some other way), and consider
    putting it into a smaller font, or using a grey background, or something
    else to indicate that it's a digression. Orange would draw attention to
    it; you want the opposite.

    Gary
  • John J. Lee at Aug 18, 2003 at 8:47 pm

    Gary Feldman <gafStopSpamData at ziplink.stopallspam.net> writes:
    On 18 Aug 2003 14:43:06 +0100, jjl at pobox.com (John J. Lee) wrote:

    Hmm, good idea, but I really don't want to split the documentation up
    -- one page is simpler -- and the example is instructive for people
    Then definitely blockquote it (or indent it some other way), and consider
    putting it into a smaller font, or using a grey background, or something
    else to indicate that it's a digression.
    Again, that would be a good idea if it *were* a digression, but it's
    necessary for understanding what the module gets up to. Without that
    understanding in the reader's mind, it's hard to explain the code that
    one uses in practice if it's any more complicated than urlopen. And
    the very top of the page says:
    import ClientCookie
    response = ClientCookie.urlopen("http://foo.bar.com/")

    This function behaves identically to urllib2.urlopen, except that it
    deals with cookies automatically. That's probably all you need to
    know.
    So you don't even have to read further than that for most purposes. I
    can't see how to improve on that, but I'm happy to learn how!

    Orange would draw attention to
    it; you want the opposite.
    Only the comment is in emacs-orange (well, my copy of python-mode uses
    that kind of rust-orange for Python comments), so it doesn't
    particularly draw attention to that block of code more than the rest.
    And I *do* want to draw attention to the comment, so people read the
    comment before the code.

    Admittedly, it doesn't seem to work ;-) (on a sample of one
    misinterpreter, Mark, so far -- I only just added that comment
    recently, though there are several other warnings that cover the same
    ground elsewhere).


    John
  • Mark Carter at Aug 18, 2003 at 9:45 am

    You want something like this:

    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_cookie_data("hemscott-cookie.bin")
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = ClientCookie.urlopen(url)

    print response.read()
    response.close()

    It doesn't work on my XP machine at least. This is proboably why
    people have been doing it all wrong, allegedly.

    I'll investigate further. Apologies for the time lag.
  • John J. Lee at Aug 18, 2003 at 1:33 pm

    cartermark46 at ukmail.com (Mark Carter) writes:

    You want something like this:

    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_cookie_data("hemscott-cookie.bin")
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = ClientCookie.urlopen(url)

    print response.read()
    response.close()

    It doesn't work on my XP machine at least. This is proboably why
    people have been doing it all wrong, allegedly.
    I don't see how -- the examples in question (the one everybody seems
    driven to copy from, and the ones people should copy from) does not
    include any reference to any CookieJar.

    I'll investigate further. Apologies for the time lag.
    Thanks


    John
  • John J. Lee at Aug 18, 2003 at 6:01 pm

    jjl at pobox.com (John J. Lee) writes:

    cartermark46 at ukmail.com (Mark Carter) writes:
    [...]
    It doesn't work on my XP machine at least. This is proboably why
    people have been doing it all wrong, allegedly.
    I don't see how -- the examples in question (the one everybody seems
    driven to copy from, and the ones people should copy from) does not
    include any reference to any CookieJar.
    OK, it does, now I come to actually look at it...


    John
  • Mark Carter at Aug 18, 2003 at 2:05 pm
    I'll investigate further.
    Here are the results from running tests in ClientCookie 0.4.4.a:

    def go7():
    #ClientCookie 0.4.4.a:
    #works from win xp and win 98

    # I prefer this method as a better way than using
    load_from_registry()
    #use this method!!

    import ClientCookie
    c = ClientCookie.MSIECookieJar() # do NOT set delayload
    #c.user_name = "mark carter"
    c.load_cookie_data("hemscott-cookie.bin")
    #c.load_from_registry()
    print c

    import urllib2
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    c.extract_cookies(response, request)
    #let's say this next request requires a cookie that was set in
    response
    request2 = urllib2.Request(url)
    c.add_cookie_header(request2)
    response2 = urllib2.urlopen(request2)

    print response2.geturl()
    print response2.info() # headers
    print response2.read()
    response2.close()



    def go8():
    #ClientCookie 0.4.4.a:
    #contains bug in Win98 when environ variable is commented out,
    #but works in win98 when environ variable is set
    #Works in win xp , regardless of the USERNAME line

    #this works - I can now import into Hemscott
    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    #c.user_name = "mark carter"
    #os.environ['USERNAME'] = 'mcarter' #needed by
    load_from_registry()
    c.load_from_registry()

    import urllib2
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    #c.extract_cookies(response, request)
    # let's say this next request requires a cookie that was set in
    response
    request2 = urllib2.Request(url)
    c.add_cookie_header(request2)
    response2 = urllib2.urlopen(request2)

    print response2.geturl()
    print response2.info() # headers
    for line in response2.readlines(): # body
    print line


    The upshot of this is that load_cookie_data() now works in win 98 and
    xp.
    load_from_registry() works from win xp; it works from win 98 if and
    only if you set the USERNAME environment variable.

    I appreciate that all the stuff about request2 and response2 may not
    be to your liking - but at the moment I'm just trying to figure out
    what works, and what doesn't. We can also worry about the delayload
    business later.

    What do you think about the idea of actually setting up an Aapache web
    page to test these things 'for real'?
  • John J. Lee at Aug 18, 2003 at 6:14 pm

    cartermark46 at ukmail.com (Mark Carter) writes:

    I'll investigate further.
    Here are the results from running tests in ClientCookie 0.4.4.a: [...]
    The upshot of this is that load_cookie_data() now works in win 98 and
    xp.
    load_from_registry() works from win xp; it works from win 98 if and
    only if you set the USERNAME environment variable.
    You missed the username argument.

    cookiejar.load_from_registry(username="mark")

    (should only be required for win9x family)

    I appreciate that all the stuff about request2 and response2 may not
    be to your liking - but at the moment I'm just trying to figure out
    what works, and what doesn't. We can also worry about the delayload
    business later.
    No really, I wasn't joking: you *never* need to use add_cookie_header
    / extract_cookies if you're using urllib2 (at least, I can't think of
    any possible reason to do so). It can only break things.

    What do you think about the idea of actually setting up an Aapache web
    page to test these things 'for real'?
    I've done limited testing on Windows with 'fake' cookies from a local
    Apache server, and on wine on linux. As I said, though, I don't have
    a networked Windows OS, so it's inconvenient to test these things in a
    'real' situation. And my machine currently doesn't boot into Windows
    without physically switching cables around (security & obscure
    hardware issues, not software ones), which means I currently can't be
    bothered to test it on Windows :-P. So, your feedback is appreciated.


    John
  • Anand Pillai at Aug 19, 2003 at 6:07 am
    I am working on a Cookie module which works *with* urllib2 rather
    than on top of it like the existing ClientCookie module. It uses
    the Cookie module which comes with python standard library.

    This module is written as an extension of my Harvestman webcrawler.
    The alpha code is ready. We are doing testing right now.

    Details will be posted to my website at
    http://members.lycos.co.uk/anandpillai within say 2 weeks or so.

    -Anand


    jjl at pobox.com (John J. Lee) wrote in message news:<87k79a25pi.fsf at pobox.com>...
    cartermark46 at ukmail.com (Mark Carter) writes:
    I'll investigate further.
    Here are the results from running tests in ClientCookie 0.4.4.a: [...]
    The upshot of this is that load_cookie_data() now works in win 98 and
    xp.
    load_from_registry() works from win xp; it works from win 98 if and
    only if you set the USERNAME environment variable.
    You missed the username argument.

    cookiejar.load_from_registry(username="mark")

    (should only be required for win9x family)

    I appreciate that all the stuff about request2 and response2 may not
    be to your liking - but at the moment I'm just trying to figure out
    what works, and what doesn't. We can also worry about the delayload
    business later.
    No really, I wasn't joking: you *never* need to use add_cookie_header
    / extract_cookies if you're using urllib2 (at least, I can't think of
    any possible reason to do so). It can only break things.

    What do you think about the idea of actually setting up an Aapache web
    page to test these things 'for real'?
    I've done limited testing on Windows with 'fake' cookies from a local
    Apache server, and on wine on linux. As I said, though, I don't have
    a networked Windows OS, so it's inconvenient to test these things in a
    'real' situation. And my machine currently doesn't boot into Windows
    without physically switching cables around (security & obscure
    hardware issues, not software ones), which means I currently can't be
    bothered to test it on Windows :-P. So, your feedback is appreciated.


    John
  • John J. Lee at Aug 19, 2003 at 12:30 pm

    pythonguy at Hotpop.com (Anand Pillai) writes:

    I am working on a Cookie module which works *with* urllib2 rather
    than on top of it like the existing ClientCookie module. It uses
    the Cookie module which comes with python standard library.
    Interesting, though I don't know quite what you mean.

    First, if there's a way to work more closely with urllib2 than I've
    figured out (which is quite possible), this patch needs to know about
    it, so please post a comment:

    http://www.python.org/sf/759792

    If I understand what you mean, ClientCookie only works 'on top of'
    rather than 'with' urllib2 to the extent that it currently has to
    cut-n-paste code to add cookie handling to urllib2. That patch is
    designed to remove the need to cut-n-paste, which would mean you'd do
    urllib2.urlopen (after building an OpenerDirector that has an
    HTTPCookieProcessor from ClientCookie) instead of ClientCookie.urlopen
    as is required at present.

    Second: is your module intended to do what ClientCookie does
    (ie. figure out what cookies should be set and returned, and do so),
    or is it just a more OO way of getting and returning Cookie headers?
    I guess the latter?

    This module is written as an extension of my Harvestman webcrawler.
    The alpha code is ready. We are doing testing right now.
    Is this the Royal We? ;-)

    Details will be posted to my website at
    http://members.lycos.co.uk/anandpillai within say 2 weeks or so.
    [...]

    Please do post an announcement to c.l.py.announce, or I'll forget.


    John
  • Anand Pillai at Aug 20, 2003 at 10:42 am
    Hi John

    I wanted to add cookies support to harvestman, your module
    looked ideal for it.

    'We', nothing royal about it. It is just me and my friend
    & co-developer Nirmal Chidambaram. Apparently he has found
    a way around some of the bugs in Clientcookie. He has written
    a new module using the existing Cookie module of python &
    urllib2. One of the problems 'we' had with Clientcookie is that
    it uses its own 'urlopen' methods which does not fit our
    applications needs, so 'we' had to find a way around it.

    Once the code is ready, I will post it on my webpage, and
    of course it is not a module in itself, so I think an
    announcement to c.l.py is out of place.

    Regards

    -Anand


    jjl at pobox.com (John J. Lee) wrote in message news:<87ada5de3j.fsf at pobox.com>...
    pythonguy at Hotpop.com (Anand Pillai) writes:
    I am working on a Cookie module which works *with* urllib2 rather
    than on top of it like the existing ClientCookie module. It uses
    the Cookie module which comes with python standard library.
    Interesting, though I don't know quite what you mean.

    First, if there's a way to work more closely with urllib2 than I've
    figured out (which is quite possible), this patch needs to know about
    it, so please post a comment:

    http://www.python.org/sf/759792

    If I understand what you mean, ClientCookie only works 'on top of'
    rather than 'with' urllib2 to the extent that it currently has to
    cut-n-paste code to add cookie handling to urllib2. That patch is
    designed to remove the need to cut-n-paste, which would mean you'd do
    urllib2.urlopen (after building an OpenerDirector that has an
    HTTPCookieProcessor from ClientCookie) instead of ClientCookie.urlopen
    as is required at present.

    Seco
    nd: is your module intended to do what ClientCookie does
    (ie. figure out what cookies should be set and returned, and do so),
    or is it just a more OO way of getting and returning Cookie headers?
    I guess the latter?

    This module is written as an extension of my Harvestman webcrawler.
    The alpha code is ready. We are doing testing right now.
    Is this the Royal We? ;-)

    Details will be posted to my website at
    http://members.lycos.co.uk/anandpillai within say 2 weeks or so.
    [...]

    Please do post an announcement to c.l.py.announce, or I'll forget.


    John
  • John J. Lee at Aug 20, 2003 at 1:15 pm
    pythonguy at Hotpop.com (Anand Pillai) writes:
    [...]
    'We', nothing royal about it. It is just me and my friend
    & co-developer Nirmal Chidambaram. Apparently he has found
    a way around some of the bugs in Clientcookie. He has written
    It'd be great if you made me aware what those bugs are!

    (BTW, no intent to offend with my comment about your plurality, or
    lack thereof -- it's just that the convention of using 'we' in source
    code comments is common enough that I've sometimes found myself using
    it even when writing code alone, which is funny.)

    a new module using the existing Cookie module of python &
    urllib2. One of the problems 'we' had with Clientcookie is that
    it uses its own 'urlopen' methods which does not fit our
    applications needs, so 'we' had to find a way around it.
    As I said before, if you know how to do that, please comment on the
    RFE I referenced in my last post. Jeremy Hylton is planning to look
    at the patch associated with that RFE in detail sometime, and you
    could save him some time if you know a way to do this without patching
    urllib2. And I'd like to know how to do it, too :-)

    Once the code is ready, I will post it on my webpage, and
    of course it is not a module in itself, so I think an
    announcement to c.l.py is out of place.
    [...]

    Would you mind sending me an email?

    Thanks


    John
  • Anand Pillai at Aug 20, 2003 at 6:56 pm
    In fact, I am not aware of how Nirmal has done it.
    He is wrapping up the coding soon, when I will get to
    see how he has done it. He is doing the persistency part
    of the cookies, and I guess he is using anydbm for it. Details
    later.

    I will send a mail to you, once I get the code from him
    and review it. Meanwhile you can visit his homepage here

    http://nirmalc.freezope.org

    There might be some details on the work there.

    Regards

    -Anand

    jjl at pobox.com (John J. Lee) wrote in message news:<87smnwh3lp.fsf at pobox.com>...
    pythonguy at Hotpop.com (Anand Pillai) writes:
    [...]
    'We', nothing royal about it. It is just me and my friend
    & co-developer Nirmal Chidambaram. Apparently he has found
    a way around some of the bugs in Clientcookie. He has written
    It'd be great if you made me aware what those bugs are!

    (BTW, no intent to offend with my comment about your plurality, or
    lack thereof -- it's just that the convention of using 'we' in source
    code comments is common enough that I've sometimes found myself using
    it even when writing code alone, which is funny.)

    a new module using the existing Cookie module of python &
    urllib2. One of the problems 'we' had with Clientcookie is that
    it uses its own 'urlopen' methods which does not fit our
    applications needs, so 'we' had to find a way around it.
    As I said before, if you know how to do that, please comment on the
    RFE I referenced in my last post. Jeremy Hylton is planning to look
    at the patch associated with that RFE in detail sometime, and you
    could save him some time if you know a way to do this without patching
    urllib2. And I'd like to know how to do it, too :-)

    Once the code is ready, I will post it on my webpage, and
    of course it is not a module in itself, so I think an
    announcement to c.l.py is out of place.
    [...]

    Would you mind sending me an email?

    Thanks


    John
  • Robin Munn at Aug 21, 2003 at 6:55 pm

    John J. Lee wrote:
    pythonguy at Hotpop.com (Anand Pillai) writes:
    [...]
    'We', nothing royal about it. It is just me and my friend
    & co-developer Nirmal Chidambaram. Apparently he has found
    a way around some of the bugs in Clientcookie. He has written
    It'd be great if you made me aware what those bugs are!

    (BTW, no intent to offend with my comment about your plurality, or
    lack thereof -- it's just that the convention of using 'we' in source
    code comments is common enough that I've sometimes found myself using
    it even when writing code alone, which is funny.)
    I do that all the time; "we" is the default in my code comments and I
    have rarely, I think never, used "I". I usually think of it as myself
    reading through the code with another coder, who is tasked with taking
    over code maintenance from me. I'm explaining to him what the code is
    intended to do: "So here, we want to search through the list for..."

    --
    Robin Munn <rmunn at pobox.com> | http://www.rmunn.com/ | PGP key 0x6AFB6838
    -----------------------------+-----------------------+----------------------
    "Remember, when it comes to commercial TV, the program is not the product.
    YOU are the product, and the advertiser is the customer." - Mark W. Schumann
  • Steven Taschuk at Aug 21, 2003 at 9:43 pm
    Quoth Robin Munn:
    [use of "we" in code comments]
    [...] I usually think of it as myself
    reading through the code with another coder, who is tasked with taking
    over code maintenance from me. I'm explaining to him what the code is
    intended to do: "So here, we want to search through the list for..."
    This convention is also typical in mathematical writing. I
    interpret it much as you do.

    --
    Steven Taschuk "If you've learned anything from what I've said
    staschuk at telusplanet.net today, you haven't been paying attention."
    -- "The Power of Ignorance", Chris Gibbs
  • John J. Lee at Aug 21, 2003 at 10:18 pm

    Robin Munn <rmunn at pobox.com> writes:

    John J. Lee wrote:
    [...]
    lack thereof -- it's just that the convention of using 'we' in source
    code comments is common enough that I've sometimes found myself using
    it even when writing code alone, which is funny.)
    I do that all the time; "we" is the default in my code comments and I
    have rarely, I think never, used "I". I usually think of it as myself
    reading through the code with another coder, who is tasked with taking
    over code maintenance from me. I'm explaining to him what the code is
    intended to do: "So here, we want to search through the list for..."
    That's a good excuse. :-)


    John
  • Anand Pillai at Aug 22, 2003 at 6:50 am
    Does it mean that all coders have split personalities? ;-)

    -Anand

    jjl at pobox.com (John J. Lee) wrote in message news:<87wud6ejtw.fsf at pobox.com>...
    Robin Munn <rmunn at pobox.com> writes:
    John J. Lee wrote:
    [...]
    lack thereof -- it's just that the convention of using 'we' in source
    code comments is common enough that I've sometimes found myself using
    it even when writing code alone, which is funny.)
    I do that all the time; "we" is the default in my code comments and I
    have rarely, I think never, used "I". I usually think of it as myself
    reading through the code with another coder, who is tasked with taking
    over code maintenance from me. I'm explaining to him what the code is
    intended to do: "So here, we want to search through the list for..."
    That's a good excuse. :-)


    John
  • Mark Carter at Aug 19, 2003 at 8:41 am

    No really, I wasn't joking: you *never* need to use add_cookie_header
    / extract_cookies if you're using urllib2 (at least, I can't think of
    any possible reason to do so). It can only break things.
    I must admit that I don't really know what I am doing. How would you
    simplify the following code:

    def go8():
    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry(username='mcarter') #only need username for
    win9x

    import urllib2
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    request2 = urllib2.Request(url)
    c.add_cookie_header(request2)
    response2 = urllib2.urlopen(request2)

    print response2.geturl()
    print response2.info() # headers
    for line in response2.readlines(): # body
    print line
  • John J. Lee at Aug 20, 2003 at 9:47 pm

    cartermark46 at ukmail.com (Mark Carter) writes:

    No really, I wasn't joking: you *never* need to use add_cookie_header
    / extract_cookies if you're using urllib2 (at least, I can't think of
    any possible reason to do so). It can only break things.
    I must admit that I don't really know what I am doing. How would you
    simplify the following code:
    [...]

    It doesn't tend to encourage people to help you if you don't even
    *try* to apply their advice, and just expect them to do all your work
    for you. Read that paragraph you quoted (and the rest I posted in
    that message and previously), try doing what it suggests, then come
    back here if you need to.


    John
  • Mark Carter at Aug 25, 2003 at 3:08 pm
    jjl at pobox.com (John J. Lee) wrote in message news:<87isos0zng.fsf at pobox.com>...
    cartermark46 at ukmail.com (Mark Carter) writes:
    No really, I wasn't joking: you *never* need to use add_cookie_header
    / extract_cookies if you're using urllib2 (at least, I can't think of
    any possible reason to do so). It can only break things.
    I must admit that I don't really know what I am doing. How would you
    simplify the following code:
    [...]

    It doesn't tend to encourage people to help you if you don't even
    *try* to apply their advice, and just expect them to do all your work
    for you. Read that paragraph you quoted (and the rest I posted in
    that message and previously), try doing what it suggests, then come
    back here if you need to.
    I've looked again, and add_cookie_header() definitely(!) is required
    for the following code to work:

    def go10():
    #works with win xp
    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry(username='mcarter') #only need username for
    win9x

    import urllib2
    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    request2 = urllib2.Request(url)
    c.add_cookie_header(request2)
    response2 = urllib2.urlopen(request2)

    print response2.geturl()
    print response2.info() # headers
    for line in response2.readlines(): # body
    print line

    If I take out add_cookie_header(), then the wrong page (i.e. the
    registration page) is returned.

    I will investigate further and let you know what I find.
  • John J. Lee at Aug 26, 2003 at 12:08 pm

    cartermark46 at ukmail.com (Mark Carter) writes:

    jjl at pobox.com (John J. Lee) wrote in message news:<87isos0zng.fsf at pobox.com>...
    cartermark46 at ukmail.com (Mark Carter) writes:
    [...]
    I've looked again, and add_cookie_header() definitely(!) is required
    for the following code to work:
    Oh, sorry, I missed something: you're using urllib.urlopen. You need
    to call ClientCookie.urlopen instead (this is still using urllib2 --
    ClientCookie uses urllib2 to do most of the work).

    You seem to be opening the same URL twice for no obvious reason, so I
    removed that below:


    def go10():
    #works with win xp
    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry(username='mcarter') #only need username for win9x

    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = ClientCookie.urlopen(url)

    print response.info() # headers
    print response.read() # body


    John
  • Mark Carter at Aug 26, 2003 at 5:50 pm

    Oh, sorry, I missed something: you're using urllib.urlopen. You need
    to call ClientCookie.urlopen instead (this is still using urllib2 --
    ClientCookie uses urllib2 to do most of the work).

    You seem to be opening the same URL twice for no obvious reason, so I
    removed that below:


    def go10():
    #works with win xp
    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry(username='mcarter') #only need username for win9x

    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = ClientCookie.urlopen(url)

    print response.info() # headers
    print response.read() # body

    At the risk of sounding thick ... this doesn't work either! I couldn't
    get load_cookie_data() to work using the above template, either. What
    seems odd for starters is the line
    response = ClientCookie.urlopen(url)

    What makes it odd is that to an offhand observer, there is nothing
    visibly connecting the urlopen() command to c. Sure, some fancy stuff
    may be going on under the hood; but like I say, ClientCookie.urlopen()
    doesn't seem connected with the MSIECookieJar instance just by looking
    at it. It looks like I really want to be saying something like
    response = c.urlopen(url)
    but that just gives me
    AttributeError: MSIECookieJar instance has no attribute 'urlopen'
  • John J. Lee at Aug 26, 2003 at 8:19 pm
    cartermark46 at ukmail.com (Mark Carter) writes:
    [...]
    At the risk of sounding thick ... this doesn't work either! I couldn't
    get load_cookie_data() to work using the above template, either. What
    seems odd for starters is the line
    response = ClientCookie.urlopen(url)

    What makes it odd is that to an offhand observer, there is nothing
    visibly connecting the urlopen() command to c. Sure, some fancy stuff
    [...]

    D'Oh!

    def go10():
    #works with win xp
    import ClientCookie
    c = ClientCookie.MSIECookieJar(delayload=1)
    c.load_from_registry(username='mcarter') #only need username for win9x
    opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(c))

    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = opener.open(url)

    print response.info() # headers
    print response.read() # body


    All of this is explained in the web page, though. [attempts to looks
    at web page] Well, it would be if the link weren't broken or I could
    log in to my SF account to fix it... Sigh.

    BTW, if you want to use ClientCookie.urlopen rather than opener.open,
    you can call ClientCookie.install_opener(opener) to install a global
    opener, just as you would with urllib2.


    John
  • Mark Carter at Aug 27, 2003 at 9:00 am

    opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(c))

    url = 'http://businessplus.hemscott.net/corp/crp03733.htm'
    response = opener.open(url)
    Huzzah, we have liftoff!
    All of this is explained in the web page, though.
    You are right! All the technical information is there. But can I
    suggest a
    change in the docs that might spell it out a bit more obviously to
    someone
    coming to the module from fresh? Something like:

    h1: Examples

    h2: Simple example

    import ClientCookie
    response = ClientCookie.urlopen("http://foo.bar.com/")

    This function behaves identically to urllib2.urlopen, except that it
    deals with cookies automatically. That's probably all you need to
    know.

    h2: Using Internet Explorer or Netscape/Mozilla cookies

    c = ClientCookie.MSIECookieJar(delayload=1) # see note 1
    c.load_from_registry(username='joe bloggs') # see note 2
    opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(c))

    url = 'http://foo.bar.com/'
    response = opener.open(url)

    print response.info() # headers
    print response.read() # body

    h3: Notes
    1. Mozilla users should use MozillaCookieJar() instead of
    MSIECookieJar
    2. Usually only win98 and 95 users need supply the username. As an
    alternative
    to loading from the registry, consider using the line
    c.load_cookie_data('cookie-file.txt')
    instead.


    I think that the above is much more up-front, and is likely to satisfy
    the
    majority of users, who will probably just want basic cookie handling,
    without
    pouring over the documentation. Ideally, it should be at the top,
    where people
    will see it first, and before those more complicated examples.


    Hope this is useful!
  • John J. Lee at Aug 27, 2003 at 11:54 am
    cartermark46 at ukmail.com (Mark Carter) writes:

    [...]
    All of this is explained in the web page, though.
    You are right! All the technical information is there. But can I
    suggest a change in the docs that might spell it out a bit more
    obviously to someone coming to the module from fresh? Something
    like:
    Certainly!


    [...]
    h2: Using Internet Explorer or Netscape/Mozilla cookies

    c = ClientCookie.MSIECookieJar(delayload=1) # see note 1
    c.load_from_registry(username='joe bloggs') # see note 2
    opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(c))

    url = 'http://foo.bar.com/'
    response = opener.open(url)

    print response.info() # headers
    print response.read() # body
    Possibly, but there's a balance to be struck between convenience and
    documentation bloat. Since "Cooperating with Netscape/Mozilla and
    Internet Explorer" explains the CookieJar part, and the very next
    section is "Using your own CookieJar instance", it seems to flow quite
    well to me. The first sentence of that section is kind of misleading
    though, so I'll change that.

    [...]
    2. Usually only win98 and 95 users need supply the username. As an
    alternative
    to loading from the registry, consider using the line
    c.load_cookie_data('cookie-file.txt')
    instead.
    Thanks -- worthwhile addition.

    [...]
    should be at the top, where people will see it first, and before
    those more complicated examples.
    It is at the top -- after an example of the simplest usages and an
    explanation of what CookieJar is actually for, which I think have to
    come first.


    John
  • Mark Carter at Aug 14, 2003 at 2:09 pm

    I am using Windows 98, python 2.3, ClientCookie 0.4.3a.

    When I do:
    c.load_from_registry()

    I get the response:
    KeyError: 'USERNAME'
    I suspect this only works on "real" win32 (the NT family) operating
    I have a fix/workaround for this - and would like it included in ClientCookie.
    It my problems out real nice! Interested?

    BTW, I can't seem to locate the project on Sourceforge.
  • JanC at Aug 16, 2003 at 2:58 am

    cartermark46 at ukmail.com (Mark Carter) schreef:

    BTW, I can't seem to locate the project on Sourceforge.
    <http://wwwsearch.sourceforge.net/>

    --
    JanC

    "Be strict when sending and tolerant when receiving."
    RFC 1958 - Architectural Principles of the Internet - section 3.9

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedAug 14, '03 at 9:37a
activeAug 27, '03 at 11:54a
posts37
users8
websitepython.org

People

Translate

site design / logo © 2022 Grokbase