Grokbase › Groups › Python › python-list › June 2009
FAQ
Sir,
Is there any module in python to open https sites through a
proxy.I am connectyed to the net via a proxy server and i am unable to
crawl https sites.

Search Discussions

  • Shawn Milochik at Jun 26, 2009 at 4:28 pm ⇧

    On Jun 26, 2009, at 12:08 PM, padfoot wrote:

    Sir,
    Is there any module in python to open https sites through a
    proxy.I am connectyed to the net via a proxy server and i am unable to
    crawl https sites.
    --
    http://mail.python.org/mailman/listinfo/python-list
    Check out the "Scrape the Web" series from the 2009 PyCon:

    http://advocacy.python.org/podcasts/
  • Cgoldberg at Jun 26, 2009 at 4:34 pm ⇧

    Is there any module in python to open https
    sites through a proxy.
    yes, you can use "urllib2".

    from the urllib2 docs:
    "The default is to read the list of proxies from the environment
    variables"

    So if you have a proxy setup, it should detect it from your
    environment vars. If you want to specify something different, you
    want to build an opener with a ProxyHandler:

    http://docs.python.org/library/urllib2.html#urllib2.ProxyHandler

    -Corey
  • Cameron Simpson at Jun 30, 2009 at 10:39 pm ⇧

    On 26Jun2009 09:34, cgoldberg wrote:
    Is there any module in python to open https
    sites through a proxy.
    yes, you can use "urllib2".

    from the urllib2 docs:
    "The default is to read the list of proxies from the environment
    variables"

    So if you have a proxy setup, it should detect it from your
    environment vars. If you want to specify something different, you
    want to build an opener with a ProxyHandler:

    http://docs.python.org/library/urllib2.html#urllib2.ProxyHandler
    Except that HTTPS through a proxy doesn't work:
    http://bugs.python.org/issue1424152

    There's a patch under review.
    --
    Cameron Simpson <cs at zip.com.au> DoD#743
    http://www.cskk.ezoshosting.com/cs/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 26, '09 at 4:08p
activeJun 30, '09 at 10:39p
posts4
users4
websitepython.org

4 users in discussion

Padfoot: 1 post Cameron Simpson: 1 post Shawn Milochik: 1 post Cgoldberg: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase