FAQ
hi ,

I trying to use urllib2 in my script , but the problem is lets say a domains resolves to multiple IPs , If the URL is served by plain http , I can add ?Host: domain? header and check whether all IPs are returning proper responses or not , but in case of https , I have to trust on my local machines dns resolver and I can?t apply host header in the request .

Is it possible to override dns resolver in urllib2 or something else ?
req = urllib2.Request(url='https://google.com/')
f = urllib2.urlopen(req)
print f.read()
I wanted to control the IPs which urllib2 is trying to connect .


I Hope I am some sense out of my question ,

Thanks in advance.

~saurabh

Search Discussions

  • Chris Angelico at Jun 15, 2011 at 12:43 am

    On Wed, Jun 15, 2011 at 4:34 AM, saurabh verma wrote:
    hi ,

    I trying to use urllib2 in my script , but the problem is lets say a domains resolves to multiple IPs , If the URL is served by plain http , I can add ?Host: domain? header and check whether all IPs are returning proper responses or not , but in case of https , I have to trust on my local machines dns resolver and I can?t apply host header in the request .
    Regarding Host: headers, experimentation showed that urllib2 did
    indeed send one (I tested using Python 2.7.1 on Windows, talking to a
    snooping HTTPS server running on a Linux box beside me - source code
    available if you're curious, but it's not Python).
    import urllib2
    req=urllib2.Request(url='https://mirlemont/')
    f=urllib2.urlopen(req)
    f.read()
    'Hello, world!'

    Meanwhile the snoop server reports:

    conn = Protocols.HTTP.Server.Request("GET" "/")
    GET / HTTP/1.1
    Accept-Encoding: identity
    Host: mirlemont
    Connection: close
    User-Agent: Python-urllib/2.7

    (Yes, my computer's name is Mirlemont. What's yours'? :) )

    You could control the selection of IP address using a hosts file. In
    Unix, that's /etc/hosts; in Windows,
    c:\windows\system32\drivers\etc\hosts; in OS/2, c:\mptn\etc\hosts;
    etc. The urllib2 resolver should respect that.

    Chris Angelico
  • Saurabh verma at Jun 15, 2011 at 6:23 am

    On 15-Jun-2011, at 6:13 AM, Chris Angelico wrote:
    On Wed, Jun 15, 2011 at 4:34 AM, saurabh verma wrote:
    hi ,

    I trying to use urllib2 in my script , but the problem is lets say a domains resolves to multiple IPs , If the URL is served by plain http , I can add ?Host: domain? header and check whether all IPs are returning proper responses or not , but in case of https , I have to trust on my local machines dns resolver and I can?t apply host header in the request .
    Regarding Host: headers, experimentation showed that urllib2 did
    indeed send one (I tested using Python 2.7.1 on Windows, talking to a
    snooping HTTPS server running on a Linux box beside me - source code
    available if you're curious, but it's not Python).
    Ok thats informative , thanks :) . But my problem lies in testing https based virtual served by nginx when the domain is load balanced by DNS .

    Example ,

    I have ?http://something.com? and ?https://something.com? both served by two servers and something.com resolves to two ips IPA,IPB . lets say i want to test both servers on http i can do the following

    curl ?http://IPA/? -H ?Host: something.com"
    curl ?http://IPB/? -H ?Host: something.com?


    But in the case of https , I can do above because https handshake is based on the domain i am trying to connect , so lets say I want to following inside a python script using libcurl2 but without touching /etc/hosts ,

    curl ?https://something.com? , now something.com will try to connect to either IPA or IPB which I don?t have control over , I know internally it must be calling a DNS resolver libarary of python , I want to control over that , may be libcurl2 exposing a function to do some DNS altering .

    Is it worth hacking libcurl2 code for something like , Also i?m just not more than 1 week exprienced in python , but i do enjoy your mailing list , awesome participation .


    Thanks ,

    Saurabh Verma
  • Chris Angelico at Jun 15, 2011 at 6:45 am

    On Wed, Jun 15, 2011 at 4:23 PM, saurabh verma wrote:
    But in the case of https , I can do above because https handshake is based on the domain i am trying to connect , so lets say I want to following inside a python ?script using libcurl2 but without touching /etc/hosts ,

    curl ?https://something.com? , now something.com will try to connect to either IPA or IPB which I don?t have control over , I know internally it must be calling a DNS resolver libarary of python , I want to control over that , may be libcurl2 exposing a function to do some DNS altering .
    If you edit your hosts file, it will affect where something.com points
    - you can force it to be IPA and then test, then force it to IPB and
    test. You'll still be downloading https://something.com so the HTTPS
    handshake should work exactly the same way.

    Chris Angelico
  • Saurabh verma at Jun 15, 2011 at 7:50 am



    If you edit your hosts file, it will affect where something.com points
    - you can force it to be IPA and then test, then force it to IPB and
    test. You'll still be downloading https://something.com so the HTTPS
    handshake should work exactly the same way.
    there are issues with editing /etc/hosts , what if i don't have sudo access
    on the target server ? I need a way out programmatically to fix this problem
    .
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-list/attachments/20110615/de482117/attachment.html>
  • Chris Angelico at Jun 15, 2011 at 8:28 am

    On Wed, Jun 15, 2011 at 5:50 PM, saurabh verma wrote:
    If you edit your hosts file, it will affect where something.com points
    - you can force it to be IPA and then test, then force it to IPB and
    test. You'll still be downloading https://something.com so the HTTPS
    handshake should work exactly the same way.
    there are issues with editing /etc/hosts , what if i don't have sudo access
    on the target server ? I need a way out programmatically to fix this problem
    You don't need to edit it on the server; just use any handy computer.
    You need only tinker with the configuration on the client, not the
    server.

    Chris Angelico
  • Saurabh verma at Jun 15, 2011 at 9:26 am

    You don't need to edit it on the server; just use any handy computer.
    You need only tinker with the configuration on the client, not the
    server.
    Hmm true , Ok i can widen the problem statement but editing /etc/hosts still looks Ok for my test server . ( Thinking of putting this as a nagios plugin on the servers .. that will not be a viable right now )

    Thanks chris.

    ~saurabh
  • Michael Hrivnak at Jun 18, 2011 at 6:03 pm
    The latest libcurl includes the CURLOPTS_RESOLVE option
    (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html) that will do
    what you want. It may not have made its way into pycurl yet, but you
    could just call the command-line curl binary with the --resolve
    option. This feature was introduced in version 7.21.3.

    Michael
    On Tue, Jun 14, 2011 at 2:34 PM, saurabh verma wrote:
    hi ,

    I trying to use urllib2 in my script , but the problem is lets say a domains resolves to multiple IPs , If the URL is served by plain http , I can add ?Host: domain? header and check whether all IPs are returning proper responses or not , but in case of https , I have to trust on my local machines dns resolver and I can?t apply host header in the request .

    Is it possible to override dns resolver in urllib2 or something else ?
    req = urllib2.Request(url='https://google.com/')
    f = urllib2.urlopen(req)
    print f.read()
    I wanted to control the IPs which urllib2 is trying to connect .


    I Hope I am some sense out of my question ,

    Thanks in advance.

    ~saurabh
    --
    http://mail.python.org/mailman/listinfo/python-list
  • Saurabh verma at Jun 24, 2011 at 10:17 am

    Michael Hrivnak wrote:
    The latest libcurl includes the CURLOPTS_RESOLVE option
    (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html) that will do
    what you want. It may not have made its way into pycurl yet, but you
    could just call the command-line curl binary with the --resolve
    option. This feature was introduced in version 7.21.3.

    Michael
    Hey Michael , Thanks for the response . Thats exactly what i was looking
    for.

    "curl binary with the --resolve" ?

    --(0)> curl --resolve
    curl: option --resolve: is unknown
    curl: try 'curl --help' or 'curl --manual' for more information
    --(saurabhve at sa-mac-saurabh)-(~)--
    --(2)> curl -V
    curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l
    zlib/1.2.3


    Couldn't find it .

    ~saurabh
  • Michael Hrivnak at Jun 25, 2011 at 8:27 pm
    As I noted, the feature was introduced in version 7.21.3. From your
    post, it looks like you're running version 7.19.7. Perhaps you should
    look into getting a newer version of curl.

    Or if you're feeling brave and have some C skills, you could try
    implementing this feature in pycurl! That would be a help to you, and
    a way to give back to the open source community.

    Best of luck,
    Michael
    On Fri, Jun 24, 2011 at 6:17 AM, saurabh verma wrote:
    Michael Hrivnak wrote:
    The latest libcurl includes the CURLOPTS_RESOLVE option
    (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html) that will do
    what you want. ?It may not have made its way into pycurl yet, but you
    could just call the command-line curl binary with the --resolve
    option. ?This feature was introduced in version 7.21.3.

    Michael
    Hey Michael , Thanks for the response . Thats exactly what i was looking
    for.

    "curl binary with the --resolve" ?

    --(0)> curl --resolve
    curl: option --resolve: is unknown
    curl: try 'curl --help' or 'curl --manual' for more information
    --(saurabhve at sa-mac-saurabh)-(~)--
    --(2)> curl -V
    curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l
    zlib/1.2.3


    Couldn't find it .

    ~saurabh

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 14, '11 at 6:34p
activeJun 25, '11 at 8:27p
posts10
users3
websitepython.org

People

Translate

site design / logo © 2022 Grokbase