FAQ
Very slow perfomance with multiple http.Get (Windows 7).
Some urls is not exists and it looks like something blocking.

Please, help.


package main

import (

     "fmt"

     "io/ioutil"

     "net/http"

     "runtime"

     "strings"

     "time"

)

func goroutine(task string) {

     _, err := http.Get(task)

     if err != nil { fmt.Print("-") } else { fmt.Print("+") }

}

func main() {

     // use all available processors

     runtime.GOMAXPROCS(runtime.NumCPU())

      // read file with strings (urls)

     f_byte, err := ioutil.ReadFile("u.txt")

     if err != nil {

         fmt.Println(err)

         return

     }

     f_string := strings.Replace(string(f_byte), "\r\n", "\n", -1)

     f_strings := strings.Split(f_string, "\n")



     // run goroutines

     for i := range f_strings {

         go goroutine(f_strings[i])

     }

     time.Sleep(60 * time.Second)

}


--

Search Discussions

  • Dave Cheney at Jan 18, 2013 at 3:23 am
    You are leaking response bodies, in goroutine(), this will be leaking
    file descriptors. Please read the documention for http.Get

    You are also kicking off a goroutine per site concurrently. Your
    machine will be attempting to do a Get from each site simultaneously.

    Instead of a sleep 60 at the bottom of your program, maybe you want a
    wait group.

    Cheers

    Dave
    On Fri, Jan 18, 2013 at 12:45 PM, wrote:
    Very slow perfomance with multiple http.Get (Windows 7).
    Some urls is not exists and it looks like something blocking.

    Please, help.


    package main

    import (

    "fmt"

    "io/ioutil"

    "net/http"

    "runtime"

    "strings"

    "time"

    )

    func goroutine(task string) {

    _, err := http.Get(task)

    if err != nil { fmt.Print("-") } else { fmt.Print("+") }

    }

    func main() {

    // use all available processors

    runtime.GOMAXPROCS(runtime.NumCPU())



    // read file with strings (urls)

    f_byte, err := ioutil.ReadFile("u.txt")

    if err != nil {

    fmt.Println(err)

    return

    }

    f_string := strings.Replace(string(f_byte), "\r\n", "\n", -1)

    f_strings := strings.Split(f_string, "\n")



    // run goroutines

    for i := range f_strings {

    go goroutine(f_strings[i])

    }

    time.Sleep(60 * time.Second)

    }


    --
    --
  • Htdesign at Jan 19, 2013 at 12:07 am

    On Friday, January 18, 2013 6:23:26 AM UTC+3, Dave Cheney wrote:
    You are leaking response bodies, in goroutine(), this will be leaking
    file descriptors. Please read the documention for http.Get

    This is just a simple example to show GO problem.
    Changed goroutine code (but performance is still very slow):

    func goroutine(task string) {

         res, err := http.Get(task)

         if err != nil {

             fmt.Print("-")

         } else {

             fmt.Print("+")

             res.Body.Close()

         }

    }
    On Friday, January 18, 2013 6:23:26 AM UTC+3, Dave Cheney wrote:

    You are also kicking off a goroutine per site concurrently. Your
    machine will be attempting to do a Get from each site simultaneously.

    There are only 1000 urls if file, so hope that is shouldn't be a problem
    for GO on the Win Server 2003 with 100 Mbit channel.
    On Friday, January 18, 2013 6:23:26 AM UTC+3, Dave Cheney wrote:

    Instead of a sleep 60 at the bottom of your program, maybe you want a
    wait group.

    60s is enough to see GO problen and it is also enough to complete this task
    with multithreaded Delphi application.

    p.s. Sorry for my English.

    --
  • Htdesign at Jan 19, 2013 at 12:07 am
    very is my post?

    --
  • Htdesign at Jan 19, 2013 at 7:58 am
    I wonder, does GO block when trying to get IP address of host in Windows?

    --
  • Brainman at Jan 19, 2013 at 10:27 pm
    Go does block when it asks windows to convert hostname to ip address. If that is inappropriate for your program, you should do the conversion yourself. If it is easy enough to do we could change net package accordingly.

    Alex

    --
  • Htdesign at Jan 20, 2013 at 10:29 am
    Can somebody run this code in FreeBSD or Linux to compare speed?

    As I know libuv (Node.js) support async DNS lookup in Windows (based on
    C-Areas), so why GO not?

    --
  • Dave Cheney at Jan 20, 2013 at 10:31 am
    How would async DNS lookups help here ?

    Node uses async DNS lookups to avoid blocking it's single event loop.

    In your program, the dns lookup is done in the context of the
    goroutine executing the http.Get.

    The result is the same.
    On Sun, Jan 20, 2013 at 9:29 PM, wrote:

    Can somebody run this code in FreeBSD or Linux to compare speed?

    As I know libuv (Node.js) support async DNS lookup in Windows (based on
    C-Areas), so why GO not?

    --
    --
  • Htdesign at Jan 20, 2013 at 4:50 pm
    ok

    C++ applitation with 1000 working threads (Boost::Threads) can do 1000 GET
    reqests to different hostnames in some seconds.
    It will take a lot of memory and will load processor to much, but it can do
    it fast.

    Go has amazing performance for HTTP servers (even faster then Gevent on
    Python), but does Go has good perfrormance for HTTP clietns?
    If yes, then please tell me how to get status code for 1 million hosts with
    Go on Windows.

    --
  • Dave Cheney at Jan 20, 2013 at 8:04 pm
    Can you post your previous, non go code, which is much faster? Maybe they are not doing the same thing.

    Dave
    On 21/01/2013, at 3:50, htdesign@tut.by wrote:

    ok

    C++ applitation with 1000 working threads (Boost::Threads) can do 1000 GET reqests to different hostnames in some seconds.
    It will take a lot of memory and will load processor to much, but it can do it fast.

    Go has amazing performance for HTTP servers (even faster then Gevent on Python), but does Go has good perfrormance for HTTP clietns?
    If yes, then please tell me how to get status code for 1 million hosts with Go on Windows.
    --
    --
  • Htdesign at Jan 20, 2013 at 10:00 pm
    Here is an example on Python. I made it just for test, because threads in
    Python are slow (thanx GIL) when one of them is blocked )
    My test: 100 processes working with 110 urls/second.
    On Sunday, January 20, 2013 11:04:41 PM UTC+3, Dave Cheney wrote:

    Can you post your previous, non go code, which is much faster? Maybe they
    are not doing the same thing.

    Dave

    On 21/01/2013, at 3:50, htde...@tut.by <javascript:> wrote:

    ok

    C++ applitation with 1000 working threads (Boost::Threads) can do 1000 GET
    reqests to different hostnames in some seconds.
    It will take a lot of memory and will load processor to much, but it can
    do it fast.

    Go has amazing performance for HTTP servers (even faster then Gevent on
    Python), but does Go has good perfrormance for HTTP clietns?
    If yes, then please tell me how to get status code for 1 million hosts
    with Go on Windows.
    --


    --
  • Dave Cheney at Jan 20, 2013 at 10:07 pm
    Your python program is doing something different to the Go program.

    In python you are manually sending GET / HTTP/1.1 to the remote host
    then reading at most 12 bytes from the host before slamming the
    connection closed . You aren't even sending a Host: header.

    In the Go program you are using the http client to do a GET, which
    will wait for the remote server and the whole response body, before
    returning from the method.

    I would suggest that if you want to make this a fair comparison,

    a. use HEAD, rather than GET, as this what you are doing
    b. a (i hear there are several) python http libraries to issue your request
    c. or, do not use the http package, and instead use the net pkg
    directly, as you do in python.

    At that point, I think some more valid comparisons can be made about
    the efficiency of Go vs Python.
    On Mon, Jan 21, 2013 at 9:00 AM, wrote:
    Here is an example on Python. I made it just for test, because threads in
    Python are slow (thanx GIL) when one of them is blocked )
    My test: 100 processes working with 110 urls/second.
    On Sunday, January 20, 2013 11:04:41 PM UTC+3, Dave Cheney wrote:

    Can you post your previous, non go code, which is much faster? Maybe they
    are not doing the same thing.

    Dave


    On 21/01/2013, at 3:50, htde...@tut.by wrote:

    ok

    C++ applitation with 1000 working threads (Boost::Threads) can do 1000 GET
    reqests to different hostnames in some seconds.
    It will take a lot of memory and will load processor to much, but it can
    do it fast.

    Go has amazing performance for HTTP servers (even faster then Gevent on
    Python), but does Go has good perfrormance for HTTP clietns?
    If yes, then please tell me how to get status code for 1 million hosts
    with Go on Windows.
    --
    --
    --
  • Htdesign at Jan 20, 2013 at 11:06 pm
    a. use HEAD, rather than GET, as this what you are doing
    With 100 mbit channel it doesn't matter. I'm already did it and got about
    the same speed (main problem not in response size here).

    b. a (i hear there are several) python http libraries to issue your request

    A lot of them (Gevent, Tornado, Twisted, Cogen, Eventlet etc.), but no one
    work fast on Windows (no one has fully supported IOCP in Windows).They are
    very fast under Linux and FreeBSD, but slow on Windows with this topic task.

    c. or, do not use the http package, and instead use the net pkg
    >
    directly, as you do in python.

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.

    My logic is:
    Gevent can parse 1000 urls/second under Ubuntu. This is already tested
    fact.
    I found benchmark where Go HTTP server work faster then Gevent HTTP server
    (whick is fastest on Python now).
    I found that Go support Windows and thought that Go will show me excelent
    performance.



    --
  • Dave Cheney at Jan 20, 2013 at 11:33 pm

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave

    --
  • Htdesign at Jan 21, 2013 at 12:34 am
    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
         u, err := url.Parse("http://tut.by/")
         c, err := net.Dial("tcp", u.Host + ":80")
         if err != nil { return }
             _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
         buf := make([]byte, 12)
         _, err = io.ReadFull(c, buf)
         if err != nil {
             return
         } else {
             fmt.Println(string(buf)[9:12])
         }
    }

    Did you see now that main problem not in size of response. This application
    load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
  • Wei Cheng at Jan 21, 2013 at 10:08 am
    As Alex has already pointed out, the host name to ip resolving in Windows
    implementation is blocking.
    See func lookupIP in src/pkg/net/lookup_windows.go.

    Therefore, it means no matter how many goroutines we created, there are at
    most n (number of threads, equals to the CPU number in your code) host name
    resolving are running. I guess that's the main reason.

    In Python or C++ version, 1000 threads are created, so there are 1000 host
    name resolving. That will be much faster.

    It seems there's no easy way to concurrently resolve host inside one thread
    in Windows,

    struct hostent* FAR gethostbyname(
       _In_ const char *name
    );

    The hostent * is thread local. Therefore, it is safe to run gethostbyname()
    in multiple threads, but not safe to call gethostbyname() multiple times in
    one thread, since the later value will overwrite the previous value.

    Cheng Wei
    On Monday, 21 January 2013 08:34:22 UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
  • Wei Cheng at Jan 21, 2013 at 10:16 am
    One mistake in my previous post:

    Actually, no matter how many goroutines we created, there is only one host
    name resolving is running due to the global lock. Therefore, the DNS
    resolving is always sequential. That's even slower.

    Cheng Wei
    On Monday, 21 January 2013 18:08:22 UTC+8, Wei Cheng wrote:

    As Alex has already pointed out, the host name to ip resolving in Windows
    implementation is blocking.
    See func lookupIP in src/pkg/net/lookup_windows.go.

    Therefore, it means no matter how many goroutines we created, there are at
    most n (number of threads, equals to the CPU number in your code) host name
    resolving are running. I guess that's the main reason.

    In Python or C++ version, 1000 threads are created, so there are 1000 host
    name resolving. That will be much faster.

    It seems there's no easy way to concurrently resolve host inside one
    thread in Windows,

    struct hostent* FAR gethostbyname(
    _In_ const char *name
    );

    The hostent * is thread local. Therefore, it is safe to run
    gethostbyname() in multiple threads, but not safe to call gethostbyname()
    multiple times in one thread, since the later value will overwrite the
    previous value.

    Cheng Wei
    On Monday, 21 January 2013 08:34:22 UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
  • Wei Cheng at Jan 21, 2013 at 10:21 am
    Async_resolve in latest boost::asio library is completely asynchronous on
    Windows. Therefore, maybe we can change the lookupIP implementation in a
    similar way?

    Cheng Wei
    On Monday, 21 January 2013 18:08:22 UTC+8, Wei Cheng wrote:

    As Alex has already pointed out, the host name to ip resolving in Windows
    implementation is blocking.
    See func lookupIP in src/pkg/net/lookup_windows.go.

    Therefore, it means no matter how many goroutines we created, there are at
    most n (number of threads, equals to the CPU number in your code) host name
    resolving are running. I guess that's the main reason.

    In Python or C++ version, 1000 threads are created, so there are 1000 host
    name resolving. That will be much faster.

    It seems there's no easy way to concurrently resolve host inside one
    thread in Windows,

    struct hostent* FAR gethostbyname(
    _In_ const char *name
    );

    The hostent * is thread local. Therefore, it is safe to run
    gethostbyname() in multiple threads, but not safe to call gethostbyname()
    multiple times in one thread, since the later value will overwrite the
    previous value.

    Cheng Wei
    On Monday, 21 January 2013 08:34:22 UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
  • Dave Cheney at Jan 21, 2013 at 10:23 am
    Looks like it is already implemented, you just need to make the switch

    https://code.google.com/p/go/source/browse/src/pkg/net/lookup_windows.go#43
    On Mon, Jan 21, 2013 at 9:21 PM, Wei Cheng wrote:
    Async_resolve in latest boost::asio library is completely asynchronous on
    Windows. Therefore, maybe we can change the lookupIP implementation in a
    similar way?

    Cheng Wei
    On Monday, 21 January 2013 18:08:22 UTC+8, Wei Cheng wrote:

    As Alex has already pointed out, the host name to ip resolving in Windows
    implementation is blocking.
    See func lookupIP in src/pkg/net/lookup_windows.go.

    Therefore, it means no matter how many goroutines we created, there are at
    most n (number of threads, equals to the CPU number in your code) host name
    resolving are running. I guess that's the main reason.

    In Python or C++ version, 1000 threads are created, so there are 1000 host
    name resolving. That will be much faster.

    It seems there's no easy way to concurrently resolve host inside one
    thread in Windows,

    struct hostent* FAR gethostbyname(
    _In_ const char *name
    );

    The hostent * is thread local. Therefore, it is safe to run
    gethostbyname() in multiple threads, but not safe to call gethostbyname()
    multiple times in one thread, since the later value will overwrite the
    previous value.

    Cheng Wei
    On Monday, 21 January 2013 08:34:22 UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is
    just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
    --
  • Wei Cheng at Jan 21, 2013 at 10:49 am
    That's great. Just wish the default is NewLookup. :)
    On Monday, 21 January 2013 18:23:03 UTC+8, Dave Cheney wrote:

    Looks like it is already implemented, you just need to make the switch

    https://code.google.com/p/go/source/browse/src/pkg/net/lookup_windows.go#43
    On Mon, Jan 21, 2013 at 9:21 PM, Wei Cheng wrote:
    Async_resolve in latest boost::asio library is completely asynchronous on
    Windows. Therefore, maybe we can change the lookupIP implementation in a
    similar way?

    Cheng Wei
    On Monday, 21 January 2013 18:08:22 UTC+8, Wei Cheng wrote:

    As Alex has already pointed out, the host name to ip resolving in
    Windows
    implementation is blocking.
    See func lookupIP in src/pkg/net/lookup_windows.go.

    Therefore, it means no matter how many goroutines we created, there are
    at
    most n (number of threads, equals to the CPU number in your code) host
    name
    resolving are running. I guess that's the main reason.

    In Python or C++ version, 1000 threads are created, so there are 1000
    host
    name resolving. That will be much faster.

    It seems there's no easy way to concurrently resolve host inside one
    thread in Windows,

    struct hostent* FAR gethostbyname(
    _In_ const char *name
    );

    The hostent * is thread local. Therefore, it is safe to run
    gethostbyname() in multiple threads, but not safe to call
    gethostbyname()
    multiple times in one thread, since the later value will overwrite the
    previous value.

    Cheng Wei
    On Monday, 21 January 2013 08:34:22 UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is
    just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
    --
  • Htdesign at Jan 21, 2013 at 11:32 am
    Can anybody tell me is it possible and easy to use it now?
    On Monday, January 21, 2013 1:49:03 PM UTC+3, Wei Cheng wrote:

    That's great. Just wish the default is NewLookup. :)

    --
  • Alexey Borzenkov at Jan 21, 2013 at 6:49 pm
    I find it very strange that locks are used there like this. Wouldn't it be
    better to just use runtime.LockOSThread() and runtime.UnlockOSThread()?
    Granted, since runtime.LockOSThread() does not "increment" the lock count,
    and runtime.UnlockOSThread() just unlocks it no matter what, this makes it
    unusable in this case. Otherwise, instead of locks it could have been:

    func oldLookupIP(name string) (addrs []IP, err error) {
    runtime.LockOSThread()
    defer runtime.UnlockOSThread()
    ...

    Wouldn't it be so much better?


    On Mon, Jan 21, 2013 at 2:23 PM, Dave Cheney wrote:

    Looks like it is already implemented, you just need to make the switch

    https://code.google.com/p/go/source/browse/src/pkg/net/lookup_windows.go#43
    On Mon, Jan 21, 2013 at 9:21 PM, Wei Cheng wrote:
    Async_resolve in latest boost::asio library is completely asynchronous on
    Windows. Therefore, maybe we can change the lookupIP implementation in a
    similar way?

    Cheng Wei
    On Monday, 21 January 2013 18:08:22 UTC+8, Wei Cheng wrote:

    As Alex has already pointed out, the host name to ip resolving in
    Windows
    implementation is blocking.
    See func lookupIP in src/pkg/net/lookup_windows.go.

    Therefore, it means no matter how many goroutines we created, there are
    at
    most n (number of threads, equals to the CPU number in your code) host
    name
    resolving are running. I guess that's the main reason.

    In Python or C++ version, 1000 threads are created, so there are 1000
    host
    name resolving. That will be much faster.

    It seems there's no easy way to concurrently resolve host inside one
    thread in Windows,

    struct hostent* FAR gethostbyname(
    _In_ const char *name
    );

    The hostent * is thread local. Therefore, it is safe to run
    gethostbyname() in multiple threads, but not safe to call
    gethostbyname()
    multiple times in one thread, since the later value will overwrite the
    previous value.

    Cheng Wei
    On Monday, 21 January 2013 08:34:22 UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is
    just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
    --

    --
  • Brainman at Jan 22, 2013 at 10:46 pm
    Alexey,

    I think you proposal will work - it should relax the contention. I will try to make this change next week when I have time.

    But I am surprised that so many people still use old windows APIs. I will check that next week too.

    Alex

    --
  • Alexey Borzenkov at Jan 23, 2013 at 5:54 am
    Hi brainman,

    I'm not actually sure whether people use it or not, new API should be
    available since WinXP SP2, so should not be many. But new lookup is not in
    go1, so users of go1 are of course using old code (I'm using tip, btw). If
    you (or me) make LockOSThread reentrant and switch to it, will it make it
    into go1? (or is it go 1.1 soon? I kinda lost track)

    P.S. Even if the chance of people using old lookup is small I think it's
    still best to make this change.

    On Wed, Jan 23, 2013 at 2:46 AM, brainman wrote:

    Alexey,

    I think you proposal will work - it should relax the contention. I will
    try to make this change next week when I have time.

    But I am surprised that so many people still use old windows APIs. I will
    check that next week too.

    Alex

    --

    --
  • Brainman at Jan 23, 2013 at 6:59 am
    Whatever changes we will make - they will go into 1.1 - it should be not far away.

    Alex

    --
  • Alexey Borzenkov at Jan 23, 2013 at 8:00 pm
    Hi brainman,

    I've submitted two CLs:

    https://codereview.appspot.com/7197050/
    https://codereview.appspot.com/7202045/

    On Wed, Jan 23, 2013 at 10:59 AM, brainman wrote:

    Whatever changes we will make - they will go into 1.1 - it should be not
    far away.

    Alex

    --

    --
  • Htdesign at Jan 21, 2013 at 10:45 am
    thanx.

    In Python or C++ version, 1000 threads are created, so there are 1000 host
    name resolving. That will be much faster.
    Python version with threads is also slow because CPython use GIL.

    --
  • Wei Cheng at Jan 21, 2013 at 10:48 am
    Yes, you are right. It should be " In your python version, 100 processes
    are created."

    Cheng Wei
    On Monday, 21 January 2013 18:35:08 UTC+8, htde...@tut.by wrote:

    thanx.

    In Python or C++ version, 1000 threads are created, so there are 1000 host
    name resolving. That will be much faster.
    Python version with threads is also slow because CPython use GIL.
    --
  • Dustin Sallings at Jan 21, 2013 at 5:31 pm

    htdesign@tut.by writes:

    Python version with threads is also slow because CPython use GIL.
       I filed a bug against python wherein running the resolver from
    multiple threads would result in incorrect values. Oddly, I can't find
    it now, but I included a python script that would fail on every version
    of python I had. Their response was that the lock should keep that from
    happening. :/

    --
    dustin

    --
  • Chengw at Jan 21, 2013 at 5:16 pm
    I guess the slowness is caused by the dns resolving. No matter how many
    goroutines we create, in Windows Go Implementation, the dns resolving is
    done sequentially.
    Maybe you can try another test with IP list to confirm this.

    gopath\src\pkg\net\lookup_windows.go:
    func lookupIP(name string) (addrs []IP, err error) {
          hostentLock.lock()
          defer hostentLock.unlock()
          ...
    }

    On Monday, January 21, 2013 8:34:22 AM UTC+8, htde...@tut.by wrote:

    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {
    u, err := url.Parse("http://tut.by/")
    c, err := net.Dial("tcp", u.Host + ":80")
    if err != nil { return }
    _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
    buf := make([]byte, 12)
    _, err = io.ReadFull(c, buf)
    if err != nil {
    return
    } else {
    fmt.Println(string(buf)[9:12])
    }
    }

    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:

    Exuse me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)

    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err := io.ReadAll(c, buf)

    Do you see how this is the same as your Python version, and different
    to your Go version ?

    Dave
    --
  • Onchannel at Feb 28, 2014 at 10:10 am
    <chengw@...> writes:
      same issue with http://www.onchannel.net . any suggestions?

    I guess the slowness is caused by the dns resolving. No matter how many
    goroutines we create, in Windows Go Implementation, the dns resolving is
    done sequentially.
    Maybe you can try another test with IP list to confirm this.


    gopath\src\pkg\net\lookup_windows.go:
    func lookupIP(name string) (addrs []IP, err error)  {
    hostentLock.lock()
    defer hostentLock.unlock()
    ...
    }


    On Monday, January 21, 2013 8:34:22 AM UTC+8,
    htde...@tut.by wrote:
    Yes, I see.
    I tried new goroutine code, but got same low speed results.

    func goroutine(task string) {    u, err := url.Parse("http://tut.by/")
     c, err := net.Dial("tcp", u.Host + ":80")    if err != nil { return }
     _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))    buf := make([]byte,
    12)    _, err = io.ReadFull(c, buf)    if err != nil {         return    }
    else {         fmt.Println(string(buf)[9:12])    }}
    Did you see now that main problem not in size of response. This
    application load 100 mbit channel up to 1%.
    On Monday, January 21, 2013 2:33:42 AM UTC+3, Dave Cheney wrote:> Exuse
    me, I don't understabd what you mean. I thought that http is just a
    part of net package (net/http).

    So I think that this comparrison doesn't matter.
    Here is the Go version of the core of your Python program (not
    checked, contains errors)
    u, err := url.Parse(...)
    c, err := net.Dial("tcp", u.Host)
    n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
    buf := make([]byte, 12)
    n, err :=  io.ReadAll(c, buf)
    Do you see how this is the same as your Python version, and different
    to your Go version ?
    Dave









    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Watchkart at Jun 30, 2014 at 12:10 pm

    Dave Cheney writes:



    Can you post your previous, non go code, which is much faster? Maybe they
    are not doing the same thing.
    DaveOn 21/01/2013, at 3:50, htdesign@tut.by wrote:



    ok

    C++ applitation with 1000 working threads (Boost::Threads) can do 1000 GET
    reqests to different hostnames in some seconds.
    It will take a lot of memory and will load processor to much, but it can do it fast.
    Go has amazing performance for HTTP servers (even faster then Gevent on
    Python), but does Go has good perfrormance for HTTP clietns?
    If yes, then please tell me how to get status code for 1 million hosts with
    Go on Windows.
    watchkart.com


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nudemv at Mar 11, 2015 at 5:50 pm
    the video clip sites: http://www.nudemv.com is ok for desktop, but very slow for mobile,
    waht is the best way to speed up mobile pages loading?

    thanks




    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 18, '13 at 3:16a
activeMar 11, '15 at 5:50p
posts33
users10
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase