[go-nuts] Slow performance with multiple http.Get (get status code for top 1 million sites by Alexa.com) in Windows 7.

You are leaking response bodies, in goroutine(), this will be leaking
file descriptors. Please read the documention for http.Get

You are also kicking off a goroutine per site concurrently. Your
machine will be attempting to do a Get from each site simultaneously.

Instead of a sleep 60 at the bottom of your program, maybe you want a
wait group.

Cheers

Dave

This is just a simple example to show GO problem.
Changed goroutine code (but performance is still very slow):

func goroutine(task string) {

     res, err := http.Get(task)

     if err != nil {

         fmt.Print("-")

     } else {

         fmt.Print("+")

         res.Body.Close()

     }

}

There are only 1000 urls if file, so hope that is shouldn't be a problem
for GO on the Win Server 2003 with 100 Mbit channel.

60s is enough to see GO problen and it is also enough to complete this task
with multithreaded Delphi application.

p.s. Sorry for my English.

--

very is my post?

--

I wonder, does GO block when trying to get IP address of host in Windows?

--

Go does block when it asks windows to convert hostname to ip address. If that is inappropriate for your program, you should do the conversion yourself. If it is easy enough to do we could change net package accordingly.

Alex

--

Can somebody run this code in FreeBSD or Linux to compare speed?

As I know libuv (Node.js) support async DNS lookup in Windows (based on
C-Areas), so why GO not?

--

How would async DNS lookups help here ?

Node uses async DNS lookups to avoid blocking it's single event loop.

In your program, the dns lookup is done in the context of the
goroutine executing the http.Get.

The result is the same.

ok

C++ applitation with 1000 working threads (Boost::Threads) can do 1000 GET
reqests to different hostnames in some seconds.
It will take a lot of memory and will load processor to much, but it can do
it fast.

Go has amazing performance for HTTP servers (even faster then Gevent on
Python), but does Go has good perfrormance for HTTP clietns?
If yes, then please tell me how to get status code for 1 million hosts with
Go on Windows.

--

Can you post your previous, non go code, which is much faster? Maybe they are not doing the same thing.

Dave

Here is an example on Python. I made it just for test, because threads in
Python are slow (thanx GIL) when one of them is blocked )
My test: 100 processes working with 110 urls/second.

Your python program is doing something different to the Go program.

In python you are manually sending GET / HTTP/1.1 to the remote host
then reading at most 12 bytes from the host before slamming the
connection closed . You aren't even sending a Host: header.

In the Go program you are using the http client to do a GET, which
will wait for the remote server and the whole response body, before
returning from the method.

I would suggest that if you want to make this a fair comparison,

a. use HEAD, rather than GET, as this what you are doing
b. a (i hear there are several) python http libraries to issue your request
c. or, do not use the http package, and instead use the net pkg
directly, as you do in python.

At that point, I think some more valid comparisons can be made about
the efficiency of Go vs Python.

With 100 mbit channel it doesn't matter. I'm already did it and got about
the same speed (main problem not in response size here).

b. a (i hear there are several) python http libraries to issue your request

A lot of them (Gevent, Tornado, Twisted, Cogen, Eventlet etc.), but no one
work fast on Windows (no one has fully supported IOCP in Windows).They are
very fast under Linux and FreeBSD, but slow on Windows with this topic task.

c. or, do not use the http package, and instead use the net pkg
>
directly, as you do in python.

Exuse me, I don't understabd what you mean. I thought that http is just a
part of net package (net/http).

So I think that this comparrison doesn't matter.

My logic is:
Gevent can parse 1000 urls/second under Ubuntu. This is already tested
fact.
I found benchmark where Go HTTP server work faster then Gevent HTTP server
(whick is fastest on Python now).
I found that Go support Windows and thought that Go will show me excelent
performance.



--

Here is the Go version of the core of your Python program (not
checked, contains errors)

u, err := url.Parse(...)
c, err := net.Dial("tcp", u.Host)
n, err := c.Write([]byte("GET / HTTP/1.1\r\n\r\n")
buf := make([]byte, 12)
n, err := io.ReadAll(c, buf)

Do you see how this is the same as your Python version, and different
to your Go version ?

Dave

--

Yes, I see.
I tried new goroutine code, but got same low speed results.

func goroutine(task string) {
     u, err := url.Parse("http://tut.by/")
     c, err := net.Dial("tcp", u.Host + ":80")
     if err != nil { return }
         _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
     buf := make([]byte, 12)
     _, err = io.ReadFull(c, buf)
     if err != nil {
         return
     } else {
         fmt.Println(string(buf)[9:12])
     }
}

Did you see now that main problem not in size of response. This application
load 100 mbit channel up to 1%.

As Alex has already pointed out, the host name to ip resolving in Windows
implementation is blocking.
See func lookupIP in src/pkg/net/lookup_windows.go.

Therefore, it means no matter how many goroutines we created, there are at
most n (number of threads, equals to the CPU number in your code) host name
resolving are running. I guess that's the main reason.

In Python or C++ version, 1000 threads are created, so there are 1000 host
name resolving. That will be much faster.

It seems there's no easy way to concurrently resolve host inside one thread
in Windows,

struct hostent* FAR gethostbyname(
   _In_ const char *name
);

The hostent * is thread local. Therefore, it is safe to run gethostbyname()
in multiple threads, but not safe to call gethostbyname() multiple times in
one thread, since the later value will overwrite the previous value.

Cheng Wei

One mistake in my previous post:

Actually, no matter how many goroutines we created, there is only one host
name resolving is running due to the global lock. Therefore, the DNS
resolving is always sequential. That's even slower.

Cheng Wei

Async_resolve in latest boost::asio library is completely asynchronous on
Windows. Therefore, maybe we can change the lookupIP implementation in a
similar way?

Cheng Wei

Looks like it is already implemented, you just need to make the switch

https://code.google.com/p/go/source/browse/src/pkg/net/lookup_windows.go#43

That's great. Just wish the default is NewLookup. :)

Can anybody tell me is it possible and easy to use it now?

I find it very strange that locks are used there like this. Wouldn't it be
better to just use runtime.LockOSThread() and runtime.UnlockOSThread()?
Granted, since runtime.LockOSThread() does not "increment" the lock count,
and runtime.UnlockOSThread() just unlocks it no matter what, this makes it
unusable in this case. Otherwise, instead of locks it could have been:

func oldLookupIP(name string) (addrs []IP, err error) {
runtime.LockOSThread()
defer runtime.UnlockOSThread()
...

Wouldn't it be so much better?

Alexey,

I think you proposal will work - it should relax the contention. I will try to make this change next week when I have time.

But I am surprised that so many people still use old windows APIs. I will check that next week too.

Alex

--

Hi brainman,

I'm not actually sure whether people use it or not, new API should be
available since WinXP SP2, so should not be many. But new lookup is not in
go1, so users of go1 are of course using old code (I'm using tip, btw). If
you (or me) make LockOSThread reentrant and switch to it, will it make it
into go1? (or is it go 1.1 soon? I kinda lost track)

P.S. Even if the chance of people using old lookup is small I think it's
still best to make this change.

Whatever changes we will make - they will go into 1.1 - it should be not far away.

Alex

--

Hi brainman,

I've submitted two CLs:

https://codereview.appspot.com/7197050/
https://codereview.appspot.com/7202045/

thanx.

In Python or C++ version, 1000 threads are created, so there are 1000 host
Python version with threads is also slow because CPython use GIL.

--

Yes, you are right. It should be " In your python version, 100 processes
are created."

Cheng Wei

   I filed a bug against python wherein running the resolver from
multiple threads would result in incorrect values. Oddly, I can't find
it now, but I included a python script that would fail on every version
of python I had. Their response was that the lock should keep that from
happening. :/

I guess the slowness is caused by the dns resolving. No matter how many
goroutines we create, in Windows Go Implementation, the dns resolving is
done sequentially.
Maybe you can try another test with IP list to confirm this.

gopath\src\pkg\net\lookup_windows.go:
func lookupIP(name string) (addrs []IP, err error) {
      hostentLock.lock()
      defer hostentLock.unlock()
      ...
}

<chengw@...> writes:
  same issue with http://www.onchannel.net . any suggestions?
goroutines we create, in Windows Go Implementation, the dns resolving is
done sequentially.
 c, err := net.Dial("tcp", u.Host + ":80")    if err != nil { return }
 _, err = c.Write([]byte("GET / HTTP/1.0\r\n\r\n"))    buf := make([]byte,
12)    _, err = io.ReadFull(c, buf)    if err != nil {         return    }
else {         fmt.Println(string(buf)[9:12])    }}
me, I don't understabd what you mean. I thought that http is just a

watchkart.com

the video clip sites: http://www.nudemv.com is ok for desktop, but very slow for mobile,
waht is the best way to speed up mobile pages loading?

thanks