|| at Oct 5, 2013 at 3:35 am
On 10/4/2013 3:29 PM, YellowJacket wrote:
I am learning Go and working through the Tour. I solved (I think) the web
crawler exercise on page 71 of the Tour <http://tour.golang.org/#71
I am checking a map of fetched URLs in a function closure. And the
function closure is called by goroutines. Was wondering if I need to add
some kind of locking when accessing the URL map to avoid race conditions
involving multiple goroutines accessing the map the same time.
Yes. Go maps are not thread-safe.
Also, you're firing off a goroutine for every URL you find.
That will rapidly generate a huge number of simultaneous
HTTP connections. It's generally considered bad form to do that to
external web sites, and some sites will block your IP address if you
do that. The big search engines deliberately crawl much more slowly
than they could.
The other way to approach this is to have a channel which is
a to-do list of URLs. Then you have a fixed number of goroutines
getting work from the to-do list. They pass newly found URLs
back along a channel to the controlling goroutine, which throws out the
duplicates and puts new ones on the to-do list. Then you're not
sharing data across goroutines.
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
For more options, visit https://groups.google.com/groups/opt_out.