FAQ
I am learning Go and working through the Tour. I solved (I think) the web
crawler exercise on page 71 of the Tour <http://tour.golang.org/#71> like
this:

http://play.golang.org/p/J7d58BcdVR

I am checking a map of fetched URLs in a function closure. And the
function closure is called by goroutines. Was wondering if I need to add
some kind of locking when accessing the URL map to avoid race conditions
involving multiple goroutines accessing the map the same time.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • John Nagle at Oct 5, 2013 at 3:35 am

    On 10/4/2013 3:29 PM, YellowJacket wrote:
    I am learning Go and working through the Tour. I solved (I think) the web
    crawler exercise on page 71 of the Tour <http://tour.golang.org/#71> like
    this:

    http://play.golang.org/p/J7d58BcdVR

    I am checking a map of fetched URLs in a function closure. And the
    function closure is called by goroutines. Was wondering if I need to add
    some kind of locking when accessing the URL map to avoid race conditions
    involving multiple goroutines accessing the map the same time.
         Yes. Go maps are not thread-safe.

         Also, you're firing off a goroutine for every URL you find.
    That will rapidly generate a huge number of simultaneous
    HTTP connections. It's generally considered bad form to do that to
    external web sites, and some sites will block your IP address if you
    do that. The big search engines deliberately crawl much more slowly
    than they could.

         The other way to approach this is to have a channel which is
    a to-do list of URLs. Then you have a fixed number of goroutines
    getting work from the to-do list. They pass newly found URLs
    back along a channel to the controlling goroutine, which throws out the
    duplicates and puts new ones on the to-do list. Then you're not
    sharing data across goroutines.

          John Nagle



    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • YellowJacket at Oct 5, 2013 at 3:27 pm
    Makes sense, thank you!

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedOct 5, '13 at 12:52a
activeOct 5, '13 at 3:27p
posts3
users2
websitegolang.org

2 users in discussion

YellowJacket: 2 posts John Nagle: 1 post

People

Translate

site design / logo © 2021 Grokbase