FAQ
Hello,

I'm trying to mimic the time.NewTicker behavior which returns a channel when
called. I want to use this in the DNS package, so that a client can send
multiple queries and then async. wait for a result. However this leads me to the
case where I have to wait for multiple channels. See this example:
http://play.golang.org/p/e7vQISWw-y
(doesn't work on play, does run locally)

However I don't like this code, so I could do two things 1) use a global
channel, which all goroutines use, so that there is only one channel. 2) use
something more elegant than the current code...

Regards,

--
Miek Gieben http://miek.nl

Search Discussions

  • Rémy Oudompheng at Nov 18, 2012 at 11:17 pm

    On 2012/11/18 Miek Gieben wrote:
    Hello,

    I'm trying to mimic the time.NewTicker behavior which returns a channel when
    called. I want to use this in the DNS package, so that a client can send
    multiple queries and then async. wait for a result. However this leads me to the
    case where I have to wait for multiple channels.
    Can you detail the pattern that leads you to wait on an indefinite
    number of channels? I'm not sure why that happens.

    Rémy.

    --
  • Miek Gieben at Nov 19, 2012 at 11:20 am
    [ Quoting in "Re: [go-nuts] slice of channels and..." ]
    I'm trying to mimic the time.NewTicker behavior which returns a channel when
    called. I want to use this in the DNS package, so that a client can send
    multiple queries and then async. wait for a result. However this leads me to the
    case where I have to wait for multiple channels.
    Can you detail the pattern that leads you to wait on an indefinite
    number of channels? I'm not sure why that happens.
    Well, I have a util that acts like 'dig':

    ./q mx @<nameserver> qname1 qname2 ... qnameN

    Each qnameX will result in a query, each query will return a channel. At the
    end I loop through all channels showing the replies. That would currently
    lead me to use N channels.

    I think just returning a global channels is the best way forward.

    Regards,

    --
    Miek Gieben http://miek.nl
  • Mjarco at Nov 19, 2012 at 11:39 am
    Use sync.WaitGroup:
    http://play.golang.org/p/1srl9yavab

    In above code you don't even need this slice of chans... You may collect
    results in goroutine spawned inside Add method and return them as a result
    of Wait
    On Monday, November 19, 2012 12:20:39 PM UTC+1, Miek Gieben wrote:

    [ Quoting <remyoud...@gmail.com <javascript:>> in "Re: [go-nuts] slice of
    channels and..." ]
    I'm trying to mimic the time.NewTicker behavior which returns a
    channel when
    called. I want to use this in the DNS package, so that a client can
    send
    multiple queries and then async. wait for a result. However this leads
    me to the
    case where I have to wait for multiple channels.
    Can you detail the pattern that leads you to wait on an indefinite
    number of channels? I'm not sure why that happens.
    Well, I have a util that acts like 'dig':

    ./q mx @<nameserver> qname1 qname2 ... qnameN

    Each qnameX will result in a query, each query will return a channel. At
    the
    end I loop through all channels showing the replies. That would currently
    lead me to use N channels.

    I think just returning a global channels is the best way forward.

    Regards,

    --
    Miek Gieben
    http://miek.nl
    --
  • Jan Mercl at Nov 19, 2012 at 11:49 am

    On Mon, Nov 19, 2012 at 12:20 PM, Miek Gieben wrote:
    [ Quoting <remyoudompheng@gmail.com> in "Re: [go-nuts] slice of channels and..." ]
    I'm trying to mimic the time.NewTicker behavior which returns a channel when
    called. I want to use this in the DNS package, so that a client can send
    multiple queries and then async. wait for a result. However this leads me to the
    case where I have to wait for multiple channels.
    Can you detail the pattern that leads you to wait on an indefinite
    number of channels? I'm not sure why that happens.
    Well, I have a util that acts like 'dig':

    ./q mx @<nameserver> qname1 qname2 ... qnameN

    Each qnameX will result in a query, each query will return a channel. At the
    end I loop through all channels showing the replies. That would currently
    lead me to use N channels.

    I think just returning a global channels is the best way forward.
    I use a different approach[1] which allows to use 1..N channels for N
    concurrent queries (i.e. also ony 1 channel for all of them):

    ch := msg1.GoExchange(conn1, limit, nil)
    msg2.GoExchange(conn2, limit, ch)
    ...
    msgN.GoExchange(conn3, limit, ch)

    // later or elsewhere
    reply <- ch // first remote server to answer wins

    -j

    [1]: http://go.pkgdoc.org/github.com/cznic/dns/msg#Message.GoExchange

    --
  • John Beisley at Nov 19, 2012 at 12:44 pm

    On 19 November 2012 11:49, Jan Mercl wrote:
    On Mon, Nov 19, 2012 at 12:20 PM, Miek Gieben wrote:
    [ Quoting <remyoudompheng@gmail.com> in "Re: [go-nuts] slice of channels and..." ]
    I'm trying to mimic the time.NewTicker behavior which returns a channel when
    called. I want to use this in the DNS package, so that a client can send
    multiple queries and then async. wait for a result. However this leads me to the
    case where I have to wait for multiple channels.
    Can you detail the pattern that leads you to wait on an indefinite
    number of channels? I'm not sure why that happens.
    Well, I have a util that acts like 'dig':

    ./q mx @<nameserver> qname1 qname2 ... qnameN

    Each qnameX will result in a query, each query will return a channel. At the
    end I loop through all channels showing the replies. That would currently
    lead me to use N channels.

    I think just returning a global channels is the best way forward.
    I use a different approach[1] which allows to use 1..N channels for N
    concurrent queries (i.e. also ony 1 channel for all of them):

    ch := msg1.GoExchange(conn1, limit, nil)
    msg2.GoExchange(conn2, limit, ch)
    ...
    msgN.GoExchange(conn3, limit, ch)

    // later or elsewhere
    reply <- ch // first remote server to answer wins
    Note that by default (as I understand the code from my brief reading
    of it), the channel has a buffer size of 100 [2], so if you want to
    avoid leaving goroutines hanging indefinitely then you'll either need
    to:

    1) Consume all replies on the channel.
    2) Allocate enough channel buffer size for all but one goroutine.
  • Jan Mercl at Nov 19, 2012 at 1:23 pm

    On Mon, Nov 19, 2012 at 1:44 PM, John Beisley wrote:
    Note that by default (as I understand the code from my brief reading
    of it), the channel has a buffer size of 100 [2], so if you want to
    avoid leaving goroutines hanging indefinitely then you'll either need
    to:

    1) Consume all replies on the channel.
    2) Allocate enough channel buffer size for all but one goroutine.

    [2]: https://github.com/cznic/dns/blob/master/msg/message.go#L537
    You're right. The default 100 channel items is presumed safe for
    typical usage, which is 1-4 concurrent DNS servers being asked the
    same question (and thus possibly responding to the same channel). The
    average for, for example the .cz DNS zone, is slightly bellow 2, IIRC.

    -j

    --
  • Roger peppe at Nov 19, 2012 at 3:12 pm

    On 19 November 2012 13:22, Jan Mercl wrote:
    On Mon, Nov 19, 2012 at 1:44 PM, John Beisley wrote:

    Note that by default (as I understand the code from my brief reading
    of it), the channel has a buffer size of 100 [2], so if you want to
    avoid leaving goroutines hanging indefinitely then you'll either need
    to:

    1) Consume all replies on the channel.
    2) Allocate enough channel buffer size for all but one goroutine.

    [2]: https://github.com/cznic/dns/blob/master/msg/message.go#L537
    You're right. The default 100 channel items is presumed safe for
    typical usage, which is 1-4 concurrent DNS servers being asked the
    same question (and thus possibly responding to the same channel). The
    average for, for example the .cz DNS zone, is slightly bellow 2, IIRC.
    I can't see how the buffer size of 100 items helps.
    It looks to me as if GoExchangeBuf can send a maximum of
    one message.

    To be honest, I wouldn't bother defining a function like that - it's
    trivial to implement on top of ExchangeBuf and there are at
    least two ways to do it, depending on the caller's needs (pass
    in a channel or return a channel).

    As for Miek's original problem, the easiest way to solve it
    would be to pass a channel to the query rather than having
    the query return a channel. Then it's possible to pass the
    same channel to all the queries and return the first reply
    received. It's the same pattern used in various Go talks
    for receiving the first reply from a web server.

    --
  • Jan Mercl at Nov 19, 2012 at 3:43 pm

    On Mon, Nov 19, 2012 at 4:12 PM, roger peppe wrote:
    I can't see how the buffer size of 100 items helps.
    It looks to me as if GoExchangeBuf can send a maximum of
    one message.
    Correct, and N `GoExchangeBuf`s can send N messages, possibly to the
    same channel - if the client code wants that.
    To be honest, I wouldn't bother defining a function like that - it's
    trivial to implement on top of ExchangeBuf and there are at
    least two ways to do it, depending on the caller's needs (pass
    in a channel or return a channel).
    `GoExchangeBuf` supports two ways to do it, depending on the caller's
    needs (pass in a channel or return a _new_ channel _on demand_).
    As for Miek's original problem, the easiest way to solve it
    would be to pass a channel to the query rather than having
    the query return a channel.
    Yeah, that's what I was trying to communicate (quoting my from earlier
    post in this thread and commenting now):

    ch := msg1.GoExchange(conn1, limit, nil) // pass nil -> create
    a chanel for me please
    msg2.GoExchange(conn2, limit, ch) // pass `ch` -> please reply to `ch`
    ...
    msgN.GoExchange(conn3, limit, ch)
    Then it's possible to pass the
    same channel to all the queries and return the first reply
    received.
    // later or elsewhere
    reply <- ch // first remote server to answer wins

    It's the same pattern used in various Go talks
    for receiving the first reply from a web server.
    It's a naturally occurring pattern, agreed. That's why I suggested it
    to Miek for his OP problem.

    -j

    --
  • Roger peppe at Nov 19, 2012 at 4:16 pm

    On 19 November 2012 15:36, Jan Mercl wrote:
    On Mon, Nov 19, 2012 at 4:12 PM, roger peppe wrote:
    I can't see how the buffer size of 100 items helps.
    It looks to me as if GoExchangeBuf can send a maximum of
    one message.
    Correct, and N `GoExchangeBuf`s can send N messages, possibly to the
    same channel - if the client code wants that.
    To be honest, I wouldn't bother defining a function like that - it's
    trivial to implement on top of ExchangeBuf and there are at
    least two ways to do it, depending on the caller's needs (pass
    in a channel or return a channel).
    `GoExchangeBuf` supports two ways to do it, depending on the caller's
    needs (pass in a channel or return a _new_ channel _on demand_).
    As for Miek's original problem, the easiest way to solve it
    would be to pass a channel to the query rather than having
    the query return a channel.
    Yeah, that's what I was trying to communicate (quoting my from earlier
    post in this thread and commenting now):

    ch := msg1.GoExchange(conn1, limit, nil) // pass nil -> create
    a chanel for me please
    msg2.GoExchange(conn2, limit, ch) // pass `ch` -> please reply to `ch`
    ...
    msgN.GoExchange(conn3, limit, ch)
    Then it's possible to pass the
    same channel to all the queries and return the first reply
    received.
    // later or elsewhere
    reply <- ch // first remote server to answer wins

    It's the same pattern used in various Go talks
    for receiving the first reply from a web server.
    It's a naturally occurring pattern, agreed. That's why I suggested it
    to Miek for his OP problem.

    -j
    --
  • Miek Gieben at Nov 19, 2012 at 4:23 pm
    [ Quoting in "Re: [go-nuts] slice of channels and..." ]
    `GoExchangeBuf` supports two ways to do it, depending on the caller's
    needs (pass in a channel or return a _new_ channel _on demand_).
    I do that now too, but I like Roger's suggestion: why not always
    ask for a channel? Makes the whole thing simple and obeys the
    principle of least surprise.

    Regards,

    --
    Miek Gieben http://miek.nl
  • Roger peppe at Nov 19, 2012 at 4:34 pm

    On 19 November 2012 15:36, Jan Mercl wrote:
    On Mon, Nov 19, 2012 at 4:12 PM, roger peppe wrote:
    I can't see how the buffer size of 100 items helps.
    It looks to me as if GoExchangeBuf can send a maximum of
    one message.
    Correct, and N `GoExchangeBuf`s can send N messages, possibly to the
    same channel - if the client code wants that.
    To be honest, I wouldn't bother defining a function like that - it's
    trivial to implement on top of ExchangeBuf and there are at
    least two ways to do it, depending on the caller's needs (pass
    in a channel or return a channel).
    `GoExchangeBuf` supports two ways to do it, depending on the caller's
    needs (pass in a channel or return a _new_ channel _on demand_).
    The fact that passing a nil channel results in a new channel
    with a buffer size of 100 is asking for trouble AFAICS.
    Usually the caller will know how many requests they're
    issuing and so can size the channel accordingly - 100 is
    either way too many or perhaps too few.

    The former is just inefficient - the latter is potentially leaky.
    As for Miek's original problem, the easiest way to solve it
    would be to pass a channel to the query rather than having
    the query return a channel.
    Yeah, that's what I was trying to communicate (quoting my from earlier
    post in this thread and commenting now):

    ch := msg1.GoExchange(conn1, limit, nil) // pass nil -> create
    a chanel for me please
    msg2.GoExchange(conn2, limit, ch) // pass `ch` -> please reply to `ch`
    ...
    msgN.GoExchange(conn3, limit, ch)
    I'd suggest that:

    ch := make(msg.ExchangeReply, N)
    msg1.GoExchange(conn1, limit, ch)
    msg1.GoExchange(conn2, limit, ch)
    ...
    msgN.GoExchange(connN, limit, ch)

    is more natural and more obvious to the reader.

    However, even better, leaving all the automatic concurrency stuff out of the
    package would be more in keeping with Go's usual style. We don't have
    http.GoGet or net.GoDial.

    --
  • Jan Mercl at Nov 19, 2012 at 5:04 pm

    On Mon, Nov 19, 2012 at 5:26 PM, roger peppe wrote:
    The fact that passing a nil channel results in a new channel
    with a buffer size of 100 is asking for trouble AFAICS.
    Usually the caller will know how many requests they're
    issuing and so can size the channel accordingly - 100 is
    either way too many or perhaps too few.
    Too few it is not, typical zone has two (reasonable number) name
    servers. 100 items is an overkill on the safe side, but it costs only
    some kilobytes, i.e. it's comparable to the size of the buffers which
    have to be used for the communication anyway. And you can alway pass
    in a smaller (shorter) channel if you don't want the default size for
    any reason. However, 20 instead of 100 would

    There is probably some misunderstanding here. The message exchange is
    for one specific DNS query. If the _same_ query is to be sent by the
    client to more than one server (typically 1 to 4, the average being
    less than 2) under certain circumstances then those 2 (sometimes 3 or
    4) queries are sent to different servers and the client has the option
    to collect all the responses in one channel only. But only if she wish
    to do so. It's all configurable in the client's code.

    Different queries for different questions are not typically reported
    back on the same channel (no good reason for that). Walking the DNS
    tree is mostly not an async process at all (at least not as depicted
    in the RFCs). The exception to this rule happens when a name server
    provides an authority without glue records. Then it is (see the RFC
    again), legitimate, or even reasonable, to search for them in
    parallel. Again, usually/typically two questions are sent concurrently
    in that situation.
    The former is just inefficient - the latter is potentially leaky. ???
    I'd suggest that:

    ch := make(msg.ExchangeReply, N)
    msg1.GoExchange(conn1, limit, ch)
    msg1.GoExchange(conn2, limit, ch)
    ...
    msgN.GoExchange(connN, limit, ch)

    is more natural and more obvious to the reader.
    You can write exactly that, it _is_ supported by `GoExchange`. BTW,
    the confusion perhaps steams from that only the mechanism was
    suggested (the channel "chaíning" by the functions), not the
    implementation. Actually the "msg" package is a very low level one.
    Normally DNS clients use the "resolver" package instead. The "msg"
    package contains factored out stuff common to a DNS client and a DNS
    server, roughly said.
    However, even better, leaving all the automatic concurrency stuff out of the
    package would be more in keeping with Go's usual style. We don't have
    http.GoGet or net.GoDial.
    You're right, I would not write that in the standard library either.
    In a specific project's support package it is a matter of preferences.
    One usually wants to expose the reusable parts for own/other's
    (re)use, but also adds on top of that helpers based on patterns
    occurring repeatedly in the client code (and yes, your boss wants the
    code no later than week ago). And then you use the exposed helpers and
    cannot remove them w/o breaking backwards compatibility.

    -j

    PS: Also it should be probably noted that the code in question, the
    "dns" package was started and written mostly in 2010. For sure I would
    today, in my fourth year of Go hacking, design (some) things in a
    different way. But one usually doesn't simply later change/break
    company's production code only b/c it's not nice (but working ;-)

    --
  • Miek Gieben at Nov 19, 2012 at 5:13 pm
    [ Quoting in "Re: [go-nuts] slice of channels and..." ]
    "dns" package was started and written mostly in 2010. For sure I would
    today, in my fourth year of Go hacking, design (some) things in a
    different way. But one usually doesn't simply later change/break
    company's production code only b/c it's not nice (but working ;-)
    looking at the 4-line function I have now, it is indeed obvious how to
    do this for experienced Go programmers.

    So ... on third thought I might kill the whole function. Less is more.

    Regards,

    --
    Miek Gieben http://miek.nl
  • Miek Gieben at Nov 19, 2012 at 4:23 pm
    [ Quoting in "Re: [go-nuts] slice of channels and..." ]
    of it), the channel has a buffer size of 100 [2], so if you want to
    avoid leaving goroutines hanging indefinitely then you'll either need
    to:

    1) Consume all replies on the channel.
    2) Allocate enough channel buffer size for all but one goroutine.

    [2]: https://github.com/cznic/dns/blob/master/msg/message.go#L537
    You're right. The default 100 channel items is presumed safe for
    typical usage, which is 1-4 concurrent DNS servers being asked the
    same question (and thus possibly responding to the same channel). The
    average for, for example the .cz DNS zone, is slightly bellow 2, IIRC.
    Interesting approach. Worth copying :-) Why can't you make the channel
    blocking?

    Regards,

    --
    Miek Gieben http://miek.nl
  • Li at Nov 19, 2012 at 6:18 am
    There's a function called reflect.Select can do this, but it only exists in
    tip but not 1.0.3

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 18, '12 at 11:06p
activeNov 19, '12 at 5:13p
posts16
users7
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase