[go-nuts] slice of channels and select

Can you detail the pattern that leads you to wait on an indefinite
number of channels? I'm not sure why that happens.

Rémy.

--

[ Quoting in "Re: [go-nuts] slice of channels and..." ]
Well, I have a util that acts like 'dig':

./q mx @<nameserver> qname1 qname2 ... qnameN

Each qnameX will result in a query, each query will return a channel. At the
end I loop through all channels showing the replies. That would currently
lead me to use N channels.

I think just returning a global channels is the best way forward.

Regards,

Use sync.WaitGroup:
http://play.golang.org/p/1srl9yavab

In above code you don't even need this slice of chans... You may collect
results in goroutine spawned inside Add method and return them as a result
of Wait

I use a different approach[1] which allows to use 1..N channels for N
concurrent queries (i.e. also ony 1 channel for all of them):

ch := msg1.GoExchange(conn1, limit, nil)
msg2.GoExchange(conn2, limit, ch)
...
msgN.GoExchange(conn3, limit, ch)

// later or elsewhere
reply <- ch // first remote server to answer wins

-j

[1]: http://go.pkgdoc.org/github.com/cznic/dns/msg#Message.GoExchange

--

Note that by default (as I understand the code from my brief reading
of it), the channel has a buffer size of 100 [2], so if you want to
avoid leaving goroutines hanging indefinitely then you'll either need
to:

1) Consume all replies on the channel.
2) Allocate enough channel buffer size for all but one goroutine.

You're right. The default 100 channel items is presumed safe for
typical usage, which is 1-4 concurrent DNS servers being asked the
same question (and thus possibly responding to the same channel). The
average for, for example the .cz DNS zone, is slightly bellow 2, IIRC.

-j

--

I can't see how the buffer size of 100 items helps.
It looks to me as if GoExchangeBuf can send a maximum of
one message.

To be honest, I wouldn't bother defining a function like that - it's
trivial to implement on top of ExchangeBuf and there are at
least two ways to do it, depending on the caller's needs (pass
in a channel or return a channel).

As for Miek's original problem, the easiest way to solve it
would be to pass a channel to the query rather than having
the query return a channel. Then it's possible to pass the
same channel to all the queries and return the first reply
received. It's the same pattern used in various Go talks
for receiving the first reply from a web server.

--

Correct, and N `GoExchangeBuf`s can send N messages, possibly to the
same channel - if the client code wants that.
`GoExchangeBuf` supports two ways to do it, depending on the caller's
needs (pass in a channel or return a _new_ channel _on demand_).
Yeah, that's what I was trying to communicate (quoting my from earlier
post in this thread and commenting now):

ch := msg1.GoExchange(conn1, limit, nil) // pass nil -> create
a chanel for me please
msg2.GoExchange(conn2, limit, ch) // pass `ch` -> please reply to `ch`
...
msgN.GoExchange(conn3, limit, ch)
// later or elsewhere
reply <- ch // first remote server to answer wins

It's a naturally occurring pattern, agreed. That's why I suggested it
to Miek for his OP problem.

-j

--

--

[ Quoting in "Re: [go-nuts] slice of channels and..." ]
I do that now too, but I like Roger's suggestion: why not always
ask for a channel? Makes the whole thing simple and obeys the
principle of least surprise.

Regards,

The fact that passing a nil channel results in a new channel
with a buffer size of 100 is asking for trouble AFAICS.
Usually the caller will know how many requests they're
issuing and so can size the channel accordingly - 100 is
either way too many or perhaps too few.

The former is just inefficient - the latter is potentially leaky.
I'd suggest that:

ch := make(msg.ExchangeReply, N)
msg1.GoExchange(conn1, limit, ch)
msg1.GoExchange(conn2, limit, ch)
...
msgN.GoExchange(connN, limit, ch)

is more natural and more obvious to the reader.

However, even better, leaving all the automatic concurrency stuff out of the
package would be more in keeping with Go's usual style. We don't have
http.GoGet or net.GoDial.

--

Too few it is not, typical zone has two (reasonable number) name
servers. 100 items is an overkill on the safe side, but it costs only
some kilobytes, i.e. it's comparable to the size of the buffers which
have to be used for the communication anyway. And you can alway pass
in a smaller (shorter) channel if you don't want the default size for
any reason. However, 20 instead of 100 would

There is probably some misunderstanding here. The message exchange is
for one specific DNS query. If the _same_ query is to be sent by the
client to more than one server (typically 1 to 4, the average being
less than 2) under certain circumstances then those 2 (sometimes 3 or
4) queries are sent to different servers and the client has the option
to collect all the responses in one channel only. But only if she wish
to do so. It's all configurable in the client's code.

Different queries for different questions are not typically reported
back on the same channel (no good reason for that). Walking the DNS
tree is mostly not an async process at all (at least not as depicted
in the RFCs). The exception to this rule happens when a name server
provides an authority without glue records. Then it is (see the RFC
again), legitimate, or even reasonable, to search for them in
parallel. Again, usually/typically two questions are sent concurrently
in that situation.
You can write exactly that, it _is_ supported by `GoExchange`. BTW,
the confusion perhaps steams from that only the mechanism was
suggested (the channel "chaíning" by the functions), not the
implementation. Actually the "msg" package is a very low level one.
Normally DNS clients use the "resolver" package instead. The "msg"
package contains factored out stuff common to a DNS client and a DNS
server, roughly said.
You're right, I would not write that in the standard library either.
In a specific project's support package it is a matter of preferences.
One usually wants to expose the reusable parts for own/other's
(re)use, but also adds on top of that helpers based on patterns
occurring repeatedly in the client code (and yes, your boss wants the
code no later than week ago). And then you use the exposed helpers and
cannot remove them w/o breaking backwards compatibility.

-j

PS: Also it should be probably noted that the code in question, the
"dns" package was started and written mostly in 2010. For sure I would
today, in my fourth year of Go hacking, design (some) things in a
different way. But one usually doesn't simply later change/break
company's production code only b/c it's not nice (but working ;-)

--

[ Quoting in "Re: [go-nuts] slice of channels and..." ]
looking at the 4-line function I have now, it is indeed obvious how to
do this for experienced Go programmers.

So ... on third thought I might kill the whole function. Less is more.

Regards,

[ Quoting in "Re: [go-nuts] slice of channels and..." ]
Interesting approach. Worth copying :-) Why can't you make the channel
blocking?

Regards,

There's a function called reflect.Select can do this, but it only exists in
tip but not 1.0.3

--