[go-nuts] Different buffer implementations for channels

On Tue, 14 Jan 2014 13:41:36 +0100 Aram Hăvărneanu wrote: [...]
This is a queue that drops old elements:
http://play.golang.org/p/VxYNSFmqj4 For a stack that drops elements
(why would you need that?)

Why not? For instance, it's what network routers are supposed to do
with UDP datagrams when the outbound link is congested.
UDP is typically used to transmit data which has high redundancy such
as video streams. If a Go server is used for handling something like
this, using a queue discipline which only queues until the buffer is
full and then starts to drop incoming data appears to be reasonable.

On Tuesday, January 14, 2014 2:07:48 PM UTC+2, Henrik Johansson wrote:
Hi,

This may have been discussed before but would it not be practical to be
able to control the behavior of the underlying channel buffer?

Say you don't want the buffer to block when it is full but simply drop one
of the ends in an either fifo or lifo way.

It could probably be implemented in a wholly backwards compatible way as
perhaps an extra argument to make of type chan.Buffer (or whatever makes
sense)?

What do you think?

/ Henrik

On Tue, Jan 14, 2014 at 1:45 PM, egon wrote:


http://play.golang.org/p/iiktjv_l26

You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Tuesday, January 14, 2014 2:58:34 PM UTC+2, Henrik Johansson wrote:
My question is not at all related to any real need I have now.
It was merely a thought I had when refreshing my message passing...

For example ttl functionality is something easily implemented on a case by
case but and the ttl on a message clearly belong to the message.
The dropping of the message feels like it belongs to the queue though.

Maybe more advanced example would be priorities?

We are always advocating separation of concerns and I just think that some
of these approaches may belong in the queuing rather than with the user.

On Tue, Jan 14, 2014 at 2:12 PM, egon wrote:


Or rather a separate library.

I don't think chan was meant for IPC or as a "does everything related to
queueing and messaging". If you are having these kinds of problems then you
should probably use a dedicated messaging library/framework... e.g.
RabbitMQ.

+ egon

You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Tue, Jan 14, 2014 at 8:20 AM, Henrik Johansson wrote:

For networking I would go for something else at least until/if net chans
come back.

My point is simply that some of these things may belong in the queue.

It is just food for thought and I am sure there must have been much
discussion when channels were first conceived.

On Jan 15, 2014 3:02 AM, "minux" wrote:

but the queue doesn't belong to the language, it could (and should) be
implemented as
a package.
All these problems should be fixed in user programs, not on the Go
language level.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, Jan 15, 2014 at 9:11 AM, Henrik Johansson wrote:
What queue? I am talking about the channels underlying queue.

The whole point is to modify the behavior of the channel. If I can do it via
a library, fine but it is not possible right?

There would need to be all sorts of intermediary buffers and a number of
channels. This can easily be implemented in a library, sure but why not have
it in the native channels? There is already the distinction between buffered
and non-buffered channels that you have to consider actively when you code.
This is no different.

I think it belongs in the language but there could be implementation details
too destructive to make it worth it of course.

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, Jan 15, 2014 at 10:32 AM, Peter Bourgon wrote:
Language primitives like channels are building blocks, not prepackaged
solutions to classes of problems.

Systems with the semantics you describe are straightforward to
implement using channels as they exist today.

http://play.golang.org/p/JyTXqkBxVX
(Disclaimer: not thoroughly tested.)

chans
via
Groups
an

On Wednesday, January 15, 2014 11:50:02 AM UTC+2, Henrik Johansson wrote:
I think the dropping case(s) lifo/fifo is a pretty natural extension to
the current channels.
It provides a bit more control of the behavior without sacrificing easy
and nice way that channels are used.

1. Unbuffered channels - Rendezvous
2. Buffered, blocking
3. Buffered non-blocking - drop newest
4. Buffered non-blocking - drop oldest

And sure these can always be implemented (everything can) but that does
not make it a bad evolution of the channels.

On Wed, Jan 15, 2014 at 10:32 AM, Peter Bourgon <pe...@bourgon.org<javascript:>

On Wed, Jan 15, 2014 at 11:04 AM, egon wrote:


...and then out of order delivery, then prioritized delivery, then UDP
style channels that can accept dropped messages... Where should the line be
drawn?

If it goes into the language it must have a lot of use-cases; otherwise
it's better to implement it in a library. If something is really necessary
in the language you should be able to demonstrate it:
i.e. a list of 5-15 practical usefulness examples would be a good starting
point, and then a pro/con analysis for adding that. And not just some
examples that can be handled by a library... but by examples that will be
often needed in many variations.

+ egon

You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Am Mittwoch, 15. Januar 2014 10:50:02 UTC+1 schrieb Henrik Johansson:
I think the dropping case(s) lifo/fifo is a pretty natural extension to
the current channels.
It provides a bit more control of the behavior without sacrificing easy
and nice way that channels are used.

1. Unbuffered channels - Rendezvous
2. Buffered, blocking
3. Buffered non-blocking - drop newest
4. Buffered non-blocking - drop oldest

And sure these can always be implemented (everything can) but that does
not make it a bad evolution of the channels.

Am Mittwoch, 15. Januar 2014 10:50:02 UTC+1 schrieb Henrik Johansson:
Seeing code like
select {
case channel <- value:
default:
}
I know by looking at 4 local lines of code that value
won't be sent if nobody is reading from channel or
channel's buffer is full.

With some magic internal non-blocking flag I would
have to consult the definition of channel (which might
be somewhere else, maybe not even under my control).

I prefer the local variant and do not see any evolutionary
advantage of these non-blocking channels.

V.

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Tue, Jan 14, 2014 at 4:07 PM, Henrik Johansson wrote:

Hi,

This may have been discussed before but would it not be practical to be
able to control the behavior of the underlying channel buffer?

Say you don't want the buffer to block when it is full but simply drop one
of the ends in an either fifo or lifo way.

It could probably be implemented in a wholly backwards compatible way as
perhaps an extra argument to make of type chan.Buffer (or whatever makes
sense)?

What do you think?

On Wed, Jan 15, 2014 at 11:30 AM, Dmitry Vyukov wrote:
You can drop messages using "non-blocking select" today.

An arbitrary complex overload policies must/can not be implemented in
runtime. In this case you need an intermidiate dispatcher that will
reorder/drop messages:

inc := make(chan *Req, 1000)
outc := make(chan chan *Req, 1000)
go func() {
// dispatcher
reqs := MakeHeapOrWhatever()
for {
select {
case r := <-inc:
reqs.Add(r)
case c := <-outc:
c <- reqs.Get()
case <-time.After(...):
reqs.Tick()
}
}
}()

for i := 0; i < 10; i++ {
go func() {
// consumer
c := make(chan *Req, 1)
for {
outc <- c
r := <-c
Process(r)
}
}()
}


If you are interested in other "simple" cases, then you must be very
specific and make a very strong case for the extension.

On Wed, Jan 15, 2014 at 3:17 PM, Henrik Johansson wrote:
I think I am mostly (and not even immediately) interested in the dropping
buffered scenario but I cant make a strong use case for it at this time.

Could a strong case be performance?

Since it can be solved with a buffering
dispatcher there is no real functional need to include it other than perhaps
convenience.

It mostly seems to complete the way channels work today with the two
additional cases for the buffered versions.

I will give it a thought, perhaps I can find a case that is not contrived as
they tend to be when not coming from an immediate need.

Thanks!

On Wed, Jan 15, 2014 at 3:39 PM, Dmitry Vyukov wrote:
I guess it depends on details.

And you can do "drop newest" today with good performance and w/o
dispatcher.
as
makes

On Wed, Jan 15, 2014 at 6:57 PM, Henrik Johansson wrote:
Is this what you mean?

if len(c) < cap(c) {
c <- data
}

A bit racy of course but that does not matter I guess.

On Wed, Jan 15, 2014 at 4:31 PM, Dmitry Vyukov wrote:
No, I mean:

select {
case c <- data:
default:
}

Actually, you can do "drop oldest" as well, this time indeed in a
slightly racy way:

loop: for {
select {
case c <- data:
break loop
default:
select {
case <-c:
default:
}
}
}

This racy-ness (when you recv from an already non-full chan) is fine
in producer-consumer scenarios with sufficiently buffered channel.


time.
drop
way