FAQ
I am writing an interface to a public database (NCBI entrez) that has a
policy that restricts requests to 3 per second (this is enforced after
the fact by IP blocking offending clients - this is a pain to reverse as
it is by email to their admins).

To avoid this problem with clients of the package I want to enforce this
time delay. I have two approaches in the playground below. One is a
straight ticker, which should work fine and is very simple (if the delay
were longer it would be less appealing because of potential for missing
a valid request slot, but a maximum 333ms maximum unnecessary wait is
manageable). The other uses time.After and requires additional locking
but avoids the issue of missed slots.

http://play.golang.org/p/3uWbcQ1yzA

An alternative is to use a queue with a request goroutine, but I'm less
inclined towards that as I think it will complicate things down the
track because of the nature of the returned data.

Are there any other approaches that people can think of or which of
these would people recommend?

thanks
Dan

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Kyle Lemons at Feb 8, 2013 at 12:02 am
    I'd go with the code that's simplest to read, which might look like this:
    http://play.golang.org/p/MKETaEO2mH

    type RateLimit chan struct{}

    func NewRateLimiter(count int, per time.Duration) RateLimit {
    ch := make(RateLimit, count)

    // Start out filled
    for i := 0; i < count; i++ {
    ch <- struct{}{}
    }

    go func() {
    delay := per/time.Duration(count) + 1 // +1 to avoid rounding down
    time.Sleep(per - delay) // because we started out filled

    ticker := time.NewTicker(delay)
    defer ticker.Stop()
    for _ = range ticker.C {
    select {
    case ch <- struct{}{}:
    case <-ch:
    return
    }
    }
    }()
    return ch
    }

    func (rl RateLimit) Wait() {
    <-rl
    }

    On Thu, Feb 7, 2013 at 1:54 AM, Dan Kortschak wrote:

    I am writing an interface to a public database (NCBI entrez) that has a
    policy that restricts requests to 3 per second (this is enforced after
    the fact by IP blocking offending clients - this is a pain to reverse as
    it is by email to their admins).

    To avoid this problem with clients of the package I want to enforce this
    time delay. I have two approaches in the playground below. One is a
    straight ticker, which should work fine and is very simple (if the delay
    were longer it would be less appealing because of potential for missing
    a valid request slot, but a maximum 333ms maximum unnecessary wait is
    manageable). The other uses time.After and requires additional locking
    but avoids the issue of missed slots.

    http://play.golang.org/p/3uWbcQ1yzA

    An alternative is to use a queue with a request goroutine, but I'm less
    inclined towards that as I think it will complicate things down the
    track because of the nature of the returned data.

    Are there any other approaches that people can think of or which of
    these would people recommend?

    thanks
    Dan

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Feb 8, 2013 at 12:04 am
    slight improvement:
    http://play.golang.org/p/0mhNpSiNWK

    it doesn't do the per-delay thing (it looks too magical)

    On Thu, Feb 7, 2013 at 4:02 PM, Kyle Lemons wrote:

    I'd go with the code that's simplest to read, which might look like this:
    http://play.golang.org/p/MKETaEO2mH

    type RateLimit chan struct{}

    func NewRateLimiter(count int, per time.Duration) RateLimit {
    ch := make(RateLimit, count)

    // Start out filled
    for i := 0; i < count; i++ {
    ch <- struct{}{}
    }

    go func() {
    delay := per/time.Duration(count) + 1 // +1 to avoid rounding down
    time.Sleep(per - delay) // because we started out filled

    ticker := time.NewTicker(delay)
    defer ticker.Stop()
    for _ = range ticker.C {
    select {
    case ch <- struct{}{}:
    case <-ch:
    return
    }
    }
    }()
    return ch
    }

    func (rl RateLimit) Wait() {
    <-rl
    }


    On Thu, Feb 7, 2013 at 1:54 AM, Dan Kortschak <
    dan.kortschak@adelaide.edu.au> wrote:
    I am writing an interface to a public database (NCBI entrez) that has a
    policy that restricts requests to 3 per second (this is enforced after
    the fact by IP blocking offending clients - this is a pain to reverse as
    it is by email to their admins).

    To avoid this problem with clients of the package I want to enforce this
    time delay. I have two approaches in the playground below. One is a
    straight ticker, which should work fine and is very simple (if the delay
    were longer it would be less appealing because of potential for missing
    a valid request slot, but a maximum 333ms maximum unnecessary wait is
    manageable). The other uses time.After and requires additional locking
    but avoids the issue of missed slots.

    http://play.golang.org/p/3uWbcQ1yzA

    An alternative is to use a queue with a request goroutine, but I'm less
    inclined towards that as I think it will complicate things down the
    track because of the nature of the returned data.

    Are there any other approaches that people can think of or which of
    these would people recommend?

    thanks
    Dan

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dan Kortschak at Feb 8, 2013 at 12:52 am
    I simplified the After-based code to this:

    http://play.golang.org/p/9d4Ge9U-rJ

    I'll be using that because the ticker approach hide client deadlocks,
    which is something I'd like to avoid.

    It's quite readable, I think.

    thanks
    Dan
    On Thu, 2013-02-07 at 16:02 -0800, Kyle Lemons wrote:
    I'd go with the code that's simplest to read, which might look like this:
    http://play.golang.org/p/MKETaEO2mH

    type RateLimit chan struct{}

    func NewRateLimiter(count int, per time.Duration) RateLimit {
    ch := make(RateLimit, count)

    // Start out filled
    for i := 0; i < count; i++ {
    ch <- struct{}{}
    }

    go func() {
    delay := per/time.Duration(count) + 1 // +1 to avoid rounding down
    time.Sleep(per - delay) // because we started out filled

    ticker := time.NewTicker(delay)
    defer ticker.Stop()
    for _ = range ticker.C {
    select {
    case ch <- struct{}{}:
    case <-ch:
    return
    }
    }
    }()
    return ch
    }

    func (rl RateLimit) Wait() {
    <-rl
    }


    On Thu, Feb 7, 2013 at 1:54 AM, Dan Kortschak <dan.kortschak@adelaide.edu.au
    wrote:
    I am writing an interface to a public database (NCBI entrez) that has a
    policy that restricts requests to 3 per second (this is enforced after
    the fact by IP blocking offending clients - this is a pain to reverse as
    it is by email to their admins).

    To avoid this problem with clients of the package I want to enforce this
    time delay. I have two approaches in the playground below. One is a
    straight ticker, which should work fine and is very simple (if the delay
    were longer it would be less appealing because of potential for missing
    a valid request slot, but a maximum 333ms maximum unnecessary wait is
    manageable). The other uses time.After and requires additional locking
    but avoids the issue of missed slots.

    http://play.golang.org/p/3uWbcQ1yzA

    An alternative is to use a queue with a request goroutine, but I'm less
    inclined towards that as I think it will complicate things down the
    track because of the nature of the returned data.

    Are there any other approaches that people can think of or which of
    these would people recommend?

    thanks
    Dan

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    Omnes mundum facimus.

    Dan Kortschak <dan.kortschak@adelaide.edu.au>
    F9B3 3810 C4DD E214 347C B8DA D879 B7A7 EECC 5A40
    10C7 EEF4 A467 89C9 CA00 70DF C18F 3421 A744 607C

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dan Kortschak at Feb 8, 2013 at 12:55 am

    On Fri, 2013-02-08 at 11:22 +1030, Dan Kortschak wrote:
    I simplified the After-based code to this:

    http://play.golang.org/p/9d4Ge9U-rJ

    I'll be using that because the ticker approach hide client deadlocks,
    which is something I'd like to avoid.

    It's quite readable, I think.
    Though could be improved by removing the select!

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Russ Cox at Feb 9, 2013 at 1:59 am
    time.Sleep is enough.
    http://play.golang.org/p/RLafKfr5_F

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dan Kortschak at Feb 8, 2013 at 12:44 pm
    Here Vroomfondel, why do we never think of things like that?" "Dunno," said Vroomfondel in an awed whisper, "think our brains must be too highly trained."


    On 08/02/2013, at 3:03 PM, "Russ Cox" wrote:

    time.Sleep is enough.
    http://play.golang.org/p/RLafKfr5_F


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedFeb 7, '13 at 9:54a
activeFeb 9, '13 at 1:59a
posts7
users3
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase