[go-nuts] [ANN] fn: concurrent, easy to use functional tools (right now just Map and Filter)

Please add examples of problems this package solves. Currently it just shows the usage and has some vague statements about problems of understanding wg-s and channels.

+ egon

You may find use for a rudimentary type checker: https://github.com/BurntSushi/ty

The "fun" sub package provides functions written with that type checker. (It is different from your package in that a single type assertion is required on the return value of a function.)

I believe it's possible to do something
simpler: http://play.golang.org/p/kDoOJijq3A
Also, it's more "type-safe" at compile time and faster.

+ egon

That's brilliant! It's better than mine on all fronts.

Thank you

-Tobia

I don't think it is a good idea to run as many goroutines as elements.
What was the reasoning behind that decision?


I'm copying here an example of a similar idea. See that in this example:
- the number of goroutines is an input
- and the function fn takes slices to avoid "reflecting" each element.



http://play.golang.org/p/BTUiXcRum4


package main

import (
  "fmt"
  "reflect"
  "sync"
)

func main() {
  words := []string{"Hello", "world", "here", "is", "Go"}
  lengths := make([]int, len(words))
  getLen := func(in []string, out []int) {
   for i := range in {
    out[i] = len(in[i])
   }
   fmt.Println(in, out)
  }

  Apply(getLen, words, &lengths, 2)

  fmt.Println("words =", words)
  fmt.Println("lengths =", lengths)
}

func Apply(fn interface{}, in interface{}, out interface{},
numGoroutines int) {
  inVal := reflect.ValueOf(in)
  if k := inVal.Kind(); k != reflect.Array && k != reflect.Slice {
   panic("Apply: argument 'in' must be an array or a slice")
  }

  outVal := reflect.ValueOf(out).Elem()
  if k := outVal.Kind(); k != reflect.Slice {
   panic("Apply: argument 'out' must be a slice")
  }

  inLen := inVal.Len()
  if inLen > outVal.Cap() {
   panic("Apply: capacity of argument 'out' is smaller than length of
argument 'in'")
  }
  if inLen == 0 {
   outVal.SetLen(0)
   return
  }

  fnVal := reflect.ValueOf(fn)
  if k := fnVal.Kind(); k != reflect.Func {
   panic("Apply: argument 'fn' must be a function")
  }

  inT := inVal.Type().Elem()
  outT := outVal.Type().Elem()
  if t := fnVal.Type(); t.NumIn() != 2 || t.NumOut() != 0 ||
   t.In(0).Elem() != inT || t.In(1).Elem() != outT {
   panic("Apply: type of argument 'fn' must be consistent with types of
arguments 'in' and 'out'")
  }

  if numGoroutines <= 0 {
   panic("Apply: numGoroutines <= 0")
  }

  // grow output slice to the final length
  outVal.SetLen(inLen)

  // split work into a number of goroutines
  var wg sync.WaitGroup
  task := func(in, out reflect.Value) {
   defer wg.Done()
   fnVal.Call([]reflect.Value{in, out})
  }
  taskLen := inLen / numGoroutines

  i, j := 0, taskLen
  for j < inLen {
   wg.Add(1)
   go task(inVal.Slice(i, j), outVal.Slice(i, j))
   i, j = j, j+taskLen
  }
  wg.Add(1)
  go task(inVal.Slice(i, inLen), outVal.Slice(i, inLen))

  wg.Wait()
}

For simple cases such as those in the examples, the fastest algorithm is a
simple for loop, with no concurrency at all, by orders of magnitude
compared to this particular implementation of Map / Filter. There are
benchmarks in the test file to measure how slower it is.

The point of a fully concurrent implementation (one goroutine per element)
is not being the most efficient, but being the most versatile. The Map and
Filter functions could be calling external services, making network
requests to different hosts, and so on.

I agree that more concurrency options (at least One, NumCPU, and the
current Max) are necessary before this thing can even remotely be called an
actual library, but I wanted to get some feedback on the choice of API and
on the very idea of a generic functional library for Go.

-Tobia