FAQ
Let's say I'm working on a piece of code in Go (and I'm new to Go) that
needs to make some calls to a library written in C. The library is not
trivial but also not horribly complicated: about a dozen typedefs/structs
and about 50 function calls (could potentially grow over time). The
functions when called will either get some data from somewhere else and
return it or take some data from the caller and send it/write it to some
other system.The choices I have:

(1) cgo and call directly into the C library. The problem here is that I'm
not a C expert and the C library makes heavy use of pointers to structs and
to primitive types that the caller must allocate (e.g. the typical C int*
len, data_t* data, uint8* buf). From briefly looking at cgo it seems that I
would have to allocate memory and free it on the Go side, at least in some
cases. Memory management and leaks scare me :)

(2) Suggest to the library author (coworker of mine) to provide and RPC/IPC
type service to the library that I could simply connect to from Go, e.g.
via a unix socket or some other form of RPC/IPC (e.g. gRPC). Any
suggestions here and how to deal with the authentication/authorization
part? I'd like to make sure that it's secure even though it will run on the
same machine.

Performance is not the main concern (as long as it's not horrible and too
heavy handed, e.g. HTTP/REST/JSON API would be too much). Correctness is.

Any recommendations between (1) and (2)? Have you tackled a similar problem?

Thank you.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Tamás Gulácsi at Jul 7, 2015 at 4:33 am
    For auth, its the simplest to not have, and let the main process start its plugins, and communicate with them through stdin/stdout: https://github.com/natefinch/pie

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tim K at Jul 7, 2015 at 4:27 pm
    Interesting approach to consider, thanks for replying.

    Any other thoughts (1) vs (2) ?
    On Monday, July 6, 2015 at 9:33:05 PM UTC-7, Tamás Gulácsi wrote:

    For auth, its the simplest to not have, and let the main process start its
    plugins, and communicate with them through stdin/stdout:
    https://github.com/natefinch/pie
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Hamish Ogilvy at Jul 8, 2015 at 1:10 am
    1) definitely works. Just make sure your coworker can ensure the C lib
    frees any memory it allocs. Assuming the C lib doesn't have memory leaks,
    then cgo is not difficult to get going. Even if you were to do 2), the C
    lib would still need to be able to manage it's own memory. The Go runtime
    is unaware of memory allocated by C.

    On Tuesday, 7 July 2015 10:24:03 UTC+10, Tim K wrote:

    Let's say I'm working on a piece of code in Go (and I'm new to Go) that
    needs to make some calls to a library written in C. The library is not
    trivial but also not horribly complicated: about a dozen typedefs/structs
    and about 50 function calls (could potentially grow over time). The
    functions when called will either get some data from somewhere else and
    return it or take some data from the caller and send it/write it to some
    other system.The choices I have:

    (1) cgo and call directly into the C library. The problem here is that I'm
    not a C expert and the C library makes heavy use of pointers to structs and
    to primitive types that the caller must allocate (e.g. the typical C int*
    len, data_t* data, uint8* buf). From briefly looking at cgo it seems that I
    would have to allocate memory and free it on the Go side, at least in some
    cases. Memory management and leaks scare me :)

    (2) Suggest to the library author (coworker of mine) to provide and
    RPC/IPC type service to the library that I could simply connect to from Go,
    e.g. via a unix socket or some other form of RPC/IPC (e.g. gRPC). Any
    suggestions here and how to deal with the authentication/authorization
    part? I'd like to make sure that it's secure even though it will run on the
    same machine.

    Performance is not the main concern (as long as it's not horrible and too
    heavy handed, e.g. HTTP/REST/JSON API would be too much). Correctness is.

    Any recommendations between (1) and (2)? Have you tackled a similar
    problem?

    Thank you.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tim K at Jul 8, 2015 at 2:02 am
    Yes, the C lib must do its own memory management, it will also be used by
    some other C apps. My fear is the Go/cgo side. For example there are some
    APIs where you have to receive some data, into a buffer/struct the caller
    allocates and passes a pointer to the API. This means the memory must be
    allocated from the Go side, sent to the C lib and then freed from the Go
    side. Would this be an appropriate pattern to use from Go?

    func f(...) {
       1. allocate memory for the C structs/buffers using C.malloc
       2. defer C.free
       3. make the C lib call
       4. copy the data out of the C struct into an appropriate Go struct
       5. return the Go struct (at which point defer C.free will kick in)
    }

    I'm not sure whether the above can be simplified, for example to use a Go
    struct that mirrors a C struct and that can be passed directly to the C lib
    which will fill it in with the data, to avoid the entire C.malloc/C.free
    dance. I suspect it's not possible or maybe possible only in certain cases
    depending on what the struct looks like.

    Thanks again.

    Tim
    On Tuesday, July 7, 2015 at 6:10:16 PM UTC-7, Hamish Ogilvy wrote:

    1) definitely works. Just make sure your coworker can ensure the C lib
    frees any memory it allocs. Assuming the C lib doesn't have memory leaks,
    then cgo is not difficult to get going. Even if you were to do 2), the C
    lib would still need to be able to manage it's own memory. The Go runtime
    is unaware of memory allocated by C.

    On Tuesday, 7 July 2015 10:24:03 UTC+10, Tim K wrote:

    Let's say I'm working on a piece of code in Go (and I'm new to Go) that
    needs to make some calls to a library written in C. The library is not
    trivial but also not horribly complicated: about a dozen typedefs/structs
    and about 50 function calls (could potentially grow over time). The
    functions when called will either get some data from somewhere else and
    return it or take some data from the caller and send it/write it to some
    other system.The choices I have:

    (1) cgo and call directly into the C library. The problem here is that
    I'm not a C expert and the C library makes heavy use of pointers to structs
    and to primitive types that the caller must allocate (e.g. the typical C
    int* len, data_t* data, uint8* buf). From briefly looking at cgo it seems
    that I would have to allocate memory and free it on the Go side, at least
    in some cases. Memory management and leaks scare me :)

    (2) Suggest to the library author (coworker of mine) to provide and
    RPC/IPC type service to the library that I could simply connect to from Go,
    e.g. via a unix socket or some other form of RPC/IPC (e.g. gRPC). Any
    suggestions here and how to deal with the authentication/authorization
    part? I'd like to make sure that it's secure even though it will run on the
    same machine.

    Performance is not the main concern (as long as it's not horrible and too
    heavy handed, e.g. HTTP/REST/JSON API would be too much). Correctness is.

    Any recommendations between (1) and (2)? Have you tackled a similar
    problem?

    Thank you.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Hamish Ogilvy at Jul 8, 2015 at 2:22 am
    You don't pass Go pointers into C. Go manages it's memory, so you don't
    know when that object will be freed. You want to either copy it, or if it's
    too large, get C to provide the buffer and write directly to that.

    And yes you can mirror a struct in C and can also wrap the creation within
    a Go function for convenience. Example:
    https://gist.github.com/mish15/9822474
    On 8 July 2015 at 12:02, Tim K wrote:

    Yes, the C lib must do its own memory management, it will also be used by
    some other C apps. My fear is the Go/cgo side. For example there are some
    APIs where you have to receive some data, into a buffer/struct the caller
    allocates and passes a pointer to the API. This means the memory must be
    allocated from the Go side, sent to the C lib and then freed from the Go
    side. Would this be an appropriate pattern to use from Go?

    func f(...) {
    1. allocate memory for the C structs/buffers using C.malloc
    2. defer C.free
    3. make the C lib call
    4. copy the data out of the C struct into an appropriate Go struct
    5. return the Go struct (at which point defer C.free will kick in)
    }

    I'm not sure whether the above can be simplified, for example to use a Go
    struct that mirrors a C struct and that can be passed directly to the C lib
    which will fill it in with the data, to avoid the entire C.malloc/C.free
    dance. I suspect it's not possible or maybe possible only in certain cases
    depending on what the struct looks like.

    Thanks again.

    Tim
    On Tuesday, July 7, 2015 at 6:10:16 PM UTC-7, Hamish Ogilvy wrote:

    1) definitely works. Just make sure your coworker can ensure the C lib
    frees any memory it allocs. Assuming the C lib doesn't have memory leaks,
    then cgo is not difficult to get going. Even if you were to do 2), the C
    lib would still need to be able to manage it's own memory. The Go runtime
    is unaware of memory allocated by C.

    On Tuesday, 7 July 2015 10:24:03 UTC+10, Tim K wrote:

    Let's say I'm working on a piece of code in Go (and I'm new to Go) that
    needs to make some calls to a library written in C. The library is not
    trivial but also not horribly complicated: about a dozen typedefs/structs
    and about 50 function calls (could potentially grow over time). The
    functions when called will either get some data from somewhere else and
    return it or take some data from the caller and send it/write it to some
    other system.The choices I have:

    (1) cgo and call directly into the C library. The problem here is that
    I'm not a C expert and the C library makes heavy use of pointers to structs
    and to primitive types that the caller must allocate (e.g. the typical C
    int* len, data_t* data, uint8* buf). From briefly looking at cgo it seems
    that I would have to allocate memory and free it on the Go side, at least
    in some cases. Memory management and leaks scare me :)

    (2) Suggest to the library author (coworker of mine) to provide and
    RPC/IPC type service to the library that I could simply connect to from Go,
    e.g. via a unix socket or some other form of RPC/IPC (e.g. gRPC). Any
    suggestions here and how to deal with the authentication/authorization
    part? I'd like to make sure that it's secure even though it will run on the
    same machine.

    Performance is not the main concern (as long as it's not horrible and
    too heavy handed, e.g. HTTP/REST/JSON API would be too much). Correctness
    is.

    Any recommendations between (1) and (2)? Have you tackled a similar
    problem?

    Thank you.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/mZMb492j0jg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tim K at Jul 8, 2015 at 2:35 am
    Ok, so it's basically a lot of copying back and forth between Go and C,
    with some C.malloc/C.free from the Go side, no shared memory between the
    two (which makes sense given the Go principle of "share memory by
    communicating").
    On Tuesday, July 7, 2015 at 7:22:32 PM UTC-7, Hamish Ogilvy wrote:

    You don't pass Go pointers into C. Go manages it's memory, so you don't
    know when that object will be freed. You want to either copy it, or if it's
    too large, get C to provide the buffer and write directly to that.

    And yes you can mirror a struct in C and can also wrap the creation within
    a Go function for convenience. Example:
    https://gist.github.com/mish15/9822474
    On 8 July 2015 at 12:02, Tim K <tim....@gmail.com <javascript:>> wrote:

    Yes, the C lib must do its own memory management, it will also be used by
    some other C apps. My fear is the Go/cgo side. For example there are some
    APIs where you have to receive some data, into a buffer/struct the caller
    allocates and passes a pointer to the API. This means the memory must be
    allocated from the Go side, sent to the C lib and then freed from the Go
    side. Would this be an appropriate pattern to use from Go?

    func f(...) {
    1. allocate memory for the C structs/buffers using C.malloc
    2. defer C.free
    3. make the C lib call
    4. copy the data out of the C struct into an appropriate Go struct
    5. return the Go struct (at which point defer C.free will kick in)
    }

    I'm not sure whether the above can be simplified, for example to use a Go
    struct that mirrors a C struct and that can be passed directly to the C lib
    which will fill it in with the data, to avoid the entire C.malloc/C.free
    dance. I suspect it's not possible or maybe possible only in certain cases
    depending on what the struct looks like.

    Thanks again.

    Tim
    On Tuesday, July 7, 2015 at 6:10:16 PM UTC-7, Hamish Ogilvy wrote:

    1) definitely works. Just make sure your coworker can ensure the C lib
    frees any memory it allocs. Assuming the C lib doesn't have memory leaks,
    then cgo is not difficult to get going. Even if you were to do 2), the C
    lib would still need to be able to manage it's own memory. The Go runtime
    is unaware of memory allocated by C.

    On Tuesday, 7 July 2015 10:24:03 UTC+10, Tim K wrote:

    Let's say I'm working on a piece of code in Go (and I'm new to Go) that
    needs to make some calls to a library written in C. The library is not
    trivial but also not horribly complicated: about a dozen typedefs/structs
    and about 50 function calls (could potentially grow over time). The
    functions when called will either get some data from somewhere else and
    return it or take some data from the caller and send it/write it to some
    other system.The choices I have:

    (1) cgo and call directly into the C library. The problem here is that
    I'm not a C expert and the C library makes heavy use of pointers to structs
    and to primitive types that the caller must allocate (e.g. the typical C
    int* len, data_t* data, uint8* buf). From briefly looking at cgo it seems
    that I would have to allocate memory and free it on the Go side, at least
    in some cases. Memory management and leaks scare me :)

    (2) Suggest to the library author (coworker of mine) to provide and
    RPC/IPC type service to the library that I could simply connect to from Go,
    e.g. via a unix socket or some other form of RPC/IPC (e.g. gRPC). Any
    suggestions here and how to deal with the authentication/authorization
    part? I'd like to make sure that it's secure even though it will run on the
    same machine.

    Performance is not the main concern (as long as it's not horrible and
    too heavy handed, e.g. HTTP/REST/JSON API would be too much). Correctness
    is.

    Any recommendations between (1) and (2)? Have you tackled a similar
    problem?

    Thank you.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/mZMb492j0jg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • David Chase at Jul 8, 2015 at 5:54 pm
    I am not as skilled at Go as I need to be, but I have a lot of experience doing this with other languages.
    The other-languages idiom translated into Go would be something along the lines of this:

    type ThingWithCbuffer struct {
        buf *[4096]C.uchar
        // other stuff
    }

    func finalTWCB(x *ThingWithCbuffer) {
        if x.buf != nil {
            C.free(x.buf)
            x.buf = nil // because we are paranoid
        }
    }

    func makeThingWithCbuffer() *ThingWithCBuffer {
        x := &ThingWithCBuffer{}
        runtime.setFinalizer(x, finalTWCB)
        x.buf = cast_as_necessary(C.malloc(4096))
        return x
    }

    The idea is that the pointer to the C stuff stays allocated until the Go object becomes unreachable, at which point (eventually) GC will occur and (eventually) the object will be queued for finalization.

    I use a *[4096]C.uchar to get as much type safety as I can on the Go side and provide as many hints as possible to the compiler about what is going on (right now, it ignores those hints because default behavior is C-friendly). I am told (as a Go compiler writer and officemate of a Go GC writer) that the Go GC will always be prepared to cope with pointers-to-not-Go-heap values stored in Go pointers, so it is okay to store that pointer in a Go struct. By using a pointer to an array I think we obtain type and index safety on the Go side. We prefer to allocate as a C-ish type (as much as possible) to avoid casting Go types (ever) to C types, and in the case of C structs this may eventually be a signal to use a C-friendly layout (as it happens, existing layout rules appear to match C layout rules). I.e., using pointers to and arrays of C types is future-proofing, for some possible futures. Even if it's not necessary for the futures that we end up getting, it will help make it clearer what is going on and also make it clear that problematic idioms (casting *Gotype to *Ctype) don't occur.

    I hope I got this right for Go (what I describe has worked fine in other languages in decades past) and I hope it is helpful.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJul 7, '15 at 12:24a
activeJul 8, '15 at 5:54p
posts8
users4
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase