FAQ
Given: an existing C language based system (that includes many on-disk
files).

Given: I'm using CGO to call that C code that expects those files on disk.

Goal: I'd like to copy-in/compile-in all the files that would normally live
on disk, putting them into the go executable file.

Rationale: for easy (single executable binary, in classic Go style)
distribution of all the required files and directories that attend the CGO
called code.

Constraint: Ideally the C code in the libraries wouldn't have to be
modified in order to redirect its access to the filesystem embedded inside
the binary.

Q: Is there an existing "filesystem within a file" library for Go? I'd
like OSX and linux support.

I imagine I could use LD_PRELOAD / DYLD_INSERT_LIBRARIES to hook
open/read/write calls and redirect them to the internal data. But then I
still need to have a "filesystem within a file" directory structure logic
implemented that lets me embed that filesystem in the file somewhere. The
filesystem can be read-only.

Are there any existing libraries or parts for such a job?

Caveat: I'd prefer to avoid the complexity and installation requirements of
a container/Docker/docker data volume.

I'm currently thinking I may need to resort to a single external database
file. Perhaps this file is managed by boltdb or bazil, then use LD_PRELOAD
-> hook filesystem open(),, read(), write() filesystem calls, and redirect
the filesystem operations that are to the "internal paths" to the
single-file database.

Seems like a dead-end: I could do some kind of loopback device/mount
scheme, but I don't see how to make a loopback device point to just part
(say the last half) of a file.

Thanks for your thoughts!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Tamás Gulácsi at Dec 19, 2015 at 6:17 pm
    Easy: create a zip file, append it to the executable, the you only have to locate the executable at runtime.

    On linux, you can use systemd-nspawn (see https://chimeracoder.github.io/docker-without-docker/#35). I don't know osx.

    Why can'z just extract the fs from the exexutable zip, and start in a chroot?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jason E. Aten at Dec 19, 2015 at 7:14 pm
    Tamás: thank you for those suggestions.

    Additional constraint: A run of the final executable should avoid writing
    to disk. No unpacking of a filesystem hierarchy to disk should be required.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Viktor Kojouharov at Dec 19, 2015 at 7:28 pm
    write the files' bytes in a go file and use the files as variables.
    On Saturday, December 19, 2015 at 9:14:58 PM UTC+2, Jason E. Aten wrote:

    Tamás: thank you for those suggestions.

    Additional constraint: A run of the final executable should avoid writing
    to disk. No unpacking of a filesystem hierarchy to disk should be required.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Micky at Dec 19, 2015 at 8:15 pm
    "encoding/base64"
    On Sat, Dec 19, 2015 at 10:57 PM, Jason E. Aten wrote:

    Given: an existing C language based system (that includes many on-disk
    files).

    Given: I'm using CGO to call that C code that expects those files on disk.

    Goal: I'd like to copy-in/compile-in all the files that would normally
    live on disk, putting them into the go executable file.

    Rationale: for easy (single executable binary, in classic Go style)
    distribution of all the required files and directories that attend the CGO
    called code.

    Constraint: Ideally the C code in the libraries wouldn't have to be
    modified in order to redirect its access to the filesystem embedded inside
    the binary.

    Q: Is there an existing "filesystem within a file" library for Go? I'd
    like OSX and linux support.

    I imagine I could use LD_PRELOAD / DYLD_INSERT_LIBRARIES to hook
    open/read/write calls and redirect them to the internal data. But then I
    still need to have a "filesystem within a file" directory structure logic
    implemented that lets me embed that filesystem in the file somewhere. The
    filesystem can be read-only.

    Are there any existing libraries or parts for such a job?

    Caveat: I'd prefer to avoid the complexity and installation requirements
    of a container/Docker/docker data volume.

    I'm currently thinking I may need to resort to a single external database
    file. Perhaps this file is managed by boltdb or bazil, then use LD_PRELOAD
    -> hook filesystem open(),, read(), write() filesystem calls, and redirect
    the filesystem operations that are to the "internal paths" to the
    single-file database.

    Seems like a dead-end: I could do some kind of loopback device/mount
    scheme, but I don't see how to make a loopback device point to just part
    (say the last half) of a file.

    Thanks for your thoughts!

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ziffusion at Dec 20, 2015 at 12:09 am
    This may be useful.

    https://github.com/gchaincl/gotic

    It allows you to embed resources within a go executable, which are then
    accessible as fs objects via the ioutil.ReadFile()interface.
    On Saturday, December 19, 2015 at 12:57:27 PM UTC-5, Jason E. Aten wrote:

    Given: an existing C language based system (that includes many on-disk
    files).

    Given: I'm using CGO to call that C code that expects those files on disk.

    Goal: I'd like to copy-in/compile-in all the files that would normally
    live on disk, putting them into the go executable file.

    Rationale: for easy (single executable binary, in classic Go style)
    distribution of all the required files and directories that attend the CGO
    called code.

    Constraint: Ideally the C code in the libraries wouldn't have to be
    modified in order to redirect its access to the filesystem embedded inside
    the binary.

    Q: Is there an existing "filesystem within a file" library for Go? I'd
    like OSX and linux support.

    I imagine I could use LD_PRELOAD / DYLD_INSERT_LIBRARIES to hook
    open/read/write calls and redirect them to the internal data. But then I
    still need to have a "filesystem within a file" directory structure logic
    implemented that lets me embed that filesystem in the file somewhere. The
    filesystem can be read-only.

    Are there any existing libraries or parts for such a job?

    Caveat: I'd prefer to avoid the complexity and installation requirements
    of a container/Docker/docker data volume.

    I'm currently thinking I may need to resort to a single external database
    file. Perhaps this file is managed by boltdb or bazil, then use LD_PRELOAD
    -> hook filesystem open(),, read(), write() filesystem calls, and redirect
    the filesystem operations that are to the "internal paths" to the
    single-file database.

    Seems like a dead-end: I could do some kind of loopback device/mount
    scheme, but I don't see how to make a loopback device point to just part
    (say the last half) of a file.

    Thanks for your thoughts!
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jason E. Aten at Dec 20, 2015 at 1:02 am
    Thanks guys for the suggestions. Here's what I came up with as the closest
    thing to work with:

    https://github.com/bazil/zipfs

    The zipfs demo for Tommi Virtanen's bazil.org/fuse is the closest thing
    I've found to what I had in mind. I may have to adapt it a little, but it
    is written in Go and allows legacy C code to read a zipfile that is mounted
    with fuse, and uses the Go standard library archive/zip to do the reading.
    With a little adjustment, I think this can be made to work for my purpose.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitri Shuralyov at Dec 20, 2015 at 8:45 am
    Q: Is there an existing "filesystem within a file" library for Go? I'd
    like OSX and linux support.

    I'm not sure if it'll work for your needs/constraints, but consider vfsgen:

    https://github.com/shurcooL/vfsgen

    It generates Go code that statically embeds the input virtual filesystem
    (which can be a folder on disk via http.Dir adapter). Content is gzip
    compressed (only if it makes the given file smaller), but otherwise it's a
    part of the executable code; it's not something appended to the built
    binary.

    On Saturday, December 19, 2015 at 5:03:11 PM UTC-8, Jason E. Aten wrote:

    Thanks guys for the suggestions. Here's what I came up with as the
    closest thing to work with:

    https://github.com/bazil/zipfs

    The zipfs demo for Tommi Virtanen's bazil.org/fuse is the closest thing
    I've found to what I had in mind. I may have to adapt it a little, but it
    is written in Go and allows legacy C code to read a zipfile that is mounted
    with fuse, and uses the Go standard library archive/zip to do the reading.
    With a little adjustment, I think this can be made to work for my purpose.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tamás Gulácsi at Dec 20, 2015 at 9:07 am
    Nice, but as far as I understand, the main problem is providing access to the embodied tree to the legacy C programs, as an os filesystem.

    If he can use FUSE, than he can choose from a multitude of options. If not, only extracting to a tmpfs is the way.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jason E. Aten at Dec 21, 2015 at 12:39 am
    Hi Dmitri,

    Yes, vfsgen got me thinking in this direction, it's a very innovative
    approach. I consider it inspiration.

    Tamás has it right though -- I need to supply files to non-go code as well.

    I put together a working system to solve this problem, it lets one create a
    single binary that can embed C based libraries (like R
    https://www.r-project.org/) that want to read their sub-libraries from a
    filesystem. See

    https://github.com/glycerine/libzipfs

    for the code. You can also use it quite readily to serve web media. There's
    an included utility 'libzipfs-combiner' which combines your go executable
    and a given zip archive and gives you back a file that will serve a fuse
    endpoint from the internalized zipfile.

    Enjoy!
    On Sun, Dec 20, 2015 at 1:07 AM, Tamás Gulácsi wrote:

    Nice, but as far as I understand, the main problem is providing access to
    the embodied tree to the legacy C programs, as an os filesystem.

    If he can use FUSE, than he can choose from a multitude of options. If
    not, only extracting to a tmpfs is the way.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 19, '15 at 5:57p
activeDec 21, '15 at 12:39a
posts10
users6
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase