FAQ
Hi, I am writing a package which requires some static assets (in this case
a small number of HTML template files and static javascript resources)
which I need to read at runtime (preferred) OR compile them into the binary
directly. I want the package to be easily installed and compiled, having
the assets referenced in some convenient way. Is there a pattern for this,
i.e. to distribute the package with a separate asset-directory? How would
you do?

--

Search Discussions

  • Andrew Gerrand at Nov 1, 2012 at 11:04 am
    One approach is to use the go/build package to locate the package
    directory. The Go Tour and present tool both use this technique.

    For example:
    http://code.google.com/p/go/source/browse/present/main.go?repo=talks#32

    I would like a better solution to this problem, though. This approach fails
    when the user doesn't have a Go installation.

    Andrew

    On 1 November 2012 19:30, Didrik Nordström wrote:

    Hi, I am writing a package which requires some static assets (in this case
    a small number of HTML template files and static javascript resources)
    which I need to read at runtime (preferred) OR compile them into the binary
    directly. I want the package to be easily installed and compiled, having
    the assets referenced in some convenient way. Is there a pattern for this,
    i.e. to distribute the package with a separate asset-directory? How would
    you do?

    --

    --
  • Minux at Nov 1, 2012 at 11:11 am

    On Thu, Nov 1, 2012 at 7:03 PM, Andrew Gerrand wrote:

    One approach is to use the go/build package to locate the package
    directory. The Go Tour and present tool both use this technique.

    For example:
    http://code.google.com/p/go/source/browse/present/main.go?repo=talks#32

    I would like a better solution to this problem, though. This approach
    fails when the user doesn't have a Go installation.
    you might want to review my proposal of os.Executable() (string, error) API
    here:
    https://codereview.appspot.com/6736069/

    it makes finding the real absolute path of the executable easy and portable.
    and finding asset files is one of its main use cases.

    --
  • Andrew Gerrand at Nov 1, 2012 at 11:44 am
    On Unix systems I typically put resources in a bin directory somewhere. The
    assets are rarely co-located with the binaries. So I'm not sure that
    os.Executable would help me in my typical use case.

    Maybe convention and environment variables are enough.

    On 1 November 2012 22:10, minux wrote:

    On Thu, Nov 1, 2012 at 7:03 PM, Andrew Gerrand wrote:

    One approach is to use the go/build package to locate the package
    directory. The Go Tour and present tool both use this technique.

    For example:
    http://code.google.com/p/go/source/browse/present/main.go?repo=talks#32

    I would like a better solution to this problem, though. This approach
    fails when the user doesn't have a Go installation.
    you might want to review my proposal of os.Executable() (string, error)
    API here:
    https://codereview.appspot.com/6736069/

    it makes finding the real absolute path of the executable easy and
    portable.
    and finding asset files is one of its main use cases.
    --
  • Minux at Nov 1, 2012 at 11:56 am

    On Thu, Nov 1, 2012 at 7:44 PM, Andrew Gerrand wrote:

    On Unix systems I typically put resources in a bin directory somewhere.
    The assets are rarely co-located with the binaries. So I'm not sure that
    os.Executable would help me in my typical use case.
    In fact, i think the ideal would be the go command provide a way to
    automatically pack asset files
    into the binary so that we can retain the benefit of static linking that we
    only need to deploy one file
    when it comes to asset files.

    --
  • Henrik Johansson at Nov 1, 2012 at 12:01 pm
    I agree i really like the "copy to deplo" simplicity that would bring.
    On Nov 1, 2012 12:56 PM, "minux" wrote:

    On Thu, Nov 1, 2012 at 7:44 PM, Andrew Gerrand wrote:

    On Unix systems I typically put resources in a bin directory somewhere.
    The assets are rarely co-located with the binaries. So I'm not sure that
    os.Executable would help me in my typical use case.
    In fact, i think the ideal would be the go command provide a way to
    automatically pack asset files
    into the binary so that we can retain the benefit of static linking that
    we only need to deploy one file
    when it comes to asset files.

    --

    --
  • Steve McCoy at Nov 1, 2012 at 4:15 pm

    On Thursday, November 1, 2012 7:56:34 AM UTC-4, minux wrote:
    On Thu, Nov 1, 2012 at 7:44 PM, Andrew Gerrand <a...@golang.org<javascript:>
    wrote:
    On Unix systems I typically put resources in a bin directory somewhere.
    The assets are rarely co-located with the binaries. So I'm not sure that
    os.Executable would help me in my typical use case.
    In fact, i think the ideal would be the go command provide a way to
    automatically pack asset files
    into the binary so that we can retain the benefit of static linking that
    we only need to deploy one file
    when it comes to asset files.
    This is okay for lots of things, but e.g. for a game, as either a user or
    developer, I wouldn't want every image file, audio file, etc. all inside
    the same binary blob. It would get huge and I wouldn't trust OS paging to
    keep performance acceptable. That, and it's fun and convenient to be able
    to edit assets without having to rebuild (and sometimes even without
    restarting the program) — it's okay for some things to be dynamic. On
    Windows and OSX, it's normal to keep files bundled together, and thus find
    things relative to the program location is the way to go, but on Linux &
    friends, that isn't the norm, so I'd also like to see what Andrew has
    described.

    --
  • Kevin Gillette at Nov 5, 2012 at 7:13 am
    It's a common enough problem that there are a half dozen solutions out
    there, including some that append a zip to the end of the binary.
    Standardization may be nice, but unless it covers all the use cases (_lots_
    of assets -- too big to load into mem with the prog) as well as the small
    cases (like a small website with few assets), then it will be useless to a
    lot of people.

    I think the only middle ground is a stdlib virtual filesystem interface,
    which can be configured to point to the real fs, binary-bundled assets, any
    zip file (including a zip appended to the binary), or any extendable stuff
    down the road (ssh, 9p, etc). Ideally there'd also be a file system
    unioning/overlay implementation, that takes any number of filesystems and
    overlays them in order, though that's easy enough to do in a 3rd party
    solution.
    On Thursday, November 1, 2012 5:56:34 AM UTC-6, minux wrote:


    On Thu, Nov 1, 2012 at 7:44 PM, Andrew Gerrand <a...@golang.org<javascript:>
    wrote:
    On Unix systems I typically put resources in a bin directory somewhere.
    The assets are rarely co-located with the binaries. So I'm not sure that
    os.Executable would help me in my typical use case.
    In fact, i think the ideal would be the go command provide a way to
    automatically pack asset files
    into the binary so that we can retain the benefit of static linking that
    we only need to deploy one file
    when it comes to asset files.
    --
  • Minux at Nov 5, 2012 at 7:18 am

    On Mon, Nov 5, 2012 at 3:13 PM, Kevin Gillette wrote:

    It's a common enough problem that there are a half dozen solutions out
    there, including some that append a zip to the end of the binary.
    Standardization may be nice, but unless it covers all the use cases (_lots_
    of assets -- too big to load into mem with the prog) as well as the small
    cases (like a small website with few assets), then it will be useless to a
    lot of people.
    these all depend on knowing the path to the executable. However, my attempt
    to bring that
    functionality (CL 6736069) has been reject, so it has to be migrated to a
    3rd-party package.
    I think the only middle ground is a stdlib virtual filesystem interface,
    which can be configured to point to the real fs, binary-bundled assets, any
    zip file (including a zip appended to the binary), or any extendable stuff
    down the road (ssh, 9p, etc). Ideally there'd also be a file system
    unioning/overlay implementation, that takes any number of filesystems and
    overlays them in order, though that's easy enough to do in a 3rd party
    solution.
    the stdlib does have a VFS interface, namely net/http.FileSystem.
    also, godoc has a nice VFS abstraction, and it already supports reading
    file from real FS or
    inside a zip, i think that could be a good starting point.

    --
  • Jeremy Wohl at Nov 1, 2012 at 5:19 pm
    From the perspective of large web sites, where many separate (and rapidly
    churning) projects might form the namespace, there are lots of tools and
    much thinking around colocating assets with its software. This is a sane
    practice for many.

    As for bundling within the binary, I could see that working in small
    projects, but files get too big and there are runtime updates to consider.

    I'd be really happy to see this function get committed.
    On Thursday, November 1, 2012 4:44:51 AM UTC-7, Andrew Gerrand wrote:

    On Unix systems I typically put resources in a bin directory somewhere.
    The assets are rarely co-located with the binaries. So I'm not sure that
    os.Executable would help me in my typical use case.

    Maybe convention and environment variables are enough.

    On 1 November 2012 22:10, minux <minu...@gmail.com <javascript:>> wrote:


    On Thu, Nov 1, 2012 at 7:03 PM, Andrew Gerrand <a...@golang.org<javascript:>
    wrote:
    One approach is to use the go/build package to locate the package
    directory. The Go Tour and present tool both use this technique.

    For example:

    http://code.google.com/p/go/source/browse/present/main.go?repo=talks#32

    I would like a better solution to this problem, though. This approach
    fails when the user doesn't have a Go installation.
    you might want to review my proposal of os.Executable() (string, error)
    API here:
    https://codereview.appspot.com/6736069/

    it makes finding the real absolute path of the executable easy and
    portable.
    and finding asset files is one of its main use cases.
    --
  • Bryanturley at Nov 1, 2012 at 5:44 pm
    I think that large quantities of data that could be loaded at runtime
    should not be part of your executables.
    This isn't currently a problem with other languages is it?

    Though it would be nice to know where your current program lives on the
    filesystem if that is not already possible.

    If you are having trouble with this perhaps use .deb/.rpm or create
    something like apples .dmg?


    --
  • Nate Finch at Nov 1, 2012 at 12:19 pm
    It sounds like you have to hack it for now, which probably requires an
    extra pre-compile step (assuming you want to maintain the static assets as
    standalone files in source control).
    1.) write some code to generate .go files with the contents of your static
    files as const strings.
    2.) write some code to deploy the files by unzipping the const strings and
    writing to disk at runtime.

    Then compile time looks like this:

    1.) Run code generator to create generated .go asset files
    2.) Compile main code to create .exe (which now includes the aforementioned
    .go files)

    Then, when you run the .exe, it will deploy the asset files as necessary.

    A bit fugly, but it'll work until something official comes around.
    On Thursday, November 1, 2012 4:30:41 AM UTC-4, Didrik Nordström wrote:

    Hi, I am writing a package which requires some static assets (in this case
    a small number of HTML template files and static javascript resources)
    which I need to read at runtime (preferred) OR compile them into the binary
    directly. I want the package to be easily installed and compiled, having
    the assets referenced in some convenient way. Is there a pattern for this,
    i.e. to distribute the package with a separate asset-directory? How would
    you do?
    --
  • Jim Teeuwen at Nov 1, 2012 at 1:40 pm
    I prefer embedding binary files in my program. This plays well with Go's
    static linking phylosophy, in that the single binary is entirely self
    contained. Easy to deploy.

    I wrote go-bindata for this purpose: https://github.com/jteeuwen/go-bindata

    As Nate indicated, it requires an extra pre-compile step to generate the
    .go files from the resource files.
    Since resource files rarely change, this doesn't have to happen all the
    time though.
    On Thursday, November 1, 2012 1:19:08 PM UTC+1, Nate Finch wrote:

    It sounds like you have to hack it for now, which probably requires an
    extra pre-compile step (assuming you want to maintain the static assets as
    standalone files in source control).
    1.) write some code to generate .go files with the contents of your static
    files as const strings.
    2.) write some code to deploy the files by unzipping the const strings and
    writing to disk at runtime.

    Then compile time looks like this:

    1.) Run code generator to create generated .go asset files
    2.) Compile main code to create .exe (which now includes the
    aforementioned .go files)

    Then, when you run the .exe, it will deploy the asset files as necessary.

    A bit fugly, but it'll work until something official comes around.
    On Thursday, November 1, 2012 4:30:41 AM UTC-4, Didrik Nordström wrote:

    Hi, I am writing a package which requires some static assets (in this
    case a small number of HTML template files and static javascript resources)
    which I need to read at runtime (preferred) OR compile them into the binary
    directly. I want the package to be easily installed and compiled, having
    the assets referenced in some convenient way. Is there a pattern for this,
    i.e. to distribute the package with a separate asset-directory? How would
    you do?
    --
  • Martin Bruse at Nov 1, 2012 at 3:46 pm
    I too would love to have this functionality in the go build tool. The
    simplicity of deployment would be awesome to behold.
    On Thursday, November 1, 2012 6:40:54 AM UTC-7, Jim Teeuwen wrote:

    I prefer embedding binary files in my program. This plays well with Go's
    static linking phylosophy, in that the single binary is entirely self
    contained. Easy to deploy.

    I wrote go-bindata for this purpose:
    https://github.com/jteeuwen/go-bindata

    As Nate indicated, it requires an extra pre-compile step to generate the
    .go files from the resource files.
    Since resource files rarely change, this doesn't have to happen all the
    time though.
    On Thursday, November 1, 2012 1:19:08 PM UTC+1, Nate Finch wrote:

    It sounds like you have to hack it for now, which probably requires an
    extra pre-compile step (assuming you want to maintain the static assets as
    standalone files in source control).
    1.) write some code to generate .go files with the contents of your
    static files as const strings.
    2.) write some code to deploy the files by unzipping the const strings
    and writing to disk at runtime.

    Then compile time looks like this:

    1.) Run code generator to create generated .go asset files
    2.) Compile main code to create .exe (which now includes the
    aforementioned .go files)

    Then, when you run the .exe, it will deploy the asset files as necessary.

    A bit fugly, but it'll work until something official comes around.
    On Thursday, November 1, 2012 4:30:41 AM UTC-4, Didrik Nordström wrote:

    Hi, I am writing a package which requires some static assets (in this
    case a small number of HTML template files and static javascript resources)
    which I need to read at runtime (preferred) OR compile them into the binary
    directly. I want the package to be easily installed and compiled, having
    the assets referenced in some convenient way. Is there a pattern for this,
    i.e. to distribute the package with a separate asset-directory? How would
    you do?
    --
  • Stevewang at Nov 1, 2012 at 3:57 pm
    *mahonia does it the same way.*
    http://code.google.com/p/*mahonia*/

    On Thursday, November 1, 2012 9:40:54 PM UTC+8, Jim Teeuwen wrote:

    I prefer embedding binary files in my program. This plays well with Go's
    static linking phylosophy, in that the single binary is entirely self
    contained. Easy to deploy.

    I wrote go-bindata for this purpose:
    https://github.com/jteeuwen/go-bindata

    As Nate indicated, it requires an extra pre-compile step to generate the
    .go files from the resource files.
    Since resource files rarely change, this doesn't have to happen all the
    time though.
    On Thursday, November 1, 2012 1:19:08 PM UTC+1, Nate Finch wrote:

    It sounds like you have to hack it for now, which probably requires an
    extra pre-compile step (assuming you want to maintain the static assets as
    standalone files in source control).
    1.) write some code to generate .go files with the contents of your
    static files as const strings.
    2.) write some code to deploy the files by unzipping the const strings
    and writing to disk at runtime.

    Then compile time looks like this:

    1.) Run code generator to create generated .go asset files
    2.) Compile main code to create .exe (which now includes the
    aforementioned .go files)

    Then, when you run the .exe, it will deploy the asset files as necessary.

    A bit fugly, but it'll work until something official comes around.
    On Thursday, November 1, 2012 4:30:41 AM UTC-4, Didrik Nordström wrote:

    Hi, I am writing a package which requires some static assets (in this
    case a small number of HTML template files and static javascript resources)
    which I need to read at runtime (preferred) OR compile them into the binary
    directly. I want the package to be easily installed and compiled, having
    the assets referenced in some convenient way. Is there a pattern for this,
    i.e. to distribute the package with a separate asset-directory? How would
    you do?
    --
  • Didrik Nordström at Nov 1, 2012 at 4:58 pm
    Thanks for the replies, I appreciate it a lot. I understand there are a
    couple of different solutions to this issue, and for this project in
    particular. I have tried to sum the options:

    1. *Use go/build to locate package dir:* fails, as mentioned, because
    when a developer builds a program using my package, and distributes it, the
    user won't have the assets
    2. *Compile asset files into the executable:* could work for this
    project, but fails for bigger projects, ugly, takes up unnecessary memory
    3. *Distribute asset directory along with the package and executables:*requires user setup

    I think I will go with option 3. I think I can live with the setup cost for
    the moment. Option 2 could also work for me since it's a small amount of
    files, if there is a clean way of doing it.

    Thanks again for the response
    On Thursday, November 1, 2012 7:30:41 PM UTC+11, Didrik Nordström wrote:

    Hi, I am writing a package which requires some static assets (in this case
    a small number of HTML template files and static javascript resources)
    which I need to read at runtime (preferred) OR compile them into the binary
    directly. I want the package to be easily installed and compiled, having
    the assets referenced in some convenient way. Is there a pattern for this,
    i.e. to distribute the package with a separate asset-directory? How would
    you do?
    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 1, '12 at 10:52a
activeNov 5, '12 at 7:18a
posts16
users12
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase