FAQ
Gophers,

One of the most frequent questions <http://golang.org/doc/faq#get_version>
we’ve received since Go 1 was how to deal with dependencies and their
versions. We’ve never recommended any particular answer.

In Google’s internal source tree, we vendor (copy) all our dependencies
into our source tree and have at most one copy of any given external
library. We have the equivalent of only one GOPATH and rewrite our imports
to refer to our vendored copy. For example, Go code inside Google wanting
to use “golang.org/x/crypto/openpgp” would instead import it as something
like “google/third_party/golang.org/x/crypto/openpgp”. This has worked out
very well for us and we get reproducible builds. If we ever want to update
openpgp in our vendored directory, we update it, verify that all the
affected code in the world still compiles and tests pass (making code
changes as necessary), and then check it in. The world compiles and passes
at all points in the version control system’s history, and if we bisect in
time, we see which version of an external library was in use at any point.

We’re starting to see others in the Go community do the same, with tools
like godep <https://github.com/tools/godep> and nut
<https://github.com/jingweno/nut>.

We think it’s time to start addressing the dependency & vendoring issue,
especially before too many conflicting tools arise and fragment best
practices in the Go ecosystem, unnecessarily complicating tooling. It would
be nice if the community could converge on a standard way to vendor.

Our proposal is that the Go project,


    1.

    officially recommends vendoring into an “internal
    <https://docs.google.com/a/golang.org/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit>”
    directory with import rewriting (not GOPATH modifications) as the canonical
    way to pin dependencies.
    2.

    defines a common config file format for dependencies & vendoring
    3.

    makes no code changes to cmd/go in Go 1.5. External tools such as
    “godep” or “nut” will implement 1) and 2). We can reevaluate including such
    a tool in Go 1.6+.


The important part is that as a community, we all do this the same way, so
tooling can mature and interoperate.

In Go 1.5, the “internal” package mechanism introduced in Go 1.4 for the
standard library will be extended to all go-gettable packages, so using the
“internal” directory as the root of rewritten import paths makes sense (as
opposed to “vendor” or “third_party”).

Consider an existing use of vendoring in the Go source tree:
$GOROOT/src/cmd/internal currently contains copies of “rsc.io/x86/x86asm”
and “rsc.io/arm/armasm” as $GOROOT/src/cmd/internal/rsc.io/x86/x86asm/ and
$GOROOT/src/cmd/internal/rsc.io/arm/armasm/, respectively. When we use
those inside the Go tools, however, we import them “cmd/internal/
rsc.io/x86/x86asm” and not “rsc.io/x86/x86asm”. (Although this example
comes from the Go distribution repo, the effect is the same as a local
project using $GOPATH/src/your.project/path instead of $GOROOT/src/cmd.)

We currently maintain those copies by hand. Instead, we want to write a
file (filename and syntax to be determined), such as:

      src/cmd/internal/TBDCONFIG.CFG:

              “rsc.io/x86/x86asm” with revision af2970a7819d

              “rsc.io/arm/armasm” with revision 616aea947362

And then your vendoring tool (such as “godep” or “nut”) would read
TBDCONFIG.CFG and write out,

      src/cmd/internal/rsc.io/x86/x86asm/*.go

      src/cmd/internal/rsc.io/arm/armasm/*.go

rewriting imports and import comments in these files for the new location.
It may also optionally change your source so any occurrence of

     import “rsc.io/x86/x86asm”

becomes

     import “cmd/internal/rsc.io/x86/x86asm”

The vendoring tool would be responsible for generating errors on conflicts
or missing dependencies.

The thing that we as a community need to figure out is the recommended
configuration file format.

We’d prefer something that the Go standard library can already parse
easily. That includes XML, JSON, and Go. Nobody likes XML, so that leaves
JSON and Go.

godep already uses JSON, as do Go tools themselves (e.g. “go list -json
fmt”), so we’ll probably want something like godep’s JSON format:

{

    "Deps": [

        {

            "ImportPath": "rsc.io/arm/armasm",

            "Rev": "616aea947362"

        },

        {

            "ImportPath": "rsc.io/x86/x86asm",

            "Rev": "af2970a7819d"

        }

    ]

}

We can start with that for discussion.

Note that we have rejected non-vendoring approaches that require
modifications to GOPATH or new semantics inside the go command and
toolchain. We believe it is important that the solution not require
additional effort on the part of all the tools that already understand how
to build, analyze, or modify code in the standard GOPATH hierarchy.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Aram Hăvărneanu at Mar 2, 2015 at 5:47 pm
    One of the nicest things about Go is that all the information required
    to build Go programs is in the source code, a.i. in .go files, and not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).

    --
    Aram Hăvărneanu

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Cmang at Mar 2, 2015 at 5:51 pm

    On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:
    One of the nicest things about Go is that all the information required
    to build Go programs is in the source code, a.i. in .go files, and not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).
    As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or
    TBDCONFIG.json). I'm not sure if that alleviates the problem of having
    "other files" but the other files can be other Go files.

    --
    Aram Hăvărneanu
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Kamil Kisiel at Mar 3, 2015 at 1:26 am
    I like the idea of having it be a .go file. It could be required to have a
    specific name, and would import a package which has an exported struct type
    used to describe a dependency. Over time when the requirements evolve, more
    fields could be added to the struct type. The syntax is familiar and we get
    strict checking for free via the compiler.

    eg:

    // TDBCONFIG.go
    import "deps"

    var Dependencies = []deps.D{
         {Path: "rsc.io/arm/armasm", Rev: "616aea947362"},
    }


    The dependency tool could either compile it or read the information using
    the ast package.
    On Monday, March 2, 2015 at 9:51:30 AM UTC-8, cm...@golang.org wrote:


    On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:

    One of the nicest things about Go is that all the information required
    to build Go programs is in the source code, a.i. in .go files, and not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).
    As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or
    TBDCONFIG.json). I'm not sure if that alleviates the problem of having
    "other files" but the other files can be other Go files.

    --
    Aram Hăvărneanu
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michaeljs1990 at Mar 3, 2015 at 4:14 am
    I am not sure what you are aiming for when saving the output requiring
    dependencies to the Dependencies variable. In the original post versioned
    dependencies have their own path in the vendored folder so implementing an
    entire new stdlib seems like overkill. This also allows for some odd things
    like being able to manage deps in any go file which could get very
    confusing and I see it being abused. Looking at the following JSON object I
    think this would be a much cleaner way. I can immediately tell by looking
    at this that if i want to import armasm into the project i am working on i
    would use vendor/folder/rsc.io/arm/armasm.

    {

    "destination": "vendor/folder/",

        "deps": [

            {

                "ImportPath": "rsc.io/arm/armasm",

                "Rev": "616aea947362"

            },

            {

                "ImportPath": "rsc.io/x86/x86asm",

                "Rev": "af2970a7819d"

            }

        ]

    }
    On Monday, March 2, 2015 at 8:26:05 PM UTC-5, Kamil Kisiel wrote:

    I like the idea of having it be a .go file. It could be required to have a
    specific name, and would import a package which has an exported struct type
    used to describe a dependency. Over time when the requirements evolve, more
    fields could be added to the struct type. The syntax is familiar and we get
    strict checking for free via the compiler.

    eg:

    // TDBCONFIG.go
    import "deps"

    var Dependencies = []deps.D{
    {Path: "rsc.io/arm/armasm", Rev: "616aea947362"},
    }


    The dependency tool could either compile it or read the information using
    the ast package.
    On Monday, March 2, 2015 at 9:51:30 AM UTC-8, cm...@golang.org wrote:


    On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:

    One of the nicest things about Go is that all the information required
    to build Go programs is in the source code, a.i. in .go files, and not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).
    As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or
    TBDCONFIG.json). I'm not sure if that alleviates the problem of having
    "other files" but the other files can be other Go files.

    --
    Aram Hăvărneanu
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Kamil Kisiel at Mar 3, 2015 at 5:36 am
    I think you misunderstood, I wasn't suggesting that you'd be able to have
    dependencies in any .go file, just the one specially named for
    configuration.
    Your dependencies would still ultimately go in to their own vendored path.
    I'm only discussing the format of the version definitions, using standard
    go code as an alternative to JSON syntax. I don't think the JSON is any
    cleaner, it's probably about on par with Go's syntax as far as verbosity
    goes. The advantage of using Go syntax is that it's a bit easier to
    validate via the languages type system. Maybe that's not compelling enough
    and/or parsing would be too complicated compared to JSON, that's why I'm
    asking what people think.

    I'm not sure what you mean by "implementing an entire new stdlib".
    On Monday, March 2, 2015 at 8:14:21 PM UTC-8, michae...@gmail.com wrote:

    I am not sure what you are aiming for when saving the output requiring
    dependencies to the Dependencies variable. In the original post versioned
    dependencies have their own path in the vendored folder so implementing an
    entire new stdlib seems like overkill. This also allows for some odd things
    like being able to manage deps in any go file which could get very
    confusing and I see it being abused. Looking at the following JSON object I
    think this would be a much cleaner way. I can immediately tell by looking
    at this that if i want to import armasm into the project i am working on i
    would use vendor/folder/rsc.io/arm/armasm.

    {

    "destination": "vendor/folder/",

    "deps": [

    {

    "ImportPath": "rsc.io/arm/armasm",

    "Rev": "616aea947362"

    },

    {

    "ImportPath": "rsc.io/x86/x86asm",

    "Rev": "af2970a7819d"

    }

    ]

    }
    On Monday, March 2, 2015 at 8:26:05 PM UTC-5, Kamil Kisiel wrote:

    I like the idea of having it be a .go file. It could be required to have
    a specific name, and would import a package which has an exported struct
    type used to describe a dependency. Over time when the requirements evolve,
    more fields could be added to the struct type. The syntax is familiar and
    we get strict checking for free via the compiler.

    eg:

    // TDBCONFIG.go
    import "deps"

    var Dependencies = []deps.D{
    {Path: "rsc.io/arm/armasm", Rev: "616aea947362"},
    }


    The dependency tool could either compile it or read the information using
    the ast package.
    On Monday, March 2, 2015 at 9:51:30 AM UTC-8, cm...@golang.org wrote:


    On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:

    One of the nicest things about Go is that all the information required
    to build Go programs is in the source code, a.i. in .go files, and not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).
    As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or
    TBDCONFIG.json). I'm not sure if that alleviates the problem of having
    "other files" but the other files can be other Go files.

    --
    Aram Hăvărneanu
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Dmitri Shuralyov at Mar 3, 2015 at 6:59 am
    I have one observation I'd like to share, it's one of many aspects that
    should be considered.

    The examples in discussion above seem to be all using *import paths* as
    keys.

    Perhaps it's better to allow *import path patterns* (as defined at
    https://golang.org/cmd/go/#hdr-Description_of_package_lists), since they
    can represent a VCS repository in one line. It's less likely you'd use
    different revisions for packages within one VCS repo.

    So, instead of...

    "rsc.io/arm/armasm": "616aea947362""rsc.io/x86/x86asm": "af2970a7819d"
    "github.com/gregjones/httpcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09""github.com/gregjones/httpcache/diskcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09""github.com/gregjones/httpcache/memcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09"


    It could be:

    "rsc.io/arm/armasm/...": "616aea947362"
    "rsc.io/x86/x86asm/...": "af2970a7819d"
    "github.com/gregjones/httpcache/...": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09"


    All I ask is you consider this point (that a single repository may contain
    more than one Go package) and see what works better.

    (Also note that normal import paths are valid import path patterns too, so
    it's backwards compatible.)

    On Monday, March 2, 2015 at 9:36:41 PM UTC-8, Kamil Kisiel wrote:

    I think you misunderstood, I wasn't suggesting that you'd be able to have
    dependencies in any .go file, just the one specially named for
    configuration.
    Your dependencies would still ultimately go in to their own vendored path.
    I'm only discussing the format of the version definitions, using standard
    go code as an alternative to JSON syntax. I don't think the JSON is any
    cleaner, it's probably about on par with Go's syntax as far as verbosity
    goes. The advantage of using Go syntax is that it's a bit easier to
    validate via the languages type system. Maybe that's not compelling enough
    and/or parsing would be too complicated compared to JSON, that's why I'm
    asking what people think.

    I'm not sure what you mean by "implementing an entire new stdlib".
    On Monday, March 2, 2015 at 8:14:21 PM UTC-8, michae...@gmail.com wrote:

    I am not sure what you are aiming for when saving the output requiring
    dependencies to the Dependencies variable. In the original post versioned
    dependencies have their own path in the vendored folder so implementing an
    entire new stdlib seems like overkill. This also allows for some odd things
    like being able to manage deps in any go file which could get very
    confusing and I see it being abused. Looking at the following JSON object I
    think this would be a much cleaner way. I can immediately tell by looking
    at this that if i want to import armasm into the project i am working on i
    would use vendor/folder/rsc.io/arm/armasm.

    {

    "destination": "vendor/folder/",

    "deps": [

    {

    "ImportPath": "rsc.io/arm/armasm",

    "Rev": "616aea947362"

    },

    {

    "ImportPath": "rsc.io/x86/x86asm",

    "Rev": "af2970a7819d"

    }

    ]

    }
    On Monday, March 2, 2015 at 8:26:05 PM UTC-5, Kamil Kisiel wrote:

    I like the idea of having it be a .go file. It could be required to have
    a specific name, and would import a package which has an exported struct
    type used to describe a dependency. Over time when the requirements evolve,
    more fields could be added to the struct type. The syntax is familiar and
    we get strict checking for free via the compiler.

    eg:

    // TDBCONFIG.go
    import "deps"

    var Dependencies = []deps.D{
    {Path: "rsc.io/arm/armasm", Rev: "616aea947362"},
    }


    The dependency tool could either compile it or read the information
    using the ast package.
    On Monday, March 2, 2015 at 9:51:30 AM UTC-8, cm...@golang.org wrote:


    On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:

    One of the nicest things about Go is that all the information required
    to build Go programs is in the source code, a.i. in .go files, and not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).
    As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or
    TBDCONFIG.json). I'm not sure if that alleviates the problem of having
    "other files" but the other files can be other Go files.

    --
    Aram Hăvărneanu
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nathan Youngman at Mar 4, 2015 at 7:17 am
    It is a good point that the SHA commit hashes being stored are not of the
    packages being imported, but of the repositories containing them.

    If this file is generated by a command like "godep save", it may not matter
    much, because all the commit hashes will be the same for a package in a
    given repository.

    Rather than using path patterns, a tool could just always do store the
    repository paths rather than full import paths. A savings of 4 characters!
    :-)

    Nathan.

    On 2 March 2015 at 23:59, Dmitri Shuralyov wrote:

    I have one observation I'd like to share, it's one of many aspects that
    should be considered.

    The examples in discussion above seem to be all using *import paths* as
    keys.

    Perhaps it's better to allow *import path patterns* (as defined at
    https://golang.org/cmd/go/#hdr-Description_of_package_lists), since they
    can represent a VCS repository in one line. It's less likely you'd use
    different revisions for packages within one VCS repo.

    So, instead of...

    "rsc.io/arm/armasm": "616aea947362""rsc.io/x86/x86asm": "af2970a7819d"
    "github.com/gregjones/httpcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09""github.com/gregjones/httpcache/diskcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09""github.com/gregjones/httpcache/memcache": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09"


    It could be:

    "rsc.io/arm/armasm/...": "616aea947362"
    "rsc.io/x86/x86asm/...": "af2970a7819d"
    "github.com/gregjones/httpcache/...": "4fb00a8eec008ffa8ac9ec46a93684dcbb62fc09"


    All I ask is you consider this point (that a single repository may contain
    more than one Go package) and see what works better.

    (Also note that normal import paths are valid import path patterns too, so
    it's backwards compatible.)

    On Monday, March 2, 2015 at 9:36:41 PM UTC-8, Kamil Kisiel wrote:

    I think you misunderstood, I wasn't suggesting that you'd be able to have
    dependencies in any .go file, just the one specially named for
    configuration.
    Your dependencies would still ultimately go in to their own vendored
    path. I'm only discussing the format of the version definitions, using
    standard go code as an alternative to JSON syntax. I don't think the JSON
    is any cleaner, it's probably about on par with Go's syntax as far as
    verbosity goes. The advantage of using Go syntax is that it's a bit easier
    to validate via the languages type system. Maybe that's not compelling
    enough and/or parsing would be too complicated compared to JSON, that's why
    I'm asking what people think.

    I'm not sure what you mean by "implementing an entire new stdlib".
    On Monday, March 2, 2015 at 8:14:21 PM UTC-8, michae...@gmail.com wrote:

    I am not sure what you are aiming for when saving the output requiring
    dependencies to the Dependencies variable. In the original post versioned
    dependencies have their own path in the vendored folder so implementing an
    entire new stdlib seems like overkill. This also allows for some odd things
    like being able to manage deps in any go file which could get very
    confusing and I see it being abused. Looking at the following JSON object I
    think this would be a much cleaner way. I can immediately tell by looking
    at this that if i want to import armasm into the project i am working on i
    would use vendor/folder/rsc.io/arm/armasm.

    {

    "destination": "vendor/folder/",

    "deps": [

    {

    "ImportPath": "rsc.io/arm/armasm",

    "Rev": "616aea947362"

    },

    {

    "ImportPath": "rsc.io/x86/x86asm",

    "Rev": "af2970a7819d"

    }

    ]

    }
    On Monday, March 2, 2015 at 8:26:05 PM UTC-5, Kamil Kisiel wrote:

    I like the idea of having it be a .go file. It could be required to
    have a specific name, and would import a package which has an exported
    struct type used to describe a dependency. Over time when the requirements
    evolve, more fields could be added to the struct type. The syntax is
    familiar and we get strict checking for free via the compiler.

    eg:

    // TDBCONFIG.go
    import "deps"

    var Dependencies = []deps.D{
    {Path: "rsc.io/arm/armasm", Rev: "616aea947362"},
    }


    The dependency tool could either compile it or read the information
    using the ast package.
    On Monday, March 2, 2015 at 9:51:30 AM UTC-8, cm...@golang.org wrote:


    On Monday, March 2, 2015 at 9:47:03 AM UTC-8, Aram Hăvărneanu wrote:

    One of the nicest things about Go is that all the information
    required
    to build Go programs is in the source code, a.i. in .go files, and
    not
    in any other files (Makefiles, .cfg files, etc). With this proposal,
    this would change. We'd have information in non-Go files
    (TBDCONFIG.CFG).
    As noted in the OP, TBDCONFIG.CFG could be TBDCONFIG.go (or
    TBDCONFIG.json). I'm not sure if that alleviates the problem of having
    "other files" but the other files can be other Go files.

    --
    Aram Hăvărneanu
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    Nathan Youngman
    Email: hello@nathany.com
    Web: http://www.nathany.com

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Mediocregopher at Mar 3, 2015 at 3:24 pm
    I like the idea of using a .go file as opposed to a .json file for this. Another nice property of a .go file which wasn't mentioned is that it can be commented, something which could definitely be useful if you want to inform future generations why you're using a particular commit or branch.

    One negative which we ought to address though: it's possible (though not terribly likely) that someone's project already exports a variable or type of the same name that we choose here, which would mean this would require changes to their existing package

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brad Fitzpatrick at Mar 3, 2015 at 3:26 pm

    On Tue, Mar 3, 2015 at 6:09 AM, wrote:

    I like the idea of using a .go file as opposed to a .json file for this.
    Another nice property of a .go file which wasn't mentioned is that it can
    be commented, something which could definitely be useful if you want to
    inform future generations why you're using a particular commit or branch.
    Good point. That's probably the strongest argument in favor of Go instead
    of JSON. I'm always annoyed by the lack of comments in JSON.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Russ Cox at Mar 3, 2015 at 3:31 pm
    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
         // These are for the disassemblers.
         "rsc.io/arm/armasm" // 616aea947362
         "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ian Davis at Mar 3, 2015 at 4:03 pm

    On Tue, Mar 3, 2015, at 03:31 PM, Russ Cox wrote:
    I like the idea of using a Go file, but note that it cannot be one
    that actually builds, because it needs to record the original import
    paths, not the vendored ones. I would expect that a Go config file
    would have a standard .go suffix (so that gofmt etc apply) but a build
    tag keeping it from building, a standard name (say, vendor.go), and a
    required package name (say, vendor):

    ---
    // +build ignore

    package vendor

    import ( // These are for the disassemblers. "rsc.io/arm/armasm" //
    616aea947362 "rsc.io/x86/x86asm" // af2970a7819d )
    Could use the main package if we wanted to restrict vendoring only to
    applications rather than packages.

    Ian

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Ian Davis at Mar 3, 2015 at 4:15 pm

    On Tue, Mar 3, 2015, at 03:31 PM, Russ Cox wrote:
    I like the idea of using a Go file, but note that it cannot be one
    that actually builds, because it needs to record the original import
    paths, not the vendored ones. I would expect that a Go config file
    would have a standard .go suffix (so that gofmt etc apply) but a build
    tag keeping it from building, a standard name (say, vendor.go), and a
    required package name (say, vendor):

    ---
    // +build ignore

    package vendor

    import ( // These are for the disassemblers. "rsc.io/arm/armasm" //
    616aea947362 "rsc.io/x86/x86asm" // af2970a7819d )
    ---
    I suspect go generate could be used for vendoring.

    //go:generate vendor rsc.io/arm/armasm 616aea947362 //go:generate vendor
    rsc.io/x86/x86asm[1] af2970a7819d

    Running go generate would check out those package versions, copy the
    source below an internal package and rewrite imports.

    Ian














    Links:

       1. http://rsc.io/arm/armasm

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giacomo Tartari at Mar 3, 2015 at 4:44 pm
    Has this bee proposed and ruled out already?

    package main

    import (
         //go:vendor 616aea947362
         "rsc.io/arm/armasm"

         //go:vendor af2970a7819d
         "rsc.io/x86/x86asm"
    )

    The imports reordering tools need to keep track of the "vendoring comment".


    Giacomo
    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Schuett at Mar 3, 2015 at 4:49 pm
    I feel like allowing every import to specify a version ends up with people
    in "version hell". You would also have to specify the version everytime you
    wanted to use that package. If you changed one would an error be thrown
    until you changed it in the various other places that you are specifying
    the old version?
    On Tue, Mar 3, 2015 at 11:44 AM, wrote:

    Has this bee proposed and ruled out already?

    package main

    import (
    //go:vendor 616aea947362
    "rsc.io/arm/armasm"

    //go:vendor af2970a7819d
    "rsc.io/x86/x86asm"
    )

    The imports reordering tools need to keep track of the "vendoring comment".


    Giacomo

    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giacomo Tartari at Mar 3, 2015 at 4:59 pm

    On 3Mar, 2015, at 17:49 , Michael Schuett wrote:

    I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version?
    I would assume so, but that is another easily automated task.
    Could even be made interactive:
    $go vendor
    $go vendor found mismatching version for package X: v1, v2, v3.
    do you wish to unify the versions or something? y,n
    version [1], [2], [3]?


    And it would be cool it tags are supported
      //go:vendor rev:616aea947362
      //go:vendor tag:616aea947362
    Tags are easier to read.

    Giacomo
    On Tue, Mar 3, 2015 at 11:44 AM, wrote:
    Has this bee proposed and ruled out already?

    package main

    import (
    //go:vendor 616aea947362
    "rsc.io/arm/armasm <http://rsc.io/arm/armasm>"

    //go:vendor af2970a7819d
    "rsc.io/x86/x86asm <http://rsc.io/x86/x86asm>"
    )

    The imports reordering tools need to keep track of the "vendoring comment".


    Giacomo


    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:
    I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm <http://rsc.io/arm/armasm>" // 616aea947362
    "rsc.io/x86/x86asm <http://rsc.io/x86/x86asm>" // af2970a7819d
    )
    ---

    Russ

    --
    You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe <https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe>.
    To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giacomo Tartari at Mar 3, 2015 at 5:01 pm

    On Tuesday, 3 March 2015 17:59:49 UTC+1, Giacomo Tartari wrote:

    On 3Mar, 2015, at 17:49 , Michael Schuett wrote:

    I feel like allowing every import to specify a version ends up with people
    in "version hell". You would also have to specify the version everytime you
    wanted to use that package. If you changed one would an error be thrown
    until you changed it in the various other places that you are specifying
    the old version?

    I would assume so, but that is another easily automated task.
    Could even be made interactive:
    $go vendor
    $go vendor found mismatching version for package X: v1, v2, v3.
    do you wish to unify the versions or something? y,n
    version [1], [2], [3]?


    And it would be cool it tags are supported
    //go:vendor rev:616aea947362
    //go:vendor tag:616aea947362
    this was supposed to be //go:vendor tag:v1.0
    I need more coffe/sleep, sorry.

    Tags are easier to read.
    Giacomo

    On Tue, Mar 3, 2015 at 11:44 AM, wrote:

    Has this bee proposed and ruled out already?

    package main

    import (
    //go:vendor 616aea947362
    "rsc.io/arm/armasm"

    //go:vendor af2970a7819d
    "rsc.io/x86/x86asm"
    )

    The imports reordering tools need to keep track of the "vendoring
    comment".


    Giacomo

    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Schuett at Mar 3, 2015 at 5:15 pm
    Tags would be cool but I don't really know how many packages support them I
    am guessing a very small amount since go get works off master. Some of the
    larger projects do which may be reason of enough for tag support but commit
    tag support is really all I would be looking for and would simplify things.
    But I may bias I like the simplicity of one json or go file in the root of
    the projects that manages all the dependencies I think it would also be
    familiar with people coming from pretty much any other language not that we
    need to cater to them. I am leaning towards a .json file because the
    vendoring file should not be considered to be part of your go project it's
    just installing something that you will be using in your project.

    just my 2c.
    On Tue, Mar 3, 2015 at 12:01 PM, wrote:


    On Tuesday, 3 March 2015 17:59:49 UTC+1, Giacomo Tartari wrote:


    On 3Mar, 2015, at 17:49 , Michael Schuett <michaeljs1990@gmail.com>
    wrote:

    I feel like allowing every import to specify a version ends up with
    people in "version hell". You would also have to specify the version
    everytime you wanted to use that package. If you changed one would an error
    be thrown until you changed it in the various other places that you are
    specifying the old version?

    I would assume so, but that is another easily automated task.
    Could even be made interactive:
    $go vendor
    $go vendor found mismatching version for package X: v1, v2, v3.
    do you wish to unify the versions or something? y,n
    version [1], [2], [3]?


    And it would be cool it tags are supported
    //go:vendor rev:616aea947362
    //go:vendor tag:616aea947362
    this was supposed to be //go:vendor tag:v1.0
    I need more coffe/sleep, sorry.

    Tags are easier to read.
    Giacomo

    On Tue, Mar 3, 2015 at 11:44 AM, wrote:

    Has this bee proposed and ruled out already?

    package main

    import (
    //go:vendor 616aea947362
    "rsc.io/arm/armasm"

    //go:vendor af2970a7819d
    "rsc.io/x86/x86asm"
    )

    The imports reordering tools need to keep track of the "vendoring
    comment".


    Giacomo

    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/
    topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giacomo Tartari at Mar 3, 2015 at 5:29 pm
    I don’t see tags as more complicated, to the contrary they would make it easier to distinguish form major versions(and backward incompatibilities):

    //go:vendor rev:af2970a7819d
    "rsc.io/x86/x86asm”

    vs

    //go:vendor tag:v2.1
    "rsc.io/x86/x86asm”

    As for the json file, I prefer to keep it to go.
    Eventually a json file will be filled up with other configurations for other incompatible third party tools and stuff will break.
    Keeping it in control of the go team is a more stable choice imho.

    Giacomo
    On 3Mar, 2015, at 18:15 , Michael Schuett wrote:

    Tags would be cool but I don't really know how many packages support them I am guessing a very small amount since go get works off master. Some of the larger projects do which may be reason of enough for tag support but commit tag support is really all I would be looking for and would simplify things. But I may bias I like the simplicity of one json or go file in the root of the projects that manages all the dependencies I think it would also be familiar with people coming from pretty much any other language not that we need to cater to them. I am leaning towards a .json file because the vendoring file should not be considered to be part of your go project it's just installing something that you will be using in your project.

    just my 2c.

    On Tue, Mar 3, 2015 at 12:01 PM, wrote:

    On Tuesday, 3 March 2015 17:59:49 UTC+1, Giacomo Tartari wrote:

    On 3Mar, 2015, at 17:49 , Michael Schuett wrote:

    I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version?
    I would assume so, but that is another easily automated task.
    Could even be made interactive:
    $go vendor
    $go vendor found mismatching version for package X: v1, v2, v3.
    do you wish to unify the versions or something? y,n
    version [1], [2], [3]?


    And it would be cool it tags are supported
    //go:vendor rev:616aea947362
    //go:vendor tag:616aea947362
    this was supposed to be //go:vendor tag:v1.0
    I need more coffe/sleep, sorry.

    Tags are easier to read.

    Giacomo
    On Tue, Mar 3, 2015 at 11:44 AM, wrote:
    Has this bee proposed and ruled out already?

    package main

    import (
    //go:vendor 616aea947362
    "rsc.io/arm/armasm <http://rsc.io/arm/armasm>"

    //go:vendor af2970a7819d
    "rsc.io/x86/x86asm <http://rsc.io/x86/x86asm>"
    )

    The imports reordering tools need to keep track of the "vendoring comment".


    Giacomo


    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:
    I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm <http://rsc.io/arm/armasm>" // 616aea947362
    "rsc.io/x86/x86asm <http://rsc.io/x86/x86asm>" // af2970a7819d
    )
    ---

    Russ

    --
    You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe <https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe>.
    To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Govert Versluis at Mar 3, 2015 at 5:29 pm
    If one uses tags or commit hashes in the .go source file, how are
    conflicting versions stored in the gopath?
    e.g. package A depends on package C tag 123, package B depends on package C
    tag 456.
    They both have a magic comment in the source file to indicate the right tag
    to use.
    However, since the path in the go source file is the same ("C"), the go
    build tool (and indeed other tools) will all look in $GOPATH/src/C.

    I'm against storing this data in .go files, especially if it's in some form
    of magic comment.
    The need for structure in directives such as these stands completely
    opposed to comments (which can be anything).

    The primary approach to vendoring at the moment seems to be to place the
    vendored packages in a internal/, third_party/ or vendor/ sub-directory,
    thus import paths need to be rewritten.
    As such it seems to make more sense to store the original package path and
    the revision to use in a manner separate from the actual source files, in
    structured format. allowing all imports (in multiple files) to be rewritten
    as needed by the appropriate tool.

    I don't see an elegant way to do this in .go files.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Pieter Droogendijk at Mar 3, 2015 at 5:43 pm

    On Tuesday, March 3, 2015 at 6:29:36 PM UTC+1, Govert Versluis wrote:
    I'm against storing this data in .go files, especially if it's in some
    form of magic comment.
    The need for structure in directives such as these stands completely
    opposed to comments (which can be anything).
    I agree with this, putting it in a comment may be the wrong move. But
    aren't import paths just strings? Don't they allow somewhat more freedom?
    Didn't we do that for a reason?

    import `rsc.io/x86/x86asm, vendor, rev:"af2970a7819d"`

    Or perhaps a tag string, like struct fields. Nicely backwards-compatible,
    and it nets us something like this:

    import (
         "os"
         x86asm_v2 "rsx.io/x86/x86asm" `vendor, tag:"v2"`
    )

    >

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Giacomo Tartari at Mar 3, 2015 at 5:44 pm
    Sorry for the noise did not reply to all.
    On 3Mar, 2015, at 18:29 , Govert Versluis wrote:

    If one uses tags or commit hashes in the .go source file, how are conflicting versions stored in the gopath?
    They are not, the comment is for a vendoring tool that will checkout the right version in the internal dir.

    I would also add a go version comment to get completely reproducible builds, //go:version 1.5.
    e.g. package A depends on package C tag 123, package B depends on package C tag 456.
    They both have a magic comment in the source file to indicate the right tag to use.
    However, since the path in the go source file is the same ("C"), the go build tool (and indeed other tools) will all look in $GOPATH/src/C.

    I'm against storing this data in .go files, especially if it's in some form of magic comment.
    The need for structure in directives such as these stands completely opposed to comments (which can be anything).

    The primary approach to vendoring at the moment seems to be to place the vendored packages in a internal/, third_party/ or vendor/ sub-directory, thus import paths need to be rewritten.
    As such it seems to make more sense to store the original package path and the revision to use in a manner separate from the actual source files, in structured format. allowing all imports (in multiple files) to be rewritten as needed by the appropriate tool.

    I don't see an elegant way to do this in .go files.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Eric Myhre at Mar 3, 2015 at 6:23 pm
    Whatever comes of this discussion: Distancing any proposed model from how Google works internally is going to be difficult, but necessary.

    The open source community as a whole has been burned before by the behaviors that emerge from a company culture that attempts open source but maintains things only in their personal vendorized ecosystem: remember the release of thrift.... followed by the release of thrift? This is -- and there's no shame in this, but I remember a number of central gophers admitting their workplace basically means they never use 'go get' -- a large part of the reason 'go get' is in such a dilapidated state compared to the rest of the otherwise charming and excellent go tool.

    I don't want to see go getting hung up on root misunderstandings of how a needs of a global collaboration culture may differ from those of a single company with a single dictatorial source tree with a single snapshot view available for a single point in time. The latter is easier, to be sure. Let's focus on moving beyond that. Planet Earth does not have a single shared source tree.

    Solutions need to account for usage *without prior coordination* between package authors.

    ... And that's where a whole bunch of the proposed stuff fails. Badly.

    This whole discussion started with vendoring presumed. (Why is it even in the thread title?) Import rewriting is then proposed as a workaround.

    There's still huge trouble with this when extended to more than *literally one* project, so it's proposed (with *colossal* handwaving) that libraries and programs should be different.

    These are both ridiculous duct tape workarounds. No: import rewriting is a ridiculous ducktape workaround, and claiming that libraries and programs are clearly distinct is just lying back and thinking of England.

    I propose we take a moment and clearly define the goalposts again: everyone's concerns are oriented around isolation, repeatability, and composability. Are there better ways we can serve these goals, especially with scaling across many repositories and many uncoordinated authors?


    ### isolation

    Notice how I didn't say "import rewriting". What everyone wants is isolation. We should discuss how to implement that concept. "import rewriting" is one possible way to implement isolation; let's phrase the conversation around the concept first.

    I second every single person (there have been many) who suggest either using per-project GOPATH settings, or -- if this hasn't been suggested outloud yet, I'd like to make the proposal -- creating a new system with similar semantics. Having wholey separate directory trees for each project is, in fact, absolutely the semantic that I want. Global variables are bad; global library heaps are equally bad, particularly when shared between many different programs of wildly unrelated authorship. (Anyone from the maven world -- remember how it feels to `rm -rf ~/.m2`? Tension just flows out of the body. Tension that never should have existed.)

    Much like we gophers value our statically linked single file binaries for their ease of shipping, so too do I value a project folder that is clearly that project, only that project, and all of that project. This is what happens when I set `GOPATH=$project/.gopath/`. Much like the whole truth and nothing but the truth, it's great! This means I can sync severable parts of my work to various computers easily, etc, and is super empowering.

    I regret to say I cannot buy suggestions that this would be an overwhelmingly disruptive chance to the go ecosystem. Currently doing so with every single go library and program I've ever touched in fact seems like rather concrete proof that this is doable, since it is in fact, done.

    Import rewriting makes sense in the situation that I have a diamond dependency (A -> B -> D.v1 & A -> C -> D.v2) and I'm willing to duplicate the two different versions of that transitively required library in order to isolate them while retaining both. This is by far the exception case, not the norm. Imagine if every package that used something as common as "fmt" from the standard library re-wrote and re-vendored it!

    Note that I have little care for the size of my working tree in an active development environment -- duplicated libraries on disk there per project is fine; disk is cheap. Duplication that ends up in version control is where concern lies, because this duplication has troublesome impacts over deep time, and the trouble can be pushed to other people who are then powerless to address it.


    ### repeatability

    Notice how I didn't say "vendoring". What everyone wants is repeatability. "vendoring" is one possible way to implement repeatability; let's phrase the conversation around the concept first.

    As pointed out earlier in the threat, repeatability is also maintained by systems such as Nix: hashes do wonders. Similarly, I've been using git submodules to get perfectly reproducible builds. Bower can pin dependencies by hashes. There's a huge list of concrete examples of repeatability without vendoring. Hash-based resource references solve repeatability.

    Vendoring may score points in the immediate offline mode. And I'm deeply pleased with importance placed on offline operation -- I share this priority the extent that I label my business cards "sneakernet advocate" -- but offline operation is orthogonal to vendoring and also available through other choices. It's good to have a command after which the build is guaranted not to need network; that command need not be `git clone`, and could just as easily be some other subcommand of the go tool. I think folk coming from other languages would be perfectly unsurprised by such a feature in the go tool, and we could have very satisfying outcomes from a command that does all resource acquisition and then stops.

    Vendoring has gained traction largely because it works without additional tooling, and that makes it the most contributor-friendly thing in the current landscape where the go tool has no chosen stance and thus the community as a whole is uncommitted. In discussing additions to the go tool, this impetus is made irrelevant.

    Vendoring has a variety of serious drawbacks. It gets repeatability right and *everything else* wrong. It's not transparent to libraries; it results in globally multiple histories for files; and it results in permanent bloat to a project source repo. (This was discussed before at length in a particular golang project forum and this table of comparisons was made back in 2013, which remains fully accurate today: https://gist.github.com/heavenlyhash/6343783 )

    The most major threat to repeatability when using hash-based resource references is network available in the deep time sense: what happens when the organization behind mycompany.net goes out of business and drops their domain, or library bananapancakes is renamed to something more user-centric, etc. In tracking recent changes, this is not frequently an issue, but doing a bisect across a well-aged repo can become troublesome. Fortunately, this is solvable: The system must have a way to replace old network addresses with updated aliases. We can do this.


    ### composability

    I'm not sure this is in the discussion as such yet, but it's been mentioned in passing, and so I figured I might as well give it a name and a heading.

    As a user of go, when examining a new library for potential use, I regularly want to see if it builds. Thereafter, I want to see if its tests pass. This requires that the library specify it's dependencies; specifically, it means the library must have repeatability just like a "real" "program".

    Bundler and rubygems actually got this fairly right: specifying a semver dependency range in one file, and specifying the "locked", pre-resolved versions of everything depended on (including transitively) is a great example of unifying both composability and repeatability goals. Bundler fell short in that it still only resolves down to a precise semver string, where it should really resolve to a hash. In the javascript world, Bower did similar things, and also carried the resolve results all the way to a hash: this truly satisfies repeatability. We should take a page from these books.

    While it is possible to ship executables at the end of the day without tackling this (and that is the most important thing), as Matthew Sackman has pointed out, it's a severe cramp to developer usability if we ignore this.

    On the plus side, this is also possible to do later, since isolation and repeatability are solvable without a solution to composability -- but it may be worth thinking about it now, when discussing data formats.


    ### aside: this "libraries and programs are different" thing *doesn't work* in the field

    Treating libraries and programs differently has a hidden presumption: that I have any control over other authors.

    Regrettably, I lack total control over all programmers. (Imagine! We could forgo this entire discussion! World conquest by friday! Etc.) And even if everyone in the world reoriented their views to ask all present and future users whether a package should be a library or not, there's no guarantee we'd agree. And a time dimension is indeed a part of that discussion, unless we break free of VCS repository layouts entirely, which has not yet come to pass.

    As one example, Docker has some code that I'd like to use as a library. But they don't consider themselves a library, and as a result, they've already vendored about 5 megs of other sources (and if including history, a much larger amount even after compression). This situation is understandable -- they consider themselves a "program", and rightly so -- but nonetheless, it's become very difficult for me to reuse their code without forking (and I mean manually, by copy-paste, a highly undesirable outcome). And then AFTER copy-paste forking the sections of code I want, I have to rewrite all their imports again to fit in my project, since I'm already using several of the same libraries at the same versions. This entire process would be nonsensical made-work that actively distracts from real development, and with all due respect to all the authors in this scenario who are all understandably acting in their best interests, I want no part of this to be in my future for other golang proje
    cts.


    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Tim Tadh at Mar 4, 2015 at 8:58 pm
    + 1 million to Eric's post. Vendoring is a bad solution to dependency
    management and import rewriting makes it worse. It is true that we often
    must do these things in the Go community, but that is only because we do
    not have better tools. If you, the go authors, don't immediately want to
    solve the dependency management and version management problem then you
    should instead tackle packaging and allow binaries to depend on packages
    (during build).

    I vote no on this proposal. Vendoring is a stop gap and should not be
    formalized.

                -Tim
    On Tuesday, March 3, 2015 at 1:23:47 PM UTC-5, Eric Myhre wrote:

    Whatever comes of this discussion: Distancing any proposed model from how
    Google works internally is going to be difficult, but necessary.

    The open source community as a whole has been burned before by the
    behaviors that emerge from a company culture that attempts open source but
    maintains things only in their personal vendorized ecosystem: remember the
    release of thrift.... followed by the release of thrift? This is -- and
    there's no shame in this, but I remember a number of central gophers
    admitting their workplace basically means they never use 'go get' -- a
    large part of the reason 'go get' is in such a dilapidated state compared
    to the rest of the otherwise charming and excellent go tool.

    I don't want to see go getting hung up on root misunderstandings of how a
    needs of a global collaboration culture may differ from those of a single
    company with a single dictatorial source tree with a single snapshot view
    available for a single point in time. The latter is easier, to be sure.
    Let's focus on moving beyond that. Planet Earth does not have a single
    shared source tree.

    Solutions need to account for usage *without prior coordination* between
    package authors.

    ... And that's where a whole bunch of the proposed stuff fails. Badly.

    This whole discussion started with vendoring presumed. (Why is it even in
    the thread title?) Import rewriting is then proposed as a workaround.

    There's still huge trouble with this when extended to more than *literally
    one* project, so it's proposed (with *colossal* handwaving) that libraries
    and programs should be different.

    These are both ridiculous duct tape workarounds. No: import rewriting is
    a ridiculous ducktape workaround, and claiming that libraries and programs
    are clearly distinct is just lying back and thinking of England.

    I propose we take a moment and clearly define the goalposts again:
    everyone's concerns are oriented around isolation, repeatability, and
    composability. Are there better ways we can serve these goals, especially
    with scaling across many repositories and many uncoordinated authors?


    ### isolation

    Notice how I didn't say "import rewriting". What everyone wants is
    isolation. We should discuss how to implement that concept. "import
    rewriting" is one possible way to implement isolation; let's phrase the
    conversation around the concept first.

    I second every single person (there have been many) who suggest either
    using per-project GOPATH settings, or -- if this hasn't been suggested
    outloud yet, I'd like to make the proposal -- creating a new system with
    similar semantics. Having wholey separate directory trees for each project
    is, in fact, absolutely the semantic that I want. Global variables are
    bad; global library heaps are equally bad, particularly when shared between
    many different programs of wildly unrelated authorship. (Anyone from the
    maven world -- remember how it feels to `rm -rf ~/.m2`? Tension just flows
    out of the body. Tension that never should have existed.)

    Much like we gophers value our statically linked single file binaries for
    their ease of shipping, so too do I value a project folder that is clearly
    that project, only that project, and all of that project. This is what
    happens when I set `GOPATH=$project/.gopath/`. Much like the whole truth
    and nothing but the truth, it's great! This means I can sync severable
    parts of my work to various computers easily, etc, and is super empowering.

    I regret to say I cannot buy suggestions that this would be an
    overwhelmingly disruptive chance to the go ecosystem. Currently doing so
    with every single go library and program I've ever touched in fact seems
    like rather concrete proof that this is doable, since it is in fact, done.

    Import rewriting makes sense in the situation that I have a diamond
    dependency (A -> B -> D.v1 & A -> C -> D.v2) and I'm willing to duplicate
    the two different versions of that transitively required library in order
    to isolate them while retaining both. This is by far the exception case,
    not the norm. Imagine if every package that used something as common as
    "fmt" from the standard library re-wrote and re-vendored it!

    Note that I have little care for the size of my working tree in an active
    development environment -- duplicated libraries on disk there per project
    is fine; disk is cheap. Duplication that ends up in version control is
    where concern lies, because this duplication has troublesome impacts over
    deep time, and the trouble can be pushed to other people who are then
    powerless to address it.


    ### repeatability

    Notice how I didn't say "vendoring". What everyone wants is
    repeatability. "vendoring" is one possible way to implement repeatability;
    let's phrase the conversation around the concept first.

    As pointed out earlier in the threat, repeatability is also maintained by
    systems such as Nix: hashes do wonders. Similarly, I've been using git
    submodules to get perfectly reproducible builds. Bower can pin
    dependencies by hashes. There's a huge list of concrete examples of
    repeatability without vendoring. Hash-based resource references solve
    repeatability.

    Vendoring may score points in the immediate offline mode. And I'm deeply
    pleased with importance placed on offline operation -- I share this
    priority the extent that I label my business cards "sneakernet advocate" --
    but offline operation is orthogonal to vendoring and also available through
    other choices. It's good to have a command after which the build is
    guaranted not to need network; that command need not be `git clone`, and
    could just as easily be some other subcommand of the go tool. I think folk
    coming from other languages would be perfectly unsurprised by such a
    feature in the go tool, and we could have very satisfying outcomes from a
    command that does all resource acquisition and then stops.

    Vendoring has gained traction largely because it works without additional
    tooling, and that makes it the most contributor-friendly thing in the
    current landscape where the go tool has no chosen stance and thus the
    community as a whole is uncommitted. In discussing additions to the go
    tool, this impetus is made irrelevant.

    Vendoring has a variety of serious drawbacks. It gets repeatability right
    and *everything else* wrong. It's not transparent to libraries; it results
    in globally multiple histories for files; and it results in permanent bloat
    to a project source repo. (This was discussed before at length in a
    particular golang project forum and this table of comparisons was made back
    in 2013, which remains fully accurate today:
    https://gist.github.com/heavenlyhash/6343783 )

    The most major threat to repeatability when using hash-based resource
    references is network available in the deep time sense: what happens when
    the organization behind mycompany.net goes out of business and drops
    their domain, or library bananapancakes is renamed to something more
    user-centric, etc. In tracking recent changes, this is not frequently an
    issue, but doing a bisect across a well-aged repo can become troublesome.
    Fortunately, this is solvable: The system must have a way to replace old
    network addresses with updated aliases. We can do this.


    ### composability

    I'm not sure this is in the discussion as such yet, but it's been
    mentioned in passing, and so I figured I might as well give it a name and a
    heading.

    As a user of go, when examining a new library for potential use, I
    regularly want to see if it builds. Thereafter, I want to see if its tests
    pass. This requires that the library specify it's dependencies;
    specifically, it means the library must have repeatability just like a
    "real" "program".

    Bundler and rubygems actually got this fairly right: specifying a semver
    dependency range in one file, and specifying the "locked", pre-resolved
    versions of everything depended on (including transitively) is a great
    example of unifying both composability and repeatability goals. Bundler
    fell short in that it still only resolves down to a precise semver string,
    where it should really resolve to a hash. In the javascript world, Bower
    did similar things, and also carried the resolve results all the way to a
    hash: this truly satisfies repeatability. We should take a page from these
    books.

    While it is possible to ship executables at the end of the day without
    tackling this (and that is the most important thing), as Matthew Sackman
    has pointed out, it's a severe cramp to developer usability if we ignore
    this.

    On the plus side, this is also possible to do later, since isolation and
    repeatability are solvable without a solution to composability -- but it
    may be worth thinking about it now, when discussing data formats.


    ### aside: this "libraries and programs are different" thing *doesn't
    work* in the field

    Treating libraries and programs differently has a hidden presumption: that
    I have any control over other authors.

    Regrettably, I lack total control over all programmers. (Imagine! We
    could forgo this entire discussion! World conquest by friday! Etc.) And
    even if everyone in the world reoriented their views to ask all present and
    future users whether a package should be a library or not, there's no
    guarantee we'd agree. And a time dimension is indeed a part of that
    discussion, unless we break free of VCS repository layouts entirely, which
    has not yet come to pass.

    As one example, Docker has some code that I'd like to use as a library.
    But they don't consider themselves a library, and as a result, they've
    already vendored about 5 megs of other sources (and if including history, a
    much larger amount even after compression). This situation is
    understandable -- they consider themselves a "program", and rightly so --
    but nonetheless, it's become very difficult for me to reuse their code
    without forking (and I mean manually, by copy-paste, a highly undesirable
    outcome). And then AFTER copy-paste forking the sections of code I want, I
    have to rewrite all their imports again to fit in my project, since I'm
    already using several of the same libraries at the same versions. This
    entire process would be nonsensical made-work that actively distracts from
    real development, and with all due respect to all the authors in this
    scenario who are all understandably acting in their best interests, I want
    no part of this to be in my future for other golang proje
    cts.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Davidn at Mar 5, 2015 at 3:44 pm
    I'd also like to add a vote to the "vendoring is a bad idea" camp.
    Reproducible builds and composability are critical, but vendoring isn't the
    only way to achieve that, it's just the only one that fills the peculiar
    shape created by what "go get" does and doesn't do. The go tool does both
    too much and too little to solve these problems: too little because it
    provides no mechanism for reproducible builds, and too much because its
    conventions, like using a repo path (but no version!) as an import
    identifier, make it impossible to properly solve them and maintain full
    compatibility without breaking the conventions.

    Rather than rehashing all the problems with vendoring (well-summarized in
    Eric's post), I'll just jump straight to the direction I'd rather see
    things go:

    The solution I've created for go builds at my workplace is a tool called
    begot[1]. You record your dependencies in a yaml file that points to git
    repos, optionally pinning to a branch or tag or commit (sorta like a
    Gemfile). It fetches them for you into a private cache, which you're not
    supposed to look at. Commit ids are recorded in another file (sorta like a
    Gemfile.lock). When building, it ensures that all the cached repos are at
    the right version, rewrites imports as necessary (but the rewrites are only
    local and never pushed anywhere!), constructs a temporary workspace with
    symlinks pointing to the right places (sorta like Blaze), then sets GOPATH
    and calls the go command to do the build. When you declare a dependency on
    another repo that uses begot, it checks that all the versions match.

    This decouples import paths used in code from the origin of the code, so
    you can feel free to use short and convenient names for your imports, with
    the actual origin recorded in the dependency file. It also means it's easy
    to switch a whole dependency to a private fork (e.g. while you're waiting
    for a patch to get merged upstream) with a just one-line change to the
    dependency file. (You can even do this for indirect dependencies without
    forking all the intermediate repos.)

    It also allows for extra metadata to be included with dependencies, such as
    which git transport to use, so private repos with ssh just work.

    Here's what I think this approach does right:
    - Fully deterministic and reproducible builds.
    - Continues the go tradition of using scm tools as a "package system"
    without binary packages (even more than the vendoring approach).
    - Decouples import paths from code origin.
    - Allows more metadata than can fit in an import path (versions, aliases,
    transports, etc.).
    - Always collapses multiple copies of a library.
    - Doesn't bloat repos with full copies of their dependencies.

    Here's where it needs some more work:

    - It requires that all versions of a repo referenced in a project's
    dependency tree be locked to the same commit id. This is probably too
    strict for fast-moving projects. The approach could be extended to be more
    flexible, e.g. with semantic versions in git tags.

    - In order to depend on a library that uses begot, you have to use begot
    too. Obviously, if an approach like this becomes the standard and is built
    into the go tool, this isn't a concern anymore.

    - It doesn't provide quite as strong of a reproducibility guarantee as
    vendoring: if you refer to a third-party's repo on say, GitHub, and they
    delete it or rewrite their history such that a commit id no longer exists,
    you might not be able to reconstruct it. My plan for solving this is a
    separate tool (not yet written) that makes it easy to mirror git/other scm
    repos so that you have a copy under your organization's control. This could
    integrate with the build tool so that using a mirror in place of the origin
    is completely transparent.

    ---

    The go tool and vendoring conventions have created a sort of local maximum
    that we can't break out of without sacrificing something, but is far from
    the ideal. I suggest we think about how to sacrifice some compatibility
    with the present world so we can get to a better one in the future.


    [1] https://github.com/solanolabs/begot

    On Wednesday, March 4, 2015 at 12:58:17 PM UTC-8, tim....@gmail.com wrote:

    + 1 million to Eric's post. Vendoring is a bad solution to dependency
    management and import rewriting makes it worse. It is true that we often
    must do these things in the Go community, but that is only because we do
    not have better tools. If you, the go authors, don't immediately want to
    solve the dependency management and version management problem then you
    should instead tackle packaging and allow binaries to depend on packages
    (during build).

    I vote no on this proposal. Vendoring is a stop gap and should not be
    formalized.

    -Tim
    On Tuesday, March 3, 2015 at 1:23:47 PM UTC-5, Eric Myhre wrote:

    Whatever comes of this discussion: Distancing any proposed model from how
    Google works internally is going to be difficult, but necessary.

    The open source community as a whole has been burned before by the
    behaviors that emerge from a company culture that attempts open source but
    maintains things only in their personal vendorized ecosystem: remember the
    release of thrift.... followed by the release of thrift? This is -- and
    there's no shame in this, but I remember a number of central gophers
    admitting their workplace basically means they never use 'go get' -- a
    large part of the reason 'go get' is in such a dilapidated state compared
    to the rest of the otherwise charming and excellent go tool.

    I don't want to see go getting hung up on root misunderstandings of how a
    needs of a global collaboration culture may differ from those of a single
    company with a single dictatorial source tree with a single snapshot view
    available for a single point in time. The latter is easier, to be sure.
    Let's focus on moving beyond that. Planet Earth does not have a single
    shared source tree.

    Solutions need to account for usage *without prior coordination* between
    package authors.

    ... And that's where a whole bunch of the proposed stuff fails. Badly.

    This whole discussion started with vendoring presumed. (Why is it even
    in the thread title?) Import rewriting is then proposed as a workaround.

    There's still huge trouble with this when extended to more than
    *literally one* project, so it's proposed (with *colossal* handwaving) that
    libraries and programs should be different.

    These are both ridiculous duct tape workarounds. No: import rewriting is
    a ridiculous ducktape workaround, and claiming that libraries and programs
    are clearly distinct is just lying back and thinking of England.

    I propose we take a moment and clearly define the goalposts again:
    everyone's concerns are oriented around isolation, repeatability, and
    composability. Are there better ways we can serve these goals, especially
    with scaling across many repositories and many uncoordinated authors?


    ### isolation

    Notice how I didn't say "import rewriting". What everyone wants is
    isolation. We should discuss how to implement that concept. "import
    rewriting" is one possible way to implement isolation; let's phrase the
    conversation around the concept first.

    I second every single person (there have been many) who suggest either
    using per-project GOPATH settings, or -- if this hasn't been suggested
    outloud yet, I'd like to make the proposal -- creating a new system with
    similar semantics. Having wholey separate directory trees for each project
    is, in fact, absolutely the semantic that I want. Global variables are
    bad; global library heaps are equally bad, particularly when shared between
    many different programs of wildly unrelated authorship. (Anyone from the
    maven world -- remember how it feels to `rm -rf ~/.m2`? Tension just flows
    out of the body. Tension that never should have existed.)

    Much like we gophers value our statically linked single file binaries for
    their ease of shipping, so too do I value a project folder that is clearly
    that project, only that project, and all of that project. This is what
    happens when I set `GOPATH=$project/.gopath/`. Much like the whole truth
    and nothing but the truth, it's great! This means I can sync severable
    parts of my work to various computers easily, etc, and is super empowering.

    I regret to say I cannot buy suggestions that this would be an
    overwhelmingly disruptive chance to the go ecosystem. Currently doing so
    with every single go library and program I've ever touched in fact seems
    like rather concrete proof that this is doable, since it is in fact, done.

    Import rewriting makes sense in the situation that I have a diamond
    dependency (A -> B -> D.v1 & A -> C -> D.v2) and I'm willing to duplicate
    the two different versions of that transitively required library in order
    to isolate them while retaining both. This is by far the exception case,
    not the norm. Imagine if every package that used something as common as
    "fmt" from the standard library re-wrote and re-vendored it!

    Note that I have little care for the size of my working tree in an active
    development environment -- duplicated libraries on disk there per project
    is fine; disk is cheap. Duplication that ends up in version control is
    where concern lies, because this duplication has troublesome impacts over
    deep time, and the trouble can be pushed to other people who are then
    powerless to address it.


    ### repeatability

    Notice how I didn't say "vendoring". What everyone wants is
    repeatability. "vendoring" is one possible way to implement repeatability;
    let's phrase the conversation around the concept first.

    As pointed out earlier in the threat, repeatability is also maintained by
    systems such as Nix: hashes do wonders. Similarly, I've been using git
    submodules to get perfectly reproducible builds. Bower can pin
    dependencies by hashes. There's a huge list of concrete examples of
    repeatability without vendoring. Hash-based resource references solve
    repeatability.

    Vendoring may score points in the immediate offline mode. And I'm deeply
    pleased with importance placed on offline operation -- I share this
    priority the extent that I label my business cards "sneakernet advocate" --
    but offline operation is orthogonal to vendoring and also available through
    other choices. It's good to have a command after which the build is
    guaranted not to need network; that command need not be `git clone`, and
    could just as easily be some other subcommand of the go tool. I think folk
    coming from other languages would be perfectly unsurprised by such a
    feature in the go tool, and we could have very satisfying outcomes from a
    command that does all resource acquisition and then stops.

    Vendoring has gained traction largely because it works without additional
    tooling, and that makes it the most contributor-friendly thing in the
    current landscape where the go tool has no chosen stance and thus the
    community as a whole is uncommitted. In discussing additions to the go
    tool, this impetus is made irrelevant.

    Vendoring has a variety of serious drawbacks. It gets repeatability
    right and *everything else* wrong. It's not transparent to libraries; it
    results in globally multiple histories for files; and it results in
    permanent bloat to a project source repo. (This was discussed before at
    length in a particular golang project forum and this table of comparisons
    was made back in 2013, which remains fully accurate today:
    https://gist.github.com/heavenlyhash/6343783 )

    The most major threat to repeatability when using hash-based resource
    references is network available in the deep time sense: what happens when
    the organization behind mycompany.net goes out of business and drops
    their domain, or library bananapancakes is renamed to something more
    user-centric, etc. In tracking recent changes, this is not frequently an
    issue, but doing a bisect across a well-aged repo can become troublesome.
    Fortunately, this is solvable: The system must have a way to replace old
    network addresses with updated aliases. We can do this.


    ### composability

    I'm not sure this is in the discussion as such yet, but it's been
    mentioned in passing, and so I figured I might as well give it a name and a
    heading.

    As a user of go, when examining a new library for potential use, I
    regularly want to see if it builds. Thereafter, I want to see if its tests
    pass. This requires that the library specify it's dependencies;
    specifically, it means the library must have repeatability just like a
    "real" "program".

    Bundler and rubygems actually got this fairly right: specifying a semver
    dependency range in one file, and specifying the "locked", pre-resolved
    versions of everything depended on (including transitively) is a great
    example of unifying both composability and repeatability goals. Bundler
    fell short in that it still only resolves down to a precise semver string,
    where it should really resolve to a hash. In the javascript world, Bower
    did similar things, and also carried the resolve results all the way to a
    hash: this truly satisfies repeatability. We should take a page from these
    books.

    While it is possible to ship executables at the end of the day without
    tackling this (and that is the most important thing), as Matthew Sackman
    has pointed out, it's a severe cramp to developer usability if we ignore
    this.

    On the plus side, this is also possible to do later, since isolation and
    repeatability are solvable without a solution to composability -- but it
    may be worth thinking about it now, when discussing data formats.


    ### aside: this "libraries and programs are different" thing *doesn't
    work* in the field

    Treating libraries and programs differently has a hidden presumption:
    that I have any control over other authors.

    Regrettably, I lack total control over all programmers. (Imagine! We
    could forgo this entire discussion! World conquest by friday! Etc.) And
    even if everyone in the world reoriented their views to ask all present and
    future users whether a package should be a library or not, there's no
    guarantee we'd agree. And a time dimension is indeed a part of that
    discussion, unless we break free of VCS repository layouts entirely, which
    has not yet come to pass.

    As one example, Docker has some code that I'd like to use as a library.
    But they don't consider themselves a library, and as a result, they've
    already vendored about 5 megs of other sources (and if including history, a
    much larger amount even after compression). This situation is
    understandable -- they consider themselves a "program", and rightly so --
    but nonetheless, it's become very difficult for me to reuse their code
    without forking (and I mean manually, by copy-paste, a highly undesirable
    outcome). And then AFTER copy-paste forking the sections of code I want, I
    have to rewrite all their imports again to fit in my project, since I'm
    already using several of the same libraries at the same versions. This
    entire process would be nonsensical made-work that actively distracts from
    real development, and with all due respect to all the authors in this
    scenario who are all understandably acting in their best interests, I want
    no part of this to be in my future for other golang proje
    cts.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alexandre Bourget at Mar 5, 2015 at 5:33 pm
    Sorry folks, I tried to post to the list, but sent it only privately. But
    now Google Groups doesn't allow me to find that post back.. can you mail it
    to me privately please ? :)

    Alexandre

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Bourget Alexandre at Mar 5, 2015 at 9:05 pm
    I dislike calling other tools instead of `go build`. It's not as
    discoverable.

    I'd prefer having hooks à-la `go generate` that would calls the right
    tools, from instructions in the source code, when running `go build`. At
    least `go build` could display error messages saying you need to install
    this or that dependency management tool.

    maybe just allowing notation of imports like `import
    "github.com/bob/project v1.0"` or something, those pluggable dependency
    management tools could extract the info they need, and act upon.

    It would stay simple for beginners, you'd have hooks for more advanced use
    cases, and flexibility of tooling for different projects

    Alexandre

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Mat Evans at Mar 6, 2015 at 2:56 pm

    On Thursday, March 5, 2015 at 3:44:32 PM UTC, dav...@gmail.com wrote:
    I'd also like to add a vote to the "vendoring is a bad idea" camp.
    Reproducible builds and composability are critical, but vendoring isn't the
    only way to achieve that, it's just the only one that fills the peculiar
    shape created by what "go get" does and doesn't do. The go tool does both
    too much and too little to solve these problems: too little because it
    provides no mechanism for reproducible builds, and too much because its
    conventions, like using a repo path (but no version!) as an import
    identifier, make it impossible to properly solve them and maintain full
    compatibility without breaking the conventions.

    Rather than rehashing all the problems with vendoring (well-summarized in
    Eric's post), I'll just jump straight to the direction I'd rather see
    things go:

    The solution I've created for go builds at my workplace is a tool called
    begot[1]. You record your dependencies in a yaml file that points to git
    repos, optionally pinning to a branch or tag or commit (sorta like a
    Gemfile). It fetches them for you into a private cache, which you're not
    supposed to look at. Commit ids are recorded in another file (sorta like a
    Gemfile.lock). When building, it ensures that all the cached repos are at
    the right version, rewrites imports as necessary (but the rewrites are only
    local and never pushed anywhere!), constructs a temporary workspace with
    symlinks pointing to the right places (sorta like Blaze), then sets GOPATH
    and calls the go command to do the build. When you declare a dependency on
    another repo that uses begot, it checks that all the versions match.
    I have a problem with that sort of setup - how do you guarantee a good
    build on a new server node that has no previous knowledge of the app you're
    about the build? You can't guarantee that the 'cached repos' are all going
    to be available or even that if they are the right versions will be
    available at that time. The point of a vendor folder for us at least, is
    that we control a copy of all the code that we will ever need to build an
    app anywhere.
    Unless i'm missing the point?

    This decouples import paths used in code from the origin of the code, so
    you can feel free to use short and convenient names for your imports, with
    the actual origin recorded in the dependency file. It also means it's easy
    to switch a whole dependency to a private fork (e.g. while you're waiting
    for a patch to get merged upstream) with a just one-line change to the
    dependency file. (You can even do this for indirect dependencies without
    forking all the intermediate repos.)
    It also allows for extra metadata to be included with dependencies, such
    as which git transport to use, so private repos with ssh just work.

    Here's what I think this approach does right:
    - Fully deterministic and reproducible builds.
    not sure about that - see above.. ^

    - Continues the go tradition of using scm tools as a "package system"
    without binary packages (even more than the vendoring approach).
    - Decouples import paths from code origin.
    - Allows more metadata than can fit in an import path (versions, aliases,
    transports, etc.).
    - Always collapses multiple copies of a library.
    I don't think in general any of us need to worry about having duplicate
    packages hanging around - how do you upgrade an old app safely if you
    update a package for all apps at once?

    - Doesn't bloat repos with full copies of their dependencies.

    Here's where it needs some more work:

    - It requires that all versions of a repo referenced in a project's
    dependency tree be locked to the same commit id. This is probably too
    strict for fast-moving projects. The approach could be extended to be more
    flexible, e.g. with semantic versions in git tags.

    - In order to depend on a library that uses begot, you have to use begot
    too. Obviously, if an approach like this becomes the standard and is built
    into the go tool, this isn't a concern anymore.

    - It doesn't provide quite as strong of a reproducibility guarantee as
    vendoring: if you refer to a third-party's repo on say, GitHub, and they
    delete it or rewrite their history such that a commit id no longer exists,
    you might not be able to reconstruct it. My plan for solving this is a
    separate tool (not yet written) that makes it easy to mirror git/other scm
    repos so that you have a copy under your organization's control. This could
    integrate with the build tool so that using a mirror in place of the origin
    is completely transparent.
    More tools - where as the vendoring solution is one that doens't need a
    tool as such, just the ability to read a config file. You can use anything.
    Shell/Go/manually get them all - once you have them you have them = a app
    that can be built anywhere and everywhere.

    ---

    The go tool and vendoring conventions have created a sort of local maximum
    that we can't break out of without sacrificing something, but is far from
    the ideal. I suggest we think about how to sacrifice some compatibility
    with the present world so we can get to a better one in the future.


    [1] https://github.com/solanolabs/begot

    Again, just some of my thoughts - but from my point of view, the simpler
    the better. Always.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • David Reiss at Mar 7, 2015 at 12:25 am

    On Fri, Mar 6, 2015 at 6:56 AM, wrote:
    On Thursday, March 5, 2015 at 3:44:32 PM UTC, dav...@gmail.com wrote:

    The solution I've created for go builds at my workplace is a tool called
    begot[1]. You record your dependencies in a yaml file that points to git
    repos, optionally pinning to a branch or tag or commit (sorta like a
    Gemfile). It fetches them for you into a private cache, which you're not
    supposed to look at. Commit ids are recorded in another file (sorta like a
    Gemfile.lock). When building, it ensures that all the cached repos are at
    the right version, rewrites imports as necessary (but the rewrites are only
    local and never pushed anywhere!), constructs a temporary workspace with
    symlinks pointing to the right places (sorta like Blaze), then sets GOPATH
    and calls the go command to do the build. When you declare a dependency on
    another repo that uses begot, it checks that all the versions match.
    I have a problem with that sort of setup - how do you guarantee a good
    build on a new server node that has no previous knowledge of the app you're
    about the build? You can't guarantee that the 'cached repos' are all going
    to be available or even that if they are the right versions will be
    available at that time. The point of a vendor folder for us at least, is
    that we control a copy of all the code that we will ever need to build an
    app anywhere.
    Unless i'm missing the point?
    Sorry, I left out a few details in an attempt to be concise: The commit ids
    (hashes) of all dependencies are recorded in a file and that file is
    checked into version control. After fetching a project repo on a new
    machine, one command will fetch all the dependent repos and reset them to
    the recorded commit ids. Semantically, you can imagine it fetching and
    resetting all dependencies for every build. The cache is just an
    optimization to avoid fetching over and over again.

    As for dependencies being unavailable, I suggested solving that with a tool
    to mirror repos. It's an important concern, but an orthogonal concern to
    getting the right versions of things, so it makes sense to solve it with a
    different (but integrated) tool.

    - Always collapses multiple copies of a library.
    I don't think in general any of us need to worry about having duplicate
    packages hanging around - how do you upgrade an old app safely if you
    update a package for all apps at once?
    I meant that it collapses multiple copies of a library within one binary.
    All projects have independent dependency metadata and can be upgraded
    independently (you can also upgrade specific packages for a project without
    upgrading everything). Again, any potential cross-project sharing through a
    cache is an optimization and doesn't affect the semantics.

    More tools - where as the vendoring solution is one that doens't need a
    tool as such, just the ability to read a config file. You can use anything.
    Shell/Go/manually get them all - once you have them you have them = a app
    that can be built anywhere and everywhere.
    Again, just some of my thoughts - but from my point of view, the simpler
    the better. Always.
    The popularity of vendoring tools seems to suggest that people certainly
    prefer a tool to help manage the complexity of vendoring, even if it's not
    strictly necessary.

    I am a huge fan of simplicity as well, perhaps to a fault. All these design
    decisions involve tradeoffs, and will make one thing simpler while quietly
    pushing complexity into another. It took me several months of trying and
    thinking about various approaches, seeing what worked well and what didn't,
    and conversations with colleagues, to convince myself that what I described
    has the least overall complexity (of solutions that solve the problems I
    set out to solve), even if in a few aspects, it is more complex than the
    status quo.

    I'll also note that my tool, begot <https://github.com/solanolabs/begot>,
    has fewer (non-test) lines of code than godep, nut, or glide. This may be a
    cheap shot, as it's less mature and will surely grow a little over time,
    and LOC is a crude measure, but I want to make the point that it's not
    inherently more complicated than those. And of course all of these depend
    on the go command, which is much larger than any of them.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • David Reiss at Mar 7, 2015 at 1:03 am

    On Fri, Mar 6, 2015 at 4:24 PM, David Reiss wrote:
    On Fri, Mar 6, 2015 at 6:56 AM, wrote:
    On Thursday, March 5, 2015 at 3:44:32 PM UTC, dav...@gmail.com wrote:

    The solution I've created for go builds at my workplace is a tool called
    begot[1]. You record your dependencies in a yaml file that points to git
    repos, optionally pinning to a branch or tag or commit (sorta like a
    Gemfile). It fetches them for you into a private cache, which you're not
    supposed to look at. Commit ids are recorded in another file (sorta like a
    Gemfile.lock). When building, it ensures that all the cached repos are at
    the right version, rewrites imports as necessary (but the rewrites are only
    local and never pushed anywhere!), constructs a temporary workspace with
    symlinks pointing to the right places (sorta like Blaze), then sets GOPATH
    and calls the go command to do the build. When you declare a dependency on
    another repo that uses begot, it checks that all the versions match.
    I have a problem with that sort of setup - how do you guarantee a good
    build on a new server node that has no previous knowledge of the app you're
    about the build? You can't guarantee that the 'cached repos' are all going
    to be available or even that if they are the right versions will be
    available at that time. The point of a vendor folder for us at least, is
    that we control a copy of all the code that we will ever need to build an
    app anywhere.
    Unless i'm missing the point?
    Sorry, I left out a few details in an attempt to be concise: The commit
    ids (hashes) of all dependencies are recorded in a file and that file is
    checked into version control. After fetching a project repo on a new
    machine, one command will fetch all the dependent repos and reset them to
    the recorded commit ids. Semantically, you can imagine it fetching and
    resetting all dependencies for every build. The cache is just an
    optimization to avoid fetching over and over again.
    I'd like to expand on a point that I don't think has been made explicit so
    far in this thread:

    What we actually want is two dependency metadata files, not one. The first
    file (the "request file") is where a human writes a request for a package,
    which obviously includes a repository identifier and may also include some
    sort of version specifier. Often this will just be "master"/"go1",
    sometimes another branch, and rarely a specific commit id. Or maybe some
    sort of semantic version specifier. The second file (the "lock file") is
    where a tool will record the actual commit ids that it got when resolving
    the version specifiers at a specific time. The request file is meant to be
    human editable and the lock file is not. The lock file contains all the
    metadata needed to reproduce a build (not the data, which could either be
    embedded in the project repo or not, depending on your vendoring
    preference).

    In the status quo, the request file isn't actually a separate file but is
    spread across the imports of all .go files, and it doesn't support version
    specifiers.

    Thinking about two files rather than one might help clarify some of the
    disagreements in this thread:

    - Commit ids are necessary to reproduce builds exactly, but they're not
    user-friendly and shouldn't go in the request file (most of the time).
    - Branch names shouldn't end up in the lock file, because they may resolve
    to different things at different times.
    - The lock file doesn't have to be human-editable (except for very rare
    situations), so it can be something like JSON.
    - The request file contains human intentions, which often benefit from
    accompanying explanations, so it should be in a format that allows comments
    (i.e. not JSON).
    - The request file doesn't have to be a separate file (although I believe
    it's more powerful and flexible if it is).
    - The lock file should be checked into version control to allow
    reproducible builds, even for libraries (although consumers of those
    libraries could choose to override it with versions recorded in their own
    lock file).
    - This is all orthogonal to vendoring vs. not vendoring (as long as you
    don't rewrite import paths).

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Eric Myhre at Mar 7, 2015 at 1:19 am

    On 03/06/2015 07:03 PM, David Reiss wrote:
    What we actually want is two dependency metadata files, not one. The first
    file (the "request file") is where a human writes a request for a package,
    which obviously includes a repository identifier and may also include some
    sort of version specifier. Often this will just be "master"/"go1",
    sometimes another branch, and rarely a specific commit id. Or maybe some
    sort of semantic version specifier. The second file (the "lock file") is
    where a tool will record the actual commit ids that it got when resolving
    the version specifiers at a specific time. The request file is meant to be
    human editable and the lock file is not. The lock file contains all the
    metadata needed to reproduce a build (not the data, which could either be
    embedded in the project repo or not, depending on your vendoring
    preference).
    +1 to this entire post; Well said David, that's far more precisely stated about the distinction between "request file" and "lock file" than my gesticulations.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Roger peppe at Mar 8, 2015 at 11:38 pm

    On 7 March 2015 at 01:03, David Reiss wrote:
    What we actually want is two dependency metadata files, not one. The first
    file (the "request file") is where a human writes a request for a package,
    which obviously includes a repository identifier and may also include some
    sort of version specifier. Often this will just be "master"/"go1", sometimes
    another branch, and rarely a specific commit id. Or maybe some sort of
    semantic version specifier. The second file (the "lock file") is where a
    tool will record the actual commit ids that it got when resolving the
    version specifiers at a specific time. The request file is meant to be human
    editable and the lock file is not. The lock file contains all the metadata
    needed to reproduce a build (not the data, which could either be embedded in
    the project repo or not, depending on your vendoring preference).

    In the status quo, the request file isn't actually a separate file but is
    spread across the imports of all .go files, and it doesn't support version
    specifiers.
    This seems about right to me. The network location implied by the
    import paths may be sufficient in most cases but not all, and it's
    quite reasonable for some other tool to be responsible for fetching
    repositories instead of the Go tool. Any such tool would likely have a
    configuration file, but the details of the tool and its configuration
    are beyond the scope of the current discussion, I think.

    That said, I would see the role of such a tool as
    providing of access to the required repositories or source
    code and this allowing the "lock file" constraints to be
    satisfied, but not of specifying versions, otherwise
    ISTM that the two files would be at cross-purposes.

    - Import paths specify the required API (potentially including a major
    version number);

    - The "lock file" specifies exact dependency versions, chosen by a
    repository owner;

    - The "request file" could specify a network location
    where repositories containing the chosen versions can be find. Vendoring
    is an alternative (and possibly preferable) approach here.

    I find myself thinking that although a "request file" may be a necessary
    evil in some circumstances (for instance if there's much-used buggy repository
    that won't accept bugfixes), it's probably almost always better
    to just explicitly fork the repository and let the import paths
    mean what they say.

       cheers,
         rog.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Transfire at Jun 12, 2015 at 1:17 am

    On Thursday, March 5, 2015 at 10:44:32 AM UTC-5, dav...@gmail.com wrote:

    This decouples import paths used in code from the origin of the code, so
    you can feel free to use short and convenient names for your imports, with
    the actual origin recorded in the dependency file. It also means it's easy
    to switch a whole dependency to a private fork (e.g. while you're waiting
    for a patch to get merged upstream) with a just one-line change to the
    dependency file. (You can even do this for indirect dependencies without
    forking all the intermediate repos.)
    I like this part of your idea, and have often thought that would be nice
    approach.

    I can't quite agree about being against vendoring though -- all the cached
    repo and symlinking stuff seems more complicated than it need be.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Joshua Marsh at Mar 3, 2015 at 5:23 pm

    On Tue, Mar 3, 2015 at 9:49 AM, Michael Schuett wrote:

    I feel like allowing every import to specify a version ends up with people
    in "version hell". You would also have to specify the version everytime you
    wanted to use that package. If you changed one would an error be thrown
    until you changed it in the various other places that you are specifying
    the old version?
    I think this might actually be no different than how a config might produce
    a "version hell". If one developer was working on one revision and another
    developer a different revision, once the two sources were brought together
    there would be a build failure that could be resolved. Using a config won't
    fix this. You could possibly catch it early if you looked at the config
    during the merge, but otherwise, you'd probably still find it during the
    testing after the merge. If the a revision was already in place local
    testing would catch it just as fast as the command that parses the config.

    This way seems more inline with the go ecosystem as opposed to some
    external configuration file in my mind. Using a configuration feels like
    building an additional room to your house with some plywood and duct tape.
    Using something like a build constraint or struct tag like I suggested
    earlier feels like you called a contractor and got it done right.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Sean Bowman at Mar 3, 2015 at 7:06 pm
    Aren't we already at the gates of version hell? I feel like the only reason the import system has worked so far is because we don't have a lot of "v2" libraries out there yet. The ecosystem is too young.

    I'm already used to repeating myself with Go. It's not the DRYist language in the world. And yes, I'd expect the compiler to generate an error if I use conflicting versions in my code.
    On Mar 3, 2015, at 9:49 AM, Michael Schuett wrote:

    I feel like allowing every import to specify a version ends up with people in "version hell". You would also have to specify the version everytime you wanted to use that package. If you changed one would an error be thrown until you changed it in the various other places that you are specifying the old version?
    On Tue, Mar 3, 2015 at 11:44 AM, wrote:
    Has this bee proposed and ruled out already?

    package main

    import (
    //go:vendor 616aea947362
    "rsc.io/arm/armasm"

    //go:vendor af2970a7819d
    "rsc.io/x86/x86asm"
    )

    The imports reordering tools need to keep track of the "vendoring comment".


    Giacomo

    On Tuesday, 3 March 2015 16:31:58 UTC+1, rsc wrote:
    I like the idea of using a Go file, but note that it cannot be one that actually builds, because it needs to record the original import paths, not the vendored ones. I would expect that a Go config file would have a standard .go suffix (so that gofmt etc apply) but a build tag keeping it from building, a standard name (say, vendor.go), and a required package name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Chris Hines at Mar 3, 2015 at 6:48 pm

    On Tuesday, March 3, 2015 at 10:31:58 AM UTC-5, rsc wrote:
    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    It could be valuable to have the vendoring metadata available at runtime. I
    am thinking of asking a binary to dump out its version and the versions of
    its build dependencies. This would be trivial if the metadata was compiled
    into the binary in an accessible Go data structure. I don't think this is a
    slam dunk argument, but something to consider.

    Chris

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Collingbourne at Mar 3, 2015 at 8:44 pm
    Maybe something like

      import (
       “cmd/internal/rsc.io/arm/armasm” // "rsc.io/arm/armasm" @ 616aea947362
    )

    would work?
    On Tue, Mar 3, 2015 at 7:31 AM, Russ Cox wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ

    --
    You received this message because you are subscribed to the Google Groups
    "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jonathan Roes at Mar 4, 2015 at 5:25 am
    If we're going to call the file vendor.go, the package vendor, and we're
    going to be using "vendored" as a verb so often, I can't help but think
    that we might as well be consistent and use "vendor" in the path instead of
    "internal."
    On Tuesday, March 3, 2015 at 10:31:58 AM UTC-5, rsc wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nathan Youngman at Mar 4, 2015 at 6:32 am
    I'm not sold on using a Go file to store SHA commit hashes. Perhaps this is
    just the way I use godep currently.

    When working on an app, I use godep to save the state of the world inside
    my main source tree. I'm not manually editing the config file, just like I
    never would manually edit a Gemfile.lock in Ruby-land (excepting merge
    conflicts :-( ).

    When patching dependencies, I leave my hermetically sealed bubble and work
    on a feature branch of that library. This allows me to push my changes
    upstream for hopeful inclusion (yay, collaboration, win!). I treat the
    Godeps subdirectory (now internal/github.com/*) as "DO NOT EDIT", simply a
    snapshot of work I did out in the world. This particular workflow makes
    "godep restore" quite necessary (implying bidirectional rewriting of import
    paths).

    But the point is, I'm not mucking about with SHA commit hashes, that's what
    we have tools for. I'm also not providing tags or branches in some config
    file. If I want to vendor a particular branch, I use commands like "cd" and
    "git checkout" and then take a snapshot.

    For me, this is the simplest thing that could possibly work, but no simpler.

    Mind you, I'm relatively new to this whole concept of copying third-party
    libraries into an internal/ folder. I'm sure I have much yet to learn,
    especially when applied in large teams.

    Nathan.

    On Tuesday, 3 March 2015 08:31:58 UTC-7, rsc wrote:

    I like the idea of using a Go file, but note that it cannot be one that
    actually builds, because it needs to record the original import paths, not
    the vendored ones. I would expect that a Go config file would have a
    standard .go suffix (so that gofmt etc apply) but a build tag keeping it
    from building, a standard name (say, vendor.go), and a required package
    name (say, vendor):

    ---
    // +build ignore

    package vendor

    import (
    // These are for the disassemblers.
    "rsc.io/arm/armasm" // 616aea947362
    "rsc.io/x86/x86asm" // af2970a7819d
    )
    ---

    Russ
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Brian Picciano at Mar 4, 2015 at 6:40 am
    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the file
    the vendor information is stored in and the basic details of how vendoring
    will work. It's likely that tools will sprout up around this that will
    allow for what you describe, regardless of what format is chosen.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Schuett at Mar 4, 2015 at 6:57 am
    So it seems like we are a little all over the place with ideas right now
    but I would like to get one thing ironed out if we can and I think it will
    greatly help move the conversation forward.
    https://docs.google.com/forms/d/1y9jmJBHk2-mml1vwTvVVTYkSM-YgxkzmLU9A1zuL26w/viewanalytics?usp=already_responded
    shows that just over half of the people actively contributing to this
    thread would like to stick with keeping this in a go file. From reading
    above people proposing .ini did not seem opposed to using go as well but
    just preferred using the .ini format. I prefer .json just because all my
    vendor toolings in the past have used it but it's just preference and I do
    see the strength of using a go file for this.

    Does anyone feel strongly enough to argue why we should not be using a .go
    file for vendor config?
    On Wed, Mar 4, 2015 at 1:40 AM, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the file
    the vendor information is stored in and the basic details of how vendoring
    will work. It's likely that tools will sprout up around this that will
    allow for what you describe, regardless of what format is chosen.

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nathan Youngman at Mar 4, 2015 at 6:58 am
    If this is file is maintained by tools, *.go doesn't seem like the most
    natural format.

    In this thread I've seen considerations like comments and commas as issues
    with JSON and the ability to limit to package main as a potential benefit
    of *.go. None of this is enough to win me over.

    A *.go file feels like it should be capable of doing more, like there is
    some reason we decided to use .go as a data file, but it seems unlikely
    that this would be the case because of +build ignore. A *.go file also
    seems like it should be able to go anywhere, but in practice it would
    probably need to be internal/vendor.go as this special case just to store
    the SHAs of transient dependencies as comments? Hm.

    Appending comments to import lines in a centralized file is hardly
    extensible. What if we want to track more than just the SHA commit in the
    future? Having a standard set of keys in a simple JSON file with some means
    for tool-specific extensions would provide this. The JSON deserialization
    libraries are already happy to ignore keys that a particular tool doesn't
    use.

    Nathan.


    On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the file
    the vendor information is stored in and the basic details of how vendoring
    will work. It's likely that tools will sprout up around this that will
    allow for what you describe, regardless of what format is chosen.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alex Smith at Mar 4, 2015 at 7:08 am
    I take 0 offense to any of this, just throwing idea out. Backwards
    comparability is much more important IMHO.
    On Wed, Mar 4, 2015 at 12:58 AM, Nathan Youngman wrote:


    If this is file is maintained by tools, *.go doesn't seem like the most
    natural format.

    In this thread I've seen considerations like comments and commas as issues
    with JSON and the ability to limit to package main as a potential benefit
    of *.go. None of this is enough to win me over.

    A *.go file feels like it should be capable of doing more, like there is
    some reason we decided to use .go as a data file, but it seems unlikely
    that this would be the case because of +build ignore. A *.go file also
    seems like it should be able to go anywhere, but in practice it would
    probably need to be internal/vendor.go as this special case just to store
    the SHAs of transient dependencies as comments? Hm.

    Appending comments to import lines in a centralized file is hardly
    extensible. What if we want to track more than just the SHA commit in the
    future? Having a standard set of keys in a simple JSON file with some means
    for tool-specific extensions would provide this. The JSON deserialization
    libraries are already happy to ignore keys that a particular tool doesn't
    use.

    Nathan.


    On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the
    file the vendor information is stored in and the basic details of how
    vendoring will work. It's likely that tools will sprout up around this that
    will allow for what you describe, regardless of what format is chosen.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Schuett at Mar 4, 2015 at 7:13 am
    I do agree with you Nathan. I was actually just trying to workout what
    exactly a *.go implementation would look like and everything seems like it
    would be easy to abuse and syntax wise is just a pain. I think the
    following json allows for a simple and extensible way of managing
    dependencies without having to litter your imports with comments which I
    think will make our otherwise clean *.go files start to look a little more
    nasty. This also has the advantage of looking in one file and seeing what
    the package is all pulling in. How often do you really need to add a
    comment for a dependency? I know I have never run into this issue.

    {
       "require": [
         {
           "rsc.io/arm/armasm": "616aea947362"
         },
         {
           "rsc.io/x86/x86asm": "af2970a7819d"
         },
         {
           "rsc.io/x86/x86asm": "7e021db6589f"
         }
       ]
    }
    On Wed, Mar 4, 2015 at 1:58 AM, Nathan Youngman wrote:


    If this is file is maintained by tools, *.go doesn't seem like the most
    natural format.

    In this thread I've seen considerations like comments and commas as issues
    with JSON and the ability to limit to package main as a potential benefit
    of *.go. None of this is enough to win me over.

    A *.go file feels like it should be capable of doing more, like there is
    some reason we decided to use .go as a data file, but it seems unlikely
    that this would be the case because of +build ignore. A *.go file also
    seems like it should be able to go anywhere, but in practice it would
    probably need to be internal/vendor.go as this special case just to store
    the SHAs of transient dependencies as comments? Hm.

    Appending comments to import lines in a centralized file is hardly
    extensible. What if we want to track more than just the SHA commit in the
    future? Having a standard set of keys in a simple JSON file with some means
    for tool-specific extensions would provide this. The JSON deserialization
    libraries are already happy to ignore keys that a particular tool doesn't
    use.

    Nathan.


    On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the
    file the vendor information is stored in and the basic details of how
    vendoring will work. It's likely that tools will sprout up around this that
    will allow for what you describe, regardless of what format is chosen.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alex Smith at Mar 4, 2015 at 7:18 am
    I agree. I just see that this as being a tumbling block for new users,
    this can always be fixed or cleaned up later on.

    Alex
    On Wed, Mar 4, 2015 at 1:13 AM, Michael Schuett wrote:

    I do agree with you Nathan. I was actually just trying to workout what
    exactly a *.go implementation would look like and everything seems like it
    would be easy to abuse and syntax wise is just a pain. I think the
    following json allows for a simple and extensible way of managing
    dependencies without having to litter your imports with comments which I
    think will make our otherwise clean *.go files start to look a little more
    nasty. This also has the advantage of looking in one file and seeing what
    the package is all pulling in. How often do you really need to add a
    comment for a dependency? I know I have never run into this issue.

    {
    "require": [
    {
    "rsc.io/arm/armasm": "616aea947362"
    },
    {
    "rsc.io/x86/x86asm": "af2970a7819d"
    },
    {
    "rsc.io/x86/x86asm": "7e021db6589f"
    }
    ]
    }
    On Wed, Mar 4, 2015 at 1:58 AM, Nathan Youngman wrote:


    If this is file is maintained by tools, *.go doesn't seem like the most
    natural format.

    In this thread I've seen considerations like comments and commas as
    issues with JSON and the ability to limit to package main as a potential
    benefit of *.go. None of this is enough to win me over.

    A *.go file feels like it should be capable of doing more, like there is
    some reason we decided to use .go as a data file, but it seems unlikely
    that this would be the case because of +build ignore. A *.go file also
    seems like it should be able to go anywhere, but in practice it would
    probably need to be internal/vendor.go as this special case just to store
    the SHAs of transient dependencies as comments? Hm.

    Appending comments to import lines in a centralized file is hardly
    extensible. What if we want to track more than just the SHA commit in the
    future? Having a standard set of keys in a simple JSON file with some means
    for tool-specific extensions would provide this. The JSON deserialization
    libraries are already happy to ignore keys that a particular tool doesn't
    use.

    Nathan.


    On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the
    file the vendor information is stored in and the basic details of how
    vendoring will work. It's likely that tools will sprout up around this that
    will allow for what you describe, regardless of what format is chosen.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nathan Youngman at Mar 4, 2015 at 7:31 am
    I would lean towards the more verbose key/value pairs like {"repo":
    rsc.io/arm/armasm", "commit": "616aea947362"}. It's ugly and seems
    wasteful, but it provides more flexibilty down the road.

    Since we already have all the code in our repo, I assume the reason why we
    want this metadata *at all *is to restore the world outside our hermetic
    bubble to match up?

    Say I submit a patch but it hasn't been merged upstream yet. Maybe a tool
    would like to store the fact that I had these other "remotes": ["
    github.com/libgo/arm"] when I took my snapshot. Then if said tool can't
    find "616aea947362" on a colleagues computer, it can add the remotes and
    see if that helps.

    That's just an example. The point is simplicity and extensibility.

    Nathan.

    P.S. I have add comments for dependencies before. Usually that I can't use
    some new thing until such and such. It's fine to put those comments in the
    code alongside the import statement though. No need to stuff comments in
    the JSON.


    On 4 March 2015 at 00:13, Michael Schuett wrote:

    I do agree with you Nathan. I was actually just trying to workout what
    exactly a *.go implementation would look like and everything seems like it
    would be easy to abuse and syntax wise is just a pain. I think the
    following json allows for a simple and extensible way of managing
    dependencies without having to litter your imports with comments which I
    think will make our otherwise clean *.go files start to look a little more
    nasty. This also has the advantage of looking in one file and seeing what
    the package is all pulling in. How often do you really need to add a
    comment for a dependency? I know I have never run into this issue.

    {
    "require": [
    {
    "rsc.io/arm/armasm": "616aea947362"
    },
    {
    "rsc.io/x86/x86asm": "af2970a7819d"
    },
    {
    "rsc.io/x86/x86asm": "7e021db6589f"
    }
    ]
    }
    On Wed, Mar 4, 2015 at 1:58 AM, Nathan Youngman wrote:


    If this is file is maintained by tools, *.go doesn't seem like the most
    natural format.

    In this thread I've seen considerations like comments and commas as
    issues with JSON and the ability to limit to package main as a potential
    benefit of *.go. None of this is enough to win me over.

    A *.go file feels like it should be capable of doing more, like there is
    some reason we decided to use .go as a data file, but it seems unlikely
    that this would be the case because of +build ignore. A *.go file also
    seems like it should be able to go anywhere, but in practice it would
    probably need to be internal/vendor.go as this special case just to store
    the SHAs of transient dependencies as comments? Hm.

    Appending comments to import lines in a centralized file is hardly
    extensible. What if we want to track more than just the SHA commit in the
    future? Having a standard set of keys in a simple JSON file with some means
    for tool-specific extensions would provide this. The JSON deserialization
    libraries are already happy to ignore keys that a particular tool doesn't
    use.

    Nathan.


    On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes, that's
    what we have tools for

    My understanding is that we're primarily discussing the format of the
    file the vendor information is stored in and the basic details of how
    vendoring will work. It's likely that tools will sprout up around this that
    will allow for what you describe, regardless of what format is chosen.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

    --
    Nathan Youngman
    Email: hello@nathany.com
    Web: http://www.nathany.com

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Alex Smith at Mar 4, 2015 at 7:40 am
    While I like this idea (it is useful, especially for alpha/beta) how would
    backwards comparability work? Is it a tag, branch, etc. with standardized
    naming? This is *exactly* why I started using that weird go alias, it
    allowed me to "go get" a given library at a given time.
    On Wed, Mar 4, 2015 at 1:31 AM, Nathan Youngman wrote:


    I would lean towards the more verbose key/value pairs like {"repo":
    rsc.io/arm/armasm", "commit": "616aea947362"}. It's ugly and seems
    wasteful, but it provides more flexibilty down the road.

    Since we already have all the code in our repo, I assume the reason why we
    want this metadata *at all *is to restore the world outside our hermetic
    bubble to match up?

    Say I submit a patch but it hasn't been merged upstream yet. Maybe a tool
    would like to store the fact that I had these other "remotes": ["
    github.com/libgo/arm"] when I took my snapshot. Then if said tool can't
    find "616aea947362" on a colleagues computer, it can add the remotes and
    see if that helps.

    That's just an example. The point is simplicity and extensibility.

    Nathan.

    P.S. I have add comments for dependencies before. Usually that I can't use
    some new thing until such and such. It's fine to put those comments in the
    code alongside the import statement though. No need to stuff comments in
    the JSON.


    On 4 March 2015 at 00:13, Michael Schuett wrote:

    I do agree with you Nathan. I was actually just trying to workout what
    exactly a *.go implementation would look like and everything seems like it
    would be easy to abuse and syntax wise is just a pain. I think the
    following json allows for a simple and extensible way of managing
    dependencies without having to litter your imports with comments which I
    think will make our otherwise clean *.go files start to look a little more
    nasty. This also has the advantage of looking in one file and seeing what
    the package is all pulling in. How often do you really need to add a
    comment for a dependency? I know I have never run into this issue.

    {
    "require": [
    {
    "rsc.io/arm/armasm": "616aea947362"
    },
    {
    "rsc.io/x86/x86asm": "af2970a7819d"
    },
    {
    "rsc.io/x86/x86asm": "7e021db6589f"
    }
    ]
    }

    On Wed, Mar 4, 2015 at 1:58 AM, Nathan Youngman <hello@nathany.com>
    wrote:
    If this is file is maintained by tools, *.go doesn't seem like the most
    natural format.

    In this thread I've seen considerations like comments and commas as
    issues with JSON and the ability to limit to package main as a potential
    benefit of *.go. None of this is enough to win me over.

    A *.go file feels like it should be capable of doing more, like there is
    some reason we decided to use .go as a data file, but it seems unlikely
    that this would be the case because of +build ignore. A *.go file also
    seems like it should be able to go anywhere, but in practice it would
    probably need to be internal/vendor.go as this special case just to store
    the SHAs of transient dependencies as comments? Hm.

    Appending comments to import lines in a centralized file is hardly
    extensible. What if we want to track more than just the SHA commit in the
    future? Having a standard set of keys in a simple JSON file with some means
    for tool-specific extensions would provide this. The JSON deserialization
    libraries are already happy to ignore keys that a particular tool doesn't
    use.

    Nathan.


    On Tuesday, 3 March 2015 23:40:39 UTC-7, Brian Picciano wrote:

    But the point is, I'm not mucking about with SHA commit hashes,
    that's what we have tools for

    My understanding is that we're primarily discussing the format of the
    file the vendor information is stored in and the basic details of how
    vendoring will work. It's likely that tools will sprout up around this that
    will allow for what you describe, regardless of what format is chosen.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

    --
    Nathan Youngman
    Email: hello@nathany.com
    Web: http://www.nathany.com

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nathan Youngman at Mar 4, 2015 at 4:42 pm
    Hi Alex,

    I'm not totally sure if I understand your question.

    In my mind, this configuration is a generated file just like godep. It is
    based on the state of the world when using "save" to vendor all the code.

    If you want to use a particular tag or branch of a third-party repo, you
    just check it out. Then godep save saves the commit hash (Rev) that is
    currently checked out along with all the code currently checked out.

    On a another computer, someone can already build and use the app because it
    contains all the code. But if they want to collaborate upstream, they can
    run a single command that puts external repositories into a state matching
    what has been vendored (assuming none of those repositories is dirty). Then
    they can make changes, send pull requests, and snapshot those changes back
    into the main repo.

    This is how godep works, or at least, how I work with godep presently.

    All this gets back to what the purpose of this file is in the first place.
    :-)

    Other people have a different idea as to what this is for, and I'd really
    like to hear the Go Team's take on why they feel the need to store commit
    hash (Rev) at all. What would it enable in their workflow that necessitates
    having this extra file and the potential merge conflicts to go along with
    it?

    Nathan.

    On Wednesday, 4 March 2015 00:40:26 UTC-7, Alex Smith wrote:

    While I like this idea (it is useful, especially for alpha/beta) how would
    backwards comparability work? Is it a tag, branch, etc. with standardized
    naming? This is *exactly* why I started using that weird go alias, it
    allowed me to "go get" a given library at a given time.

    On Wed, Mar 4, 2015 at 1:31 AM, Nathan Youngman <he...@nathany.com
    <javascript:>> wrote:
    I would lean towards the more verbose key/value pairs like {"repo":
    rsc.io/arm/armasm", "commit": "616aea947362"}. It's ugly and seems
    wasteful, but it provides more flexibilty down the road.

    Since we already have all the code in our repo, I assume the reason why
    we want this metadata *at all *is to restore the world outside our
    hermetic bubble to match up?

    Say I submit a patch but it hasn't been merged upstream yet. Maybe a tool
    would like to store the fact that I had these other "remotes": ["
    github.com/libgo/arm"] when I took my snapshot. Then if said tool can't
    find "616aea947362" on a colleagues computer, it can add the remotes and
    see if that helps.

    That's just an example. The point is simplicity and extensibility.

    Nathan.

    P.S. I have add comments for dependencies before. Usually that I can't
    use some new thing until such and such. It's fine to put those comments in
    the code alongside the import statement though. No need to stuff comments
    in the JSON.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Nate Finch at Mar 4, 2015 at 7:50 pm
    In general, I like that Go has no configuration files, however, I think
    this deps.cfg file or whatever, is ok, because it's not used for building,
    and it doesn't exist in your main code.

    The layout of your application's code will look something like this:

    github.com/natefinch/myapp/main.go
    github.com/natefinch/myapp/pkg1/pkg1.go
    github.com/natefinch/myapp/internal/deps.cfg
    github.com/natefinch/myapp/internal/github.com/notme/somepkg/somepkg.go

    *go build* and *go install* and *go get* don't need to know anything about
    deps.cfg. They don't read it at all. You can still *go get* this code,
    and it'll Just Work™.

    The only tool that needs to understand it and work with it is *go deps* (or
    whatever it'll be called), and then you only need to use that command when
    you're vendoring dependencies for the first time, or updating to a newer
    version of a dependency. In general, updating should be rare (if it's not,
    you may want to consider why the external code you're depending is churning
    so much... it's probably an indication that you shouldn't depend on it).

    As for the format, I have to agree with those that say that JSON makes for
    a terrible config language. In theory, this will always be machine-written
    and -read, so it doesn't *really* matter.... but having something human
    readable and writable can make a big difference in real life. However, I
    also agree that the configuration file should be in a format that can be
    read using the std lib.

    I don't think that making it a runnable .go file is a good idea. For one,
    runnable code during project configuration is how Ruby has gotten into
    trouble so many times with gems doing nasty things. For another, the go
    tool would have to know to *not* build that code when doing `go build
    ./...` from the root directory.

    So... unless we're willing to put a different configuration language in the
    std lib (like TOML, which I love), then I think JSON is the least bad
    choice. But really, let's put TOML in the stdlib. That would be amazing.

    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Peter Waldschmidt at Mar 4, 2015 at 8:58 pm
    The reason for storing the git hash is that it documents which version of
    the particular dependency you are using.

    In the case of vendoring it helps the tool maintain the right version of
    files in the local directories. In the absence of vendoring, the toolchain
    can check out the matching versions of dependencies rather than whatever
    happens to be on the master branch that day.

    In the case of conflicts, the toolchain can choose to select or pin
    particular dependencies to resolve them.
    On Mar 4, 2015 11:42 AM, "Nathan Youngman" wrote:

    Hi Alex,

    I'm not totally sure if I understand your question.

    In my mind, this configuration is a generated file just like godep. It is
    based on the state of the world when using "save" to vendor all the code.

    If you want to use a particular tag or branch of a third-party repo, you
    just check it out. Then godep save saves the commit hash (Rev) that is
    currently checked out along with all the code currently checked out.

    On a another computer, someone can already build and use the app because
    it contains all the code. But if they want to collaborate upstream, they
    can run a single command that puts external repositories into a state
    matching what has been vendored (assuming none of those repositories is
    dirty). Then they can make changes, send pull requests, and snapshot those
    changes back into the main repo.

    This is how godep works, or at least, how I work with godep presently.

    All this gets back to what the purpose of this file is in the first place.
    :-)

    Other people have a different idea as to what this is for, and I'd really
    like to hear the Go Team's take on why they feel the need to store commit
    hash (Rev) at all. What would it enable in their workflow that necessitates
    having this extra file and the potential merge conflicts to go along with
    it?

    Nathan.

    On Wednesday, 4 March 2015 00:40:26 UTC-7, Alex Smith wrote:

    While I like this idea (it is useful, especially for alpha/beta) how
    would backwards comparability work? Is it a tag, branch, etc. with
    standardized naming? This is *exactly* why I started using that weird go
    alias, it allowed me to "go get" a given library at a given time.

    On Wed, Mar 4, 2015 at 1:31 AM, Nathan Youngman <he...@nathany.com>
    wrote:
    I would lean towards the more verbose key/value pairs like {"repo":
    rsc.io/arm/armasm", "commit": "616aea947362"}. It's ugly and seems
    wasteful, but it provides more flexibilty down the road.

    Since we already have all the code in our repo, I assume the reason why
    we want this metadata *at all *is to restore the world outside our
    hermetic bubble to match up?

    Say I submit a patch but it hasn't been merged upstream yet. Maybe a
    tool would like to store the fact that I had these other "remotes": ["
    github.com/libgo/arm"] when I took my snapshot. Then if said tool can't
    find "616aea947362" on a colleagues computer, it can add the remotes and
    see if that helps.

    That's just an example. The point is simplicity and extensibility.

    Nathan.

    P.S. I have add comments for dependencies before. Usually that I can't
    use some new thing until such and such. It's fine to put those comments in
    the code alongside the import statement though. No need to stuff comments
    in the JSON.

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Michael Schuett at Mar 4, 2015 at 5:32 pm
    No attacking anyone as I know tone can be lost over email. Under what case
    would we want conditional dependencies I just see it as becoming a huge
    mess. Go already handles making sure test imports aren't included in your
    production binary and that is the only conditional dependency I can think
    of. It is needless complication in my opinion. In what way does a *.go
    offer better revision locking or vendoring that you would not get out of a
    simple config? Maybe if you are using tags? however this would have to be
    solved with writing another file in any case that would keep track of the
    hash of the installed version. I think that is a little out of scope of
    OP's proposal though.
    On Wed, Mar 4, 2015 at 12:07 PM, Peter Waldschmidt wrote:

    The reason for storing the git hash is that it documents which version of
    the particular dependency you are using.

    In the case of vendoring it helps the tool maintain the right version of
    files in the local directories. In the absence of vendoring, the toolchain
    can check out the matching versions of dependencies rather than whatever
    happens to be on the master branch that day.

    In the case of conflicts, the toolchain can choose to select or pin
    particular dependencies to resolve them.
    On Mar 4, 2015 11:42 AM, "Nathan Youngman" wrote:

    Hi Alex,

    I'm not totally sure if I understand your question.

    In my mind, this configuration is a generated file just like godep. It is
    based on the state of the world when using "save" to vendor all the code.

    If you want to use a particular tag or branch of a third-party repo, you
    just check it out. Then godep save saves the commit hash (Rev) that is
    currently checked out along with all the code currently checked out.

    On a another computer, someone can already build and use the app because
    it contains all the code. But if they want to collaborate upstream, they
    can run a single command that puts external repositories into a state
    matching what has been vendored (assuming none of those repositories is
    dirty). Then they can make changes, send pull requests, and snapshot those
    changes back into the main repo.

    This is how godep works, or at least, how I work with godep presently.

    All this gets back to what the purpose of this file is in the first
    place. :-)

    Other people have a different idea as to what this is for, and I'd really
    like to hear the Go Team's take on why they feel the need to store commit
    hash (Rev) at all. What would it enable in their workflow that necessitates
    having this extra file and the potential merge conflicts to go along with
    it?

    Nathan.

    On Wednesday, 4 March 2015 00:40:26 UTC-7, Alex Smith wrote:

    While I like this idea (it is useful, especially for alpha/beta) how
    would backwards comparability work? Is it a tag, branch, etc. with
    standardized naming? This is *exactly* why I started using that weird go
    alias, it allowed me to "go get" a given library at a given time.

    On Wed, Mar 4, 2015 at 1:31 AM, Nathan Youngman <he...@nathany.com>
    wrote:
    I would lean towards the more verbose key/value pairs like {"repo":
    rsc.io/arm/armasm", "commit": "616aea947362"}. It's ugly and seems
    wasteful, but it provides more flexibilty down the road.

    Since we already have all the code in our repo, I assume the reason why
    we want this metadata *at all *is to restore the world outside our
    hermetic bubble to match up?

    Say I submit a patch but it hasn't been merged upstream yet. Maybe a
    tool would like to store the fact that I had these other "remotes": ["
    github.com/libgo/arm"] when I took my snapshot. Then if said tool
    can't find "616aea947362" on a colleagues computer, it can add the remotes
    and see if that helps.

    That's just an example. The point is simplicity and extensibility.

    Nathan.

    P.S. I have add comments for dependencies before. Usually that I can't
    use some new thing until such and such. It's fine to put those comments in
    the code alongside the import statement though. No need to stuff comments
    in the JSON.

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-dev" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-dev/nMWoEAG55v8/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-dev" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-dev @
categoriesgo
postedMar 2, '15 at 5:38p
activeJul 3, '15 at 1:29p
posts219
users98
websitegolang.org

98 users in discussion

Alex Smith: 14 posts Michael Schuett: 10 posts Nathan Youngman: 10 posts Andrew Gerrand: 9 posts Giacomo Tartari: 7 posts Matthew Sackman: 6 posts Humberto S. N. dos Anjos: 6 posts Ian Davis: 6 posts Eric Myhre: 5 posts Richard Fliam: 5 posts Brad Fitzpatrick: 5 posts Keith Rarick: 5 posts Burcu Dogan: 4 posts Tianon Gravi: 4 posts Axel Wagner: 4 posts Roger peppe: 4 posts Owen Ou: 3 posts Russ Cox: 3 posts Ulrich Schreiner: 3 posts Mediocregopher: 3 posts Ben Darnell: 3 posts Pablo Astigarraga: 3 posts Mat Evans: 3 posts David Reiss: 3 posts Samm: 2 posts Craig: 2 posts Joshua Marsh: 2 posts Anlhord: 2 posts Brendan Tracey: 2 posts Kamil Kisiel: 2 posts Lars Seipel: 2 posts Mihai: 2 posts Pico303: 2 posts Rodrigo Kochenburger: 2 posts Bourget Alexandre: 2 posts Daniel Theophanes: 2 posts Alex Sl: 2 posts Martin Bruse: 2 posts Murillodaviziko: 2 posts Psimyth: 2 posts Sam Dodrill: 2 posts Peter Waldschmidt: 1 post Tim Tadh: 1 post Yiyus: 1 post Peter Collingbourne: 1 post Benjamin Measures: 1 post Agrondin: 1 post Markcol: 1 post Gustavo Niemeyer: 1 post Adrien Delorme: 1 post Julius: 1 post Luna Duclos: 1 post Jeremiaslongo: 1 post Egon Elbre: 1 post Rob Fielding: 1 post Jeanepaul: 1 post Chris Hines: 1 post Aram Hăvărneanu: 1 post Chris dollin: 1 post Peter Vessenes: 1 post Sébastien Douche: 1 post Minux: 1 post Bits128: 1 post Nate Finch: 1 post Prepin: 1 post Ralph Corderoy: 1 post Evan Shaw: 1 post Poteland: 1 post Pieter Droogendijk: 1 post Zellyn: 1 post Transfire: 1 post Daniel Skinner: 1 post Peter Waldschmidt: 1 post Matthijs Mullender: 1 post Lucio De Re: 1 post Emil2k: 1 post Normandiggs: 1 post Andrewchamberss: 1 post Dmitri Savintsev: 1 post Manlio Perillo: 1 post R W Johnstone: 1 post Karan Misra: 1 post Andyster: 1 post Pierre Durand: 1 post Leiser 1960: 1 post Ben Darnell: 1 post Govert Versluis: 1 post Dmitri Shuralyov: 1 post Eric T Anderton: 1 post Rob Figueiredo: 1 post Justin Scheiber: 1 post Salman Aljammaz: 1 post Cudmore Mb: 1 post Kyle Lemons: 1 post Jacek Masiulaniec: 1 post Cmang: 1 post Info: 1 post Jonathan Roes: 1 post

People

Translate

site design / logo © 2021 Grokbase