[go-nuts] GOPATH for a project written in several languages (for example Go and Python)?

On Wed, Aug 14, 2013 at 4:01 PM, Nicolas Grilly wrote:

What is the recommended way to organize the source tree of a project
written in several programming languages (for example Go and Python)?

Ideally, we would want to keep everything in the same git repository
because 1) it makes easier to keep the Python and Go logic in sync and 2) some
assets are shared by the Python and Go code.

One obvious way is to checkout the project source in some arbitrary
directory like $HOME/projects/myproject, create a subdirectory go
containing all the Go code, and make it accessible in $GOPATH with ln -s
$HOME/projects/myproject/go $GOPATH/src/myproject. But this solution looks
like a dead-end because symbolic links were declared "not supported" many
times on golang-nuts and golang-dev.

Another way is to checkout the full project source (Go and Python) in
$GOPATH/src/myproject and import "myproject/go" instead of "myproject".
It's perfect and idiomatic from Go's point of view. But from Python's point
of view, it's weird to be forced to live in $GOPATH. Go requires the
project source to live in a directory src under $GOPATH. Now let's imagine
that Python would require the project source to be rooted in a directory
lib (for example) under its own $PYPATH. That would make Go's and Python's
requirements contradictory and impossible to satisfy at the same time.

Another way is to create a Go workspace directly in the project source
tree and add it to the $GOPATH. This implies myproject would have its own
src/bin/pkg directories and this approach was explicitly not recommended on
this mailing list in the past (
https://groups.google.com/forum/#!topic/golang-nuts/dxOFYPpJEXo).

I'm not sure how to solve this in a simple and clean way. What is your
experience solving this issue in your own projects?

Nicolas

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, Aug 15, 2013 at 6:51 AM, Kyle Lemons wrote:

Since Go is almost always the primary language for such projects of mine,
I just make all of the source live in $GOPATH/src/
github.com/kylelemons/project. The Go code imports itself as
github.com/kylelemons/project/whatever and the rest of the code doesn't
really care where it lives.

On Thursday, August 15, 2013 6:51:13 AM UTC+2, Kyle Lemons wrote:

Since Go is almost always the primary language for such projects of mine,
I just make all of the source live in $GOPATH/src/
github.com/kylelemons/project. The Go code imports itself as
github.com/kylelemons/project/whatever and the rest of the code doesn't
really care where it lives.

On Sat, Jan 11, 2014 at 12:22 PM, wrote:

I'm having a similar problem with GOPATH and compiling. The issue I'm
having is with multiple branches of code. A typical project looks like:

project_foo/
trunk/
branch1/
branch2/

I don't have too many branches going at once, but even a single branch is
causing problems because if I have this in the src/ directory of the
GOPATH, then the branches are assumed to be part of the package names. It
seems that the root location of a project or a branch of the project is
tied to the package name. Is there any way around this?

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Saturday, January 11, 2014 at 2:22:27 PM UTC-5, fmcc...@gmail.com wrote:

I'm having a similar problem with GOPATH and compiling. The issue I'm
having is with multiple branches of code. A typical project looks like:

project_foo/
trunk/
branch1/
branch2/

I don't have too many branches going at once, but even a single branch is
causing problems because if I have this in the src/ directory of the
GOPATH, then the branches are assumed to be part of the package names. It
seems that the root location of a project or a branch of the project is
tied to the package name. Is there any way around this?

On Thursday, 15 August 2013 00:01:01 UTC+1, Nicolas Grilly wrote:

What is the recommended way to organize the source tree of a project
written in several programming languages (for example Go and Python)?

Ideally, we would want to keep everything in the same git repository
because 1) it makes easier to keep the Python and Go logic in sync and 2) some
assets are shared by the Python and Go code.

One obvious way is to checkout the project source in some arbitrary
directory like $HOME/projects/myproject, create a subdirectory go
containing all the Go code, and make it accessible in $GOPATH with ln -s
$HOME/projects/myproject/go $GOPATH/src/myproject. But this solution looks
like a dead-end because symbolic links were declared "not supported" many
times on golang-nuts and golang-dev.

Another way is to checkout the full project source (Go and Python) in
$GOPATH/src/myproject and import "myproject/go" instead of "myproject".
It's perfect and idiomatic from Go's point of view. But from Python's point
of view, it's weird to be forced to live in $GOPATH. Go requires the
project source to live in a directory src under $GOPATH. Now let's imagine
that Python would require the project source to be rooted in a directory
lib (for example) under its own $PYPATH. That would make Go's and Python's
requirements contradictory and impossible to satisfy at the same time.

Another way is to create a Go workspace directly in the project source
tree and add it to the $GOPATH. This implies myproject would have its own
src/bin/pkg directories and this approach was explicitly not recommended on
this mailing list in the past (
https://groups.google.com/forum/#!topic/golang-nuts/dxOFYPpJEXo).

I'm not sure how to solve this in a simple and clean way. What is your
experience solving this issue in your own projects?

Nicolas

On Thursday, August 15, 2013 2:21:21 PM UTC+2, frou wrote:

The symlink approach does work, doesn't it? If it stops working then
you'll know about it and can find an alternative.

On Thursday, 15 August 2013 14:50:30 UTC+1, Nicolas Grilly wrote:
No, it does not.

In my experience, many things don't work with symlinks:

- go build/fmt/test myproject/... does not find packages rooted under the
symlinked directory (this is because the related code in the Go standard
library and tools uses filepath.Walk which does not follow symbolic links)
- Packages rooted under the symlinked directory do not appear in godoc
- go install myproject does not know in which directory to install the
resulting binary
- etc.

And according to some of Go authors on this mailing list, symlinks are
avoided by design because their usage solves some issues (like mine) but
also brings a lot of problems (like the possibility of creating circular
references and having several different paths leading to the same file,
none of them being canonical).

But maybe I'm missing something. Do you use symlinks in your own projects,
without the problems I listed above?

On Thursday, August 15, 2013 4:26:32 PM UTC+2, frou wrote:

go {build,install,fmt,test} proj/a/b

work for me where proj is a symlink in GOPATH to outside of GOPATH.

Indeed those packages don't show up in godoc.

I may be missing something myself. I have only experimented with one such
multi-language project and haven't read the ML discussions you mention.

On Aug 14, 2013 10:46 PM, "Nicolas Grilly" wrote:

What is the recommended way to organize the source tree of a project
written in several programming languages (for example Go and Python)?

Ideally, we would want to keep everything in the same git repository
because 1) it makes easier to keep the Python and Go logic in sync and 2) some
assets are shared by the Python and Go code.

One obvious way is to checkout the project source in some arbitrary
directory like $HOME/projects/myproject, create a subdirectory go
containing all the Go code, and make it accessible in $GOPATH with ln -s
$HOME/projects/myproject/go $GOPATH/src/myproject. But this solution looks
like a dead-end because symbolic links were declared "not supported" many
times on golang-nuts and golang-dev.

Another way is to checkout the full project source (Go and Python) in
$GOPATH/src/myproject and import "myproject/go" instead of "myproject".
It's perfect and idiomatic from Go's point of view. But from Python's point
of view, it's weird to be forced to live in $GOPATH. Go requires the
project source to live in a directory src under $GOPATH. Now let's imagine
that Python would require the project source to be rooted in a directory
lib (for example) under its own $PYPATH. That would make Go's and Python's
requirements contradictory and impossible to satisfy at the same time.

Another way is to create a Go workspace directly in the project source
tree and add it to the $GOPATH. This implies myproject would have its own
src/bin/pkg directories and this approach was explicitly not recommended on
this mailing list in the past (
https://groups.google.com/forum/#!topic/golang-nuts/dxOFYPpJEXo).

I'm not sure how to solve this in a simple and clean way. What is your
experience solving this issue in your own projects?

Nicolas

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, August 15, 2013 3:55:47 PM UTC+2, Jeremy Wall wrote:

You can have multiple GOPATHs. Why not just add an entry for the path
where your go code in the larger project lives?

On Thu, Aug 15, 2013 at 9:24 AM, Nicolas Grilly wrote:
Just to make sure I understand your suggestion, let's say the Go files of
my project are in github.com/ngrilly/myproject/go/*.go. What would you
add to GOPATH?

You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, August 15, 2013 5:32:30 PM UTC+2, Jeremy Wall wrote:
Well first you'd have to do some rearranging. Add a gopath like directory
with a src/ and pkg/ directory in your repo.
github.com/ngrilly/myproject/go/{src,pkg}<http://github.com/ngrilly/myproject/go/%7Bsrc,pkg%7D>for instance

Then move the *.go files into a reasonable directory in the src/
directory. .../go/src/myproject/*.go for instance.

Then in your build script which you will need anyway since this is a multi
language project add e.g. github.com/ngrilly/myproject/go/src/ to your
GOPATH so all go builds of myproject/... will work.

This will work for all your developers and packagers. It will not work for
making the the go code go gettable but this is a multilanguage project
where Go is not the primarly language so that is likely not really a goal
in the first place.

if go gettability *is* a goal then you should probably have the go code in
a separate repo instead.

On Thursday, August 15, 2013 9:32:33 PM UTC+2, TR NS wrote:

In the mean time, to work around I've had to do what you call the
"obvious" way, but instead of using a symlink I have a small shell script
in my project that reassigns the $GOPATH, which I can run before working on
the project. Annoying but effective. Perhaps a more ideal solution would be
to do the exporting when running a local build tool (e.g. Rake). By
exporting the path in such a script it would only be temporary while the
build is taking place, but then of course you have to write a bunch of
build tasks to route to the go tool.

You're right: what I called the "obvious way" is more hacky than obvious :)

Could you give some details about your setup? What is the layout of your
source tree? Is the root of the Go workspace inside or outside your source
tree? To which directory $GOPATH points?

-- Nicolas

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, August 15, 2013 9:54:50 PM UTC+2, Rodrigo Kochenburger wrote:

By the way, guys. If you need more flexibility you can always call the
tools directly (compiler, linker, etc) instead of using the higher level
tool and build your own build script.

On Thursday, August 15, 2013 10:22:05 PM UTC+2, andrey mirtchovski wrote:

I don't understand the issue here. GOPATH is only really needed when
you're doing "go get" or "go install". as long as you have a default
GOPATH for the stuff you "go get", then the rest of your projects can
be wherever you want. use relative imports to build library
hierarchies completely outside the canon, but still use the go tool to
compile them. I have plenty of projects outside GOPATH (including
stuff with relative imports), the go tool manages them just fine*.

The problem is that the use of relative import paths is heavily
discouraged by most members of the Go team, according to what I've read in
this list.

On Thursday, August 15, 2013 11:43:27 PM UTC+2, andrey mirtchovski wrote:

that said, the relative import trick is not for everybody, i agree. if
i understand your constraints correctly, what you really are asking
for is to have your Go code live in two places at the same time:
inside GOPATH when convenient, and outside GOPATH the rest of the
time. you can do with some degree of success with union mounts on
unix-y operating systems, or you can even use sshfs to do it. just
mount at the right path and, even though you'll be editing outside
GOPATH the go tool will find your code. that gives you a fine control
over naming libraries as only the last part of the path must agree
with the stated package name. here's an example:

http://play.golang.org/p/twUCNd2UhJ

Your suggestion reminds me of Plan 9 name spaces and the paper "Lexical File
Names in Plan 9 or Getting Dot-Dot Right" by Rob Pike
(http://plan9.bell-labs.com/sys/doc/lexnames.html).

On Thursday, August 15, 2013 3:47:48 PM UTC-4, Nicolas Grilly wrote:
You're right: what I called the "obvious way" is more hacky than obvious :)

Could you give some details about your setup? What is the layout of your
source tree? Is the root of the Go workspace inside or outside your source
tree? To which directory $GOPATH points?

On Sunday, August 18, 2013 2:12:01 AM UTC+2, TR NS wrote:

First, let me qualify things by saying I am new to Go, so I am still
learning how to do things too. Having said that, do you know about Ruby
Gemsets? We'll its basically like that. By changing the $GOPATH to point to
each project as I work on it, I have an isolated set of Go installs all
geared for use by the project I am working on. Of course the downside it
that It creates redundancy, but I can live with it.

On Thursday, 15 August 2013 00:01:01 UTC+1, Nicolas Grilly wrote:

Another way is to checkout the full project source (Go and Python) in
$GOPATH/src/myproject and import "myproject/go" instead of "myproject".
It's perfect and idiomatic from Go's point of view. But from Python's point
of view, it's weird to be forced to live in $GOPATH. Go requires the
project source to live in a directory src under $GOPATH. Now let's imagine
that Python would require the project source to be rooted in a directory
lib (for example) under its own $PYPATH. That would make Go's and Python's
requirements contradictory and impossible to satisfy at the same time.

On Sun, Aug 18, 2013 at 2:55 AM, Benjamin Measures wrote:

It's only impossible if you've really completely ruled out symlinks. If
you can't symlink go, why not python? Eg.
ln -s $GOPATH/src/myproject/python $PYPATH/lib

What is the recommended way to organize the source tree of a project written
in several programming languages (for example Go and Python)?

Ideally, we would want to keep everything in the same git repository because
1) it makes easier to keep the Python and Go logic in sync and 2) some
assets are shared by the Python and Go code.

One obvious way is to checkout the project source in some arbitrary
directory like $HOME/projects/myproject, create a subdirectory go containing
all the Go code, and make it accessible in $GOPATH with ln -s
$HOME/projects/myproject/go $GOPATH/src/myproject. But this solution looks
like a dead-end because symbolic links were declared "not supported" many
times on golang-nuts and golang-dev.

Another way is to checkout the full project source (Go and Python) in
$GOPATH/src/myproject and import "myproject/go" instead of "myproject". It's
perfect and idiomatic from Go's point of view. But from Python's point of
view, it's weird to be forced to live in $GOPATH. Go requires the project
source to live in a directory src under $GOPATH. Now let's imagine that
Python would require the project source to be rooted in a directory lib (for
example) under its own $PYPATH. That would make Go's and Python's
requirements contradictory and impossible to satisfy at the same time.

Another way is to create a Go workspace directly in the project source tree
and add it to the $GOPATH. This implies myproject would have its own
src/bin/pkg directories and this approach was explicitly not recommended on
this mailing list in the past
(https://groups.google.com/forum/#!topic/golang-nuts/dxOFYPpJEXo).

I'm not sure how to solve this in a simple and clean way. What is your
experience solving this issue in your own projects?

Nicolas

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.