FAQ
I'm nearly complete with my first Go program but have stumbled into an odd
behavior in how exec.Command handles arguments sent to it which contains a
space. They appear to be treated differently - such as --proxy
127.0.0.1:8080 is handled differently then -H "Accept: application/json".
The reason for my program is to create a wrapper for curl which allows for
repeated arguments to live in a config file making repeated curl command
much shorter.

Below is an abridged version of what I'm doing in my Go program.

// mail-list-post

package main

import (

"bytes"

"fmt"

"io"

"os"

"os/exec"

)

func main() {

// #1 version without requiring a proxy

//finalCmd := []string{"--insecure", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}

// #2 actual version I was testing with

//finalCmd := []string{"--insecure", "--proxy 127.0.0.1:8080", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}

// #3 while this one works where I've broken up the --proxy argument into 2 values

finalCmd := []string{"--insecure", "--proxy", "127.0.0.1:8080", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}


cmd := exec.Command("curl", finalCmd...)


// Catch stdout

stdout, err := cmd.StdoutPipe()

if err != nil {

fmt.Printf("Error:\n\t%v", err)

os.Exit(0)

}


// Catch stderr

errorReader, err := cmd.StderrPipe()

if err != nil {

fmt.Printf("Error:\n\t%v", err)

os.Exit(0)

}


// Start executing the curl command

err = cmd.Start()

if err != nil {

fmt.Printf("Error occurred: %v\n", err)

os.Exit(0)

}


// Use IO copy to print curl's stdout

io.Copy(os.Stdout, stdout)

// Buffer stderr for possible future display

buf := new(bytes.Buffer)

buf.ReadFrom(errorReader)


// Wait for curl to complete

cmd.Wait()


// Check for a curl error

if cmd.ProcessState.String() != "exit status 0" {

fmt.Println("A curl error occurred:\n")

fmt.Printf("\t%s\n", buf.String())

}

}


Everything works fine with the version above - curl ends up requesting
www.google.com's web page and displaying it from standard out.
(assuming you have a proxy listening on 8080)

The strange thing is that if you comment out the 3rd version of finalCmd
and uncomment #2, you get the the following:

A curl error occurred:

curl: option --proxy 127.0.0.1:8080: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

If you take those same options and manually type the command in a terminal,
it works fine.

For the curious, the full program is on GitHub
here: https://github.com/mtesauro/jerry-curl/blob/master/jerry-curl.go

I removed the code which takes command line arguments plus arguments from a
config file and sets up the situation above. It allowed me to prove that
the issue was isolated to the bit of code above as the issue occurs in both
the long and short versions.

If I need to split command line arguments on space and change how I create
the slice sent to exec.Command, that's not a problem but I find it very odd
that the space is not treated the same for a double dashed argument and a
single dashed argument. (Yes, I tried quoting the IP/port for the proxy)

So is this the expected behavior for exec.Command?

Thanks for any help/suggestions/advice.

Cheers!

Matt Tesauro

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dan Kortschak at Jan 30, 2013 at 5:11 am
    The arguments passed by exec.Command are not split further than you have
    by having them in a slice of strings. There is no shell between the Go
    program and the exec'd command.
    On Tue, 2013-01-29 at 20:15 -0800, Matt Tesauro wrote:
    //finalCmd := []string{"--insecure", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}

    // #2 actual version I was testing with

    //finalCmd := []string{"--insecure", "--proxy 127.0.0.1:8080", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}

    // #3 while this one works where I've broken up the --proxy argument into 2 values

    finalCmd := []string{"--insecure", "--proxy", "127.0.0.1:8080", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Andrew Gerrand at Jan 30, 2013 at 5:17 am

    On 30 January 2013 15:15, Matt Tesauro wrote:

    If you take those same options and manually type the command in a
    terminal, it works fine.

    When you type:

    $ echo foo bar

    at a shell, the shell breaks up your command into arguments, and passes
    them as an array of strings to the exec syscall. This array is known as
    argv to C programs, or os.Args to Go programs.

    The important part here is that the *shell* splits the arguments at the
    spaces. When you quote arguments,

    $ echo "foo bar"

    it's the shell that makes sure that "foo bar" is passed as a single
    argument. Note that it doesn't pass the quotes.

    So when you type:

    $ echo -n "foo bar"

    the argument array is []string{"echo", "-n", "foo bar"}.

    Now when you use the exec.Command function, it does no interpretation of
    the arguments (except the first one, the program that should be executed).
    It just passes the args straight to the process as it execs it.

    That's why "--proxy" and "127.0.0.1:8080" must be specified as separate
    args in your Go program, so that it works just as the shell does.

    Andrew

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nate Finch at Jan 30, 2013 at 12:34 pm
    Yep, I hit this too. Took me a minute to figure it out my first time.

    Just think of it this way - the args given to exec.Command directly
    populate the os.Args (argv) of the target application as-is.
    Related - Anyone know of a go package that'll take a command line style
    string and split it up into arguments, like the shell does? I'd like to be
    able to read in commandline-like strings and pass them to exec.Command...
    and the more I think about the syntax, the more I hope someone else has
    already done this :)

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse McNelis at Jan 30, 2013 at 12:51 pm

    On Wed, Jan 30, 2013 at 11:34 PM, Nate Finch wrote:

    Related - Anyone know of a go package that'll take a command line style
    string and split it up into arguments, like the shell does? I'd like to be
    able to read in commandline-like strings and pass them to exec.Command...
    and the more I think about the syntax, the more I hope someone else has
    already done this :)
    The format of a "command line like string" really depends on the shell you
    use, as each shell is different.
    The simplest solution to this is to have exec start a shell and pass it
    that string.
    eg.
    exec.Command("/bin/sh", "echo 'pizza' | wc -c")

    --
    =====================
    http://jessta.id.au

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nate Finch at Jan 30, 2013 at 1:49 pm
    Ahh, of course. That's a good workaround, and one I've even used before,
    just didn't think of it. You have a good point that different shells parse
    command lines in different ways... this is probably the best bet, and just
    make different implementations per OS.
    On Wednesday, January 30, 2013 7:51:15 AM UTC-5, Jesse McNelis wrote:

    On Wed, Jan 30, 2013 at 11:34 PM, Nate Finch <nate....@gmail.com<javascript:>
    wrote:
    Related - Anyone know of a go package that'll take a command line style
    string and split it up into arguments, like the shell does? I'd like to be
    able to read in commandline-like strings and pass them to exec.Command...
    and the more I think about the syntax, the more I hope someone else has
    already done this :)
    The format of a "command line like string" really depends on the shell you
    use, as each shell is different.
    The simplest solution to this is to have exec start a shell and pass it
    that string.
    eg.
    exec.Command("/bin/sh", "echo 'pizza' | wc -c")

    --
    =====================
    http://jessta.id.au
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Matt Tesauro at Jan 30, 2013 at 2:26 pm
    @Nate

    I did something like what you're talking about in my full program. I take
    the command line arguments to jerry-curl and parse them to sort out those
    for jerry-curl and those to pass through to curl. You can see where I did
    this on GitHub in the parseArgs function on line 278
    here: https://github.com/mtesauro/jerry-curl/blob/master/jerry-curl.go

    That's not quite what your asking but quite close.
    On Wednesday, January 30, 2013 7:49:56 AM UTC-6, Nate Finch wrote:

    Ahh, of course. That's a good workaround, and one I've even used before,
    just didn't think of it. You have a good point that different shells parse
    command lines in different ways... this is probably the best bet, and just
    make different implementations per OS.
    On Wednesday, January 30, 2013 7:51:15 AM UTC-5, Jesse McNelis wrote:
    On Wed, Jan 30, 2013 at 11:34 PM, Nate Finch wrote:

    Related - Anyone know of a go package that'll take a command line style
    string and split it up into arguments, like the shell does? I'd like to be
    able to read in commandline-like strings and pass them to exec.Command...
    and the more I think about the syntax, the more I hope someone else has
    already done this :)
    The format of a "command line like string" really depends on the shell
    you use, as each shell is different.
    The simplest solution to this is to have exec start a shell and pass it
    that string.
    eg.
    exec.Command("/bin/sh", "echo 'pizza' | wc -c")

    --
    =====================
    http://jessta.id.au
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • John Asmuth at Jan 30, 2013 at 2:57 pm
    https://github.com/kballard/go-shellquote
    On Wednesday, January 30, 2013 7:34:24 AM UTC-5, Nate Finch wrote:

    Yep, I hit this too. Took me a minute to figure it out my first time.

    Just think of it this way - the args given to exec.Command directly
    populate the os.Args (argv) of the target application as-is.
    Related - Anyone know of a go package that'll take a command line style
    string and split it up into arguments, like the shell does? I'd like to be
    able to read in commandline-like strings and pass them to exec.Command...
    and the more I think about the syntax, the more I hope someone else has
    already done this :)
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Matt Tesauro at Jan 30, 2013 at 3:02 pm
    First, props to the golang group/golang community for sever very quick
    answers. The answers informed my further experimentation today and now I
    see what's happening.

    While I get why quoting is needed in certain circumstances in a shell, the
    thing confusing me was why I need to split the --proxy and its value but
    don't need to for -H. I've been playing with it more this AM and it
    finally clicked. My code is not only broken for --proxy, but -H isn't
    working as I thought. For example, if finalCmd is this:

    finalCmd := []string{"--proxy", "127.0.0.1:8080", "-H \"Accept:
    application/json\"", "http://www.google.com/"}

    it works - or at least appears to work. However, looking in my local
    proxy, I see that the Accept header is also wrong:

    GET http://www.google.com/ HTTP/1.1
    User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1
    zlib/1.2.3.4 libidn/1.23 librtmp/2.3
    Host: www.google.com
    Accept: */*
    Proxy-Connection: Keep-Alive
    "Accept: application/json"

    Quoted headers - DOH! If you do the same thing manually in a shell (bash
    on Linux), it works - headers and all:

    $ curl --proxy 127.0.0.1:8080 -H "Accept: application/json"
    http://www.google.com

    My mistake was to think of "--proxy 127.0.0.1:8080" and "-H \"Accept:
    application/json\"" as atomic units rather then items from a command line
    that need to be separated by space. From how -H is handled, I see that any
    string in my slice which has a space is quoted while constructing the
    command.

    Since the proxy error was very apparent with an error from curl (non-zero
    exit), the -H was quietly sent in HTTP which I missed until this AM.

    It may help to have a more complex example for exec.Command in the docs
    since the tr example doesn't have command line options with following
    values.

    Thanks all for your help - i'll add a bit of code to split any command line
    argument with an space into two strings in my slice.

    Cheers!
    On Tuesday, January 29, 2013 11:16:29 PM UTC-6, Andrew Gerrand wrote:


    On 30 January 2013 15:15, Matt Tesauro <mtes...@gmail.com <javascript:>>wrote:
    If you take those same options and manually type the command in a
    terminal, it works fine.

    When you type:

    $ echo foo bar

    at a shell, the shell breaks up your command into arguments, and passes
    them as an array of strings to the exec syscall. This array is known as
    argv to C programs, or os.Args to Go programs.

    The important part here is that the *shell* splits the arguments at the
    spaces. When you quote arguments,

    $ echo "foo bar"

    it's the shell that makes sure that "foo bar" is passed as a single
    argument. Note that it doesn't pass the quotes.

    So when you type:

    $ echo -n "foo bar"

    the argument array is []string{"echo", "-n", "foo bar"}.

    Now when you use the exec.Command function, it does no interpretation of
    the arguments (except the first one, the program that should be executed).
    It just passes the args straight to the process as it execs it.

    That's why "--proxy" and "127.0.0.1:8080" must be specified as separate
    args in your Go program, so that it works just as the shell does.

    Andrew
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 30, '13 at 5:07a
activeJan 30, '13 at 3:02p
posts9
users6
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase