FAQ
Requesting help to identify if my usage of os/exec has issues of if there
are limitations I am not recognizing.

I need to read from cmd stdout/stderr and output to os.Stdout & logfile.
I am using multireader (created from cmd.StdoutPipe() and cmd.StderrPipe())
and multiwriter (os.Stdout and bytes.Buffer) for this.
The bytes.Buffer is later written to Logger.

Code is as at:
http://play.golang.org/p/KPoAZvBg2T

I see sporadic hang especially with large outputs.

Thanks.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Dave Cheney at Nov 20, 2013 at 1:49 am
    I don't think you can use mutlireader like that

    http://godoc.org/io#MultiReader

    The readers are drained sequentialy, so if stdout is idle and stderr
    has filled it's pipe, your process will block.
    On Wed, Nov 20, 2013 at 12:47 PM, Shivakumar GN wrote:
    Requesting help to identify if my usage of os/exec has issues of if there
    are limitations I am not recognizing.

    I need to read from cmd stdout/stderr and output to os.Stdout & logfile.
    I am using multireader (created from cmd.StdoutPipe() and cmd.StderrPipe())
    and multiwriter (os.Stdout and bytes.Buffer) for this.
    The bytes.Buffer is later written to Logger.

    Code is as at:
    http://play.golang.org/p/KPoAZvBg2T

    I see sporadic hang especially with large outputs.

    Thanks.


    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Shivakumar GN at Nov 20, 2013 at 1:54 am
    Isn't the block temporary since any command will either write to stdout or
    exit resulting in EOF on Stdout?

    On Wed, Nov 20, 2013 at 7:19 AM, Dave Cheney wrote:

    I don't think you can use mutlireader like that

    http://godoc.org/io#MultiReader

    The readers are drained sequentialy, so if stdout is idle and stderr
    has filled it's pipe, your process will block.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Shivakumar GN at Nov 20, 2013 at 2:08 am
    I now understand the dependency it creates due to sequential reading.

    So, how to achieve what I need?

    cmd.Stdout and cmd.Stderr must appear in near real time on os.Stdout.
    And it must be written to log once cmd is completed.

    Is this possible at all?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Nov 20, 2013 at 2:12 am
    That sounds like a lot of assumptions. What are the processes you are running which are failing ?


    On 20 Nov 2013, at 12:54, Shivakumar GN wrote:

    Isn't the block temporary since any command will either write to stdout or exit resulting in EOF on Stdout?

    On Wed, Nov 20, 2013 at 7:19 AM, Dave Cheney wrote:
    I don't think you can use mutlireader like that

    http://godoc.org/io#MultiReader

    The readers are drained sequentialy, so if stdout is idle and stderr
    has filled it's pipe, your process will block.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Shivakumar GN at Nov 20, 2013 at 3:52 am

    On Wed, Nov 20, 2013 at 7:42 AM, Dave Cheney wrote:

    That sounds like a lot of assumptions. What are the processes you are
    running which are failing ?

    Had a look at http://golang.org/src/pkg/io/multi.go?s=217:275#L10.

    As you suspect, the assumption about the processes is not right. These are
    custom scripts that may live for 1s to 20mins written by other people and I
    have little control on these.

    How to achieve what I need:
             cmd.Stdout and cmd.Stderr must appear in near real time on
    os.Stdout.
             The output must be logged into a file once cmd is completed.

    Should I implement my own multireader that is _not_ sequential?

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Nov 20, 2013 at 3:57 am
    Sounds like the best course of action. Be sure to make the readers of
    stdout and stderr line buffered so you don't corrupt the merged
    output.
    On Wed, Nov 20, 2013 at 2:52 PM, Shivakumar GN wrote:
    On Wed, Nov 20, 2013 at 7:42 AM, Dave Cheney wrote:

    That sounds like a lot of assumptions. What are the processes you are
    running which are failing ?


    Had a look at http://golang.org/src/pkg/io/multi.go?s=217:275#L10.

    As you suspect, the assumption about the processes is not right. These are
    custom scripts that may live for 1s to 20mins written by other people and I
    have little control on these.

    How to achieve what I need:
    cmd.Stdout and cmd.Stderr must appear in near real time on
    os.Stdout.
    The output must be logged into a file once cmd is completed.

    Should I implement my own multireader that is _not_ sequential?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Carlos Castillo at Nov 21, 2013 at 12:12 am
    Does this do what you are looking for:

    cmd := exec.Cmd(...)

    output := io.MultiWriter(os.Stdout, file)
    cmd.Stdout, cmd.Stderr = output, output

    cmd.Start()

    This should redirect both the stdout and stderr of the command to both your
    program's stdout and a file.
    On Tuesday, November 19, 2013 7:52:21 PM UTC-8, Shivakumar GN wrote:


    On Wed, Nov 20, 2013 at 7:42 AM, Dave Cheney <da...@cheney.net<javascript:>
    wrote:
    That sounds like a lot of assumptions. What are the processes you are
    running which are failing ?

    Had a look at http://golang.org/src/pkg/io/multi.go?s=217:275#L10.

    As you suspect, the assumption about the processes is not right. These are
    custom scripts that may live for 1s to 20mins written by other people and I
    have little control on these.

    How to achieve what I need:
    cmd.Stdout and cmd.Stderr must appear in near real time on
    os.Stdout.
    The output must be logged into a file once cmd is completed.

    Should I implement my own multireader that is _not_ sequential?
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Shivakumar GN at Nov 21, 2013 at 1:04 am
    The intent is right. Seems like exec.Cmd does not have the issue of
    sequential read as that of multireader.
    Very likely this will work for me.

    Will check. Thanks.


    On Thu, Nov 21, 2013 at 5:42 AM, Carlos Castillo wrote:

    Does this do what you are looking for:

    cmd := exec.Cmd(...)

    output := io.MultiWriter(os.Stdout, file)
    cmd.Stdout, cmd.Stderr = output, output

    cmd.Start()

    This should redirect both the stdout and stderr of the command to both
    your program's stdout and a file.
    On Tuesday, November 19, 2013 7:52:21 PM UTC-8, Shivakumar GN wrote:

    On Wed, Nov 20, 2013 at 7:42 AM, Dave Cheney wrote:

    That sounds like a lot of assumptions. What are the processes you are
    running which are failing ?

    Had a look at http://golang.org/src/pkg/io/multi.go?s=217:275#L10.

    As you suspect, the assumption about the processes is not right. These
    are custom scripts that may live for 1s to 20mins written by other people
    and I have little control on these.

    How to achieve what I need:
    cmd.Stdout and cmd.Stderr must appear in near real time on
    os.Stdout.
    The output must be logged into a file once cmd is completed.

    Should I implement my own multireader that is _not_ sequential?

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 20, '13 at 1:47a
activeNov 21, '13 at 1:04a
posts9
users3
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase