FAQ
In a quest to build a file upload proxy (http://goo.gl/AgCMc9), I've run
into a situation where my go program needs to take an incoming file upload
from a user and stream it in real time to N other HTTP servers. I can't
wait until the user finishes sending his file - I need to stream the data
to the HTTP servers as it comes in from the user. What I'd really like is
an io.Writer interface to http.Client or http.Request, but I can't find the
best way to do it.

The only way to send a POST body using the http package seems to be by
assigning an io.ReadCloser to a new http.Request object. I could implement
a ReadCloser that reads data off the file upload stream and feeds it to the
http client (or better yet, pass the original file upload Request.Body
directly into the new Request), but this doesn't work as well when I want
to duplicate the stream to multiple file servers with throttling and
monitoring of the transfers.

The ideal interface that I can imagine is an io.Writer interface to the TCP
connections, where I copy a []byte for each HTTP server transfer. But if I
do that, do I have to completely stop using the http package and manage the
entire http connection myself?

Thanks!

--
Sam

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Jesse McNelis at May 9, 2014 at 9:33 am

    On Fri, May 9, 2014 at 5:57 PM, Sam Ghods wrote:
    The only way to send a POST body using the http package seems to be by
    assigning an io.ReadCloser to a new http.Request object. I could implement a
    ReadCloser that reads data off the file upload stream and feeds it to the
    http client (or better yet, pass the original file upload Request.Body
    directly into the new Request), but this doesn't work as well when I want to
    duplicate the stream to multiple file servers with throttling and monitoring
    of the transfers.
    You want,
    http://golang.org/pkg/io/#Pipe
    Write() to one end of the pipe and give the other end of the pipe to
    the http.Request.

    If you need to write to many of these you can put the write end of the pipe in a
    http://golang.org/pkg/io/#MultiWriter
    and write to that.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
  • Sam Ghods at May 9, 2014 at 10:15 am
    Yes, exactly what I was looking for. Crazy how much stuff is in the
    standard lib. Thanks!

    On Fri, May 9, 2014 at 2:33 AM, Jesse McNelis wrote:
    On Fri, May 9, 2014 at 5:57 PM, Sam Ghods wrote:
    The only way to send a POST body using the http package seems to be by
    assigning an io.ReadCloser to a new http.Request object. I could
    implement a
    ReadCloser that reads data off the file upload stream and feeds it to the
    http client (or better yet, pass the original file upload Request.Body
    directly into the new Request), but this doesn't work as well when I want to
    duplicate the stream to multiple file servers with throttling and
    monitoring
    of the transfers.
    You want,
    http://golang.org/pkg/io/#Pipe
    Write() to one end of the pipe and give the other end of the pipe to
    the http.Request.

    If you need to write to many of these you can put the write end of the
    pipe in a
    http://golang.org/pkg/io/#MultiWriter
    and write to that.


    --
    Sam

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
  • Rui Ueyama at May 9, 2014 at 4:52 pm
    It's worth noting that PipeReader/PipeWriter are synchronous, and there's
    no buffering in the pipe. So upload throughput would be limited by download
    time of the other client if you copy traffic using it, and vice versa.

    If you fan out incoming traffic to multiple clients using MultiWriter,
    throughput would be limited to the slowest client in the group, as there's
    also no buffering in MultiWriter.

    On Fri, May 9, 2014 at 3:14 AM, Sam Ghods wrote:

    Yes, exactly what I was looking for. Crazy how much stuff is in the
    standard lib. Thanks!

    On Fri, May 9, 2014 at 2:33 AM, Jesse McNelis wrote:
    On Fri, May 9, 2014 at 5:57 PM, Sam Ghods wrote:
    The only way to send a POST body using the http package seems to be by
    assigning an io.ReadCloser to a new http.Request object. I could
    implement a
    ReadCloser that reads data off the file upload stream and feeds it to the
    http client (or better yet, pass the original file upload Request.Body
    directly into the new Request), but this doesn't work as well when I want to
    duplicate the stream to multiple file servers with throttling and
    monitoring
    of the transfers.
    You want,
    http://golang.org/pkg/io/#Pipe
    Write() to one end of the pipe and give the other end of the pipe to
    the http.Request.

    If you need to write to many of these you can put the write end of the
    pipe in a
    http://golang.org/pkg/io/#MultiWriter
    and write to that.


    --
    Sam

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to [email protected].
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedMay 9, '14 at 7:57a
activeMay 9, '14 at 4:52p
posts4
users3
websitegolang.org

People

Translate

site design / logo © 2023 Grokbase