FAQ
Howdy -

I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
Cloud Storage and am able to list files in a bucket, get metadata on a
single file, etc. but I can't figure out how to download a file. As I
would expect, I can't simply GET the MediaLink that is returned with each
file's metadata (it gives me a "Login Required" error). But I don't see
any functions in v1beta2/storage-gen.go for requesting/downloading file
contents. I do see the reverse (uploading file contents) however...

Thanks very much!
- Ian


Apologies if this is the wrong forum for this question (please direct me to
the right place!)...

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Kyle Lemons at Jan 27, 2014 at 8:50 pm
    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using
    your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please direct me
    to the right place!)...

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Rose at Jan 27, 2014 at 9:01 pm
    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n", err)
    os.Exit(1)
    }

    filename :=
    "5105650963054592/2014-01-27-15-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.


    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose <ianr...@gmail.com<javascript:>
    wrote:
    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please direct me
    to the right place!)...

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Carlos Castillo at Jan 28, 2014 at 2:12 am
    Are you using an oauth2 authenticated http.Client to make the request? The
    one created on line 1 of the main function should be authenticated if the
    API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n",
    err)
    os.Exit(1)
    }

    filename :=
    "5105650963054592/2014-01-27-15-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please direct me
    to the right place!)...

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Rose at Jan 28, 2014 at 3:00 am
    Yes, as you can see in my example code, I am using an authenticated
    http.Client obtained from serviceaccount.NewClient(). And I know that this
    client works because I can use it to do things like list the objects in a
    bucket.

    If I try client.Get(obj.MediaLink), then I receive a 404 from the server.
      To test things out, I tried using this same client to hit one of my local
    servers and from this I was able to confirm that the request includes a
    "Authorization: Bearer <token>" header, as expected. However I'm wondering
    if there are some other headers that I need to set in my request?



    On Mon, Jan 27, 2014 at 9:12 PM, Carlos Castillo wrote:

    Are you using an oauth2 authenticated http.Client to make the request? The
    one created on line 1 of the main function should be authenticated if the
    API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n",
    err)
    os.Exit(1)
    }

    filename := "5105650963054592/2014-01-27-15-01-29-01-04-146bc100000000-
    12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename,
    err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please direct
    me to the right place!)...

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Carlos Castillo at Jan 28, 2014 at 3:12 am
    Just covering the bases, since your posted code didn't show how it
    performed the actual download (your code doesn't show the line that you
    claim "fails"). BTW, your posted code ignores the error from storage.New().
    On Monday, January 27, 2014 6:59:54 PM UTC-8, Ian Rose wrote:

    Yes, as you can see in my example code, I am using an authenticated
    http.Client obtained from serviceaccount.NewClient(). And I know that this
    client works because I can use it to do things like list the objects in a
    bucket.

    If I try client.Get(obj.MediaLink), then I receive a 404 from the server.
    To test things out, I tried using this same client to hit one of my local
    servers and from this I was able to confirm that the request includes a
    "Authorization: Bearer <token>" header, as expected. However I'm wondering
    if there are some other headers that I need to set in my request?




    On Mon, Jan 27, 2014 at 9:12 PM, Carlos Castillo <cook...@gmail.com<javascript:>
    wrote:
    Are you using an oauth2 authenticated http.Client to make the request?
    The one created on line 1 of the main function should be authenticated if
    the API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n",
    err)
    os.Exit(1)
    }

    filename := "5105650963054592/2014-01-27-15-01-29-01-04-146bc100000000-
    12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename,
    err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please direct
    me to the right place!)...

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Rose at Jan 28, 2014 at 3:22 am
    Good point - I probably copy-and-pasted the storage.New line and forgot to
    add back the err variable. I'll fix that.

    In other news, I just tried with a different bucket & filename that are
    much shorter than the "real ones" (as shown in my sample code) and doing
    client.Get worked! So this might be user error on my part, or maybe there
    is some weird interaction with long file names? Stay tuned...

    On Mon, Jan 27, 2014 at 10:12 PM, Carlos Castillo wrote:

    Just covering the bases, since your posted code didn't show how it
    performed the actual download (your code doesn't show the line that you
    claim "fails"). BTW, your posted code ignores the error from storage.New().

    On Monday, January 27, 2014 6:59:54 PM UTC-8, Ian Rose wrote:

    Yes, as you can see in my example code, I am using an authenticated
    http.Client obtained from serviceaccount.NewClient(). And I know that this
    client works because I can use it to do things like list the objects in a
    bucket.

    If I try client.Get(obj.MediaLink), then I receive a 404 from the server.
    To test things out, I tried using this same client to hit one of my local
    servers and from this I was able to confirm that the request includes a
    "Authorization: Bearer <token>" header, as expected. However I'm wondering
    if there are some other headers that I need to set in my request?



    On Mon, Jan 27, 2014 at 9:12 PM, Carlos Castillo wrote:

    Are you using an oauth2 authenticated http.Client to make the request?
    The one created on line 1 of the main function should be authenticated if
    the API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n",
    err)
    os.Exit(1)
    }

    filename := "5105650963054592/2014-01-27-1
    5-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename,
    err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please direct
    me to the right place!)...

    --
    You received this message because you are subscribed to the Google
    Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to golang-nuts...@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "golang-nuts" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/golang-nuts/juguXl-ss2Q/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    golang-nuts+unsubscribe@googlegroups.com.

    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Rose at Jan 28, 2014 at 4:05 am
    I believe I have narrowed the problem down to the fact that I am trying to
    download files from within a folder. As a test case, I created a new
    bucket "test987" and uploaded two json files, one without an enclosing
    folder and one to a folder named "foo". So the structure is like:

    test987 bucket:
       - outside.json
       - foo/
         - inside.json


    I can fetch metadata (including the MediaLink) on both outside.json and
    foo/inside.json. However if I think client.Get() the MediaLink in each
    case, it works on outside.json but returns a 404 on foo/inside.json. Here
    is my updated program:

    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "io/ioutil"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n", err)
    os.Exit(1)
    }

    filename := "foo/inside.json"
    bucket := "test987"

    service, err := storage.New(client)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create GCS client client: %v\n", err)
    os.Exit(1)
    }

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)

    url := obj.MediaLink
    fmt.Printf("Fetching %s\n", url)

    rsp, err := client.Get(url)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to direct fetch file: %v\n", err)
    os.Exit(1)
    }

    fmt.Printf("Status code: %d\n", rsp.StatusCode)

    fmt.Printf("Reading %d bytes of response body...\n", rsp.ContentLength)
    content, err := ioutil.ReadAll(rsp.Body)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to ready body of fetch-file response: %v\n",
    err)
    os.Exit(1)
    }

    fmt.Printf("Read %d bytes from response body...\n", len(content))

    fmt.Println("")
    fmt.Println(string(content))
    }

    And here is the output when filename is set to "foo/inside.json":

    Found test987/foo/inside.json...
    MediaLink:
    https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
    Fetching
    https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
    Status code: 404
    Reading -1 bytes of response body...
    Read 9 bytes from response body...

    Not Found

    And here is the output when filename is set to "outside.json":

    Found test987/outside.json...
    MediaLink:
    https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
    Fetching
    https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
    Status code: 200
    Reading 798 bytes of response body...
    Read 798 bytes from response body...

    {
       "albums": [
         (...content truncated for brevity...)
       ]
    }



    On Monday, January 27, 2014 10:22:08 PM UTC-5, Ian Rose wrote:

    Good point - I probably copy-and-pasted the storage.New line and forgot to
    add back the err variable. I'll fix that.

    In other news, I just tried with a different bucket & filename that are
    much shorter than the "real ones" (as shown in my sample code) and doing
    client.Get worked! So this might be user error on my part, or maybe there
    is some weird interaction with long file names? Stay tuned...

    On Mon, Jan 27, 2014 at 10:12 PM, Carlos Castillo wrote:

    Just covering the bases, since your posted code didn't show how it
    performed the actual download (your code doesn't show the line that you
    claim "fails"). BTW, your posted code ignores the error from storage.New().

    On Monday, January 27, 2014 6:59:54 PM UTC-8, Ian Rose wrote:

    Yes, as you can see in my example code, I am using an authenticated
    http.Client obtained from serviceaccount.NewClient(). And I know that this
    client works because I can use it to do things like list the objects in a
    bucket.

    If I try client.Get(obj.MediaLink), then I receive a 404 from the
    server. To test things out, I tried using this same client to hit one of
    my local servers and from this I was able to confirm that the request
    includes a "Authorization: Bearer <token>" header, as expected. However
    I'm wondering if there are some other headers that I need to set in my
    request?



    On Mon, Jan 27, 2014 at 9:12 PM, Carlos Castillo wrote:

    Are you using an oauth2 authenticated http.Client to make the request?
    The one created on line 1 of the main function should be authenticated if
    the API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client:
    %v\n", err)
    os.Exit(1)
    }

    filename := "5105650963054592/2014-01-27-1
    5-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket,
    filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please
    direct me to the right place!)...
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kyle Lemons at Feb 1, 2014 at 1:28 am

    On Mon, Jan 27, 2014 at 8:05 PM, Ian Rose wrote:

    I believe I have narrowed the problem down to the fact that I am trying to
    download files from within a folder. As a test case, I created a new
    bucket "test987" and uploaded two json files, one without an enclosing
    folder and one to a folder named "foo". So the structure is like:

    test987 bucket:
    - outside.json
    - foo/
    - inside.json


    I can fetch metadata (including the MediaLink) on both outside.json and
    foo/inside.json. However if I think client.Get() the MediaLink in each
    case, it works on outside.json but returns a 404 on foo/inside.json. Here
    is my updated program:

    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "io/ioutil"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n",
    err)
    os.Exit(1)
    }

    filename := "foo/inside.json"
    bucket := "test987"

    service, err := storage.New(client)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create GCS client client: %v\n", err)
    os.Exit(1)
    }

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)

    url := obj.MediaLink
    fmt.Printf("Fetching %s\n", url)

    rsp, err := client.Get(url)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to direct fetch file: %v\n", err)
    os.Exit(1)
    }

    fmt.Printf("Status code: %d\n", rsp.StatusCode)

    fmt.Printf("Reading %d bytes of response body...\n", rsp.ContentLength)
    content, err := ioutil.ReadAll(rsp.Body)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to ready body of fetch-file response:
    %v\n", err)
    os.Exit(1)
    }

    fmt.Printf("Read %d bytes from response body...\n", len(content))

    fmt.Println("")
    fmt.Println(string(content))
    }

    And here is the output when filename is set to "foo/inside.json":

    Found test987/foo/inside.json...
    MediaLink:
    https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
    Fetching
    https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
    Status code: 404
    Reading -1 bytes of response body...
    Read 9 bytes from response body...

    Not Found
    Shot in the dark; if that %2F is a /, does it work?

    And here is the output when filename is set to "outside.json":

    Found test987/outside.json...
    MediaLink:
    https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
    Fetching
    https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
    Status code: 200
    Reading 798 bytes of response body...
    Read 798 bytes from response body...

    {
    "albums": [
    (...content truncated for brevity...)
    ]
    }
    On Monday, January 27, 2014 10:22:08 PM UTC-5, Ian Rose wrote:

    Good point - I probably copy-and-pasted the storage.New line and forgot
    to add back the err variable. I'll fix that.

    In other news, I just tried with a different bucket & filename that are
    much shorter than the "real ones" (as shown in my sample code) and doing
    client.Get worked! So this might be user error on my part, or maybe there
    is some weird interaction with long file names? Stay tuned...

    On Mon, Jan 27, 2014 at 10:12 PM, Carlos Castillo wrote:

    Just covering the bases, since your posted code didn't show how it
    performed the actual download (your code doesn't show the line that you
    claim "fails"). BTW, your posted code ignores the error from storage.New().

    On Monday, January 27, 2014 6:59:54 PM UTC-8, Ian Rose wrote:

    Yes, as you can see in my example code, I am using an authenticated
    http.Client obtained from serviceaccount.NewClient(). And I know that this
    client works because I can use it to do things like list the objects in a
    bucket.

    If I try client.Get(obj.MediaLink), then I receive a 404 from the
    server. To test things out, I tried using this same client to hit one of
    my local servers and from this I was able to confirm that the request
    includes a "Authorization: Bearer <token>" header, as expected. However
    I'm wondering if there are some other headers that I need to set in my
    request?



    On Mon, Jan 27, 2014 at 9:12 PM, Carlos Castillo wrote:

    Are you using an oauth2 authenticated http.Client to make the request?
    The one created on line 1 of the main function should be authenticated if
    the API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google Compute
    Engine, so I can make use of service accounts, which means that I don't
    manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client:
    %v\n", err)
    os.Exit(1)
    }

    filename := "5105650963054592/2014-01-27-1
    5-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket,
    filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please
    direct me to the right place!)...
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.

    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ian Rose at Feb 2, 2014 at 3:48 am
    On the contrary - it *has* to be %2F and in fact therein lays the problem;
    net/url (as used by an http.Client) "normalizes" escape sequences which
    means that it will actually replace the %2F in the URL with a '/' and thus
    the request fails. The solution is to make use of the Opaque property.

    Full discussion here:
    https://plus.google.com/111198546322070924637/posts/Tzw3QZqEQZk

    - Ian



    On Fri, Jan 31, 2014 at 8:27 PM, Kyle Lemons wrote:
    On Mon, Jan 27, 2014 at 8:05 PM, Ian Rose wrote:

    I believe I have narrowed the problem down to the fact that I am trying
    to download files from within a folder. As a test case, I created a new
    bucket "test987" and uploaded two json files, one without an enclosing
    folder and one to a folder named "foo". So the structure is like:

    test987 bucket:
    - outside.json
    - foo/
    - inside.json


    I can fetch metadata (including the MediaLink) on both outside.json and
    foo/inside.json. However if I think client.Get() the MediaLink in each
    case, it works on outside.json but returns a 404 on foo/inside.json. Here
    is my updated program:

    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "io/ioutil"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n",
    err)
    os.Exit(1)
    }

    filename := "foo/inside.json"
    bucket := "test987"

    service, err := storage.New(client)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create GCS client client: %v\n", err)
    os.Exit(1)
    }

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)

    url := obj.MediaLink
    fmt.Printf("Fetching %s\n", url)

    rsp, err := client.Get(url)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to direct fetch file: %v\n", err)
    os.Exit(1)
    }

    fmt.Printf("Status code: %d\n", rsp.StatusCode)

    fmt.Printf("Reading %d bytes of response body...\n", rsp.ContentLength)
    content, err := ioutil.ReadAll(rsp.Body)
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to ready body of fetch-file response:
    %v\n", err)
    os.Exit(1)
    }

    fmt.Printf("Read %d bytes from response body...\n", len(content))

    fmt.Println("")
    fmt.Println(string(content))
    }

    And here is the output when filename is set to "foo/inside.json":

    Found test987/foo/inside.json...
    MediaLink:
    https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
    Fetching
    https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
    Status code: 404
    Reading -1 bytes of response body...
    Read 9 bytes from response body...

    Not Found
    Shot in the dark; if that %2F is a /, does it work?

    And here is the output when filename is set to "outside.json":

    Found test987/outside.json...
    MediaLink:
    https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
    Fetching
    https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
    Status code: 200
    Reading 798 bytes of response body...
    Read 798 bytes from response body...

    {
    "albums": [
    (...content truncated for brevity...)
    ]
    }
    On Monday, January 27, 2014 10:22:08 PM UTC-5, Ian Rose wrote:

    Good point - I probably copy-and-pasted the storage.New line and forgot
    to add back the err variable. I'll fix that.

    In other news, I just tried with a different bucket & filename that are
    much shorter than the "real ones" (as shown in my sample code) and doing
    client.Get worked! So this might be user error on my part, or maybe there
    is some weird interaction with long file names? Stay tuned...

    On Mon, Jan 27, 2014 at 10:12 PM, Carlos Castillo wrote:

    Just covering the bases, since your posted code didn't show how it
    performed the actual download (your code doesn't show the line that you
    claim "fails"). BTW, your posted code ignores the error from storage.New().

    On Monday, January 27, 2014 6:59:54 PM UTC-8, Ian Rose wrote:

    Yes, as you can see in my example code, I am using an authenticated
    http.Client obtained from serviceaccount.NewClient(). And I know that this
    client works because I can use it to do things like list the objects in a
    bucket.

    If I try client.Get(obj.MediaLink), then I receive a 404 from the
    server. To test things out, I tried using this same client to hit one of
    my local servers and from this I was able to confirm that the request
    includes a "Authorization: Bearer <token>" header, as expected. However
    I'm wondering if there are some other headers that I need to set in my
    request?



    On Mon, Jan 27, 2014 at 9:12 PM, Carlos Castillo wrote:

    Are you using an oauth2 authenticated http.Client to make the
    request? The one created on line 1 of the main function should be
    authenticated if the API calls work.

    eg, did you do:

    client.Get(obj.MediaLink)

    or:

    http.Get(obj.MediaLink)

    On Monday, January 27, 2014 1:01:51 PM UTC-8, Ian Rose wrote:

    Here is an example program. Note that I am running on Google
    Compute Engine, so I can make use of service accounts, which means that I
    don't manipulate the access-token directly at all. Although I could make a
    generic REST call to the JSON API to accomplish what I want, but I'd of
    course rather use the client library just for convenience.


    package main

    import (
    "code.google.com/p/goauth2/compute/serviceaccount"
    "code.google.com/p/google-api-go-client/storage/v1beta2"
    "fmt"
    "os"
    )

    func main() {
    client, err := serviceaccount.NewClient(&serviceaccount.Options{})
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to create service account client:
    %v\n", err)
    os.Exit(1)
    }

    filename := "5105650963054592/2014-01-27-1
    5-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
    bucket := "fs-staging-cooked-events"

    service, _ := storage.New(client)

    obj, err := service.Objects.Get(bucket, filename).Do()
    if err != nil {
    fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket,
    filename, err)
    os.Exit(1)
    }

    fmt.Printf("Found %s/%s...\n", bucket, filename)
    fmt.Printf("MediaLink: %s\n", obj.MediaLink)
    fmt.Printf("SelfLink: %s\n", obj.SelfLink)
    }

    On Monday, January 27, 2014 3:50:36 PM UTC-5, Kyle Lemons wrote:

    Can you provide source code? You should be able to make an authenticated
    GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using your oauth token.

    On Mon, Jan 27, 2014 at 12:30 PM, Ian Rose wrote:

    Howdy -

    I'm just getting started using the google-api-go-client<https://code.google.com/p/google-api-go-client/> with
    Cloud Storage and am able to list files in a bucket, get metadata on a
    single file, etc. but I can't figure out how to download a file. As I
    would expect, I can't simply GET the MediaLink that is returned with each
    file's metadata (it gives me a "Login Required" error). But I don't see
    any functions in v1beta2/storage-gen.go for requesting/downloading file
    contents. I do see the reverse (uploading file contents) however...

    Thanks very much!
    - Ian


    Apologies if this is the wrong forum for this question (please
    direct me to the right place!)...
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.

    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedJan 27, '14 at 8:30p
activeFeb 2, '14 at 3:48a
posts10
users3
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase