[go-nuts] Google API client: downloading files from Cloud Storage

Can you provide source code? You should be able to make an authenticated
GET request<https://developers.google.com/storage/docs/reference-methods#getobject>using
your oauth token.

Here is an example program. Note that I am running on Google Compute
Engine, so I can make use of service accounts, which means that I don't
manipulate the access-token directly at all. Although I could make a
generic REST call to the JSON API to accomplish what I want, but I'd of
course rather use the client library just for convenience.


package main

import (
"code.google.com/p/goauth2/compute/serviceaccount"
"code.google.com/p/google-api-go-client/storage/v1beta2"
"fmt"
"os"
)

func main() {
client, err := serviceaccount.NewClient(&serviceaccount.Options{})
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n", err)
os.Exit(1)
}

filename :=
"5105650963054592/2014-01-27-15-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"
bucket := "fs-staging-cooked-events"

service, _ := storage.New(client)

obj, err := service.Objects.Get(bucket, filename).Do()
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
os.Exit(1)
}

fmt.Printf("Found %s/%s...\n", bucket, filename)
fmt.Printf("MediaLink: %s\n", obj.MediaLink)
fmt.Printf("SelfLink: %s\n", obj.SelfLink)
}

Are you using an oauth2 authenticated http.Client to make the request? The
one created on line 1 of the main function should be authenticated if the
API calls work.

eg, did you do:

client.Get(obj.MediaLink)

or:

http.Get(obj.MediaLink)

Yes, as you can see in my example code, I am using an authenticated
http.Client obtained from serviceaccount.NewClient(). And I know that this
client works because I can use it to do things like list the objects in a
bucket.

If I try client.Get(obj.MediaLink), then I receive a 404 from the server.
  To test things out, I tried using this same client to hit one of my local
servers and from this I was able to confirm that the request includes a
"Authorization: Bearer <token>" header, as expected. However I'm wondering
if there are some other headers that I need to set in my request?

Just covering the bases, since your posted code didn't show how it
performed the actual download (your code doesn't show the line that you
claim "fails"). BTW, your posted code ignores the error from storage.New().

Good point - I probably copy-and-pasted the storage.New line and forgot to
add back the err variable. I'll fix that.

In other news, I just tried with a different bucket & filename that are
much shorter than the "real ones" (as shown in my sample code) and doing
client.Get worked! So this might be user error on my part, or maybe there
is some weird interaction with long file names? Stay tuned...

I believe I have narrowed the problem down to the fact that I am trying to
download files from within a folder. As a test case, I created a new
bucket "test987" and uploaded two json files, one without an enclosing
folder and one to a folder named "foo". So the structure is like:

test987 bucket:
   - outside.json
   - foo/
     - inside.json


I can fetch metadata (including the MediaLink) on both outside.json and
foo/inside.json. However if I think client.Get() the MediaLink in each
case, it works on outside.json but returns a 404 on foo/inside.json. Here
is my updated program:

package main

import (
"code.google.com/p/goauth2/compute/serviceaccount"
"code.google.com/p/google-api-go-client/storage/v1beta2"
"fmt"
"io/ioutil"
"os"
)

func main() {
client, err := serviceaccount.NewClient(&serviceaccount.Options{})
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n", err)
os.Exit(1)
}

filename := "foo/inside.json"
bucket := "test987"

service, err := storage.New(client)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to create GCS client client: %v\n", err)
os.Exit(1)
}

obj, err := service.Objects.Get(bucket, filename).Do()
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)
os.Exit(1)
}

fmt.Printf("Found %s/%s...\n", bucket, filename)
fmt.Printf("MediaLink: %s\n", obj.MediaLink)

url := obj.MediaLink
fmt.Printf("Fetching %s\n", url)

rsp, err := client.Get(url)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to direct fetch file: %v\n", err)
os.Exit(1)
}

fmt.Printf("Status code: %d\n", rsp.StatusCode)

fmt.Printf("Reading %d bytes of response body...\n", rsp.ContentLength)
content, err := ioutil.ReadAll(rsp.Body)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to ready body of fetch-file response: %v\n",
err)
os.Exit(1)
}

fmt.Printf("Read %d bytes from response body...\n", len(content))

fmt.Println("")
fmt.Println(string(content))
}

And here is the output when filename is set to "foo/inside.json":

Found test987/foo/inside.json...
MediaLink:
https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
Fetching
https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media
Status code: 404
Reading -1 bytes of response body...
Read 9 bytes from response body...

Not Found

And here is the output when filename is set to "outside.json":

Found test987/outside.json...
MediaLink:
https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
Fetching
https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media
Status code: 200
Reading 798 bytes of response body...
Read 798 bytes from response body...

{
   "albums": [
     (...content truncated for brevity...)
   ]
}

Shot in the dark; if that %2F is a /, does it work?

On the contrary - it *has* to be %2F and in fact therein lays the problem;
net/url (as used by an http.Client) "normalizes" escape sequences which
means that it will actually replace the %2F in the URL with a '/' and thus
the request fails. The solution is to make use of the Opaque property.

Full discussion here:
https://plus.google.com/111198546322070924637/posts/Tzw3QZqEQZk

- Ian