FAQ
Hello,

I've been recently looking at the goamz library to manipulate resources on
S3. The library has good support for your general CRUD on an S3 bucket.
  However, S3 has since evolved to include things like ACLs, Versioning,
Static Hosting, etc. In order to manipulate these resources, one must
access, what AWS calls, sub-resources.

These sub-resources take the form of /?resource. For example, /?website.
  Creating a URL instance to access this is not impossible and quite
feasible within the go standard sdk. However, goamz needs to parse a URL
to generate an Authorization code based on AWS specific canonicalization
rules. This signature has very specific requirements that must be emulated
on the client as it is performed on the server.

In goamz today, you provide a URL in the form of /?website. In the process
of decoding the Values of the URL, a '=' character will be appended to the
key as in '/?website='. The extra = will case the Authentication code to
not match what the server generates and thus using goamz today it is not
possible without working around the net/url package. It seems like a waste
not to improve on the standard library to handle instances where query
params are just keys with no value and correctly handle the inclusion of =
or not based on the presence of a value. Have some time on the airplane, I
was able to control the inclusion of the '=' based on a value being present
or not. Something along the lines of:

Values{"foo": {}, "bar": {}} will render to "foo&bar"
Values{"foo": "", "bar": ""} will render to "foo=&bar="

Is there interest in updating the Encode() func to handle these types of
URLs? Parsing would also need to be updated so a parsed query string would
correctly encode the query string as it found. I'm happy to share what I
have and I'm also happy if there is existing way to do it that I may have
overlooked.

Thank you,
-Noah

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Gijs at Nov 18, 2013 at 4:48 pm
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:

    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite
    feasible within the go standard sdk. However, goamz needs to parse a URL
    to generate an Authorization code based on AWS specific canonicalization
    rules. This signature has very specific requirements that must be emulated
    on the client as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it
    is not possible without working around the net/url package. It seems like
    a waste not to improve on the standard library to handle instances where
    query params are just keys with no value and correctly handle the inclusion
    of = or not based on the presence of a value. Have some time on the
    airplane, I was able to control the inclusion of the '=' based on a value
    being present or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Nov 19, 2013 at 12:32 am
    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite feasible
    within the go standard sdk. However, goamz needs to parse a URL to generate
    an Authorization code based on AWS specific canonicalization rules. This
    signature has very specific requirements that must be emulated on the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it is
    not possible without working around the net/url package. It seems like a
    waste not to improve on the standard library to handle instances where query
    params are just keys with no value and correctly handle the inclusion of =
    or not based on the presence of a value. Have some time on the airplane, I
    was able to control the inclusion of the '=' based on a value being present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Noah Campbell at Nov 20, 2013 at 5:06 am
    I’m working on getting the failing example. In doing so I’ve gone back through and looked at the process for signing a request. It seems that the query params are handled in the sing routine within goamz: http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84

    I overlooked that goamz has the capabilities to set the params for the request. I think my original position still hold water. To highlight the issue at its most basic level, I put together this: http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used by amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be parsed and encoded into the same byte for byte representation would be useful within the standard library.

    Let me know if you need a more concrete example. I also have the change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite feasible
    within the go standard sdk. However, goamz needs to parse a URL to generate
    an Authorization code based on AWS specific canonicalization rules. This
    signature has very specific requirements that must be emulated on the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it is
    not possible without working around the net/url package. It seems like a
    waste not to improve on the standard library to handle instances where query
    params are just keys with no value and correctly handle the inclusion of =
    or not based on the presence of a value. Have some time on the airplane, I
    was able to control the inclusion of the '=' based on a value being present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com
  • Dave Cheney at Nov 20, 2013 at 5:25 am
    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave


    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell wrote:
    I’m working on getting the failing example. In doing so I’ve gone back through and looked at the process for signing a request. It seems that the query params are handled in the sing routine within goamz: http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84

    I overlooked that goamz has the capabilities to set the params for the request. I think my original position still hold water. To highlight the issue at its most basic level, I put together this: http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used by amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be parsed and encoded into the same byte for byte representation would be useful within the standard library.

    Let me know if you need a more concrete example. I also have the change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite feasible
    within the go standard sdk. However, goamz needs to parse a URL to generate
    an Authorization code based on AWS specific canonicalization rules. This
    signature has very specific requirements that must be emulated on the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it is
    not possible without working around the net/url package. It seems like a
    waste not to improve on the standard library to handle instances where query
    params are just keys with no value and correctly handle the inclusion of =
    or not based on the presence of a value. Have some time on the airplane, I
    was able to control the inclusion of the '=' based on a value being present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Noah Campbell at Nov 20, 2013 at 9:57 pm
    Hi Dave,

    I here what you're saying about the format being odd. However, even if I
    were able to influence AWS to make the change, it'd be years before the API
    is deprecated given they still support the original API when the service
    was launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to
    find an RFC state that a query parameter had to be key value pairs
    exclusively. RFC 3986 says that the query is everything in between the ?
    and the #. But, you're probably more familiar with this than I am.
      Regardless of the RFC, it doesn't necessarily address the reality of AWS
    not changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string, not
    map[string]string. http://golang.org/pkg/net/url/#Values. In this case
    there is room for a key in a query to exist without any values. In my
    local version, it results in: {"website": {}} for no value and {"webiste:
    {""}} for an empty string value. This provides enough information to
    correctly Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so we
    don't see folks working around it. The change I have doesn't break
    existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah


    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:

    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave


    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell wrote:
    I’m working on getting the failing example. In doing so I’ve gone back
    through and looked at the process for signing a request. It seems that the
    query params are handled in the sing routine within goamz:
    http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84
    I overlooked that goamz has the capabilities to set the params for the
    request. I think my original position still hold water. To highlight the
    issue at its most basic level, I put together this:
    http://play.golang.org/p/jMmSc6O0Yg
    While, semantically they are equivalent, the signing algorithm used by
    amazon (and it’s clones) must rely on byte for byte equivalence.
    Anyway, I think the ability for a querystring like ?website to be parsed
    and encoded into the same byte for byte representation would be useful
    within the standard library.
    Let me know if you need a more concrete example. I also have the change
    made in my local golang dev branch if you’re interested.
    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both
    versions
    of the signature process
    (
    http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
    )
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign
    character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's
    signing
    process?

    - Gijs


    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.comwrote:
    Hello,

    I've been recently looking at the goamz library to manipulate
    resources on
    S3. The library has good support for your general CRUD on an S3
    bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example,
    /?website.
    Creating a URL instance to access this is not impossible and quite
    feasible
    within the go standard sdk. However, goamz needs to parse a URL to
    generate
    an Authorization code based on AWS specific canonicalization rules.
    This
    signature has very specific requirements that must be emulated on the
    client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be
    appended
    to the key as in '/?website='. The extra = will case the
    Authentication
    code to not match what the server generates and thus using goamz
    today it is
    not possible without working around the net/url package. It seems
    like a
    waste not to improve on the standard library to handle instances
    where query
    params are just keys with no value and correctly handle the inclusion
    of =
    or not based on the presence of a value. Have some time on the
    airplane, I
    was able to control the inclusion of the '=' based on a value being
    present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types
    of
    URLs? Parsing would also need to be updated so a parsed query string
    would
    correctly encode the query string as it found. I'm happy to share
    what I
    have and I'm also happy if there is existing way to do it that I may
    have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Nov 20, 2013 at 10:02 pm
    Hi Noah,

    Can you please tell em the full URL you are trying to sign with goamz.

    Cheers

    Dave
    On 21 Nov 2013, at 8:57, Noah Campbell wrote:

    Hi Dave,

    I here what you're saying about the format being odd. However, even if I were able to influence AWS to make the change, it'd be years before the API is deprecated given they still support the original API when the service was launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to find an RFC state that a query parameter had to be key value pairs exclusively. RFC 3986 says that the query is everything in between the ? and the #. But, you're probably more familiar with this than I am. Regardless of the RFC, it doesn't necessarily address the reality of AWS not changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string, not map[string]string. http://golang.org/pkg/net/url/#Values. In this case there is room for a key in a query to exist without any values. In my local version, it results in: {"website": {}} for no value and {"webiste: {""}} for an empty string value. This provides enough information to correctly Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so we don't see folks working around it. The change I have doesn't break existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah


    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:
    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave


    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell wrote:
    I’m working on getting the failing example. In doing so I’ve gone back through and looked at the process for signing a request. It seems that the query params are handled in the sing routine within goamz: http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84

    I overlooked that goamz has the capabilities to set the params for the request. I think my original position still hold water. To highlight the issue at its most basic level, I put together this: http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used by amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be parsed and encoded into the same byte for byte representation would be useful within the standard library.

    Let me know if you need a more concrete example. I also have the change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite feasible
    within the go standard sdk. However, goamz needs to parse a URL to generate
    an Authorization code based on AWS specific canonicalization rules. This
    signature has very specific requirements that must be emulated on the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it is
    not possible without working around the net/url package. It seems like a
    waste not to improve on the standard library to handle instances where query
    params are just keys with no value and correctly handle the inclusion of =
    or not based on the presence of a value. Have some time on the airplane, I
    was able to control the inclusion of the '=' based on a value being present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Noah Campbell at Nov 20, 2013 at 10:34 pm
    Here’s the path that needs to be signed (whitespace included)

    "PUT


    Wed, 20 Nov 2013 04:46:50 UTC
    /my-s3-bucket/?website”

    In the goamz I created a PutConfiguration func that takes “/?website” as the input.

    The resulting url includes /my-s3-bucket/?website= which causes Amazon to reject the request because the signatures don’t map.

    -Noah

    On Nov 20, 2013, at 2:01 PM, Dave Cheney wrote:

    Hi Noah,

    Can you please tell em the full URL you are trying to sign with goamz.

    Cheers

    Dave
    On 21 Nov 2013, at 8:57, Noah Campbell wrote:

    Hi Dave,

    I here what you're saying about the format being odd. However, even if I were able to influence AWS to make the change, it'd be years before the API is deprecated given they still support the original API when the service was launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to find an RFC state that a query parameter had to be key value pairs exclusively. RFC 3986 says that the query is everything in between the ? and the #. But, you're probably more familiar with this than I am. Regardless of the RFC, it doesn't necessarily address the reality of AWS not changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string, not map[string]string. http://golang.org/pkg/net/url/#Values. In this case there is room for a key in a query to exist without any values. In my local version, it results in: {"website": {}} for no value and {"webiste: {""}} for an empty string value. This provides enough information to correctly Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so we don't see folks working around it. The change I have doesn't break existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah



    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:
    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave


    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell wrote:
    I’m working on getting the failing example. In doing so I’ve gone back through and looked at the process for signing a request. It seems that the query params are handled in the sing routine within goamz: http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84

    I overlooked that goamz has the capabilities to set the params for the request. I think my original position still hold water. To highlight the issue at its most basic level, I put together this: http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used by amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be parsed and encoded into the same byte for byte representation would be useful within the standard library.

    Let me know if you need a more concrete example. I also have the change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite feasible
    within the go standard sdk. However, goamz needs to parse a URL to generate
    an Authorization code based on AWS specific canonicalization rules. This
    signature has very specific requirements that must be emulated on the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it is
    not possible without working around the net/url package. It seems like a
    waste not to improve on the standard library to handle instances where query
    params are just keys with no value and correctly handle the inclusion of =
    or not based on the presence of a value. Have some time on the airplane, I
    was able to control the inclusion of the '=' based on a value being present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com
  • Dave Cheney at Nov 20, 2013 at 10:36 pm
    Hi Noah,

    Who is generating that URL? If it is required by amazon, can you show me the documentation for that.

    Cheers

    Dave
    On 21 Nov 2013, at 9:34, Noah Campbell wrote:

    Here’s the path that needs to be signed (whitespace included)

    "PUT


    Wed, 20 Nov 2013 04:46:50 UTC
    /my-s3-bucket/?website”

    In the goamz I created a PutConfiguration func that takes “/?website” as the input.

    The resulting url includes /my-s3-bucket/?website= which causes Amazon to reject the request because the signatures don’t map.

    -Noah

    On Nov 20, 2013, at 2:01 PM, Dave Cheney wrote:

    Hi Noah,

    Can you please tell em the full URL you are trying to sign with goamz.

    Cheers

    Dave
    On 21 Nov 2013, at 8:57, Noah Campbell wrote:

    Hi Dave,

    I here what you're saying about the format being odd. However, even if I were able to influence AWS to make the change, it'd be years before the API is deprecated given they still support the original API when the service was launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to find an RFC state that a query parameter had to be key value pairs exclusively. RFC 3986 says that the query is everything in between the ? and the #. But, you're probably more familiar with this than I am. Regardless of the RFC, it doesn't necessarily address the reality of AWS not changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string, not map[string]string. http://golang.org/pkg/net/url/#Values. In this case there is room for a key in a query to exist without any values. In my local version, it results in: {"website": {}} for no value and {"webiste: {""}} for an empty string value. This provides enough information to correctly Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so we don't see folks working around it. The change I have doesn't break existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah


    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:
    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave


    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell wrote:
    I’m working on getting the failing example. In doing so I’ve gone back through and looked at the process for signing a request. It seems that the query params are handled in the sing routine within goamz: http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84

    I overlooked that goamz has the capabilities to set the params for the request. I think my original position still hold water. To highlight the issue at its most basic level, I put together this: http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used by amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be parsed and encoded into the same byte for byte representation would be useful within the standard library.

    Let me know if you need a more concrete example. I also have the change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both versions
    of the signature process
    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's signing
    process?

    - Gijs

    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite feasible
    within the go standard sdk. However, goamz needs to parse a URL to generate
    an Authorization code based on AWS specific canonicalization rules. This
    signature has very specific requirements that must be emulated on the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it is
    not possible without working around the net/url package. It seems like a
    waste not to improve on the standard library to handle instances where query
    params are just keys with no value and correctly handle the inclusion of =
    or not based on the presence of a value. Have some time on the airplane, I
    was able to control the inclusion of the '=' based on a value being present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Noah Campbell at Nov 20, 2013 at 10:42 pm
    http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html

    On Wed, Nov 20, 2013 at 2:35 PM, Dave Cheney wrote:

    Hi Noah,

    Who is generating that URL? If it is required by amazon, can you show me
    the documentation for that.

    Cheers

    Dave

    On 21 Nov 2013, at 9:34, Noah Campbell wrote:

    Here’s the path that needs to be signed (whitespace included)

    "PUT


    Wed, 20 Nov 2013 04:46:50 UTC
    /my-s3-bucket/?website”

    In the goamz I created a PutConfiguration func that takes “/?website” as
    the input.

    The resulting url includes /my-s3-bucket/?website= which causes Amazon to
    reject the request because the signatures don’t map.

    -Noah


    On Nov 20, 2013, at 2:01 PM, Dave Cheney wrote:

    Hi Noah,

    Can you please tell em the full URL you are trying to sign with goamz.

    Cheers

    Dave

    On 21 Nov 2013, at 8:57, Noah Campbell wrote:

    Hi Dave,

    I here what you're saying about the format being odd. However, even if I
    were able to influence AWS to make the change, it'd be years before the API
    is deprecated given they still support the original API when the service
    was launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to
    find an RFC state that a query parameter had to be key value pairs
    exclusively. RFC 3986 says that the query is everything in between the ?
    and the #. But, you're probably more familiar with this than I am.
    Regardless of the RFC, it doesn't necessarily address the reality of AWS
    not changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string, not
    map[string]string. http://golang.org/pkg/net/url/#Values. In this case
    there is room for a key in a query to exist without any values. In my
    local version, it results in: {"website": {}} for no value and {"webiste:
    {""}} for an empty string value. This provides enough information to
    correctly Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so
    we don't see folks working around it. The change I have doesn't break
    existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah


    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:

    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave



    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell <noahcampbell@gmail.com>
    wrote:
    I’m working on getting the failing example. In doing so I’ve gone back
    through and looked at the process for signing a request. It seems that the
    query params are handled in the sing routine within goamz:
    http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84
    I overlooked that goamz has the capabilities to set the params for the
    request. I think my original position still hold water. To highlight the
    issue at its most basic level, I put together this:
    http://play.golang.org/p/jMmSc6O0Yg
    While, semantically they are equivalent, the signing algorithm used by
    amazon (and it’s clones) must rely on byte for byte equivalence.
    Anyway, I think the ability for a querystring like ?website to be
    parsed and encoded into the same byte for byte representation would be
    useful within the standard library.
    Let me know if you need a more concrete example. I also have the
    change made in my local golang dev branch if you’re interested.
    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both
    versions
    of the signature process
    (
    http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
    )
    create a canonicalized version of the query string in which every
    empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign
    character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's
    signing
    process?

    - Gijs


    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.comwrote:
    Hello,

    I've been recently looking at the goamz library to manipulate
    resources on
    S3. The library has good support for your general CRUD on an S3
    bucket.
    However, S3 has since evolved to include things like ACLs,
    Versioning,
    Static Hosting, etc. In order to manipulate these resources, one
    must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example,
    /?website.
    Creating a URL instance to access this is not impossible and quite
    feasible
    within the go standard sdk. However, goamz needs to parse a URL to
    generate
    an Authorization code based on AWS specific canonicalization rules.
    This
    signature has very specific requirements that must be emulated on
    the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be
    appended
    to the key as in '/?website='. The extra = will case the
    Authentication
    code to not match what the server generates and thus using goamz
    today it is
    not possible without working around the net/url package. It seems
    like a
    waste not to improve on the standard library to handle instances
    where query
    params are just keys with no value and correctly handle the
    inclusion of =
    or not based on the presence of a value. Have some time on the
    airplane, I
    was able to control the inclusion of the '=' based on a value being
    present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these
    types of
    URLs? Parsing would also need to be updated so a parsed query
    string would
    correctly encode the query string as it found. I'm happy to share
    what I
    have and I'm also happy if there is existing way to do it that I may
    have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com


    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com


    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Dave Cheney at Nov 20, 2013 at 11:38 pm
    Thanks Noah,

    I see where that query param is coming from,

    "f the request addresses a subresource, such as ?versioning,
    ?location, ?acl, ?torrent, ?lifecycle, or ?versionid, append the
    subresource, its value if it has one, and the question mark. Note that
    in case of multiple subresources, subresources must be
    lexicographically sorted by subresource name and separated by '&',
    e.g., ?acl&versionId=value."

    I there are a few things going on here.

    1. according to RFC 3986 the data between the ? and # is the query
    section of the request, however 3986 does not define the syntax of the
    query section.

    2. RFC 1738 does define the form of the HTTP url (it's been too long
    and I can't remember the exact differnces between URL's and URIs)
    query parameter, but calls it search (i guess that is another way of
    say query). But it doesn't call it a query parameter.

    3. RFC 2616 doesn't give much detail here either.

    4. This area is under tested in the standard library, I'm mulling over
    raising an issue about this.

    Let me raise an issue on goamz (you can do this also, but it means
    having a launchpad account) and consule with the goamz author. Thanks
    for your patience.

    Cheers

    Dave
    On Thu, Nov 21, 2013 at 9:41 AM, Noah Campbell wrote:
    http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html

    On Wed, Nov 20, 2013 at 2:35 PM, Dave Cheney wrote:

    Hi Noah,

    Who is generating that URL? If it is required by amazon, can you show me
    the documentation for that.

    Cheers

    Dave

    On 21 Nov 2013, at 9:34, Noah Campbell wrote:

    Here’s the path that needs to be signed (whitespace included)

    "PUT


    Wed, 20 Nov 2013 04:46:50 UTC
    /my-s3-bucket/?website”

    In the goamz I created a PutConfiguration func that takes “/?website” as
    the input.

    The resulting url includes /my-s3-bucket/?website= which causes Amazon to
    reject the request because the signatures don’t map.

    -Noah


    On Nov 20, 2013, at 2:01 PM, Dave Cheney wrote:

    Hi Noah,

    Can you please tell em the full URL you are trying to sign with goamz.

    Cheers

    Dave

    On 21 Nov 2013, at 8:57, Noah Campbell wrote:

    Hi Dave,

    I here what you're saying about the format being odd. However, even if I
    were able to influence AWS to make the change, it'd be years before the API
    is deprecated given they still support the original API when the service was
    launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to
    find an RFC state that a query parameter had to be key value pairs
    exclusively. RFC 3986 says that the query is everything in between the ?
    and the #. But, you're probably more familiar with this than I am.
    Regardless of the RFC, it doesn't necessarily address the reality of AWS not
    changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string, not
    map[string]string. http://golang.org/pkg/net/url/#Values. In this case
    there is room for a key in a query to exist without any values. In my local
    version, it results in: {"website": {}} for no value and {"webiste: {""}}
    for an empty string value. This provides enough information to correctly
    Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so
    we don't see folks working around it. The change I have doesn't break
    existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah


    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:

    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave



    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell <noahcampbell@gmail.com>
    wrote:
    I’m working on getting the failing example. In doing so I’ve gone back
    through and looked at the process for signing a request. It seems that the
    query params are handled in the sing routine within goamz:
    http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84

    I overlooked that goamz has the capabilities to set the params for the
    request. I think my original position still hold water. To highlight the
    issue at its most basic level, I put together this:
    http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used by
    amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be
    parsed and encoded into the same byte for byte representation would be
    useful within the standard library.

    Let me know if you need a more concrete example. I also have the
    change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both
    versions
    of the signature process

    (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html)
    create a canonicalized version of the query string in which every
    empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign
    character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's
    signing
    process?

    - Gijs


    On Monday, November 18, 2013 12:35:24 AM UTC+1, noahca...@gmail.com
    wrote:
    Hello,

    I've been recently looking at the goamz library to manipulate
    resources on
    S3. The library has good support for your general CRUD on an S3
    bucket.
    However, S3 has since evolved to include things like ACLs,
    Versioning,
    Static Hosting, etc. In order to manipulate these resources, one
    must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example,
    /?website.
    Creating a URL instance to access this is not impossible and quite
    feasible
    within the go standard sdk. However, goamz needs to parse a URL to
    generate
    an Authorization code based on AWS specific canonicalization rules.
    This
    signature has very specific requirements that must be emulated on
    the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be
    appended
    to the key as in '/?website='. The extra = will case the
    Authentication
    code to not match what the server generates and thus using goamz
    today it is
    not possible without working around the net/url package. It seems
    like a
    waste not to improve on the standard library to handle instances
    where query
    params are just keys with no value and correctly handle the
    inclusion of =
    or not based on the presence of a value. Have some time on the
    airplane, I
    was able to control the inclusion of the '=' based on a value being
    present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these
    types of
    URLs? Parsing would also need to be updated so a parsed query
    string would
    correctly encode the query string as it found. I'm happy to share
    what I
    have and I'm also happy if there is existing way to do it that I may
    have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com



    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Noah Campbell at Dec 2, 2013 at 9:46 pm
    Hi Dave,

    Thanksgiving is now out of the way, I can give some attention to this. I
    didn't see an issue raised on goamz, at least in a quick pass of the
    buglist. Did you speak with the maintainers on a back channel?

    goamz can be made to work, but it seems like an issue that would not be
    specific to only goamz. As you pointed out what goes into a query is
    pretty opaque. I agree the key/value combo is definitely the norm, but
    nothing in spec directly calls out what amazon is doing as incorrect.

    The intent that I'm after with this is to make sure the query parameters
    are stable so that if you include an amazon "subresource" in a url, that it
    is honored when Encoded. It is not difficult to include this behavior and
    I think it is more in line with what an uninitiated user would expect. It
    would be less surprising, in my opinion.

    It is also right after 1.2 release so it can be put in place early in the
    dev cycle (assuming it makes sense to add this).

    Thanks,
    -Noah



    On Wed, Nov 20, 2013 at 3:38 PM, Dave Cheney wrote:

    Thanks Noah,

    I see where that query param is coming from,

    "f the request addresses a subresource, such as ?versioning,
    ?location, ?acl, ?torrent, ?lifecycle, or ?versionid, append the
    subresource, its value if it has one, and the question mark. Note that
    in case of multiple subresources, subresources must be
    lexicographically sorted by subresource name and separated by '&',
    e.g., ?acl&versionId=value."

    I there are a few things going on here.

    1. according to RFC 3986 the data between the ? and # is the query
    section of the request, however 3986 does not define the syntax of the
    query section.

    2. RFC 1738 does define the form of the HTTP url (it's been too long
    and I can't remember the exact differnces between URL's and URIs)
    query parameter, but calls it search (i guess that is another way of
    say query). But it doesn't call it a query parameter.

    3. RFC 2616 doesn't give much detail here either.

    4. This area is under tested in the standard library, I'm mulling over
    raising an issue about this.

    Let me raise an issue on goamz (you can do this also, but it means
    having a launchpad account) and consule with the goamz author. Thanks
    for your patience.

    Cheers

    Dave
    On Thu, Nov 21, 2013 at 9:41 AM, Noah Campbell wrote:
    http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html

    On Wed, Nov 20, 2013 at 2:35 PM, Dave Cheney wrote:

    Hi Noah,

    Who is generating that URL? If it is required by amazon, can you show me
    the documentation for that.

    Cheers

    Dave

    On 21 Nov 2013, at 9:34, Noah Campbell wrote:

    Here’s the path that needs to be signed (whitespace included)

    "PUT


    Wed, 20 Nov 2013 04:46:50 UTC
    /my-s3-bucket/?website”

    In the goamz I created a PutConfiguration func that takes “/?website” as
    the input.

    The resulting url includes /my-s3-bucket/?website= which causes Amazon
    to
    reject the request because the signatures don’t map.

    -Noah


    On Nov 20, 2013, at 2:01 PM, Dave Cheney wrote:

    Hi Noah,

    Can you please tell em the full URL you are trying to sign with goamz.

    Cheers

    Dave

    On 21 Nov 2013, at 8:57, Noah Campbell wrote:

    Hi Dave,

    I here what you're saying about the format being odd. However, even if
    I
    were able to influence AWS to make the change, it'd be years before the
    API
    is deprecated given they still support the original API when the
    service was
    launched back in the day.

    As you suggested this format is a violation of the RFC, I wasn't able to
    find an RFC state that a query parameter had to be key value pairs
    exclusively. RFC 3986 says that the query is everything in between the
    ?
    and the #. But, you're probably more familiar with this than I am.
    Regardless of the RFC, it doesn't necessarily address the reality of
    AWS not
    changing their interface any time soon.

    I'm sure you also meant to point out Values is a map[string][]string,
    not
    map[string]string. http://golang.org/pkg/net/url/#Values. In this
    case
    there is room for a key in a query to exist without any values. In my
    local
    version, it results in: {"website": {}} for no value and {"webiste:
    {""}}
    for an empty string value. This provides enough information to
    correctly
    Encode the query string the way it was found.

    I would like to see Golang be robust to handle these funny edge cases so
    we don't see folks working around it. The change I have doesn't break
    existing functionality according to the unit tests in net/url.

    Thoughts?

    -Noah


    On Tue, Nov 19, 2013 at 9:25 PM, Dave Cheney wrote:

    I think there are two problems here.

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.

    The second is that `website` is being silently expanded to `website=`
    inside goamz. I think this is of lesser importance if the first issue
    is resolved.

    Cheers

    Dave



    On Wed, Nov 20, 2013 at 4:06 PM, Noah Campbell <noahcampbell@gmail.com
    wrote:
    I’m working on getting the failing example. In doing so I’ve gone
    back
    through and looked at the process for signing a request. It seems
    that the
    query params are handled in the sing routine within goamz:
    http://bazaar.launchpad.net/~goamz/goamz/trunk/view/head:/s3/sign.go#L84
    I overlooked that goamz has the capabilities to set the params for
    the
    request. I think my original position still hold water. To
    highlight the
    issue at its most basic level, I put together this:
    http://play.golang.org/p/jMmSc6O0Yg

    While, semantically they are equivalent, the signing algorithm used
    by
    amazon (and it’s clones) must rely on byte for byte equivalence.

    Anyway, I think the ability for a querystring like ?website to be
    parsed and encoded into the same byte for byte representation would
    be
    useful within the standard library.

    Let me know if you need a more concrete example. I also have the
    change made in my local golang dev branch if you’re interested.

    Thank you,
    -Noah
    On Nov 18, 2013, at 4:32 PM, Dave Cheney wrote:

    Can you please show a sample request which is failing
    On Tue, Nov 19, 2013 at 3:48 AM, Gijs wrote:
    I might be mistaken, but does it really matter? According to both
    versions
    of the signature process

    (
    http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
    )
    create a canonicalized version of the query string in which every
    empty
    value has an '=' appenden:
    Separate parameter names from their values with the equal sign
    character
    (=) (ASCII character 61), even if the value is empty.

    So maybe, if you're encountering errors, there's a bug in goamz's
    signing
    process?

    - Gijs


    On Monday, November 18, 2013 12:35:24 AM UTC+1,
    noahca...@gmail.com
    wrote:
    Hello,

    I've been recently looking at the goamz library to manipulate
    resources on
    S3. The library has good support for your general CRUD on an S3
    bucket.
    However, S3 has since evolved to include things like ACLs,
    Versioning,
    Static Hosting, etc. In order to manipulate these resources, one
    must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example,
    /?website.
    Creating a URL instance to access this is not impossible and quite
    feasible
    within the go standard sdk. However, goamz needs to parse a URL
    to
    generate
    an Authorization code based on AWS specific canonicalization
    rules.
    This
    signature has very specific requirements that must be emulated on
    the client
    as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In
    the
    process of decoding the Values of the URL, a '=' character will be
    appended
    to the key as in '/?website='. The extra = will case the
    Authentication
    code to not match what the server generates and thus using goamz
    today it is
    not possible without working around the net/url package. It seems
    like a
    waste not to improve on the standard library to handle instances
    where query
    params are just keys with no value and correctly handle the
    inclusion of =
    or not based on the presence of a value. Have some time on the
    airplane, I
    was able to control the inclusion of the '=' based on a value
    being
    present
    or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these
    types of
    URLs? Parsing would also need to be updated so a parsed query
    string would
    correctly encode the query string as it found. I'm happy to share
    what I
    have and I'm also happy if there is existing way to do it that I
    may
    have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google
    Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com



    Noah Campbell
    415-513-3545
    noahcampbell@gmail.com

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kevin Gillette at Dec 2, 2013 at 10:28 pm
    Whether or not it's defined in the RFC, valueless query parameters
    (including omission of the equal-sign, such as "?x&y&z") are both common
    and typically well supported in the realm of server-side software.
    On Tuesday, November 19, 2013 10:25:09 PM UTC-7, Dave Cheney wrote:

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Noah Campbell at Dec 2, 2013 at 10:30 pm
    I would imagine they would be, but I didn't want to project my opinion.
      Can you give an example server side system that uses this convention?
      Also, do you know if they would choke if "?x=&y=&z=" were presented?

    -Noah


    On Mon, Dec 2, 2013 at 2:27 PM, Kevin Gillette
    wrote:
    Whether or not it's defined in the RFC, valueless query parameters
    (including omission of the equal-sign, such as "?x&y&z") are both common
    and typically well supported in the realm of server-side software.

    On Tuesday, November 19, 2013 10:25:09 PM UTC-7, Dave Cheney wrote:

    The first is ParseQuery("website") IMO should fail. My reading of the
    RFC says that http query strings are pairs of values concatenated with
    a =. Put another way, `website=` is map[string]string { "website": ""
    } , `website` is not valid.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kevin Gillette at Dec 2, 2013 at 10:58 pm

    On Monday, December 2, 2013 3:30:13 PM UTC-7, Noah Campbell wrote:
    I would imagine they would be, but I didn't want to project my opinion.
    Can you give an example server side system that uses this convention?
    Also, do you know if they would choke if "?x=&y=&z=" were presented?
    I haven't encountered any server side systems that would choke on either
    "?x&y&z" or "?x=&y=&z=". Simply put, they'd be rather non-robust if they
    didn't handle both forms gracefully. Some systems are unable to distinguish
    between either form, while others, for example, would consider all of the
    former to be nil valued and all of the latter to be empty-string valued. Go
    treats both forms as being identical.

    See http://play.golang.org/p/RGHKn4sBfP for an demo of this (though it
    won't run in the playground)

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Kevin Gillette at Dec 2, 2013 at 11:03 pm
    Here's an equivalent that does run in the
    playground: http://play.golang.org/p/A8Q2N_OAuC
    On Monday, December 2, 2013 3:57:57 PM UTC-7, Kevin Gillette wrote:
    On Monday, December 2, 2013 3:30:13 PM UTC-7, Noah Campbell wrote:

    I would imagine they would be, but I didn't want to project my opinion.
    Can you give an example server side system that uses this convention?
    Also, do you know if they would choke if "?x=&y=&z=" were presented?
    I haven't encountered any server side systems that would choke on either
    "?x&y&z" or "?x=&y=&z=". Simply put, they'd be rather non-robust if they
    didn't handle both forms gracefully. Some systems are unable to distinguish
    between either form, while others, for example, would consider all of the
    former to be nil valued and all of the latter to be empty-string valued. Go
    treats both forms as being identical.

    See http://play.golang.org/p/RGHKn4sBfP for an demo of this (though it
    won't run in the playground)
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Gareth at Dec 3, 2013 at 1:59 am
    fwiw I recently branched the goamz library to add support for the s3
    multi-delete operation, which expects request URLs to end in /?delete

    I had no issues sending and signing URLs ending in /?delete= instead; I
    just had to make sure that goamz knew to include the delete parameter when
    computing the signature.

    Here's my current merge proposal for the change I wrote/used:
    https://code.launchpad.net/~garethwatts/goamz/goamz/+merge/196410

    Gareth.
    On Sunday, November 17, 2013 3:35:24 PM UTC-8, noahca...@gmail.com wrote:

    Hello,

    I've been recently looking at the goamz library to manipulate resources on
    S3. The library has good support for your general CRUD on an S3 bucket.
    However, S3 has since evolved to include things like ACLs, Versioning,
    Static Hosting, etc. In order to manipulate these resources, one must
    access, what AWS calls, sub-resources.

    These sub-resources take the form of /?resource. For example, /?website.
    Creating a URL instance to access this is not impossible and quite
    feasible within the go standard sdk. However, goamz needs to parse a URL
    to generate an Authorization code based on AWS specific canonicalization
    rules. This signature has very specific requirements that must be emulated
    on the client as it is performed on the server.

    In goamz today, you provide a URL in the form of /?website. In the
    process of decoding the Values of the URL, a '=' character will be appended
    to the key as in '/?website='. The extra = will case the Authentication
    code to not match what the server generates and thus using goamz today it
    is not possible without working around the net/url package. It seems like
    a waste not to improve on the standard library to handle instances where
    query params are just keys with no value and correctly handle the inclusion
    of = or not based on the presence of a value. Have some time on the
    airplane, I was able to control the inclusion of the '=' based on a value
    being present or not. Something along the lines of:

    Values{"foo": {}, "bar": {}} will render to "foo&bar"
    Values{"foo": "", "bar": ""} will render to "foo=&bar="

    Is there interest in updating the Encode() func to handle these types of
    URLs? Parsing would also need to be updated so a parsed query string would
    correctly encode the query string as it found. I'm happy to share what I
    have and I'm also happy if there is existing way to do it that I may have
    overlooked.

    Thank you,
    -Noah
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedNov 18, '13 at 6:11a
activeDec 3, '13 at 1:59a
posts17
users5
websitegolang.org

People

Translate

site design / logo © 2022 Grokbase