[Hadoop-common-dev] [jira] Created: (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded

"s3:" URLs break when Secret Key contains a slash, even if encoded
------------------------------------------------------------------

Key: HADOOP-3733
URL: https://issues.apache.org/jira/browse/HADOOP-3733
Project: Hadoop Core
Issue Type: Bug
Components: fs/s3
Affects Versions: 0.17.1
Reporter: Stuart Sierra
Priority: Minor

When using URLs of the form s3://ID:[email protected]/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F.
Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
And your bucket is called "mybucket"
You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
But this doesn't work:
{noformat}
$ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%[email protected]/dest
08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%[email protected]/dest
08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
...
With failures, global counters are inaccurate; consider running with -i
Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
...
{noformat}

"s3:" URLs break when Secret Key contains a slash, even if encoded
------------------------------------------------------------------

Key: HADOOP-3733
URL: https://issues.apache.org/jira/browse/HADOOP-3733
Project: Hadoop Core
Issue Type: Bug
Components: fs/s3
Affects Versions: 0.17.1
Reporter: Stuart Sierra
Priority: Minor
Attachments: hadoop-3733.patch


When using URLs of the form s3://ID:[email protected]/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F.
Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
And your bucket is called "mybucket"
You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
But this doesn't work:
{noformat}
$ bin/hadoop distcp file:///source s3://RYWX12N9WCY42XVOL8WH:Xqj1%[email protected]/dest
08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%[email protected]/dest
08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
...
With failures, global counters are inaccurate; consider running with -i
Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
...
{noformat}