FAQ
Hi all,
I tried to run my hadoop program on S3 by following this wiki page:
http://wiki.apache.org/hadoop/AmazonS3
I configured the core-site.xml by adding

<property>
<name>fs.default.name</name>
<value>s3://ID:SECRET@BUCKET</value>
</property>

and I specified the accesskey and secretkey by using the URI
format:s3://ID:SECRET@BUCKET

however, it fails and datanodes reports:

NumberFormatException....
...

Is this the right way to config hadoop running on s3? if so, whats the
problem?

Regards
Song

Search Discussions

  • Rekha Joshi at Dec 18, 2009 at 5:04 am
    Not sure what the whole error is, but you can always alternatively try this -
    <property>
    <name>fs.default.name</name>
    <value>s3://BUCKET</value>
    </property>

    <property>
    <name>fs.s3.awsAccessKeyId</name>
    <value>ID</value>
    </property>

    <property>
    <name>fs.s3.awsSecretAccessKey</name>
    <value>SECRET</value>
    </property>

    And I am not sure what is the base hadoop version on S3, but possibly if S3 wiki is correct try updating conf/hadoop-site.xml

    Cheers,
    /R

    On 12/18/09 10:23 AM, "松柳" wrote:

    Hi all,
    I tried to run my hadoop program on S3 by following this wiki page:
    http://wiki.apache.org/hadoop/AmazonS3
    I configured the core-site.xml by adding

    <property>
    <name>fs.default.name</name>
    <value>s3://ID:SECRET@BUCKET</value>
    </property>

    and I specified the accesskey and secretkey by using the URI
    format:s3://ID:SECRET@BUCKET

    however, it fails and datanodes reports:

    NumberFormatException....
    ...

    Is this the right way to config hadoop running on s3? if so, whats the
    problem?

    Regards
    Song
  • Tom White at Dec 18, 2009 at 5:35 am
    If you are using S3 as your file store then you don't need to run HDFS
    (and indeed HDFS will not start up if you try).

    Cheers,
    Tom

    2009/12/17 Rekha Joshi <rekhajos@yahoo-inc.com>:
    Not sure what the whole error is, but you can always alternatively try this -
    <property>
    <name>fs.default.name</name>
    <value>s3://BUCKET</value>
    </property>

    <property>
    <name>fs.s3.awsAccessKeyId</name>
    <value>ID</value>
    </property>

    <property>
    <name>fs.s3.awsSecretAccessKey</name>
    <value>SECRET</value>
    </property>

    And I am not sure what is the base hadoop version on S3, but possibly if S3 wiki is correct try updating conf/hadoop-site.xml

    Cheers,
    /R

    On 12/18/09 10:23 AM, "松柳" wrote:

    Hi all,
    I tried to run my hadoop program on S3 by following this wiki page:
    http://wiki.apache.org/hadoop/AmazonS3
    I configured the core-site.xml by adding

    <property>
    <name>fs.default.name</name>
    <value>s3://ID:SECRET@BUCKET</value>
    </property>

    and I specified the accesskey and secretkey by using the URI
    format:s3://ID:SECRET@BUCKET

    however, it fails and datanodes reports:

    NumberFormatException....
    ...

    Is this the right way to config hadoop running on s3? if so, whats the
    problem?

    Regards
    Song

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 18, '09 at 4:53a
activeDec 18, '09 at 5:35a
posts3
users3
websitehadoop.apache.org...
irc#hadoop

3 users in discussion

Tom White: 1 post 松柳: 1 post Rekha Joshi: 1 post

People

Translate

site design / logo © 2022 Grokbase