FAQ
[ https://issues.apache.org/jira/browse/HADOOP-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595063#action_12595063 ]

Hadoop QA commented on HADOOP-930:
----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381601/hadoop-930-v2.patch
against trunk revision 654265.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 159 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

-1 release audit. The applied patch generated 208 release audit warnings (more than the trunk's current 207 warnings).

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2420/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2420/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2420/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2420/console

This message is automatically generated.
Add support for reading regular (non-block-based) files from S3 in S3FileSystem
-------------------------------------------------------------------------------

Key: HADOOP-930
URL: https://issues.apache.org/jira/browse/HADOOP-930
Project: Hadoop Core
Issue Type: New Feature
Components: fs
Affects Versions: 0.10.1
Reporter: Tom White
Assignee: Tom White
Fix For: 0.18.0

Attachments: hadoop-930-v2.patch, hadoop-930.patch, jets3t-0.6.0.jar


People often have input data on S3 that they want to use for a Map Reduce job and the current S3FileSystem implementation cannot read it since it assumes a block-based format.
We would add the following metadata to files written by S3FileSystem: an indication that it is block oriented ("S3FileSystem.type=block") and a filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files would not have the type metadata so S3FileSystem would not try to interpret them as inodes.
An extension to write regular files to S3 would not be covered by this change - we could do this as a separate piece of work (we still need to decide whether to introduce another scheme - e.g. rename block-based S3 to "s3fs" and call regular S3 "s3" - or whether to just use a configuration property to control block-based vs. regular writes).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Hadoop QA (JIRA) at May 8, 2008 at 10:53 pm
    [ https://issues.apache.org/jira/browse/HADOOP-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595415#action_12595415 ]

    Hadoop QA commented on HADOOP-930:
    ----------------------------------

    +1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12381659/hadoop-930-v3.patch
    against trunk revision 654315.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 159 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    +1 contrib tests. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2428/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2428/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2428/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2428/console

    This message is automatically generated.
    Add support for reading regular (non-block-based) files from S3 in S3FileSystem
    -------------------------------------------------------------------------------

    Key: HADOOP-930
    URL: https://issues.apache.org/jira/browse/HADOOP-930
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Affects Versions: 0.10.1
    Reporter: Tom White
    Assignee: Tom White
    Fix For: 0.18.0

    Attachments: hadoop-930-v2.patch, hadoop-930-v3.patch, hadoop-930.patch, jets3t-0.6.0.jar


    People often have input data on S3 that they want to use for a Map Reduce job and the current S3FileSystem implementation cannot read it since it assumes a block-based format.
    We would add the following metadata to files written by S3FileSystem: an indication that it is block oriented ("S3FileSystem.type=block") and a filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files would not have the type metadata so S3FileSystem would not try to interpret them as inodes.
    An extension to write regular files to S3 would not be covered by this change - we could do this as a separate piece of work (we still need to decide whether to introduce another scheme - e.g. rename block-based S3 to "s3fs" and call regular S3 "s3" - or whether to just use a configuration property to control block-based vs. regular writes).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at May 22, 2008 at 5:18 pm
    [ https://issues.apache.org/jira/browse/HADOOP-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599093#action_12599093 ]

    Owen O'Malley commented on HADOOP-930:
    --------------------------------------

    Can someone validate that this code works for them?
    Add support for reading regular (non-block-based) files from S3 in S3FileSystem
    -------------------------------------------------------------------------------

    Key: HADOOP-930
    URL: https://issues.apache.org/jira/browse/HADOOP-930
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Affects Versions: 0.10.1
    Reporter: Tom White
    Assignee: Tom White
    Fix For: 0.18.0

    Attachments: hadoop-930-v2.patch, hadoop-930-v3.patch, hadoop-930.patch, jets3t-0.6.0.jar


    People often have input data on S3 that they want to use for a Map Reduce job and the current S3FileSystem implementation cannot read it since it assumes a block-based format.
    We would add the following metadata to files written by S3FileSystem: an indication that it is block oriented ("S3FileSystem.type=block") and a filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files would not have the type metadata so S3FileSystem would not try to interpret them as inodes.
    An extension to write regular files to S3 would not be covered by this change - we could do this as a separate piece of work (we still need to decide whether to introduce another scheme - e.g. rename block-based S3 to "s3fs" and call regular S3 "s3" - or whether to just use a configuration property to control block-based vs. regular writes).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 7, '08 at 10:18p
activeMay 22, '08 at 5:18p
posts3
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Owen O'Malley (JIRA): 3 posts

People

Translate

site design / logo © 2022 Grokbase