[Hadoop-common-dev] [jira] Created: (HADOOP-3250) Extend FileSystem API to allow appending to files

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588756#action_12588756 ]

dhruba borthakur commented on HADOOP-3250:
------------------------------------------

One option for this API would be a new method in the FileSystem.open() that takes in the pathname to be opened (and permissions, etc) and returns an FSDataOutputStream. The next write into this stream will occur at the end of the file.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591055#action_12591055 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
------------------------------------------------

FileSystem.open currently returns a FSDataInputStream. Clearly, we cannot use an InputStream to do append. I guess we might want to define a new method, say "append", in the FileSystem API.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591060#action_12591060 ]

dhruba borthakur commented on HADOOP-3250:
------------------------------------------

My mistake. I wanted to that we should create another method called FileSystem.create() with an "append" flag.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591068#action_12591068 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
------------------------------------------------

- Literally, the word "create" does not mean or include append operation.

- It seems to me that permission, overwrite, replication and blockSize do not make sense in append. This is the current create header.
{code}
public abstract FSDataOutputStream create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress) throws IOException;
{code}

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591073#action_12591073 ]

Doug Cutting commented on HADOOP-3250:
--------------------------------------

I like the name 'append' for this.

{noformat}
public FSDataOutputStream append(Path f) {
return append(f, getDefaultBufferSize(), null);
}
public FSDataOutputStream append(Path f, int bufferSize) {
return append(f, buffersize, null);
}
public abstract FSDataOutputStream append(Path f, int bufferSize, Progressible progress);
{noformat}

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591116#action_12591116 ]

dhruba borthakur commented on HADOOP-3250:
------------------------------------------

+1 on Doug's proposal.

If the path exists and is not being written to, then append() will return a OutputStream to the file. If the path does not exist, then append will generate a IOException.

The DFSClient will issue ClientProtocol.append() call to the namenode. If the file was being written to by another writer but it has exceeded the soft-lease period, then the namenode would *start* a distributed lease recovery. However, it will not make the client wait for the recovery to complete. The namenode will return an AlreadyBeingCreatedException as if the soft-limit has not yet expired.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------

Attachment: 3250_20080529.patch

3250_20080529.patch: need some tests

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600972#action_12600972 ]

szetszwo edited comment on HADOOP-3250 at 5/29/08 5:05 PM:
-------------------------------------------------------------------------

3250_20080529.patch:
- Added append API to FileSystem
- Implemented RawLocalFileSystem
- throw new IOException("Not supported") for the other FileSystem subclasses
- need some tests

was (Author: szetszwo):
3250_20080529.patch: need some tests

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601188#action_12601188 ]

Doug Cutting commented on HADOOP-3250:
--------------------------------------

I think you need to add an implementation for FTPFileSystem now too. Otherwise, this looks good to me.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------

Attachment: 3250_20080530.patch

3250_20080530.patch:
- implemented FTPFileSystem
- added tests

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------

Assignee: Tsz Wo (Nicholas), SZE
Release Note: Added an optional operation, append, to the FileSystem API and an implementation in LocalFileSystem.
Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601462#action_12601462 ]

Hadoop QA commented on HADOOP-3250:
-----------------------------------

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383145/3250_20080530.patch
against trunk revision 661918.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-3250:
---------------------------------

Hadoop Flags: [Reviewed]

+1 This looks good to me!

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-3250:
----------------------------------

Resolution: Fixed
Fix Version/s: 0.18.0
Status: Resolved (was: Patch Available)

I just committed this. Thanks, Nicholas

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601928#action_12601928 ]

Hudson commented on HADOOP-3250:
--------------------------------

Integrated in Hadoop-trunk #511 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/511/])

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-3250:
------------------------------------

Release Note: (was: Added an optional operation, append, to the FileSystem API and an implementation in LocalFileSystem.)

Removed release note so as to not confuse folks about whether 1700 is done.

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637271#action_12637271 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
------------------------------------------------

[ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-3250:
-------------------------------------------

Comment: was deleted