FAQ
Extend FileSystem API to allow appending to files
-------------------------------------------------

Key: HADOOP-3250
URL: https://issues.apache.org/jira/browse/HADOOP-3250
Project: Hadoop Core
Issue Type: New Feature
Components: fs
Reporter: dhruba borthakur


Provide an API to allow applications to append data to pre-existing files.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • dhruba borthakur (JIRA) at Apr 14, 2008 at 8:08 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588756#action_12588756 ]

    dhruba borthakur commented on HADOOP-3250:
    ------------------------------------------

    One option for this API would be a new method in the FileSystem.open() that takes in the pathname to be opened (and permissions, etc) and returns an FSDataOutputStream. The next write into this stream will occur at the end of the file.


    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur

    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Apr 21, 2008 at 9:04 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591055#action_12591055 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
    ------------------------------------------------

    FileSystem.open currently returns a FSDataInputStream. Clearly, we cannot use an InputStream to do append. I guess we might want to define a new method, say "append", in the FileSystem API.
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur

    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Apr 21, 2008 at 9:08 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591060#action_12591060 ]

    dhruba borthakur commented on HADOOP-3250:
    ------------------------------------------

    My mistake. I wanted to that we should create another method called FileSystem.create() with an "append" flag.

    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur

    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Apr 21, 2008 at 9:20 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591068#action_12591068 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
    ------------------------------------------------

    - Literally, the word "create" does not mean or include append operation.

    - It seems to me that permission, overwrite, replication and blockSize do not make sense in append. This is the current create header.
    {code}
    public abstract FSDataOutputStream create(Path f,
    FsPermission permission,
    boolean overwrite,
    int bufferSize,
    short replication,
    long blockSize,
    Progressable progress) throws IOException;
    {code}



    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur

    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Apr 21, 2008 at 9:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591073#action_12591073 ]

    Doug Cutting commented on HADOOP-3250:
    --------------------------------------

    I like the name 'append' for this.

    {noformat}
    public FSDataOutputStream append(Path f) {
    return append(f, getDefaultBufferSize(), null);
    }
    public FSDataOutputStream append(Path f, int bufferSize) {
    return append(f, buffersize, null);
    }
    public abstract FSDataOutputStream append(Path f, int bufferSize, Progressible progress);
    {noformat}
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur

    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Apr 21, 2008 at 11:38 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591116#action_12591116 ]

    dhruba borthakur commented on HADOOP-3250:
    ------------------------------------------

    +1 on Doug's proposal.

    If the path exists and is not being written to, then append() will return a OutputStream to the file. If the path does not exist, then append will generate a IOException.

    The DFSClient will issue ClientProtocol.append() call to the namenode. If the file was being written to by another writer but it has exceeded the soft-lease period, then the namenode would *start* a distributed lease recovery. However, it will not make the client wait for the recovery to complete. The namenode will return an AlreadyBeingCreatedException as if the soft-limit has not yet expired.
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur

    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at May 30, 2008 at 12:04 am
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-3250:
    -------------------------------------------

    Attachment: 3250_20080529.patch

    3250_20080529.patch: need some tests
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Attachments: 3250_20080529.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at May 30, 2008 at 12:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600972#action_12600972 ]

    szetszwo edited comment on HADOOP-3250 at 5/29/08 5:05 PM:
    -------------------------------------------------------------------------

    3250_20080529.patch:
    - Added append API to FileSystem
    - Implemented RawLocalFileSystem
    - throw new IOException("Not supported") for the other FileSystem subclasses
    - need some tests

    was (Author: szetszwo):
    3250_20080529.patch: need some tests
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Attachments: 3250_20080529.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at May 30, 2008 at 4:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601188#action_12601188 ]

    Doug Cutting commented on HADOOP-3250:
    --------------------------------------

    I think you need to add an implementation for FTPFileSystem now too. Otherwise, this looks good to me.
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Attachments: 3250_20080529.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at May 30, 2008 at 9:09 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-3250:
    -------------------------------------------

    Attachment: 3250_20080530.patch

    3250_20080530.patch:
    - implemented FTPFileSystem
    - added tests
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at May 30, 2008 at 9:19 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-3250:
    -------------------------------------------

    Assignee: Tsz Wo (Nicholas), SZE
    Release Note: Added an optional operation, append, to the FileSystem API and an implementation in LocalFileSystem.
    Status: Patch Available (was: Open)
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jun 1, 2008 at 10:57 am
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601462#action_12601462 ]

    Hadoop QA commented on HADOOP-3250:
    -----------------------------------

    +1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12383145/3250_20080530.patch
    against trunk revision 661918.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 4 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    +1 contrib tests. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2527/console

    This message is automatically generated.
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Doug Cutting (JIRA) at Jun 2, 2008 at 6:42 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Doug Cutting updated HADOOP-3250:
    ---------------------------------

    Hadoop Flags: [Reviewed]

    +1 This looks good to me!
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Jun 2, 2008 at 7:54 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-3250:
    ----------------------------------

    Resolution: Fixed
    Fix Version/s: 0.18.0
    Status: Resolved (was: Patch Available)

    I just committed this. Thanks, Nicholas
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Fix For: 0.18.0

    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Jun 3, 2008 at 1:27 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601928#action_12601928 ]

    Hudson commented on HADOOP-3250:
    --------------------------------

    Integrated in Hadoop-trunk #511 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/511/])
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Fix For: 0.18.0

    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Robert Chansler (JIRA) at Jul 22, 2008 at 5:42 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-3250:
    ------------------------------------

    Release Note: (was: Added an optional operation, append, to the FileSystem API and an implementation in LocalFileSystem.)

    Removed release note so as to not confuse folks about whether 1700 is done.
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Fix For: 0.18.0

    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Oct 6, 2008 at 10:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637271#action_12637271 ]

    Tsz Wo (Nicholas), SZE commented on HADOOP-3250:
    ------------------------------------------------
    The reason that I did not propose adding truncate() to FileSystem.java is that I did not make dramatic change to FileSystem APIs.
    There will be a lot of changes. Are we going to apply all these changes to 0.18 and onwards?
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Fix For: 0.18.0

    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Oct 6, 2008 at 10:11 pm
    [ https://issues.apache.org/jira/browse/HADOOP-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-3250:
    -------------------------------------------

    Comment: was deleted
    Extend FileSystem API to allow appending to files
    -------------------------------------------------

    Key: HADOOP-3250
    URL: https://issues.apache.org/jira/browse/HADOOP-3250
    Project: Hadoop Core
    Issue Type: New Feature
    Components: fs
    Reporter: dhruba borthakur
    Assignee: Tsz Wo (Nicholas), SZE
    Fix For: 0.18.0

    Attachments: 3250_20080529.patch, 3250_20080530.patch


    Provide an API to allow applications to append data to pre-existing files.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedApr 14, '08 at 8:02p
activeOct 6, '08 at 10:11p
posts19
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Tsz Wo (Nicholas), SZE (JIRA): 19 posts

People

Translate

site design / logo © 2022 Grokbase