FAQ
Hello,



in the hdfs-default.xml, I found the warning that the property

"dfs.support.append" is false by default, because of the "append

code" containing bugs.

For our test case (which is not very complicated), I tried it out

nevertheless (in a C program using libhdfs), and I did not

encounter any problems yet. The only thing I noticed is that the

modification date of the file to which I append is not changed.



Could someone indicate which other bugs may occur? And in

which cases? Because the append could be quite useful for me,

and I would like to know the risk.



Thanks in advance

Jan

Search Discussions

  • Todd Lipcon at Jun 15, 2010 at 3:25 pm

    On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker wrote:

    Hello,



    in the hdfs-default.xml, I found the warning that the property

    "dfs.support.append" is false by default, because of the "append

    code" containing bugs.

    For our test case (which is not very complicated), I tried it out

    nevertheless (in a C program using libhdfs), and I did not

    encounter any problems yet. The only thing I noticed is that the

    modification date of the file to which I append is not changed.



    Could someone indicate which other bugs may occur? And in

    which cases? Because the append could be quite useful for me,

    and I would like to know the risk.
    Mostly in failure handling cases, but not entirely. The result is usually
    truncated files, sometimes truncated as if your append didn't happen,
    sometimes the entire last block going missing. Look for the 0.20-append fix
    version on JIRA for a more significantly list.

    I'd recommend waiting a couple weeks and using the hadoop-0.20-append branch
    which will contain fixes for many of these bugs.

    -Todd


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Dhruba Borthakur at Jun 15, 2010 at 4:53 pm
    You can start downloading and using the code form the hadoop-0.20-append.

    thanks,
    dhruba
    On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon wrote:
    On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker wrote:

    Hello,



    in the hdfs-default.xml, I found the warning that the property

    "dfs.support.append" is false by default, because of the "append

    code" containing bugs.

    For our test case (which is not very complicated), I tried it out

    nevertheless (in a C program using libhdfs), and I did not

    encounter any problems yet. The only thing I noticed is that the

    modification date of the file to which I append is not changed.



    Could someone indicate which other bugs may occur? And in

    which cases? Because the append could be quite useful for me,

    and I would like to know the risk.
    Mostly in failure handling cases, but not entirely. The result is usually
    truncated files, sometimes truncated as if your append didn't happen,
    sometimes the entire last block going missing. Look for the 0.20-append fix
    version on JIRA for a more significantly list.

    I'd recommend waiting a couple weeks and using the hadoop-0.20-append
    branch
    which will contain fixes for many of these bugs.

    -Todd


    --
    Todd Lipcon
    Software Engineer, Cloudera


    --
    Connect to me at http://www.facebook.com/dhruba
  • Todd Lipcon at Jun 15, 2010 at 4:56 pm

    On Tue, Jun 15, 2010 at 9:53 AM, Dhruba Borthakur wrote:

    You can start downloading and using the code form the hadoop-0.20-append.
    Though it's still missing some patches, so I'd recommend SVN checkouts and
    watching the branch :)

    -Todd

    thanks,
    dhruba
    On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon wrote:

    On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker <Jan.Stoecker@q2web.de>
    wrote:
    Hello,



    in the hdfs-default.xml, I found the warning that the property

    "dfs.support.append" is false by default, because of the "append

    code" containing bugs.

    For our test case (which is not very complicated), I tried it out

    nevertheless (in a C program using libhdfs), and I did not

    encounter any problems yet. The only thing I noticed is that the

    modification date of the file to which I append is not changed.



    Could someone indicate which other bugs may occur? And in

    which cases? Because the append could be quite useful for me,

    and I would like to know the risk.
    Mostly in failure handling cases, but not entirely. The result is usually
    truncated files, sometimes truncated as if your append didn't happen,
    sometimes the entire last block going missing. Look for the 0.20-append fix
    version on JIRA for a more significantly list.

    I'd recommend waiting a couple weeks and using the hadoop-0.20-append
    branch
    which will contain fixes for many of these bugs.

    -Todd


    --
    Todd Lipcon
    Software Engineer, Cloudera


    --
    Connect to me at http://www.facebook.com/dhruba


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Jan Stöcker at Jun 16, 2010 at 6:57 am
    O.k., thanks for the information & the suggestions!

    Jan

    -----Ursprüngliche Nachricht-----
    Von: Todd Lipcon
    Gesendet: Dienstag, 15. Juni 2010 18:55
    An: general@hadoop.apache.org
    Betreff: Re: HDFS file append
    On Tue, Jun 15, 2010 at 9:53 AM, Dhruba Borthakur wrote:

    You can start downloading and using the code form the hadoop-0.20-append.
    Though it's still missing some patches, so I'd recommend SVN checkouts and
    watching the branch :)

    -Todd

    thanks,
    dhruba
    On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon wrote:

    On Tue, Jun 15, 2010 at 5:31 AM, Jan Stöcker <Jan.Stoecker@q2web.de>
    wrote:
    Hello,



    in the hdfs-default.xml, I found the warning that the property

    "dfs.support.append" is false by default, because of the "append

    code" containing bugs.

    For our test case (which is not very complicated), I tried it out

    nevertheless (in a C program using libhdfs), and I did not

    encounter any problems yet. The only thing I noticed is that the

    modification date of the file to which I append is not changed.



    Could someone indicate which other bugs may occur? And in

    which cases? Because the append could be quite useful for me,

    and I would like to know the risk.
    Mostly in failure handling cases, but not entirely. The result is usually
    truncated files, sometimes truncated as if your append didn't happen,
    sometimes the entire last block going missing. Look for the 0.20-append fix
    version on JIRA for a more significantly list.

    I'd recommend waiting a couple weeks and using the hadoop-0.20-append
    branch
    which will contain fixes for many of these bugs.

    -Todd


    --
    Todd Lipcon
    Software Engineer, Cloudera


    --
    Connect to me at http://www.facebook.com/dhruba


    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgeneral @
categorieshadoop
postedJun 15, '10 at 12:32p
activeJun 16, '10 at 6:57a
posts5
users3
websitehadoop.apache.org
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase