FAQ
Folks,

Hadoop 0.19.1 is now available with the file append feature disabled.
It's time to talk about a Hadoop 0.20.0 release.

Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
few blockers are now almost fixed (should be next week) except for
HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
file append.

*** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
same disabling of file append in Hadoop 0.20.0 that we put in place to
get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

I will call a vote for 0.20.0 when blockers are fixed.

Cheers,
Nigel

Folks,

Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
branch has issues and a 0.19.1 release is needed.

Quality issues in the changes made for the file append feature have
prevented some from deploying Hadoop 0.19. One of these changes
(sync) has now been "fixed" by reducing its semantics in Hadoop
0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
branch.

I would like to propose that we apply this same "fix" to sync in
0.19.1 and 0.20.0. Since append requires the full semantics of
sync, I propose we also disable append (perhaps throw
UnsupportedOperationException from API?). Yes, this would
unfortunately be an incompatible change between 0.19.0 and 0.19.1.
We can then take the time needed to fix append properly in 0.21.0.

I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

Nigel

Search Discussions

  • Jim Kellerman (POWERSET) at Feb 25, 2009 at 12:26 am
    --1

    HBase really needs 4379. My testing to date indicates that it does work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in 0.19.2 and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
    same disabling of file append in Hadoop 0.20.0 that we put in place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel

    Folks,

    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Dhruba Borthakur at Feb 25, 2009 at 5:29 am
    Hi Jim,

    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to discuss and
    review the patches/fixes.

    Nigel: wht is the proposed deadline for 0.20?

    thanks,
    dhruba


    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) wrote:

    --1

    HBase really needs 4379. My testing to date indicates that it does work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in 0.19.2 and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
    same disabling of file append in Hadoop 0.20.0 that we put in place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel

    Folks,

    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Nigel Daley at Feb 26, 2009 at 3:35 am

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,

    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
    nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature will be fixed properly. We're building a file system. Data
    reliability and accuracy are absolutely key. We know that this
    feature has been very lightly tested.
    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige
    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:
    --1

    HBase really needs 4379. My testing to date indicates that it does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in 0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel

    Folks,

    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Dhruba Borthakur at Feb 26, 2009 at 3:52 am
    "Whipping out a patch" says nothing about its reliability.

    i would like some focus from the developer's community to properly fix this
    issue. I am willing to spend as much as time it takes ot get it fixed the
    right way, I but I would like even more constructive engagement from more
    people to get this one right. May I request you to see if you can volunteer
    to spend some time testing some of this code at scale ?(I have access to 10
    machines only for testing).

    thanks
    dhruba
    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley wrote:


    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to discuss and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this feature
    will be fixed properly. We're building a file system. Data reliability and
    accuracy are absolutely key. We know that this feature has been very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it does work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in 0.19.2 and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
    same disabling of file append in Hadoop 0.20.0 that we put in place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Nigel Daley at Feb 26, 2009 at 4:21 am

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.

    i would like some focus from the developer's community to properly
    fix this
    issue. I am willing to spend as much as time it takes ot get it
    fixed the
    right way, I but I would like even more constructive engagement from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to 10
    machines only for testing).
    Sorry, I can't commit any time/resources to this right now. Perhaps
    some hbase folks can. In the meantime, can we make append
    configurable in 0.19.2 and 0.20.0? I filed
    https://issues.apache.org/jira/browse/HADOOP-5332

    Cheers,
    Nige

    thanks
    dhruba
    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley wrote:


    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
    nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability and
    accuracy are absolutely key. We know that this feature has been very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2 and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep
    our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
    last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature
    have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Brian Bockelman at Feb 26, 2009 at 4:31 am
    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:
    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.

    i would like some focus from the developer's community to properly
    fix this
    issue. I am willing to spend as much as time it takes ot get it
    fixed the
    right way, I but I would like even more constructive engagement
    from more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to 10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you
    simply need access or folks who can run challenging jobs? Scaring up
    access to the cluster can be easy, but admin / user time isn't really
    available.
    Sorry, I can't commit any time/resources to this right now. Perhaps
    some hbase folks can. In the meantime, can we make append
    configurable in 0.19.2 and 0.20.0? I filed
    https://issues.apache.org/jira/browse/HADOOP-5332
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com>
    wrote:
    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
    nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability and
    accuracy are absolutely key. We know that this feature has been
    very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2 and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep
    our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
    last
    few blockers are now almost fixed (should be next week) except
    for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature
    have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are
    fixed.

    Nigel
  • Hemanth Yamijala at Feb 26, 2009 at 5:06 am
    +1 for HADOOP-5332. I am in the same position as Brian, as an outside
    observer. This will help us to move on Hadoop 0.20 which has a lot of
    other features as well that users are asking for.

    Thanks
    hemanth
    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.

    i would like some focus from the developer's community to properly
    fix this
    issue. I am willing to spend as much as time it takes ot get it
    fixed the
    right way, I but I would like even more constructive engagement from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to 10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you
    simply need access or folks who can run challenging jobs? Scaring up
    access to the cluster can be easy, but admin / user time isn't really
    available.
    Sorry, I can't commit any time/resources to this right now. Perhaps
    some hbase folks can. In the meantime, can we make append
    configurable in 0.19.2 and 0.20.0? I filed
    https://issues.apache.org/jira/browse/HADOOP-5332
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com>
    wrote:
    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
    nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability and
    accuracy are absolutely key. We know that this feature has been very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2 and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature
    have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Dhruba Borthakur at Feb 26, 2009 at 5:32 am
    I posted a patch for HADOOP-5332. I am suggesting that this patch be applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba
    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala wrote:

    +1 for HADOOP-5332. I am in the same position as Brian, as an outside
    observer. This will help us to move on Hadoop 0.20 which has a lot of other
    features as well that users are asking for.

    Thanks
    hemanth


    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.
    i would like some focus from the developer's community to properly fix
    this
    issue. I am willing to spend as much as time it takes ot get it fixed
    the
    right way, I but I would like even more constructive engagement from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have access to
    10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you simply
    need access or folks who can run challenging jobs? Scaring up access to the
    cluster can be easy, but admin / user time isn't really available.

    Sorry, I can't commit any time/resources to this right now. Perhaps some
    hbase folks can. In the meantime, can we make append configurable in 0.19.2
    and 0.20.0? I filed
    https://issues.apache.org/jira/browse/HADOOP-5332
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com>
    wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be nice
    if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to discuss
    and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this feature
    will be fixed properly. We're building a file system. Data reliability
    and
    accuracy are absolutely key. We know that this feature has been very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in 0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----

    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in place
    to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Raghu Angadi at Feb 26, 2009 at 5:57 am
    +1 for config option for both 0.19 and 0.20.

    Raghu.

    Dhruba Borthakur wrote:
    I posted a patch for HADOOP-5332. I am suggesting that this patch be applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba
    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala wrote:

    +1 for HADOOP-5332. I am in the same position as Brian, as an outside
    observer. This will help us to move on Hadoop 0.20 which has a lot of other
    features as well that users are asking for.

    Thanks
    hemanth
  • Jim Kellerman (POWERSET) at Feb 26, 2009 at 6:45 pm
    With the availability of HADOOP-5332 I remove my objection.
    -----Original Message-----
    From: Dhruba Borthakur
    Sent: Wednesday, February 25, 2009 9:32 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    I posted a patch for HADOOP-5332. I am suggesting that this patch be
    applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by
    default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba

    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
    inc.com>wrote:
    +1 for HADOOP-5332. I am in the same position as Brian, as an outside
    observer. This will help us to move on Hadoop 0.20 which has a lot of other
    features as well that users are asking for.

    Thanks
    hemanth


    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.
    i would like some focus from the developer's community to properly
    fix
    this
    issue. I am willing to spend as much as time it takes ot get it
    fixed
    the
    right way, I but I would like even more constructive engagement
    from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to
    10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you
    simply
    need access or folks who can run challenging jobs? Scaring up access
    to the
    cluster can be easy, but admin / user time isn't really available.

    Sorry, I can't commit any time/resources to this right now. Perhaps
    some
    hbase folks can. In the meantime, can we make append configurable in
    0.19.2
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com>
    wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
    nice
    if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss
    and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability
    and
    accuracy are absolutely key. We know that this feature has been
    very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep
    our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----

    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
    last
    few blockers are now almost fixed (should be next week) except
    for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place
    to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature
    have
    prevented some from deploying Hadoop 0.19. One of these
    changes
    (sync) has now been "fixed" by reducing its semantics in
    Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync
    in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.
    I will call a vote for 0.19.1 and 0.20.0 when blockers are
    fixed.
    Nigel
  • Nigel Daley at Feb 27, 2009 at 5:14 am
    Thanks Jim.

    Dhruba, can we move
    https://issues.apache.org/jira/browse/HADOOP-4379
    to 0.21.0?

    Nige
    On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:

    With the availability of HADOOP-5332 I remove my objection.
    -----Original Message-----
    From: Dhruba Borthakur
    Sent: Wednesday, February 25, 2009 9:32 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    I posted a patch for HADOOP-5332. I am suggesting that this patch be
    applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by
    default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in
    the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba

    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
    inc.com>wrote:
    +1 for HADOOP-5332. I am in the same position as Brian, as an
    outside
    observer. This will help us to move on Hadoop 0.20 which has a lot
    of other
    features as well that users are asking for.

    Thanks
    hemanth


    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.
    i would like some focus from the developer's community to
    properly
    fix
    this
    issue. I am willing to spend as much as time it takes ot get it
    fixed
    the
    right way, I but I would like even more constructive engagement
    from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to
    10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you
    simply
    need access or folks who can run challenging jobs? Scaring up
    access
    to the
    cluster can be easy, but admin / user time isn't really available.

    Sorry, I can't commit any time/resources to this right now. Perhaps
    some
    hbase folks can. In the meantime, can we make append
    configurable in
    0.19.2
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com
    wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix
    for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would
    be
    nice
    if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss
    and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability
    and
    accuracy are absolutely key. We know that this feature has been
    very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to
    keep
    our
    community alive. (They will go to Cassandra or something
    else to
    ensure
    reliability).

    -----Original Message-----

    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago.
    The
    last
    few blockers are now almost fixed (should be next week)
    except
    for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place
    to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly,
    the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append
    feature
    have
    prevented some from deploying Hadoop 0.19. One of these
    changes
    (sync) has now been "fixed" by reducing its semantics in
    Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the
    0.18
    branch.

    I would like to propose that we apply this same "fix" to
    sync
    in
    0.19.1 and 0.20.0. Since append requires the full
    semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.
    I will call a vote for 0.19.1 and 0.20.0 when blockers are
    fixed.
    Nigel
  • Jim Kellerman (POWERSET) at Feb 27, 2009 at 5:09 pm
    I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible.
    We are really hurting without it.

    ---
    Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
    -----Original Message-----
    From: Nigel Daley
    Sent: Thursday, February 26, 2009 9:13 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    Thanks Jim.

    Dhruba, can we move
    https://issues.apache.org/jira/browse/HADOOP-4379
    to 0.21.0?

    Nige
    On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:

    With the availability of HADOOP-5332 I remove my objection.
    -----Original Message-----
    From: Dhruba Borthakur
    Sent: Wednesday, February 25, 2009 9:32 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    I posted a patch for HADOOP-5332. I am suggesting that this patch be
    applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by
    default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in
    the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba

    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
    inc.com>wrote:
    +1 for HADOOP-5332. I am in the same position as Brian, as an
    outside
    observer. This will help us to move on Hadoop 0.20 which has a lot
    of other
    features as well that users are asking for.

    Thanks
    hemanth


    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.
    i would like some focus from the developer's community to
    properly
    fix
    this
    issue. I am willing to spend as much as time it takes ot get it
    fixed
    the
    right way, I but I would like even more constructive engagement
    from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to
    10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you
    simply
    need access or folks who can run challenging jobs? Scaring up
    access
    to the
    cluster can be easy, but admin / user time isn't really available.

    Sorry, I can't commit any time/resources to this right now. Perhaps
    some
    hbase folks can. In the meantime, can we make append
    configurable in
    0.19.2
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <ndaley@yahoo-inc.com
    wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix
    for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would
    be
    nice
    if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss
    and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability
    and
    accuracy are absolutely key. We know that this feature has been
    very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to
    keep
    our
    community alive. (They will go to Cassandra or something
    else to
    ensure
    reliability).

    -----Original Message-----

    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago.
    The
    last
    few blockers are now almost fixed (should be next week)
    except
    for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place
    to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly,
    the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append
    feature
    have
    prevented some from deploying Hadoop 0.19. One of these
    changes
    (sync) has now been "fixed" by reducing its semantics in
    Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the
    0.18
    branch.

    I would like to propose that we apply this same "fix" to
    sync
    in
    0.19.1 and 0.20.0. Since append requires the full
    semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.
    I will call a vote for 0.19.1 and 0.20.0 when blockers are
    fixed.
    Nigel
  • Doug Judd at Feb 27, 2009 at 5:20 pm
    I'd like to second that. I think it would be good to have the database
    elevated to a first class use case for HDFS. Getting fsync() working
    properly is critical for HBase, Hypertable, or any database built on top of
    HDFS.

    - Doug
    On Fri, Feb 27, 2009 at 9:08 AM, Jim Kellerman (POWERSET) wrote:

    I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible.
    We are really hurting without it.

    ---
    Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
    -----Original Message-----
    From: Nigel Daley
    Sent: Thursday, February 26, 2009 9:13 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    Thanks Jim.

    Dhruba, can we move
    https://issues.apache.org/jira/browse/HADOOP-4379
    to 0.21.0?

    Nige
    On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:

    With the availability of HADOOP-5332 I remove my objection.
    -----Original Message-----
    From: Dhruba Borthakur
    Sent: Wednesday, February 25, 2009 9:32 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    I posted a patch for HADOOP-5332. I am suggesting that this patch be
    applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by
    default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in
    the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba

    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
    inc.com>wrote:
    +1 for HADOOP-5332. I am in the same position as Brian, as an
    outside
    observer. This will help us to move on Hadoop 0.20 which has a lot
    of other
    features as well that users are asking for.

    Thanks
    hemanth


    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.
    i would like some focus from the developer's community to
    properly
    fix
    this
    issue. I am willing to spend as much as time it takes ot get it
    fixed
    the
    right way, I but I would like even more constructive engagement
    from
    more
    people to get this one right. May I request you to see if you can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to
    10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do you
    simply
    need access or folks who can run challenging jobs? Scaring up
    access
    to the
    cluster can be easy, but admin / user time isn't really available.

    Sorry, I can't commit any time/resources to this right now. Perhaps
    some
    hbase folks can. In the meantime, can we make append
    configurable in
    0.19.2
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <
    ndaley@yahoo-inc.com
    wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix
    for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would
    be
    nice
    if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss
    and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability
    and
    accuracy are absolutely key. We know that this feature has been
    very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to
    keep
    our
    community alive. (They will go to Cassandra or something
    else to
    ensure
    reliability).

    -----Original Message-----

    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago.
    The
    last
    few blockers are now almost fixed (should be next week)
    except
    for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place
    to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly,
    the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append
    feature
    have
    prevented some from deploying Hadoop 0.19. One of these
    changes
    (sync) has now been "fixed" by reducing its semantics in
    Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the
    0.18
    branch.

    I would like to propose that we apply this same "fix" to
    sync
    in
    0.19.1 and 0.20.0. Since append requires the full
    semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.
    I will call a vote for 0.19.1 and 0.20.0 when blockers are
    fixed.
    Nigel
  • Dhruba Borthakur at Feb 27, 2009 at 6:11 pm
    I would like to make HADOOP-5332 be part of the 0.19, 0.20 and trunk. This
    ensures that "append" is switched off by default. At the same time, we would
    need patches for HADOOP-4739, HADOOP-4663 and HADOOP-5027. These three are
    critical to support "appends".

    A series of offline discussions have been summarised in HADOOP-4663. I have
    not yet got comments on this summary but I am already working on it and will
    post a patch early next week.

    thanks,
    dhruba
    On Fri, Feb 27, 2009 at 9:20 AM, Doug Judd wrote:

    I'd like to second that. I think it would be good to have the database
    elevated to a first class use case for HDFS. Getting fsync() working
    properly is critical for HBase, Hypertable, or any database built on top of
    HDFS.

    - Doug

    On Fri, Feb 27, 2009 at 9:08 AM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:
    I'd really like to see 4379 in 0.19.2 and 0.20.1 if possible.
    We are really hurting without it.

    ---
    Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
    -----Original Message-----
    From: Nigel Daley
    Sent: Thursday, February 26, 2009 9:13 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    Thanks Jim.

    Dhruba, can we move
    https://issues.apache.org/jira/browse/HADOOP-4379
    to 0.21.0?

    Nige
    On Feb 26, 2009, at 10:37 AM, Jim Kellerman (POWERSET) wrote:

    With the availability of HADOOP-5332 I remove my objection.
    -----Original Message-----
    From: Dhruba Borthakur
    Sent: Wednesday, February 25, 2009 9:32 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: Hadoop 0.20.0

    I posted a patch for HADOOP-5332. I am suggesting that this patch be
    applied
    into the 0.19, 0.20 and trunk. This patch switches off "append" by
    default,
    but it can be switched on by setting the config parameter
    dfs.support.append. This does not mean that "append" is bug free in
    the
    code, it just allows developers to continue testing with append
    functionality till the bugs are fixed.

    thanks,
    dhruba

    On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhemanth@yahoo-
    inc.com>wrote:
    +1 for HADOOP-5332. I am in the same position as Brian, as an
    outside
    observer. This will help us to move on Hadoop 0.20 which has a lot
    of other
    features as well that users are asking for.

    Thanks
    hemanth


    On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote:

    On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote:

    "Whipping out a patch" says nothing about its reliability.
    i would like some focus from the developer's community to
    properly
    fix
    this
    issue. I am willing to spend as much as time it takes ot get it
    fixed
    the
    right way, I but I would like even more constructive engagement
    from
    more
    people to get this one right. May I request you to see if you
    can
    volunteer
    to spend some time testing some of this code at scale ?(I have
    access to
    10
    machines only for testing).
    Dhruba, can you define "testing some of this code at scale"? Do
    you
    simply
    need access or folks who can run challenging jobs? Scaring up
    access
    to the
    cluster can be easy, but admin / user time isn't really available.

    Sorry, I can't commit any time/resources to this right now.
    Perhaps
    some
    hbase folks can. In the meantime, can we make append
    configurable in
    0.19.2
    As an outside, irrelevant observer, I think this is a really good
    compromise. Helps out HBase but also would help prevent rushing.

    Brian

    Cheers,
    Nige

    thanks
    dhruba

    On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <
    ndaley@yahoo-inc.com
    wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,
    I can understand your problem. I can probably whip out a fix
    for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would
    be
    nice
    if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss
    and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature
    will be fixed properly. We're building a file system. Data
    reliability
    and
    accuracy are absolutely key. We know that this feature has been
    very
    lightly tested.

    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige


    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:

    --1
    HBase really needs 4379. My testing to date indicates that it
    does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in
    0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to
    keep
    our
    community alive. (They will go to Cassandra or something
    else to
    ensure
    reliability).

    -----Original Message-----

    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago.
    The
    last
    few blockers are now almost fixed (should be next week)
    except
    for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and
    apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put
    in
    place
    to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel


    Folks,
    Some Hadoop deployments have upgraded to 0.19.0. Clearly,
    the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append
    feature
    have
    prevented some from deploying Hadoop 0.19. One of these
    changes
    (sync) has now been "fixed" by reducing its semantics in
    Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the
    0.18
    branch.

    I would like to propose that we apply this same "fix" to
    sync
    in
    0.19.1 and 0.20.0. Since append requires the full
    semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and
    0.19.1.
    We can then take the time needed to fix append properly in
    0.21.0.
    I will call a vote for 0.19.1 and 0.20.0 when blockers are
    fixed.
    Nigel
  • Eric Baldeschwieler at Feb 26, 2009 at 4:46 am
    +1

    Hi Folks,

    I don't think vetoing releases until the append issue is resolved is
    constructive. I'd like to invite folks who want to fix append to work
    with us on that and support releasing 20 without it in parallel.

    Please keep in mind the spirit of this project. Putting your shoulder
    to the wheel gets features added to the project. Vetoing a release
    will not get anything added to it. It will simply delay getting those
    features we've all agreed work tested and released. Resolving the
    append issues is going to take a big investment. We all want to see
    that happen. But we also need to provide the whole hadoop community
    with stable releases while we do the hard work needed to really solve
    this problem.

    Why are we taking this position? The current append design has proven
    fix resistant. We've deployed fix after fix and have continued to see
    new append bugs that have caused data loss and cluster down time. We
    moved to disable append in our clusters because that was the only path
    we could see to delivering a stable system to our users. We believe
    the only way to solve this problem is to acknowledge that the append
    code is a critical section of the code that needs to be designed,
    documented and tested thoroughly if we expect it to function well.
    Given the pain the append issue has caused us, we are going to -1
    "quick fixes" to it. Data loss and corruption are serious issues.
    We've been burned.

    I hope folks will consider this and both support a 20 release without
    append and also work with us to really fix append. We promise to work
    hard with the community to deliver a 20.x or 21 release with append.

    Thanks,

    E14
    ---
    eric14 a.k.a. Eric Baldeschwieler
    vp grid computing
    Yahoo! Inc.
    On Feb 25, 2009, at 7:34 PM, Nigel Daley wrote:

    On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:

    Hi Jim,

    I can understand your problem. I can probably whip out a fix for
    HADOOP-4663 and HADOOP-4379 by the end of this week. It would be
    nice if
    somebody else (Hairong, Sanjay, Konstantin?) can volunteer to
    discuss and
    review the patches/fixes.
    "Whipping out a patch" doesn't give me any confidence that this
    feature will be fixed properly. We're building a file system. Data
    reliability and accuracy are absolutely key. We know that this
    feature has been very lightly tested.
    Nigel: wht is the proposed deadline for 0.20?
    March 6.

    Nige
    thanks,
    dhruba



    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
    Jim.Kellerman@microsoft.com> wrote:
    --1

    HBase really needs 4379. My testing to date indicates that it does
    work
    (although I have a bit more testing to do).

    I was ok with not putting it into 0.19.1 provided it was in 0.19.2
    and
    0.20.0.

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to
    ensure
    reliability).

    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature
    disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The
    last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly
    implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply
    the
    same disabling of file append in Hadoop 0.20.0 that we put in
    place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel

    Folks,

    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the
    0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature
    have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel
  • Ted Dunning at Feb 25, 2009 at 4:18 pm
    Indeed, I am one of these community members who are quite possibly going to
    find another alternative if hbase doesn't seem more supportable/supported
    reasonably soon. Cassandra suffers from the normal facebook open source
    syndrome of no documentation, but that begins to look good given the
    problems with append.
    On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) wrote:

    It's a big problem for us now and is hurting our ability to keep our
    community alive. (They will go to Cassandra or something else to ensure
    reliability).


    --
    Ted Dunning, CTO
    DeepDyve
  • Jonathan Gray at Feb 25, 2009 at 8:29 pm
    -1

    This is a huge deal for me as an HBase user. And as someone who does
    significant consulting and evangelism for HBase, there is probably not a
    more necessary feature at this stage of the project.

    Writes in HBase are done in memory, and we also write each edit to the HLog
    which is periodically pushed to HDFS to prevent data loss should the node go
    down in a way that does not allow us to flush everything properly. Without
    appends, we have to create a new file each time we "checkpoint". In
    practice, this was every 10,000 edits.

    With appends, we only need to sync()/flush() to the same file. In practice,
    the default number of edits between flushes became 200, and for those who
    require absolutely no data loss it can now be set to 1 without writing a new
    hdfs file each time, meaning slow but tolerable performance.

    The upcoming 0.20.0 release of HBase is going to bring with it massive
    performance boosts and should put us at or above the performance of our
    competitors. Data loss is killer and is going to be enough to deter new and
    existing users; at that point, we're no longer a fault-tolerant system.

    I typically do not vote strongly for something if I'm not able to contribute
    myself, but my impression from our testing over in HBase is that things are
    not far off from working satisfactorily for our use case. If it's only
    partially working, I'd much prefer it be slapped with warnings about not
    being fully-functional than pulling what does work out from the next
    release.

    Jonathan Gray
    -----Original Message-----
    From: Nigel Daley
    Sent: Tuesday, February 24, 2009 4:02 PM
    To: core-dev@hadoop.apache.org
    Subject: Hadoop 0.20.0

    Folks,

    Hadoop 0.19.1 is now available with the file append feature disabled.
    It's time to talk about a Hadoop 0.20.0 release.

    Hadoop 0.20.0 feature freeze date was almost 3 months ago. The last
    few blockers are now almost fixed (should be next week) except for
    HADOOP-4379. HADOOP-4379 is work that is needed to properly implement
    file append.

    *** I propose we move HADOOP-4379 off to release 0.21.0 and apply the
    same disabling of file append in Hadoop 0.20.0 that we put in place to
    get 0.19.1 released (HADOOP-5224 and HADOOP-5225).

    I will call a vote for 0.20.0 when blockers are fixed.

    Cheers,
    Nigel

    Folks,

    Some Hadoop deployments have upgraded to 0.19.0. Clearly, the 0.19
    branch has issues and a 0.19.1 release is needed.

    Quality issues in the changes made for the file append feature have
    prevented some from deploying Hadoop 0.19. One of these changes
    (sync) has now been "fixed" by reducing its semantics in Hadoop
    0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18
    branch.

    I would like to propose that we apply this same "fix" to sync in
    0.19.1 and 0.20.0. Since append requires the full semantics of
    sync, I propose we also disable append (perhaps throw
    UnsupportedOperationException from API?). Yes, this would
    unfortunately be an incompatible change between 0.19.0 and 0.19.1.
    We can then take the time needed to fix append properly in 0.21.0.

    I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.

    Nigel

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 25, '09 at 12:03a
activeFeb 27, '09 at 6:11p
posts18
users10
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase